Senator Richard Gordon, the head of the Philippine National Red Cross, told Reuters the three, who also included a Filipina, were travelling in a Red Cross vehicle on the island of Jolo when they were stopped and taken away by armed men.
Three Filipinos, who were also on the vehicle, were robbed of their possessions but were let off, police said.
The International Committee of the Red Cross (ICRC) said the abduction took place just a few hundred yards from a prison that the team had visited for a water and sanitation project.
It identified the three as Andreas Notter, a 38-year-old Swiss national; Eugenio Vagni, a 62-year-old Italian and Mary Jean Lacaba, a 37-year-old from the Philippines.
Marine Corps officers on Jolo and police said an alarm was raised when the three ICRC staff members did not arrive to catch a flight back to the mainland that they had been booked on. A search was on, they said.
"We are in a hot pursuit operation," said Julasirim Kasim, police chief in Sulu province, of which Jolo is the capital.
A military spokeswoman, Lieutenant Steffani Cacho, said troops had recovered the Red Cross vehicle and were told by locals the gunmen were taking the captives toward the mountainous interior of the island.
No claims have been made, but police said Albader Parad, a top leader of the the Abu Sayyaf, the smallest and deadliest of Muslim rebel groups operating in the Philippines, was responsible.
The group is notorious for kidnappings and is said to have about 350 members based on Jolo and the nearby island of Basilan. It has supporters among the Muslim-dominated local residents but has been largely dormant since its top leaders were killed in a series of encounters with troops in late 2006 and early 2007.
"I am appealing to the Abu Sayyaf to free those people, because they are neutral in any conflict," Gordon said. "They do not realise this but these people help them if they get wounded and get them out of the conflict areas."
A spokeswoman for President Gloria Macapagal Arroyo said the military and the police had been ordered to pursue the kidnappers and "make sure the victims are rescued unharmed, as their safety is always the utmost concern."
The Abu Sayyaf has twice attacked luxury beach resorts and taken away tourists, including Westerners. They have held them for months at a time and secured large ransoms for their release.
In 2000, Abu Sayyaf militants kidnapped over a dozen Western tourists from a beach resort in neighbouring Malaysia and brought them to Jolo. They then took millions of dollars in ransom.
A year later, the group attacked a luxury resort in the western Philippines and took away about 20 tourists and hotel workers. Most were freed after payment of ransom but three were beheaded, including an American, and an American missionary was killed in a shootout between the kidnappers and the army.
The Abu Sayyaf has also been blamed for the bombing of a ferry near Manila Bay in 2004 that killed 100 people, the worst terrorist attack in the Philippines.