The lawyers say they plan to travel through several major cities over the coming days, garnering support before staging a rally in the capital Islamabad on Thursday against the country's US-backed leader.
Black-suited lawyers chanting "Go Musharraf, Go!" began the journey with a noisy demonstration outside the mausoleum of the country's founding father Mohammad Ali Jinnah in this southern port city.
They were joined by supporters of ousted former premier Nawaz Sharif's Pakistan Muslim League-N, hardline religious party Jamaat-e-Islami and the party of cricket hero-turned politician Imran Khan.
"Around 4,000 people participated in the rally," police officer Abdul Haleem told AFP, adding that it was peaceful so far.
Pakistani lawyers have bitterly opposed Musharraf, who seized power in a coup nine years ago, since he tried to sack the country's outspoken chief justice, Iftikhar Muhammad Chaudhry, in March 2007.
That move sparked mass protests and the Supreme Court later overturned Chaudhry's suspension, but Musharraf struck back in November by imposing a state of emergency and deposing the judge and other senior colleagues.
But Musharraf has since been weakened by the defeat of his allies in general elections in February and the ruling coalition is set to introduce legislation that would curb his powers.
"This is the ultimate push to the falling wall of the tyrant military dictator," Rashid Razvi, president of the Sindh province High Court Bar Association, told the gathering at the mausoleum.
"We are on the verge of a victory and the new dawn will come in a matter of a few days."
The lawyers will leave Karachi in a caravan of buses and cars for the southern city of Sukkur, where lawyers from other towns and cities in Sindh and Baluchistan provinces were to converge, Razvi said.
The marchers will then head for the central city of Multan on Tuesday where Chaudhry will address a lawyers' convention. The caravan will then leave for Lahore and reach Islamabad on Thursday.
Musharraf on Saturday rebuffed calls for his resignation and also rejected claims that he was hatching a conspiracy against the ruling coalition, formed by Sharif's party and the party of slain ex-prime minister Benazir Bhutto.
"I won't resign in the current situation," he told local television channels. "I will live and die here, there is no other way. I don't have any house outside Pakistan."