"There were nearly 1,000 inmates in that prison. A large number of them have escaped," deputy justice minister Mohammad Qasim Hashimzai told AFP.
He declined to give a precise number, saying officials were currently "counting the remaining prisoners" in the main jail in Kandahar.
"According to the latest information I've received from the jail, a couple of hundred prisoners are still remaining," he added.
Afghan security forces have launched a massive hunt to recapture the inmates who fled in Friday's brazen attack, Hashimzai said.
"A massive operation is under way to find the escaped inmates. The Afghan security forces are searching for them within the city and along the main and secondary roads," he told AFP.
Taliban rebels blasted into the prison using a suicide car bomb and guns late Friday, freeing militant prisoners and killing at least two guards, officials said.
The raid is a blow to President Hamid Karzai, coming one day after world donors pledged 20 billion dollars to rebuild Afghanistan at a conference in Paris but also called on him to strengthen the rule of law.
The attack under cover of darkness ripped through the front entrance of the jail. Residents heard small arms fire and blasts from rockets.
"I was at home. I heard a big explosion and then there was an exchange of fire, machine-guns and rockets for a quite a while," said Abdul Bari, a resident who lives near the jail.
"I think most of the prisoners escaped," he told AFP.
Kandahar jail was the scene of a mass hunger strike by hundreds of inmates in May, during which 47 of the prisoners sewed their lips shut after complaining they had been tortured and denied fair trials.
The Taliban have been battling the government of President Karzai since they were toppled from power in a US-led operation for failing to hand over Al-Qaeda chief Osama bin Laden in 2001.