An official on the island of Sulawesi said only 20 survivors had been found so far and quoted them as saying the ferry rolled over and sank after being hit by waves of more than 5 metres (16 feet) in the early hours of Sunday.
The disaster turns the spotlight again onto the poor safety record of Indonesia's transport sector, especially ferries which link the main islands in the archipelago, the world's largest.
Accidents are common largely because of under-investment in infrastructure and a tendency to overload ferries.
An official denied the ferry that went down on Sunday had been overloaded.
Rustam Pakaya, head of the health ministry's crisis centre, said on Monday that six deaths had been confirmed in the accident by Sunday evening.
"There is a greater possibility that many more died than we expected because it happened when they were sleeping," Taufik Bulu, head of maritime safety in the port of Pare-Pare, said by telephone.
There were 250 passengers and 17 crew on board the Tertai Prima when it sank.
One survivor, Daeng Gassing, who spoke as several sobbing relatives comforted him, said that he had managed to scramble onto a life raft after clinging to a piece of wood and had pulled to safety five others but his son and father-in-law were missing.
"I grabbed my son on my back and swam to a piece of wood, but my son disappeared after being hit by a big wave," he said.
The ship's captain, who flung himself into the sea, told police that the crew could only launch two flares before the vessel went down, Metro TV reported.
The ferry was travelling from Pare-Pare on the west coast of Sulawesi to the city of Samarinda on Indonesia's side of Borneo island when it ran into bad weather.
In Pare-Pare, a Reuters photographer saw about 100 people thronging around a list of passengers.
There had been warnings of heavy seas in several parts of Indonesia in recent days and the transport minister said there had been a tropical cyclone in the area at the time.
Indonesia has come under pressure in recent years to improve its transportation sector following several serious accidents.
On December 30, 2006, a ferry with at least 600 onboard sank in a storm after it travelled between Borneo and Java. About 250 survivors were eventually found in the days after the accident.
A couple of months later at least 42 people were killed when fire broke out aboard a ferry that was heading from Jakarta to Bangka island off Sumatra.
There have also been a series of accidents involving Indonesian airlines, prompting the European Union to ban Indonesian carriers from its airspace.