Organisers claimed more than 100,000 anti-government protesters had gathered by Friday evening, though police said there were only 14,000 people present.
The protesters from the so-called People's Alliance for Democracy (PAD) claimed victory as their rally continued into the evening.
"This is victory for PAD and the peaceful methods we have employed without any bloodshed or clashes," Suriyasai Katasila, PAD's spokesman said in a speech.
"Samak's government must resign because it has totally failed to solve economic problems, it has interfered in the legal process, has helped (former prime minister) Thaksin's return to Thailand and tried to change the constitution," he said.
About 5,000 police had held protesters off for more than three hours on Friday afternoon, before a group finally pressed past police anti-riot shields to reach gates surrounding the prime minister's compound.
Security forces did not try to force them back, and more protesters soon followed, singing songs in support of King Bhumibol Adulyadej.
The PAD accused Samak of acting as a proxy for ousted prime minister Thaksin Shinawatra, who was toppled by royalist generals in a 2006 coup.
The PAD spearheaded protests against Thaksin in the months leading to the coup, raising fears of a new military intervention.
Draped in royal flags and symbols, the group is led by an unusual collection of activists, including a media baron and a retired general who now leads a sect of extremist Buddhist monks.
They consistently invoke imagery tied to Thailand's revered King Bhumibol Adulyadej, wearing yellow scarves, shirts and headbands – a colour associated with the day of the week when the king was born.
Their banners carry slogans such as "We fight for the king and our Thailand," and "Get out of Thailand, Thaksin."
Police in riot gear stood guard around Government House, waiting with teargas and water cannon, but a government spokesman said they would only be used as a last resort.
Nuthawut Saikua also told reporters Samak would not resign, saying: "This is a government of the people."
Samak moved his meetings for the day to the foreign ministry, while most Government House staffers were sent home.
The protesters have blocked a road near Government House for nearly four weeks, raising doubts about Thailand's political stability and battering the Thai stock market, which has dropped 15 per cent since the protests began.
The protesters represent an elite slice of Bangkok, drawing support from the middle class who felt alienated by Thaksin, and from Thailand's traditional power centres close to the palace and the military.