The ruling, which ended years of court battles, said it was discriminatory to consider parents' marital status. The children were mostly born to Filipina entertainers working in Japan.
"If we look at the significance of nationality in guaranteeing basic human rights, we cannot afford to ignore discrimination against children," presiding judge Niro Shimada said.
In an earlier era, "there might have been compelling reasons for parents' marriages to indicate their children's close ties with Japan."
But such rules "do not necessarily apply to current family situations," he said in the ruling.
Twelve of the 15 justices including Shimada voted to overturn the lower court ruling.
Ten children of Filipina mothers had filed suit seeking Japanese nationality because Japanese men – who were married to other women – had acknowledged they were their fathers.
"I want to make my dream come true, a dream only Japanese can achieve," said 10-year-old Masami, one of the 10 children. "I want to become a police officer."
Masami and nine other children, even though they were born in Japan and speak only Japanese, were not granted nationality because the fathers acknowledged them only after they were born.
Previously under Japanese law, the father had to confirm the child as his before an out-of-wedlock birth.
The mother of another plaintiff, 14-year-old Jurie-Anne, said, "I didn't think nationality would matter for my daughter when she was a baby."
"But from around age five, she kept asking me why she wasn't Japanese. I was always sorry for her that I wasn't married."
With tears welling up in her eyes, Jurie-Anne said, "I know I've been such a handful for my mother, but I want to say to her today, 'thank you'."
According to official statistics, approximately 2,800 children born out of wedlock from foreign mothers reside in Japan, more than 2,000 of whom have Japanese fathers.
Japan, which largely regards itself as ethnically homogeneous, bases nationality on blood ties. It has rejected the idea of large-scale immigration even though it has one of the world's lowest birthrates.
Genichi Yamaguchi, a lawyer for one of the plaintiffs, hailed the court decision as landmark.
"This ruling is very meaningful because it recognised, in a straightforward way, that there was discrimination between legitimate and illegitimate children," Yamaguchi told a news conference.
"I hope that a case with this significance, about whether or not to grant nationality, will give hope to many children in similar situations," he said.
Chief government spokesman Nobutaka Machimura said the government would come up with measures to comply with the decision.
"It is an important verdict and we need to accept it solemnly," Machimura told reporters.
"It appears to uphold equality under the law and in that sense I think that it is important."
While the plaintiffs are already living in Japan, they lacked Japanese citizenship, putting their status into legal limbo. They have Philippine nationality from their mothers.
In Japan, nationality was historically passed down through the paternal line, with a child obtaining Japanese citizenship only if his or her father was Japanese.
Japan changed the law effective from 1985 to permit either a mother or father to pass on nationality, but imposed the condition that the father must recognise paternity while the child was still in the womb.
The Tokyo High Court had last year refused to grant the plaintiffs Japanese nationality, saying that illegitimate passport seekers could abuse the law if fathers gave nationality to children already born out of wedlock.