The 28-year-old musician is now a chart-topping superstar, but he has revealed his "quirky" looks as a child made growing up difficult, as he was "instantly ripped into" by his classmates and "cried every day" throughout his school life.
He said: "I hated primary school with a passion; I cried every day. I'm ginger so I was instantly ripped into from the day I started school - ginger, had a stutter, and wore huge glasses ... just a bit odd."
The 'Castle on the Hill' hitmaker - who is now married to Cherry Seaborn - beat the bullies by making music, and now doesn't mind the way he looks.
He added: "But then as I got older I kind of loved it, I've always kind of looked a bit quirky. I never had much luck with girls and stuff like that, it was always I kind of looked a bit weird."
On top of being bullied for his looks, Ed says he wasn't "cool" in school because he "couldn't play sport", and didn't realise his musical talent until he was older.
Speaking in an interview with DJ Nihal and rapper Dave for anti-racism campaign group Love Music Hate Racism, he said: "I got to high school, started playing guitar and joined a band. Music is one of those things that can give you confidence.
"You're suddenly like 'wow, I can actually do something well'. Before I picked up a guitar I did nothing.
"I went to a primary school that was quite sporty and I couldn't play sport.That was how you were cool - if you were good at football you were cool and I wasn't good at football.
"But then when I joined high school it was such a melting pot of all different things and I started playing the guitar and suddenly started fitting in."