The star - who won the 2009 series of the singing competition - admitted that he gets annoyed with people who slate the show and is grateful for the opportunities it has given him.
"I'm always proud of where I came from on X Factor. Without that nothing else would've happened," McElderry told BBC Radio 2's Patrick Kielty.
"I've seen people who slate it and it really annoys me. It's great that people are having success but they used that show and got 20 million people watching them and that's how they're selling albums.
"Without that show I wouldn't have been able to do Popstar to Operastar or be on to this new album. I never try to get rid of it, I just try and progress from it."
The singer is currently preparing to release his fourth studio album Here's What I Believe and will distribute the title track - which he co-wrote - as the set's lead single.
"It's one of the first songs I've written for any of my albums. I've never tried songwriting before and it's one of my first attempts. I hope people like it because I can't blame it on somebody else now!" McElderry joked.
"I was really scared to do it. I think it was a confidence thing and I never wanted to do it but my new manager persuaded me to try it.
"The first song I wrote, 'What Have I Done', got picked up for the album straight away. Everybody at the label liked it and that gave me a bit of confidence. I've got three tracks on the album which is quite a big deal considering I've not written before."
Joe McElderry's new album Here's What I Believe will be released on September 10.
