The singer revealed that the Syco boss asked him to work with contestants from the talent show on his label, but he rejected the offer.
"Simon Cowell may be my manager but I'm not going to work with any of the musicians from The X Factor," Labrinth told the Daily Star.
"Simon did say a few things to me about working with some of the artists but he knows I'm not into the idea. I'm fine about always being linked to Simon because that's the way the music industry works."
Labrinth's new single 'Express Yourself' will be officially released in the UK on June 3.
The track is the fourth to be lifted from his debut album Electronic Earth.
