In May, the singer signed a $15m (£9.6m) deal to join the Fox series after reportedly initially asking for $20m (£12.7m).
However, ever since the show began holding auditions in Austin for its second season, reports have emerged suggesting that Spears is being driven to a meltdown by her judging role.
"He [Cowell] wonders if hiring her was a mistake," a source told Us Weekly. "I don't know how she's going to handle the live shows!"
A witness insisted that the 30-year-old is a poor addition to the series, saying that she refuses to criticise even the worst contestants.
"[Spears] always said nice things, even telling one mediocre singer he could be 'bigger than Justin Bieber'. Simon was shocked!" the insider said.
However, an X Factor USA source told the website that Cowell is perfectly happy with Spears.
"Simon doesn't regret hiring Britney, and as a judge, she had strong opinions," the source said.
