The creator of the ITV1 show, who still serves as executive producer on the UK series, now considers Nicole Scherzinger as head judge following her successful stint on the show, The Sun reports.
Cowell is said to be "constantly calling" the former Pussycat Doll with "flattering" comments, hinting that she will be invited back for next year's show.
"Simon's irritation with Gary is well known. He's given him the Over 25s - two of whom have been binned," a source told the paper.
"And he's been refusing to call Gary. Instead he's been buttering up Nicole on the phone - saying her comments are best and don't sound scripted like Gary's."
It was previously reported that Mel B - currently a judge on X Factor Australia - would replace Scherzinger next year.
Back in April, it was claimed in Cowell's biography that he wanted to sack Barlow as he was "too scripted" and "a bit stiff".
