He made the comment after Clark had performed his promised Egyptian-themed rendition of Spandau Ballet's 'Gold'.
Louis Walsh was full of praise, calling Clark "amazing", and Tulisa Contostavlos said: "I don't think this competition would be the same without you."
However, Barlow commented: "I really was having fun until you started singing tonight... I'm embarrassed to be sat here."
He added: "I said 'no' at the first audition, which means you [Contostavlos] should be ashamed of yourself, you [Walsh] should be ashamed of yourself."
He then criticised Nicole Scherzinger for picking Clark when there had been "five amazing singers" for her to pick from.
Telling the Take That singer not to be "an old, grumpy fart", Scherzinger teased him by referring to the band's risqué video for old track 'Do What You Like'.
Clark admitted that he did not have the "best voice" in the competition, but insisted that there was a "lot more to the show than the voice" and declared: "This is what I want to do."
