The former Fame Academy coach told Sky News that Clark is "unbelievably fame-hungry".
"You have had in the many years of X Factor many people desperate for success. But this is the first person I have ever seen that would dress up and cross the street to be at the opening of an envelope," he said.
"Rylan is the man who has had the greatest songwriter of our generation told he is deaf by someone who manages Jedward. Every time you think X Factor is a talent show, something like this happens to remind us that it's not."
Grant added that talented contestants always leave before acts such as Clark.
"If you are a circus act and you are somebody that people are going to look at because it is car crash TV and on the other hand you have the really good singer, the really good singer goes home because of the circus act. It's not a talent show."
Clark has been surrounded by controversy since he escaped elimination and Carolynne Poole was sent home in the first week of the live shows.
He was reportedly evicted from the Corinthia Hotel along with fellow contestant Lucy Spraggan after a drunken night out yesterday.
