The PR professional was cast in the show for its first series last year and returned for its second and third runs on E4.
"I'm quitting," Taylor told Now. "It's a big gamble but I think I might have outgrown the show.
"Originally the producers only wanted me as a peripheral character and that's all I wanted to contribute.
"I could wind up having absolutely everything or I could end up with nothing. But I've become lazy and comfortable on the show."
Asked if he was concerned about being branded 'a reality TV star', he added: "I worry about it every day. It's one of my main motivations for not doing a reality show anymore.
"I often look at what I've done and think, 'S**t, did I completely sell out?' Staying on Made in Chelsea and bitching about the colour of someone's dress has started to become unimportant to me.
"I'm 26 - the show works better with people under 24. It becomes harder to do these shows as you get older. I started thinking I need to do something credible that I'm proud of."
Taylor said that he was "passionate" about film and hoped for a career in acting or directing.
"I plan to study at the Lee Strasberg Theatre And Film Institute [in New York], which teaches method acting," he said.
"It's expensive and you have to have some talent. I don't think they've accepted anyone from reality TV before."
