James told Spy that he wants the BBC fantasy drama to end "on a high", adding that he would only continue on the show "if everyone still has the passion for it".
"I think we're in a place of taking it year-by-year at the moment," said the actor, who plays King Arthur. "We've got two weeks left of filming [on series five], we'll all get a break and see how we feel."
The 28-year-old added that he would be "happy to move on" from Merlin if the series does not return for a sixth run, "because it's been a fantastic experience".
"We probably need to sit down and see if everyone still has the passion for it," he suggested. "When the passion goes, that's when you need to stop."
A trilogy of Merlin films - first announced at this year's San Diego Comic-Con - is also still in the early stages of development, James confirmed.
"I don't think it's moved forward from what they were talking about at Comic-Con," he claimed "That again boils down to what's on the table, what they want to do with the story.
"Again, if the passion's there for it, I'm sure it will come about. There are a few elements that need to be in the right place for that to happen. Again, we shall see. Watch this space."
Merlin returns for a fifth series this Saturday (October 6) at 7.45pm on BBC One.
