The first live performances were mired in controversy after Poole and Rylan Clark were left in the bottom two this weekend.
With Gary Barlow and Tulisa Contostavlos voting to save Poole and Nicole Scherzinger voting to save her act Clark, the final decision was left to Louis Walsh.
However, after Walsh initially said that he wanted to keep Poole in the competition, he changed his mind and took the vote to deadlock - which resulted in Poole leaving the competition.
Since Sunday, viewers and audience members have questioned why producer Holloway was seen speaking to Walsh prior to the controversial decision.
One attendee claimed he overheard Walsh telling Contostavlos he had been instructed to go to deadlock.
However, Holloway told The Sun that he was not involved in any fixing on the show.
"We never tell the judges what to do," he said, adding that the conversation was simply to let Walsh know he would be voting last. Holloway also said it would be "impossible" for anyone to overhear the conversation over the music on set.
Holloway additionally told The Mirror: "We regularly chat to the judges during the show, they don't wear earpieces like Dermot [O'Leary], so we have to speak to them on anything from timings to running order changes.
"On Sunday I was telling Louis the order the judges would vote in. We don't tell judges how to vote."
