Woodward's evidence is from his own Plan of Attack and from the 9/11 Commission Report. Former counterterrorism director Richard A. Clarke's Against All Enemies provides corroboration that Rumsfeld was fixated on Iraq from the beginning. Clarke implies that driving Rumsfeld's focus was Wolfowitz:
On the morning of the 12th DOD's focus was already beginning to shift from al Qaeda. CIA was explicit now that al Qaeda was guilty of the attacks, but Paul Wolfowitz, Rumsfeld's deputy, was not yet persuaded. It was too sophisticated and complicated an operation, he said, for a terrorist group to have pulled off by itself, without a state sponsor--Iraq must have been helping them...This account is consistent with Woodward's, which recounts that on Sept. 15, "Bush rejected going after Iraq," but that on Nov. 21, Bush likely did ask Rumsfeld for contingency plans.
By the afternoon on Wednesday, Secretary Rumsfeld was talking about broadening the objectives of our response and "getting Iraq." Secretary Powell pushed back, urging a focus on al Qaeda...
Later in the day, Secretary Rumsfeld complained that there were no decent targets for bombing in Afghanistan and that we should consider bombing Iraq, which, he said, had better targets. At first I thought Rumsfeld was joking. But he was serious and the President did not reject out of hand the idea of attacking Iraq. Instead, he noted that what we needed to do with Iraq was change the government, not just hit it with more cruise missiles, as Rumsfeld had implied.
On the 12th and 13th the discussions wandered: what was our objective, who was the enemy, was our reaction to be a war on terrorism in general or al Qaeda in specific? If it was all terrorism we would fight, did we have to attack the anti-government forces in Colombia's jungles too? Gradually, the obvious prevailed: we would go to war with al Qaeda and the Taliban. The compromise consensus, however, was that the struggle against al Qaeda and the Taliban would be the first stage in a broader war on terrorism. It was also clear that there would be a second stage (pp 30-31).