Among the many things Saruman does in The Lord of the Rings, not dealing with the Ents when he had the chance is considered one of the biggest plot holes in The Lord of the Rings: The Two Towers.

During the attack on Isengard, they storm through the Wizards ... They still have voices, and can speak with the Ents, but they have become queer and wild. Dangerous. I should be terrified ...