Tower + Gate is a lot of mana (both up front and committed to future actions), preventing you from using higher cost attack spells, especially if you are putting out a relatively expensive arcane creature every other round. I think the tower is more powerful in a solo or buddy Wizard than one that uses the Gate, since you'll have enough mana to take full advantage of the extra attack spell per round. But I'm no expert, and not much of a Wizard player in general (e.g. have only used him 4 or 5 times total).
Also, did you immediately summon a second (and third) Tower when the first (and second) were destroyed? One of the biggest complaints about the Tower is that it's a major threat that takes resources to take down, and when you do take it down, another one gets cast immediately, then another, then another - and each time, it can cast an attack spell upon being cast. Hence, a commonly proposed nerf is to make it Epic.
I see the Priestess/Priest/Paladin becoming the best counter to the Wizard's Tower, since White Cloak Knight is immune to both it and Arcane Zap, and Brogan already has the right amount of Piercing for dealing with a Tower.