Interesting, I've been pondering a build similar to this too. How have you fared against other standard builds? - Heal/Temple, Swarm (as you say), Beatdown, Control etc.
I was thinking on a possible wall build too and using a lot more cheap Teleport traps. I might even try an Enfeeble or two and Gorgon Archers or Grimson to cover the rest of the Arena.
Not cheap book or mana-wise though.