I'm liking it. I've tried toying around with an imp swarm build with Pentagram, but I agree that it just doesn't work and isn't worth it.

There's really only a few suggestions I have (without playtesting it first). The first is I would find space for Adramelech's Torment. Especially with the Wildfire Imps, I feel like you definitely want one. I'd also include at least one Explode, even in place of a Dissolve. It's expensive, but I still feel like one is worth it in any Warlock build. And the last, and probably most important suggestion I have, is you probably want some number of Acid Balls and/or Rusts. Rust would probably be the better choice here since Sersiryx can cast it for you, but armor-stacking mages or Grizzlies could give you trouble.

Otherwise, it looks pretty great! I'll have to give it a run and see if I can come up with anything else.