Two question come to mind when I consider harassing the mana crystal.
How important is it to my strategy to deny my opponent mana?
If I am playing a wizard employing a mana denial strategy then the answer is very important. My goal is to disrupt my opponents strategy through resource denial. If I am playing a beastmaster attempting to swarm my opponent with creatures then it's not very important.
Are those mana crystals important enough to my opponent's strategy to make destroying them worth my effort as opposed to furthering my own strategy?
That is obviously the more difficult question as it is dependent on the situation at hand. To make the right choice requires keen insight into your opponents strategy, or maybe just some good luck.
I believe that the efficiency with which you can destroy the mana crystal, while important is of secondary concern to the necessity of doing it.
I don't even think that it is worth it if your goal is mana denial. If you are investing so much into mana denial that you are both playing mana denial cards AND going after their mana generators then you are basically settling in for a long game instead of just going for the win faster. I think that is just too much of an investment and you are basically spending a lot of your own resources in an attempt to deny your opponent theirs.
I'm not sold on the whole "spending resources to deny your opponent resources" strategy. Things like Mana Siphon are nice, but it's really expensive, which means you are denying YOURSELF a good deal of mana in order to deny your opponent some mana...at a much slower pace than the 12 you coughed up right off the bat to cast the Mana Siphon. Going after Mana Crystals/Flowers are pretty much the same IMO. You are spending resources to deny resources. Are you better off dedicating a creature to attacking a Mana Crystal...or the opposing mage? I'd say the opposing mage, because that's how you win the game. Are you better off using an attack spell to go after a Mana Flower that may or may not kill it, or using that same spell on the opposing mage? Again, it's a no brainer for me: hit the enemy mage and put them on the defensive.
When it comes to the mana denial strategy, I think the best way to go is with passive denial. Things like Mana Leeches and Staff of Arcanum are really nice because you are generating offense at the same time you are denying your opponent mana. Essence Drain and Pacify can be nice because those mana costs can really add up and they aren't that expensive to cast. Things like Cloak of Suppression and the Obelisk are nice against creature swarms, but not otherwise. Mana Drain is decent because you at least have the potential to get a large chunk of your mana spent back. Mana Siphon is ok as long as you get it out really early, otherwise you may not deny your opponent more mana than you spent yourself to cast it.
But going after Mana Crystals/Flowers? Not a fan. You're better off just dedicating those attacks at the enemy mage and trying to win the game.