Good question!
Since the Mage controlling charm is not attacking, rather the wall is, then the charm does not get destroyed. The source of the attack originates from the wall. The force push is merely an incantation that doesn't count as attacking or damage the charmed creature.
Hope that helps!