Its most likely because people are finding them soon after they pop.
The way it seems to work is destroying a mirror stunts the growth of the beastmen in that region for a period of time. It doesnt appear to be the number of mirrors that matters, rather just weather or not one has been destroyed recently.
Destroying a mirror before the last mini patch would cause the number of enemy forces to go down. Now, it just makes it temporarily stop rising.
XI Wiki

