I have had one spammer sign up to UKThrash after applying the MOD, however this was probably a human spammer as there is no conceivable way that the MOD can be compromised other than cracking the SHA1 hash of the answer.)
It is highly likely it was just a bot, as long as the software were built to recognise mathematical CAPTCHA's, it would have just done that on your site.
The Default (phpBB3 supplied) CAPTCHA is actually quite complex to crack, however, there is no CAPTCHA that is impossible to crack. The default CAPTCHA took quite a while to be cracked by applications that cost $400+ USD -- over 2 years. This kind of Mathematical CAPTCHA is actually far easier to crack, any script kiddie with 40 minutes or less of time to spare could easily create a quick script capable of cracking this CAPTCHA.
Why? just ask Google
Once phpBB comes out with a new CAPTCHA, it will again be months before that CAPTCHA is broken. But if phpBB were to implement a CAPTCHA such as this, it would be broken the very same day. In reality, this type of CAPTCHA (Mathematical based CAPTCHA’s) are rather weak, and are among the weakest CAPTCHA's that exist.
That said, any unique method - something that not everybody is using, something unique to your own site - you can use to deter bots is a good method.