Could you explain how they're not working exactly
I'm having no trouble with them; I tried uploading a torrent file on a fresh install of beta 5. I'm prompted to download the file upon clicking the attachment link (the normal behavior of my browser). I then open the torrent after download in my default client
Are you referring to the option to "open" rather than save being absent? That's really your browser's choice. If phpbb is prompting the browser to force a download, in my quick test only firefox (2.0) honored that (both ie7 and opera 9 gave the 'open' option as well)
I have tested several browsers on both windows and linux and all only allow you the option to download not open the torrent. With the exception of IE.
This was not the case with beta 4