You know, a large difference in size of backups indicates that you have a lot of tables outside of what PHPBB backs up. The admin panel utility ONLY backs up "standard" PHPBB tables (there are about 30 of them), while some MODs add several more. The only way to get ALL of them is through phpmyadmin or one of the command-line tools, like mysqldump (which is what I use on a cron job).
There are MODs that are known speed killers, like the "birthday MOD". And, if the "stop words" list in the languages directory is not right for your forum, you could have a huge search table, containing too-common words.
Knowledge Base article on backing up and restoring databases
MOD to rebuild search tables
(so you don't have to back up the search tables anymore)
Knowledge Base article on Decreasing the size of your search tables