The number of files has no affect as far as your board is concerned. If your host has a backup process in place, the number of files will slow down the backups. Another problem that having so many files in one directory is that some FTP programs will handle only 2000 files at a time, so you could be in trouble if you were to make a backup yourself of the attachment directory.
I have a gallery running with 82,000+ images at a host and they told me I had to remove some of the images because the host has a limit of 15,000 files. It didn't take me long to move to a different host.
There is a MOD to have attachments on a remote server, but I don't believe it will help with the attachments you presently have. I believe there has also been talk of allowing attachments in multiple directories, but I don't think anything ever came of it.
I don't know of any solution to your present problem other than to move or ask your present host if they have a plan that will allow you to have 80,000+ files.
Have you hugged someone today?