I had the same problem. I deleted the /install and /contrib directories however I continued to get this error.
The problem was that the /contrib folder was still on the server - even though I'd deleted it with my ftp client the folder (not the contents) remained. I removed the folder manually and the error went away. The upgrade from 2.0.RC2 to 2.0.4 is complete.