Bandwidth is basically the amount of data traveling from one place to another. This is different than latency, which is how fast it takes to get from one end to the other (pings test latency).
Bandwidth is expensive because of the massive volume of stuff that is going all over the internet. Because wires from place to place can only transfer a certain amount of information at any given time (10Mbps, 100Mbps, 1Gbps, etc.) it means that the more bandwidth that anyone is using affects everyone else, at some point. The reason that hosts limit bandwidth is that they have a connection of finite speed to the server box itself. Bandwidth limiting prevents one site from hogging all of the bandwidth for the whole box. 8)
Disclaimer: All of the above is AFAIK, and may be explained wrong. I encourage anyone to correct or add information to this!