The 16-digit number is stored in a database, isn't it? The hacker would need a method to trick the ATM or whatever into thinking that it is your card that has been inserted (assuming they figured out your PIN).
It's all in the numbers.
Even a disgruntled store clerk might be able to pull it off.
Then you're talking about hacking the bank's computer system, since hacking a simple piece of plastic isn't really possible. I've heard alot of people saying how they don't trust debit cards for security reasons, but they're not any less secure than cash. Especially since, if you're carrying cash in your pocket, and you get mugged then you've just lost all your money. If you get mugged and all they get is your debit card, they either have to beat your PIN out of you or they're out of luck. I've never worried about the security of my debit or credit cards.