This article is from the Car Audio FAQ, by Ian D. Bjorhovde (firstname.lastname@example.org) with numerous contributions by others.
Problems occur (in everyday operation) when distortion is fed to a
speaker. This occurs MUCH more often when you are dealing with an
underpowered system - typically the owner will turn up the volume too
much or set the amplifier gains too high to try and get more volume
from the system. These introduce distortion to the signal - this will
destroy *any* speaker. ((see Section 5.22).)
When a speaker is overpowered, however, it is not nearly as common to
have these kind of problems, so speakers aren't blown as much. Of
course, it is certainly possible to destroy a speaker (thermally) by
overpowering it, but you'll have a pretty hard time doing this on your
own, especially with standard car audio amplifiers.