This article is from the Car Audio FAQ, by Ian D. Bjorhovde (email@example.com) with numerous contributions by others.
Distortion is hard on speakers for two reasons.
Reason 1: Distortion causes the power spectrum to shift upwards in
frequency. A bass note, when distorted, will have lots of high
frequency energy. This will cause mid-ranges and tweeters to fry, if
the amplifier is operating full range. It doesn't harm woofers,
Reason 2: Distortion causes the average power to be much higher.
Typically, a music signal that never clips has an average power level of
1/4 the peak power level for even the most compressed speed metal or
pop. More dynamic music will be 1/8 the peak level or less on average.
When you clip the amp hard, the average output moves up to the
full-rated output of the amp or more. The peak to average ratio can be
less than 2 to 1, with the peaks being at double the rated power of the
amp, and the average being at the rated power of the amp or higher.
Thermally, the speaker can handle the average power being 1/4 the rated
power of the amp (little to no clipping), but it will have a much harder
time with the average power being the amp's rated power or more (massive
clipping). As you might expect, this is pretty hard on the amp, too.
For transients, most speakers can handle a ton of power. But for long
term signals, the power handling is much less.