"How much are we talking about due to the fuse?"

That has to do with the type of fuse and the current drawn by the speaker since it is dependent upon the low thermal mass of the fuse element. The resistance increases with temperature. All it takes is a part of that element to get warm. A large enough capacitor across it will keep that from happening. In fact if you want to risk it to save a few bucks you can get a super high capacitance with low ESR and a lower voltage rating. Of course if the fuse blows then the cap will explode.

"And how much distortion will the speaker itself create at that frequency/ level? "

People who care and buy expensive speakers usually get quite low distortion from them. My Boston Acoustics A-150s IIRC were rated 0.7 % which is phenomenally low for a speaker. Really, most of the cheap junk is more like 3 - 5 % at any decent listening level and it is mostly odd order harmonics. However with a decent crossover that is just harmonic distortion, the fuse will also create intermodulation distortion because all frequencies go through it. I would say that is much more audible.

Lower power ratings don't really help because you would use a commensurately lower value fuse and its thermal mass will be less. A slow blow is probably better but then it comes down to how much abuse to which you are willing to subject your woofers. Some of them are quite expensive.