Uhh not true at all. Different speakers have different power ratings because they have different strength magnets. You apply a weak signal to a powerful magnet and the speaker won't work at all. You apply a powerful signal to a weak magnet and you blow the speaker.
It sounds to me like somehow your speaker got damaged/blown by running it at max volume. If it sounds ok at very low volume but muffled/distorted at higher volumes, then its not a connection issue.