I used to think so as well. Then, a couple of years ago, I did a proper double-blind listening test together with a few friends (some of which consider themselves "audiophiles"). None of us could reliably tell the 256 kbps AAC or MP3 files from the uncompressed originals. Since then, I don't bother with lossless music anymore.
Proper mastering is much more important. Unfortunately many modern recordings are mastered miserably, e.g. applying too much dynamic range compression. I applaud Apple for trying to do something about that as part of their "Mastered for iTunes" best practices (although I wish they were more forceful about it).
I consider myself an "audiophile"
The difference between lossless formats (WAV, AIFF) and lossy formats (with bitrates 128, 256, 320 kbps and etc.) could be none, or day & night. Can you guess why? HARDWARE!!
If you listen to music through consumer-grade speakers, even the most expensive top-of-the-line models, there's no quality difference beyond 320kbps, perhaps even 256kbps. Hence, your double-blind observation.
But if you purchase a pair of "studio monitors" (professional reference speakers used for studio recording) and an "audio interface" capable of producing "balanced sound output" to your studio monitors through "balanced cables", then the difference will be midnight, and noon
A decent setup like this costs between $2,000-20,000
You should try it, you'll never go back!