Since no one else is jumping in, I'll hazard a guess. For reference, I have a very similar setup to you--a Mini hooked via toslink to an Onkyo 7.1 deck with the full complement of speakers.
The quality of sound you get from a DVD is going to depend a lot on how it's encoded (this is assuming that you correctly configured DVD Player to output the raw bitstream to your stereo, which takes the Mac itself out of the equation, as explained before).
CDs are straightforward--you can have good or bad original masters, but the CD itself is uncompressed (PCM) 44.1kHz, 16-bit stereo. Decent if not stellar resolution (if you have VERY picky ears), no compression artifacts.
Video DVDs, on the other hand (I assume we're talking about a video of an opera, not a DVD-Audio disc) can have a
bunch of different formats, ranging from uncompressed audio about the same as a CD, to higher-quality 96kHz, 24-bit uncompressed, to the cinema-style audio formats (DTS, Dolby AC-3, or MP2), which are all compressed and can range wildly in both quality and number of channels.
I've got three guesses as to why your opera DVD isn't sounding as good as a CD.
One is that the audio on it is encoded in one of the lower-fidelity compressed multichannel formats, and your ears are sensitive enough to pick up the compression artifacts versus the uncompressed CD audio. You can see what format the audio is in on the front panel of your Onkyo (it should have a little "DTS" or "DD" or "PCM" logo somewhere), and some software will also show you the actual bitrate and number of channels.
More likely is my second guess; that the audio is encoded in one of the multichannel formats, as above, and that your Onkyo is set to mix it up to a higher-channel surround format. In which case, the upmix may be assuming it's a movie soundtrack, which doesn't sound as good with music. If I set my Onkyo to upmix to DTS 6.1 channels, the audio has a lot of dimension (which can be good if you're watching a movie), but sounds relatively "thin" if you're just listening to music and I wouldn't want to listen to classical in that mode.
If that's the case, you could fix it by swapping through the "Music" modes on the Onkyo (which you should experiment with anyway if you haven't already, regardless of what's going on in this case). I usually find all-channel stereo to sound the best for music.
The third possibility is that the DVD's audio just wasn't recorded as well as a good CD--they might have used a multi-channel cinema-style recording setup, which might better mimic the experience of sitting in the audience of a performance, but in absolute terms just isn't as good as a simple stereo audio-only mastering.
My personal guess is #3, but even if it is fiddling with the different music-specific audio modes available (again, assuming you haven't) might make a significant difference. Even on a straight stereo CD, if you're trying to play it as multi-channel audio, you might get some surround effects, but the music will probably sound better on all-channel stereo or one of the front-channel-only modes.
