a 1/8" stereo to RCA should work. the thing to remember is that the output IS NOT A LINE LEVEL OUTPUT. this means you will also have to adjust the sound level on the Mac as well as your stereo to get optimum sound. it is recommended that you set the Macs sound to 2/3 or 3/4 full, then use the stereos sound control to get the desired level. if you set the Mac too high it can cause distortion in the signal that is being sent to the stereo. if this doesn't make sense, imagine a set of headphones. if plugged in to the jack and set real high, the headphones are blasting and the quality of the audio is degraded.
if you have the levels adjusted correctly and it still sounds bad:
1) check to make sure the plug is firmly inserted into the Mac.
2) try a different cable. the 1/8" mini to RCA cables tend to short over time, so you may have better luck with a new one.
3) not all cables are made equal. if sound quality is important, don't go cheap. yeah, anything will work, but over time, cheaper cables will go bad much sooner with constant use.
as for the digital output. the jack DOES DO double duty. in addition to a 1/8" stereo analog output it has an 1/8" mini Toslink (optical) output. there are cables available that will connect this jack on the Mac with an Optical input on a Digital receiver. if you do not have an Optical input but only Coaxial Digital inputs, there are convertors, although expensive, that you can use. unless you plan to really use the surround sound from DVDs, it's not worth paying for this particular setup.
hope this helps.