The best quality you'll get is to use the m2ts container straight off of the Blu-ray rip. Inside this container will be the video (in H.264 AVC, MPEG-2 or VC-1) and audio (usually in DTS/DTS-HD and sometimes AC3 Dolby Digital or True-HD). The reason this is the best is because you don't have to transcode it (i.e., reencode). On Macs, a few apps can play these files. Plex and XBMC are the best with fewest dropped frames and transcoding of DTS-HD or True-HD down to DTS or DD on-the-fly so that it can be sent out the optical port to your receiver. VLC can also play Blu-ray rips, but it's not as efficient.
The next best quality is to use Handbrake to transcode to any of a number of containers, including MP4 and MKV. This requires a transcode of the Blu-ray video (usually to H.264) and you can get AC3 Dolby Digital, but not DTS, 5.1 audio for passthrough of the optical port.