Specifically, OS X outputs a PCM audio stream detailed here. The only exception to this is if you are passing through a Dolby Digital (AC3) or DTS audio stream from a media player in which case the PCM stream is superseded.
In the end, the only volume control you're going to get will be altering the audio data before it's sent over HDMI which can cause undesirable side effects. With VLC for instance yes you can alter the volume which in turn simply applies gain to the audio data, but eventually that will just lead to distortion and clipping. Using a 3rd party app to lower the volume simply compresses the waveform which in turn reduces the resolution of what's actually being played.
In conclusion, if you want the best possible sound do not use software to attempt to control the volume of a PCM audio stream. You want to apply volume changes after the data has been converted to a analog signal (what you actually can hear) which means using the TV or audio amplifier/receiver volume control.
Well put.