Following up on this, I think I found the answer to my question. My Apple TV is in the same room as my computer. I just found that the remote for that can be used to turn the volume up and down on the computer. So restarting it might not be doing it. Seems like it's when I use the remote for the Apple TV, it's also controlling the Mac Mini's volume.