Great question actually and the answer depends upon the setting shown in my screenshot. If you have “Change with buttons” turned on, which it sounds like you do, the side buttons only control ringer and alert volume. In this case, the control center volume will change the system volume for all sounds besides ringer and alerts.
View attachment 761798
I have "Change with Buttons" on. While playing music, the side buttons control music volume; when not playing music or other media, the side buttons control ringer volume. I apologize if I'm missing something.