For the TV, that depends on what you have, exactly. If it has a fixed output only, you need to use the speaker's control. If there is a headphone or other variable output, the TV volume will work. And frankly, you should make sure it has any output at all, first.
The Mini will certainly control the volume with an analog output, so you can use the Apple remote for that.