This was almost a deal breaker for me. In the end I thought about it, & for my personal use, I'll have 3 volume levels, at most, for my needs; Low Noise environment (where I need ambient sounds around me, say for cycling, running along streets etc), Normal environment (like at gym or quiet work surroundings) & Noisy environment (when around loud equipment at work etc).
So, all I'd need to remember are 3 volume levels for Siri.
After using these things for a few days, telling Siri to "Set volume 40%/75%/90%" seem to be all I need to handle every situation I'm in.
It's turned out to be not as big a deal as I made it out to be.
The only awkward situation I can think of is needing to adjust the volume on crowded public transport, but since I rarely use public transport, I won't need to worry about that, but even then, I'd have the phone in my pocket, or in my hand anyway, so I'd bypass Siri..