Just leaving it here for anyone else who might be curious about this. After much efforts I was still unable to boot from my thumb drive because it said the OS was too old even though my mac mini came with 11.0.1 and I was trying to install 11.0.1 & 11.1 (both failed). Turns out there is no way to go past a certain version.
So I used the tutorial of MrMackintosh:
https://mrmacintosh.com/category/downgrade-macos/ and downloaded the oldest one I could find. At the moment of this writing it's 11.2.1 (20D74 App:16.4.06), the one for which apple download link is still available. The installation went pretty smooth after I booted from it.
View attachment 1780839
One small note though: It's a good idea to try to run the image on your existing mac
before you make a bootable usb and erase your SSD for fresh install. If you don't receive the "too old can't run" from the dmg, you won't receive the error on the recovery window. So you're good to go. Would save you hours.