Whilst I have no true, direct comparison as my iPhone's been in an Apple silicone case right from it's unboxing, I do regularly take it off at home for cleaning (microfibre round the switch, camera etc.) and it seems to be nearly immaculate with no micro-scratching etc. The only signs of very slight wear are around the ports.
I'm not saying case-scratching doesn't happen, but speaking circumstantially given the environments my phone has experienced and my phone's current condition - I can almost guarantee it could only have been in worse condition if I left it naked.
I believe that. It depends entirely on how the phone is treated. If one were to treat a naked phone the same way they treat a condom-ized phone, yes, the one in the case will be in better shape. If, however, you are willing to treat the phone the way I (and others) do that go naked (dedicated pocket -- not even paper in there with the phone, never set it down on a hard surface, etc), then you never get micro-scratching and the phone looks like it did when it came out of the box a year later. I consistently get better resale value on my phones than my wife or daughter get on theirs; theirs are in cases, mine is naked.
It absolutely is a trade off. With a case, you can be more carefree with how you handle your phone, but you have to touch and feel the case for the life of the device, instead of the phone itself. You also risk the micro-scratching situation (which you've mitigated with your habits; kudos). Without a case, you have to be much more aware of your phone and diligent with how you handle it, but you get to see, touch, feel your phone as precision-crafted device that it is and, in general, don't suffer from the wear that a case normally exacts on a phone. Either path is fine, neither one is the 'correct' way. Just depends on personal preference.