Well, a few years ago I would say yeah stop updating because new versions were getting too demanding in ressources. The iPad Mini 3 I gave to my dad is actually on iOS 12.x and I don't feel it snappy anymore, but that machine isn't properly a beast anyway.
But in the last couple of years Apple has made efforts in optimising the OS for the oldest models, specially with iOS and iPhones, so in theory they can keep up longer.
My advice would be to not install day one when a new version is out. Wait for a while and check for youtube videos demonstrating performance of a particular version in your particular device, and then judge if it feels fluid and fast enough for you. Because ultimately that's the point to consider: fluidity and speed of the interface. For everything else, updating is always a good thing (specially on security issues).