Back in the day, I used an iPhone 3GS. As it approached the time to shop for an upgrade, two things happened. One, the Galaxy Note was announced, ushering in the era of so-called phablets. The device was comically larger than the iPhone but was actually very well received. The other thing that happened was that other manufacturers were beginning coming out with with larger screen sizes in general; phones in the 4-4.5" range. While I was initially struck by the ridiculous screen size of the note, the other phones with larger screens immediately seemed right to me. The larger screens were so much more pleasant to use while the devices still felt appropriately "phone-sized". Comparatively, the 3GS (and, by extension, the original iPhone), felt like it was smaller than it really
needed to be. It is barely larger than my palm (see picture of my original iPhone for reference). I ultimately upgraded to the 4.3" Nokia Lumia 900 (Windows phone), followed by the 4.7" Google Nexus 4, then to the 4.7" iPhone 6.
I have recently upgraded from the iPhone 6 to the XS. The 6 is the thinnest iPhone ever and nearly the lightest as well, and let me tell you - it's been nice. Thus the XS represents a relatively substantial increase in bulk. Nevertheless, it's completely fine. Everything about it still feels appropriately "phone-sized". The SE strikes me as being intentionally compact. That's not a bad thing, just a matter of priority. The SE is the best (iOS-based) choice if you prioritize a compact form factor. Otherwise, I think the "standard" size of the 8/XS is the sweet spot for a phone when you are aren't interested in entering the phablet category.