My reevaluated opinion about screen sizes if you care to hear it (do take note that I'm a 6'2" male, I purchased an iPhone 6s Plus, my work gave me an iPhone 6s after that, and I used to have an iPhone 5s):
1. A 4" screen is a bit too small for me period. My girlfriend loves her iPhone SE, but it's just too cramped for me.
2. If you don't have an Apple Watch, I think it makes the most sense to get a phone with a screen size that fits your hand well- which, for me, would be the 4.7" size.
3. If you DO have an Apple Watch, however, it makes more sense in my view to go for a slightly bigger phone, so there's a more tangible difference in screen real estate when you bother to take the phone out. I have a 42mm Apple Watch, so after trying out the 6s for a while, I decided that the 5.5" screen made the most sense as a bridge between when I would look at my Watch or bother to sit down and pull out my iPad or MacBook.
This should all be taken with a grain of practicality salt. The jeans I usually wear have pretty big pockets, so I can fit my 6s Plus in them without worrying about it falling out. Women who carry purses probably have their pick of what they want, but those who can't really bring one around with them at work (like my girlfriend, again), would probably have better luck squashing an SE or maybe a 6s into the obnoxiously small pockets in women's pants.
This is quite apart from design aesthetics; as I said in my previous post, the 5s design is better in my view. But the screen size is pretty important, too.
For what it's worth, I've also noticed that several people have said the design of the 6 and 6s is very slippery without a case. I thought I'd point out that while I agree that the 5/5s/SE design is better in that regard, the 6s/6s Plus phones seem to be slightly less slick than the 6/6 Plus were (I used to have a 6 Plus). I'm not sure why, exactly. Maybe Apple used a slightly different type of aluminum or a slightly different finish or something? Either way, I thought the 6 Plus was way more slippery than the 6s Plus is.