Have you ever watched a film on a 5' screen? I find that there is a hell of a difference, YMMV. I find web surfing on a 4" screen impossible, a 5" screen is great. The screen is a big deciding factor for many.
I find that those that don't like a larger screen have never used one for any length of time. Mind, there are always those that have borrowed their friends large screen phone to be able to give an experienced opinion. Lots of those people about. Perhaps you're one
For me a 5 inch phone is too big for me to interact with. I really like the size of the Moto X. The few times I watch a feature length movie are overshadowed by the amount of time I spend doing other things when a smaller screen is more ergonomic.