It's not how big it is when you're using it it's how big it is when you're not using it. I would not want to own a phone I couldn't put in my pocket
iPhone Plus is to big, iPhone SE looks like to small for most people.
I think solution is iPhone 6s/7/7s,... But on iPhone SE video clips on youtube are visible nicely when you turn phone.
If you want really nice look than only around 5.5" is OK.
iPhone SE is perfect for pocket... But iPhone 7 fit in most pocket. I love to wear tight jackets with a lot of pockets and always have some nice place for iPhone inside or even on sleeves.