As others have mentioned, processor-wise, the SE should be supported as long as the 6s is. I've still got unlocked 5/5c's that the family uses for international travel and non-gaming performance on iOS 9 is quite decent so that's fairly respectable performance from a 3-4 year old CPU.
Mind, going from 4.3" to 4.7", you'd barely need to make adjustments at all.
That said, which size to get is entirely a matter of personal preference. I have baby hands and I like the 4.7". My brother has average-sized hands and he likes 5.5". My parents both have bigger hands than me and my brother and they prefer 4". My mom, mostly because anything bigger wouldn't fit in her jeans pocket, my dad, I have no idea why.