The iPhone SE would be a downgrade though.
Depends. Some people don't need a fancy camera or the newest hardware in their phone and instead prefer the small form factor. I love my iPhone SE and will hold onto it as long as Apple allows me to.
I have an 11" iPP and a 13" MBP (which I rarely even use since I have the iPP) and the small iPhone SE fits into that perfectly for me. I wouldn't trade it for a iPhone 11 if someone offered (unless I'm allowed to sell it and then buy another SE, who doesn't like free money?).
If they made a new ~4.5" phone with newer hardware, I'd love that as well but it doesn't look like it since the new SE2 seems to be significantly bigger than the original SE so I guess I'll stick to that for now.