Well, there's a huge market for 128GB capacity iPhones but Apple doesn't make them and I wonder why that is.
Anyway, the SE sold really well for Apple and still does. It's not that there isn't a market, it's that Apple wants you to spend more $. That's the gist of it all. It isn't that there isn't a market, it's that Apple doesn't want to waste their time on a phone that will sell for what..$399? $499? They can eliminate that price range and it will "force" probably 75% of those people into buying the Xr or Xs. Some people will wind a different make for that size and Apple deems that acceptable loss compared to the profit margin from the "forced" sales.
If there was a market for 4-inch phones, Android competitors would be all over it.
There's a market for 4-inch phones just like there's a market for 15-inch monitors and flip phones. With 7 billion people on the planet, of course there's a market. But it's not a market worth playing in for any major player.