I agree with Scepticalscribe that this is a fair point. I might quibble regarding the what the market is large enough to sustain as this doesn't have to be a huge fraction of total sales to justify its existence in a larger company, even one the size of Apple. And, as shown above, the SE was maybe 15% or so of new sales for a time.
I'd also point out that its hard to guess at the upgrade frequency of small-phone-users given that small phones haven't been updated frequently by Apple since the move to bigger sizes with the iPhone 6. So even Apple doesn't really have these numbers to work with. They might have more information then we do (ok they do, not might), but they still just have projections. Not actual data.
Another point that relates to this is that there is likely an ever increasing pent up demand for smaller phones. The iPhone SE is 2.5 years old. At its time of introduction, the iPhone 5s was also 2.5 years old. Many of those infrequent buyers of small phones probably didn't upgrade to the SE because their 5s, or even 5 generation phone, wasn't that old yet. According to the link a posted above, around 25% of iPhone users are on SE/5s/5/5c/4s/4 (as of last summer anyway). While a fair number of those users are likely not in the market for new phones, they will eventually need an upgrade path. And maybe Apple has done some research on what fraction of those users would move over to a new phone if one were offered in various sizes/prices, my suspicion would be that number increases by the day.