I agree that the iPhone needs to be larger for iOS' sake. I've always said, the desire for a larger iPhone isn't an iOS/iPhone vs. Android issue.
But to answer your question, I suppose because Android and, to some extent, Windows devices are the only ones that we can compare and gauge what the experience of a larger iPhone would be. Not just physically, but also software-wise. Even though they're different OS-es, we can get a sense of what it's like to browse, type, read, play games, etc. on a larger screen. It also inevitably comes up when people worry about non-issues like "if it's not 3.5, it won't fit in my pocket!" or "I can't reach the top of the screen" (which in most cases, is a mere few millimeters of difference), and the response from those who have actually tried it is "Well, my larger Android device fits perfectly fine into my pockets and reaching the top of the screen is only a minor adjustment."
A larger iPhone for iOS would be glorious to use in its own right.