I think apple would worry about a large screen iPhone cutting into iPad sales.
No, the last thing Apple would worry about is iPhone cutting into iPad sales. They proved with different products such as various iPods, iPhones and MacBook Air that they are willing to cut into their own sales if they feel there's a good opportunity.
Here's a relevant Tim Cook quote from 2008:
In 2008, Cook gave a memorable talk at the Goldman Sachs Technology Symposium, mostly about the iPhone. He was candid, funny and open-minded, calling unlocked iPhones “a good problem to have,” dropping hints that Apple was willing to drop its exclusive carrier relationship with AT&T and arguing that it didn’t matter if the then-new iPod Touch cannibalized iPhone sales: “I’d rather Apple cannibalize Apple than somebody else cannibalize Apple.” The next day, Apple’s stock shot up almost 5 percent.