Depends on your eyesight. Over 10 years ago, I used to read on the 3.5" OG iPhone with Stanza and I was already happy with that. Nowadays, I wouldn't want to read on the 6.1" iPhone.
I'm still perfectly fine with using the 9.7" iPads for PDF, though. I annotate letter-size PDFs side by side and 2x 9.7" iPads in portrait still offers more screen real estate than one 12.9" iPad in landscape.
My eyesight is really bad (glasses). I still use my SE to read in a pinch...literally, since it's so small!