Agreed. The net isn't up to it yet.I think that before HD content over the internet can become common place faster internet connections will either have to be the norm or people will have to get used to starting the download before going to bed or work to watch later.
I hope that Apple, when it starts working with HD, helps local ISPs cache information to minimise strain on the internet in general, as well as lowering cost for the ISP and for Apple. And if Apple uses peer-to-peer to aid the speed, hopefully they'll encourage local peering groups rather than peer to anywhere on the net.
edit:
The wait time for downloads is why I'd hoped AppleTV would work as a DVR.
Using a DVR forms new habits - you look ahead a couple of days and select what you want recorded when it airs (to watch later). This habit dovetails nicely with selecting movies you'd like to download (to watch later).