Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Earlier in the morning the better. I've downloaded Lion at 9AM EST before and it's taken as little as 40 minutes.

Last night at 10:30PM I downloaded ML for the first time. Strangely it only took about 50 minutes.
 
I have only wireless broadband (no DSL or cable available), and it is capped at 1Mbps bandwidth, but it is reliable and there are no data caps.

I have downloaded every installer since 10.7.0 and saved them. It typically takes around 10 hours. I start it before I go to bed, and save the app file after breakfast. No waiting around for it to finish :)
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.