What do you think it happening in the seconds before it plays? That's the video getting cached. Without media getting cached somewhere (either a traditional hard drive or ram) it would be a terrible bunch of heartache.
I don't think the AppleTV needs to be the end-all-be-all but I see room for improvement. If people think it is perfect I'm glad for them, I'm sure it makes for a happy purchase. I personally think that there are some basic features that would increase value without adding bloat, and having considered my options I think that I would need that value in order to happily own an AppleTV.
I think I see a Mac Mini in my future. Not perfect either, but it more closely resembles what I'd want. That is to say a larger hard drive and the potential for DVR.
I'd also like to point out that there doesn't have to be just two AppleTV's. We have iMac's and Mac Pro's, why not AppleTV's and AppleTV Pro's then?