I'm usually downloading photos at the end of the day, so it's actually more convenient for me to just plug my camera into the computer. I've never noticed a significant battery drain due to downloading photos. Compared to the power drain from all the other camera functions - especially driving autofocus lenses - I doubt reading a CF card for a few minutes is going to make any appreciable difference.
I do have a card reader - I just don't use it very often.