OK I know that mac is sort of fragment proof and one need not worry about it. But what's the story when using an external drive to back up? Does one have to defrag it regularly?
No not if it is in HFS+ format. If your using a any of the FATs or NTFS (to a lesser extent) you will need to defrag. I think UFS is as as good as HFS.
This Apple article is a good start, though it basically states "You don't need to defrag, and don't worry about it."
Aside from what Apple notes, it is really a function of how the disk writes clusters of information. OS X does everything it can to not immediately use space that is freed up. This does several things, aside from make data scavenging and reclamation easier. It speeds disk use (arguably) and it helps prevent fragmentation. The older MS FAT system (which is is still usable and efficient for many purposes) fragments, because it writes clusters all over the place. Under FAT, as soon as a small amount of space is free, it is likely filled by the os. There are a number of reasons behind this, and people often vilify FAT, but works for many applications.
Ok, I am getting a little tangential here. Basicly, it comes down to two things. 1.) The system is much more efficient at writing information to the disk so that it can be accessed quickly. 2.) A misperception as to what efficient is and what fragmentation is and does to a systems performance.