It means that email is automatically sent to your phone as soon as the mail hits the server, without having the phone (or whatever device) set up to check for mail periodically (called "pull"). For most, it's not that important, but in a business environment, push is huge.
It's one of those things on a desktop computer is essentially transparent to the user if you have your mail client set to connect and check for mail <=5 minutes. Even in a business environment, the first thing most people do when handed a computer with an Exchange mail client (Exchange is push) is turn off all those notifications because you don't necessarily want to know every time you get an email, the moment you get it. The iPhone does it subtly, though, only vibrating -- which you can turn off by turning off vibration and/or new mail notifications -- when it receives a new e-mail while it's locked in standby mode. (I have noticed, however, this is inconsistent; sometimes when my phone is in standby mode I get both the vibration and the new mail sound, sometimes just the vibration.)
On a mobile device, push saves battery life because it only connects for mail (or, in the case of MobileMe, calendar items, or contacts changes, or web bookmarks) when the device is alerted there's something new. Pull, it goes out over the cell data network or WiFi network if available every however minutes you set it, whether there's anything new or not.
Overall, it's handy for non-business purposes for iPhone users because you don't have to, if you don't want to, manually check mail or suffer the accelerated battery drain of automated regular connection to the mail server.
Note, also, the term JohnNotBeatle uses for the opposite of "push", he calls it "pull", is the industry-standard term. However, Apple calls the opposite of push on their devices and services "fetch" for whatever reason.