You absolutely should worry about it. If you are paying for service and it doesnt work, you're throwing your money away.
I would suggest checking coverage in your area with AT&T and see if the problem is connected to that. Otherwise it could be related to a lot of other things.
I just sent an email with 3 photo attachments from my ipad to another of my email accounts and that took 12 seconds.