proximity sensor was bad
I purchased a new iPhone 3G and was having the same trouble... my face or ear was constantly dialing digits, muting calls or putting them on hold. I even managed to unintentionally conference call my friend to my Dr.'s office .
The problem didn't occur on every call, but it did on most. I tried it without the rubber case, (which was designed for 3G) and thought there seemed to be some improvement, but not enough.
Since I bought my iPhone at an Apple Store 2 hours away, I called them. They advised I needed to come back for an appointment with an Apple Genius and that it would be worth my 2 hour drive if for no other reason than to get on record that I was having trouble. I just hated the idea of that kind of drive for someone to tell me I simply had to learn to hold the phone differently.
Well my advise to anyone with this problem is TAKE IT IN TO THE APPLE GENIUS. He took the phone, played with it for about 30 seconds and said, "yeah, your proximity sensor isn't working properly". He gave me a new iPhone and I have had ZERO problems since.
The upshot here is that it's not tricky at all to hold the phone and have it work properly. If you're having trouble, don't assume it's you or that you need to learn to hold the phone differently. Take it back. The difference is beautiful... not one face dialing problem since.
