That really is a matter of preference. Take it the apple store and they will tell you whether or not it is defected (quieter than it should be) or if it is normal, but just not up to your standards. The speaker you put up to your ear isn't very loud to begin with. This complaint as been posted on the forum before.
I have the same problem. I think it's an iPhoneism as I tried my buddys new 3gs and it was just as quiet. Hope that apple eventually fixes this in an update