Wow, I noticed this as well, and thought it was some type of preferences issue. I only now bothered to look it up, but I am
amazed by the number of posters that either said they were unaware of this feature, or thought it was bothersome. I can not begin to explain to you how subconscious the action of using backspace/delete as a "go back" command is for me. I assumed that everyone did this. I am going to try the workaround "dylanryan" mentioned and hope it works.
defaults write com.apple.Safari com.apple.Safari.ContentPageGroupIdentifier.WebKit2BackspaceKeyNavigationEnabled -bool YES
For those of you that are keeping score, I agree that disabling a common browser feature (that has existed since the dawn of the browser) is pretty silly. Not sure what the "gesture" for this action is, but it is not the point. Why should I have to learn a new "gesture"? Isn't this supposed to be the intuitive platform?
Also, for those of you that associate this as a "Windows" keystroke, I wouldn't know. I have been a Mac user all my life, going back to OS system 4 in the mid-80s. Any KB shortcuts I have picked up along the way have all been Mac.