Here are the instructions I got from AppleCare from an Apple Engineer. I'm not sure if the fix was specifically for my machine or if it'll work for someone else. Follow at your own risk.
1. Open the Library folder in your Home folder.
2. In the Library folder, open the Application Support folder.
3. Locate the folder named NotificationCenter. Drag this folder to the desktop.
4. Next, open the Terminal application.
Copy and paste each line of these commands into the Terminal window, in order. Press return after each line:
cd `getconf DARWIN_USER_DIR`
rm -rf com.apple.notificationcenter
killall usernoted; killall NotificationCenter
5. Close the Terminal app.
6. Restart your computer.
The notification preference changes won't stick until you restart a 2nd time after this.