I tried disconnecting all of the peripherals, but it still has the same problem.
Right now, I'm running Firefox and iTunes. The only background programs I know about is Growl.
I disabled wake-on LAN, and I do not have any Bluetooth devices paired with my Mini.