Well I came up with a solution.... Well a work around.
I ran `pmset -g assertions` and saw that it was the 'AddressBokSour' process that was keeping my machine from sleeping.
I killed the process, but it immediate came back. It must be scheduled or something.
Finally, I just renamed the application so that it couldn't be launched.
I ran these two commands:
sudo mv /System/Library/Frameworks/AddressBook.framework/Versions/A/Helpers/AddressBookSourceSync.app /System/Library/Frameworks/AddressBook.framework/Versions/A/Helpers/AddressBookSourceSync.app.BKP
sudo mv /System/Library/Frameworks/AddressBook.framework/Versions/A/Helpers/AddressBookSourceSyncScheduleHelper /System/Library/Frameworks/AddressBook.framework/Versions/A/Helpers/AddressBookSourceSyncScheduleHelper.BKP
Now my machine sleeps.
I'm not happy with that solution, but I'm happy my machine sleeps.
YMMV