I don't see anything wrong with that if the computer is not on and the drive is not being accessed.
The computer doesn't necessarily remember it had a drive connected when it starts back up. It just scans the ports to determine what is connected now so when you power it back up it looks 'around' to see what's there and will re-connect to the drive. If a drive is not plugged in, it won't show.
Don't do it while powered-up though, the disc may get corrupted.
And I would suggest powering down the drive/turning it off first too before unhooking it.