Try this at your own risk!!!
I doubt it would screw up your system (other than speech), but you have been warned.
You could always just open a terminal window, cd to the '/System/Library' directory, then do a 'mv Speech backup_Speech'. This will permanently move the speech recognizer, synthesizer and voices to a directory that won't be found by the system.
Alternatively, you could just move or delete the directory to acheive similar results. This would be a more dangerous alternative, but would be more garaunteed.
You may have to restart to clear all caches.
Taft