Real world user here.
The Anand review is great. I'd summarize it as: you won't notice, either on SSD or on a mechanical hard disk.
FileVault 2 is a marvelous technology, implemented to a tee. You can even use your machine while it sets up your disk when you turn it on (or, for that matter, if you turn it off at some point.)
It's remarkable what Apple achieved here.
And although I have no doubt the Anand review is correct in that FV2 utilizes CPU instructions built-into i-series Intel CPUs, it works great on my older Core2Duo machine too.
Absolutely no drawbacks. Just don't forget your password!