I have a Mini, not an iMac, with an SSD. FV is working well. No performance impact. The only problem I had with it had to do with multiple, USB keyboards being simultaneously attached. One directly attached, one to an external hub. That created a boot problem, doesn't sound as if that's the OP's problem.
I'm wondering if, in the OP's case, the encryption process ever completed, whether because of the "awake time" period of the computer, data amount, competition for resources, etc... If that wasn't checked, might want to try again and make sure it has actually completed before continuing any troubleshooting.
Edit: oops, the OP was in 2015. So, advice to retry and monitor is general advice.
macOS Sierra makes more efforts on this. Before Sierra, if you upgrade your macOS, you will have to input your password locally after the rebooting, as remote sever won't start until you pass FileVault. At Sierra, macOS can detect the rebooting is because of system upgrading, after the rebooting, you will get remote server started automatically, you can input your password in Remote app. So in Sierra, unless you get power losing or system failure, you won't need to input your password locally.