No. You can access it using an authorized user account. But again, these are all moot if you keep backups. If you don't have backups you can lose the data on your SSD for any number of reason other than FV. A short in a power regulator, something that affected certain late model MacBooks, will fry your SSD. Making data recovery impossible.I'm assuming Apple Silicon's Target Disk mode (Mac share something something) would also be unusable if FileVault was enabled as well. So while you can't remove the drive, you could successfully copy data without logging in.
The number one reason that users lose their data permanently is not due to machine failures. But due to user failure, to maintain at least one timely data backup.
I agree, if you have no data stored on your machine that is sensitive (primary email accounts, banking info, investment accounts, family photos, sentimental media, etc.), there's no reason for enabling FV. Otherwise, you should be backing up important/ sensitive data, therefore enabling FV is no risk.