I've found out that proper display calibration does the trick. I have a few years old calibration profile for my display, and I've just activated it after moving to Big Sur. It's outdated for professional work though, and I have to calibrate display again, but that ugly font smoothing is definitely gone.
Display calibration has no effect on font smoothing. It may have a minor effect on contrast, but it doesn't change the way the OS rasterizes the glyphs.