No, it's not. Especially if you get mid to high-end parts.
What could get annoying is building a custom PC that's 100% silent, but even that is possible.
However, if you design it right and are ok with a compromise, you can definitely design a PC isn't completely silent, but has low noise, which will be more or less masked by the environment.
I personally think you can get some very interesting smaller, no-noise, compact PC. But custom PCs still have their place because they give a far better "bang for the buck" and are much easier to repair / customize (and can get better cooling / performance to boot, because they're larger – so, better airflow).