If you have any extensions, such as adblockers, etc, try disabling all, then check for loading speed.
Many virus scanners will affect your system speed. Most folks here will point out that the benefits, if any, of using a virus scanner is not worth the loss of system performance.
But, try turning off your virus scanning to see if that makes a noticeable difference. You should use it like that for a few days to note any changes.
You can do the normal things to try for a speedier browser, such as deleting Safari cookies, deleting web history, emptying caches, etc.