I can confirm that the plugins are the culprit for the slowness when loading new links. It seems that the more plugins you have, and the bigger file size they have affects the time it takes for the page to start loading, so I assume Safari is somehow reloading/refreshing the plugins everytime a new link is requested.Deleting the plugins using the method on the website was what did the trick for me. Thanks!
You can find the plugins in the following folders:
~/Library/Internet Plug-ins/
/Library/Internet Plug-ins/