A nearly full hard drive will make the situation drastically worse, but the very fact that the OS is having to access the drive to retrieve data it would normally store in physical RAM slows the process down because physical RAM is much faster than a hard drive. According to those who've made a study of this, the VM disk flogging problem can be mitigated some by storing the VM swap files on a separate drive or partition. This sounds like a dangerous hack to me, but apparently some are motivated to do it.
Neither you nor I know precisely how OSX handles swap files, or even more importantly, why they are persistent. But they are persistent, they are not released when no longer needed, and they do multiply, and grow. Why they do this, and why it causes a performance hit on a drive with plenty of free space, I don't pretend to know. However, from actual experience, I know that deleting them does make a difference.