It's probable that you can't upgrade to a newer version of macOS because High Sierra requires a minimum of Mountain Lion (10.8) as a starting point. If you want to upgrade your OS, I suggest upgrading first to Sierra (10.12) and then to High Sierra (10.13, which is still supported with security updates until late 2020).
To install Sierra, use the direct link to the installer on this page under Step 4:
Download and install current or previous versions of the Mac operating system on compatible Mac computers.
support.apple.com
After you install Sierra, you can upgrade to High Sierra using the direct link to the installer on this page under Step 4:
Download and install current or previous versions of the Mac operating system on compatible Mac computers.
support.apple.com
Your Mac Pro can even run Mojave if you install a Metal capable graphics card. I presume you have an AMD 5770 or 5870 in your system (please correct me if I'm wrong), which are not Metal capable, so Mojave will not install on your Mac unless you replace your card.
This page lists all the graphics cards that are compatible with Mojave, if you're interested in upgrading:
Learn how to prepare your Mac Pro (Mid 2010) or Mac Pro (Mid 2012) for macOS Mojave.
support.apple.com
Other than installing High Sierra or Mojave for improved security, if the machine is meeting your needs and is providing acceptable performance for your work, then there's little reason to upgrade the hardware. If your Photoshop work would benefit from more RAM, then RAM is a fine thing to invest in.
You can also upgrade your processor if you want a bit more horsepower.
If there are specific pain points you want to resolve, let us know and we can provide more targeted advice.