The Mac Pro 4,1 and 5,1 default fan settings (like most Macs) are aiming for very silent operation but can result in (too) high Temps.
The construction of the cMP concerning airflow is very well thought and efficient, and leaving the side panel off is not recommended and will effect negatively (mainly) the CPU compartment Temps.
As you may have noticed the HDDs (which generate heat themselves too) Temps increase from bay1 going up to bay4.
This is above all caused by the heat the GPU generates.
The PSU fan blows air not only through the PSU , but through a few slots above the drive bays too.
I'd recommend the following:
-remove PCI slot covers from unused PCI slots and leave it open.
-set the PCI fan depending on PCIE ambient (see screenshot)
-set PSU depending on PSU supply 2 (see screenshot)
To keep all Temps low in the CPU compartment i'd recommend:
-Intake and Boost A fan depending on NB diode (see screenshot)
Other things to consider :
-HDDs generate much more heat than SSDs
-If possible (depending on the amount of PCIe cards and height of the GPU), you could move the GPU to slot2 instead of slot 1. Backdraw is to lose slot 3 ,if the GPU occupies 2 slots.
This way there will be airflow underneath the GPU so it stays cooler.
All fan setups are a bit personal as well, e.g. I prefer low Temps.
You could test this setup and tweak it to your taste and workload, finding a balance between low Temps and fan noise...
EDIT: This is a general recommendation without knowing any details about the hardware installed and the type of load.
Finally de-dusting the cMP regularly is recommended and can result in significantly lower Temps as well...
