Pretty sure there were a few 8GB registered DIMMs that were meant to be matched up with 16GB registered DIMMS so should avoid those if keeping the 4GB DIMMS ( which I think all unbuffered; at least for the required tech that matches the Mac Pro 4,1 ). Two unbuffered 8GB is a very low cost, incremental move.
The other unmentioned issue is whether you eventually want to jump to 6 cores ( and flash to a Mac Pro 5,1 firmware ) which takes a higher spec memory than the four cores. If not really planning on it then don't worry about it, but if on a multi-year upgrade budget process then it is a factor.
----------
For 32GB it is probably cheaper to go with 4 * 8GB unbuffered.
Again if 32GB is probably as high as you are ever going to go with this machine then it is a better move.
If just stopping at 24-32GB because all you can afford now but highly likely going to require 48GB in the future than 2 * 16GB now is a better incremental step.
If your workload isn't demanding >20GB now and hasn't been on a growth curve for last year or more then preparing for > 32GB probably isn't a good move. If it hasn't been extremely painful memory pressure wise with 12GB ( 3 * 4GB ) shooting for over 32GB probably isn't very well motivated.