I ordered 4x4GB from Crucial (ECC Unbuffered PC10600 1333MHz) for under 300$ which lends to a total of 24GB RAM for me. The Mac Pro 12-Core performs like a beast with that much RAM. (hasn't come in the mail yet still working with 8 atm.)
You can mix ECC and non-ECC Unbuffered (if you notice, both are Unbuffered, which is why they can be mixed), but it's only Unbuffered ECC that you can do this with (if non-ECC sticks are mixed in, it all runs as non-ECC).
Registered is the oddball, as it cannot be mixed with any other type (Unbuffered or standard non-ECC Unbuffered). Which is why you'll have to pull the OEM memory and hang on to it for warranty purposes, and run the Registered DIMM's by themselves (this is the right way to go if you plan on/expect to exceed 32GB).