It's nothing to do with 8G or 16G, it's UDIMM vs RDIMM. You simply can't mix them together.
OWC know it, but they didn't clearly state the fact. However, they have the following "note"
IMPORTANT OWC NOTE:
OWC p/n OWC1333D3MPE8GB (8GB modules) and OWC1333D3MPE16G (16GB modules) require that all installed modules be of the same p/n OWC 8GB and/or 16GB modules. Other existing Apple or 3rd party 1GB, 2GB and 4GB modules are not supported for use with these kits and need to be removed when these modules are added.
Since your 8G stick is not the same part number as the above mentioned. That's why it won't work.
For me, it means:
1) OWC know RDIMM cannot mix with UDIMM, and Apple stock RAM is UDIMM, therefore, they have this "notes" to tell the customer that their RAM (which most likely are RDIMM) can't work with the stock RAM.
2) Your 8G stick is UDIMM, therefore, there is no notes on that page, and you can use that stick with the Apple stock RAM.
3) Your 8G UDIMM stick cannot work with the 16G RDIMM stick, which also fit the OWC note.
Anyway, the penalty in real world should be negligible for most of the work, especially you really need the RAM. Running out of RAM always has higher penalty than not optimising the triple channel configuration.