What he said. You can go mismatched but there WILL be a performance hit.
Basically if you match memory modules in pairs, your Mac can access memory twice as fast as it can split memory access across both sticks at the same time. If you mis-match, it can't.
If you can't stretch to 2x 8 GB modules, buy another 2x 4GB modules for a total of 16 that way, you'll get better performance.
edit:
and if you do go 4+4+8+8, you may need to verify which slots "match" to ensure that you match across channels. Not sure if the iMac slots are colour coded or what, but on PCs basically you want the RAM in the white slots to match each other and the RAM in the black slots to match each other for example. If your mac came with the 4+4 from the factory, just filling the empty slots should do it.