For overclocking install MSI Afterburner. Tolerances will differ between different cards units of the same model, but so far everything up to +250 (GPU)/+375 (RAM) seems safe and has only a slight increase in temperature. +150/+250, +200/+300 are also good overclock settings. (MSI Kombustor can be used to check overclock stability.) Don't go for the highest overclock first, try out the lowest setting first, *then* go higher if successful.
What to do:
1: Install a program like HWiNFO64 to monitor temperatures, esp. CPU and GPU temperatures (and HD temperature). I prefer to use MSI Afterburner OSD (on screen display) function to have framerate and GPU temp in a top corner of my screen when playing games. This way I can keep an eye on the GPU temp, and see the effect of the overclocking - and it can make a BIG difference

.
2: If you want lower GPU and CPU temperatures (and possibly a higher stable overclock), disable CPU turbo boost in Windows 7. This is done in Control Panel: Advanced Power Options: CPU Max Speed 99%. (Don't change the minimum CPU speed setting.) I recommend this, since otherwise temperatures will get very high unless you use vsync (dual buffered). In my experience +285/+425 here works very well, but I would probably go for 250/375 in the newest games like Far Cry 3 and Crysis 3 etc. Obviously some games don't need overclocking, as they are not too demanding. Game performance with CPU turbo boost off should in most cases be near identical.
3: Use Lobbo's Fan Controller, and set a static fan speed. I prefer 2400-2500rpm for demanding games, and 1700-1800rpm for older or less demanding games. Don't go higher than 2600rpm!
4: If there are indications your iMac is straining (weird noises), or if it becomes unstable (random reboots, weird graphical glitches) lower the overclock setting. The first time you open Lubbo's make sure you lower the max fan speed setting!
5: Download RivaTuner, and extract the D3DOverrider (inside the Tools folder of the RivaTuner installation file) with a zip program (7-Zip works well), and install D3DOverrider. This way you can have triple buffered vsync in directX games. The great thing with triple buffered vsync is that you get no tearing, but also good framerates and less lagging than "normal" vsync.
What not to do:
1: Don't let the GPU heat up to much higher than 82-83C. There are differing opinions of temperature tolerance levels in this forum, but I prefer to be on the safe side. Some will find even higher temps acceptable. (With CPU turbo boost off and using Lubbo's the CPU temp usually stay in the 70s (or lower). I can get 80-82C here, if I play Crysis 3 for several hours with +250/+375.)
2: Don't mess with voltage settings.
3: Don't let the HD heat up too much, as HDs are more vulnerable to high temperatures. I've read that 60C is not good for the HD, and the temp should be well below that. I only use SSDs, so I'm not very knowledgable about HD temperature tolerances. Maybe others can shed more light on this aspect.
NB: Overclocking is done at your own risk. In most cases it should work just fine, but you never know... I've never had any issues, but I don't mess with voltage settings and I avoid very high temperatures. Enjoy!

If you have any additional questions, ask away! PS: All the programs I mentioned are for Windows 7/BootCamp. It makes no sense to overclock in OSX anyway, as games in Windows 7 perform much better.