I typically set it to mid-brightness, and let the iMac adjust the brightness according to the ambient lighting conditions. During evenings it is somewhere between 1/4 and 1/2. I can imagine during the day with bright sunlight, the screen may go up to 3/4, maybe even full, I'm not sure since I don't frequently use the iMac during the day.
Is there a downside to running the iMac at full brightness other than power consumption? Is image retention a problem for these displays? I always thought this only affected old CRTs and Plasma screen, not LCD, but I recall someone complaining about this issue on MacRumours.
Lowering the brightness may save you some energy; here are some power measurements for the iMac 27" (2015) Core i7 395x 32GB RAM and 512GB SSD running idle at login screen:
25W idle screen off,
26W idle screen min brightness,
37W idle mid-brightness,
55W idle screen at max brightness.
In comparison, the old iMac 21.5" (~2006) it replaces:
47.5W idle at min brightness,
92.4W idle at max brightness.
I used to play games on an old PC (~2006) that idled at 80W. Add to that a very old 20" LCD display that required 45W. This makes the new iMac very efficient in my opinion. Although nothing beats the Mac Mini at about 10 to 12W idle ;-)