Most that are returned were macs that were bought and returned within the 14 day window.
They all go through rigorous inspections, and the outer shell is replaced. In other words, you have a better chance at a perfect imac through the refurbished section, then buying it new. Refurbished products from apple really are like new products.
I actually have a refurbished 24" imac. If you had put it side by side with a new one, there literally is no difference.