I'm pretty sure your NOT going to be able to use just the iMac's screen as a display for the PS3 using a USB based TV tuner unless you can find a DVI to coaxial signal converter...
You either need a dedicated video capture device
http://www.epiphan.com/products/frame-grabbers/?gclid=CJbIw_6f15kCFUU_3godFkh3aw, or crack open the iMac and come up with some kind of video pass through cable which lets you bypass "iMac" part of the computer and goes straight to the screen...I have no idea how the iMac screen is connected to the mother board so am not sure if this is even possible.
Also don't forget before the PS3 will play a BlueRay disc it will check for HDCP Compliance on the display device which your iMac is guaranteed to fail so BlueRays simply wouldn't play..