but it will make controlling fast-paced games a miserable experience.
I agree the Apple TV/ iPod controllers wouldn't make sense for "fast-paced games" like "Gears of War" or "Tony Hawk", but I contend it would work perfectly well for things like "Cooking Mama" or "Enigmo", where milliseconds of latency don't really matter.
That aside, I don't understand why anyone would WANT to use iPhone as a game controller.
What makes it appealing is that you have devices with built-in high resolution, multi-touch screens with accelerometers, WiFi, speakers and, in the case of the iPhone, a microphone. If you think outside the box and don't view it as a "controller" in the conventional sense, then things get interesting.
Picture an experience where you download a quiz game from the Apple TV App Store, you choose "Start Game" from your iPod Touch and iPod screen becomes a controller with buttons 1, 2 and 3 which you can push to answer multiple-choice questions. Each press results in the speaker on the iPod sounding "bing" to the first person who gets the question right.
You decide to choose a different "cooking" game and your iPod displays a bowl of batter you stir with your finger then pour via the accelerometer into the cake mold on your TV.
You then decide for a change of pace with some racing and your iPod displays the image of a steering wheel with gas and brake pedals on either horizontal end for your thumbs. You can race down the on-TV track, steering with the accelerometer as the speaker in your iPod "controller" makes engine roars and tire screeches.
You then notice a "pirate" game just became popular on the App Store, which you download and discover that it turns your iPod into an image of a cannon where every tilt you make aims a cursor on-screen at a distant ship. Tap your cannon and the speaker sounds "boom" as the cannonball hurls toward its target.
I, for one, think there are some pretty cool possibilities with such a system if you use your imagination!