I have a sneaking suspicion that they will pick a GPU with a thermal profile that will allow them to continue to make the case thin and ventless. If the current iMac is any indication then they are prioritizing form over everything else.
The iMac 27 case is not ventless. The intake vents are on the bottom edge and the exhaust vent is on the back. If it were ventless people would not be discussing fan noise.
The M395X in the 2015 iMac 27 has a TDP of about 125 watts. The TDP on the GTX-1080m is about 150 watts, so that's probably within an achievable range on the current physical design. There will probably be an AMD Polaris GPU with good performance below 150 watts TDP that would be suitable for the current iMac 27 design. It was probably be much faster than the current M395X.