Both have exactly 36 GFLOPs/watt, and similar performance in DX12, and Vulkan, per watt. Only DX11 with high overhead on AMD GPUs is where AMD's efficiency tanks. And drives people's mindshare...
GTX 1080: 8.9 TFLOPS/166 W = 54 GFLOPS/watt
RX 480: 5.8 TFLOPS/163 W = 36 GFLOPS/watt
Power consumption is from "typical gaming" scenario at techpowerup. Clearly AMD has some ground to make up in efficiency.
Hitman (Directx 12) numbers (AMD's far and away most favorable gaming benchmark)
GTX 1080: 65 FPS/166 W = 0.39 FPS/W
RX 480: 41 FPS/163 W = 0.25 FPS/W
And here AMD loses in efficiency at their best gaming benchmark.
When Apple produced its A series chip at 14/16 nm across TSMC and global foundries it had better efficiency at TSMC. I don't think its crazy to think that some of AMD's efficiency problems may be due to the fact that Global Foundries' process is not as good.