No.
As pointed out above, it's just two different binares sharing the same icon. You can delete one and make it not universal if you so choose.
Now, the process of *coding* for a universal binary could, in theory, lead to lower perofrmance, if the programmer didn't do any architecture-specific optimization. But a good programmer could just as easily specifically optimize an app for both the G5 and Core Duo with some extra work, so again, it's not a matter of the system but of the individual app.
And note that I'm not talking about compiler optimizations--pretty sure those can be included for multiple architectures, if you want.