That means you haven't actually installed it properly. When you get that icon on the desktop, that's the disk image in it's expanded and mounted form. You then need to drag the application from there to your Applications folder, then drag that volume and the .dmg file to the Trash. Then the app is fully installed and the installer files will be gone.
jW