I have owned both a D40 and a D80. The major differences I have found are:
1) I personally found that the higher ISOs on the D40 to come out a little better than the same ISOs on the D80. I am mostly talking about the 400-600 range. The D90 (the newer replacement to the D80) does a much better job at these ISOs and higher.
2) The metering (and to a lesser extent the white balance) on the D40 in my opinion is better than the D80. This is ESPECIALLY true with matrix metering. If you do batch processes of photos (dump your pictures into Adobe Lightroom or Bridge and apply the same exposure / white balance to multiple pictures) you will find out what I'm talking about in about 5 seconds.

You can solve most of the problems by setting a specific white balance (and then fix them during post processing) and spot metering, but it can be frustrating when you are not used to these modes.
Here are the definate advantages of the d80 vs the d40 and why I purchased the d80:
1) At the time the d90 had just come out, but the darn thing was 1,000 bucks body only. The advantages (better / higher ISOs, etc.) just wasn't worth it in my opinion, although since the "newness" has gone away the d90 is a very good camera for the $.
2) If you have older lenses you can use the auto focus motor on the d80, but not the d40. The d40 doesn't have an autofocus motor so the older (and cheaper I might add) lenses such as the 1.8f 50mm (perfect for head shots / portraits with the crop factor) will work much better and you will get more / better shots instead of missing some of those low DOF moments when your sharp focus is only like 1 or 2 inches. You can find some great deals on used, older, fast lenses that will work great with your d80.
3) The d80 has a built in commander for wireless external off camera flash control. The d40 would require either purchasing an after market controller and be another piece of equipment you would need to keep track of. I have an sb600 that works great as an external flash by putting my camera's flash into commander mode.
4) The d80 has a few more megapixels than the d40, although by the time you get to the level of a dslr quality camera, this doesn't matter much unless you are doing some significant cropping of your images. In this case a few extra megapixels help somewhat.
5) The d80 has all your camera settings on the top display. It also has an extra dial and the ability to quickly change your various camera modes / settings. This you will find as a SIGNIFICANT advantage from your d40. Hunting those menus is a total pain, and drove me insane with my d40 because I would miss a great shot because my camera was set to iso 200 and the light was dim. With the d80 you just hit one button and dial the click wheel to change ISO / various settings.
My personal recommendation is this:
If you have the extra $$ to spend, get the d90, you will probably be happier with the choice, but if you are wanting to save some money and get a few more features than you have with your d40, you could do much worse than getting a d80. You will however need to play around with the camera a bit especially if you are still using the "auto" mode on your d40 most of the time. This could be an excellent opportunity to upgrade your camera and start using the other modes on that dial.
