Do they have the D50s in stock anyway? I think they're getting harder to find new, or refurbed. The difference is that the D50 is a bit bigger, more solid - more on the order of the D70/D80 build-wise. Also, it has a screw-drive AF motor built-in and a function lcd display up top, which the D40 series lack. Both are 6.1 megapixel sensor, and probably indistinguishable image quality between them.
I bought a D50 refurbished with kit lens (18-55 Nikkor) from a local dealer. It came with a 90 day warranty, but I added a three year aftermarket warranty which my retailer honors. It was less than $40. I've had the camera now for 18 months, and shot over 7000 photos - and it works perfectly. So, I'd go for the refurbished (as long as it's an official Nikon factory refurbished product, including the lens - I stress this point.) You will not be able to tell the difference between it and a brand new product, other than the box will be brown.
A Nikon rep told me that most refurbs are either rep demos, or cameras that initially failed some quality check/test during first run manufacturing, and were taken off the line, sent back through to correct the issue, which could be something rather minor. These cameras automatically become classified as refurbs, even though they're basically brand new. I'm not sure if this is true, but I gave my purchase a thorough inspection in the store, and was assured that if I didn't like it I had a couple of weeks to just bring it back for full credit. Needless to say, I still have my D50.