[This is the way it is in the US:
This has been covered in this thread already, but I'd like to point out (because this was posted after it was all covered) some things that I take issue with (because when you're dealing with the law accuracy is important)
People who are out in public have no expectation of privacy. You can take photos of anything you can normally see with your eyes so long as you did not have to do something like climb over a fence or up into a window, use a telescope or whatever. But if the subject is in normal view of everyone else there is no question you can take the photo.
You cannot legally take pictures of several government installations where it is prohibited, even if you can see it normally with your eyes.
See for example: Title 18, Part I, Chapter 37, 795 & 797 US Code
I've seen and heard mixed things about infrastructure pictures, but I can't find anything in the Patriot Act prohibiting photography. Still one must weigh the chances of having to submit to a full body cavity search and detention with the desire to capture certain images. The place I photographed my avatar image is a dam, and I was warned by the other photographers there that the local sheriff's office would become very aggressive if I were to just take pictures of the dam.
While I can't imagine prosecutorial discretion not applying- taking a picture of someone abusing an animal and then selling or intending to sell it is a crime under all but some specific circumstances (Title 18, Part I, Chapter 3, 48.)
Now the nest question: What can you do with the photo? The basic rule is that if you use the photo for any commercial work or publish it you need a release form signed by the subject. But for your own use you do not.
Again not true. You can publish it commercially for editorial use, you can sell it to someone who's going to publish it for editorial use, you can sell it for fine art, you can sell it for lots of things that aren't related directly to advertising without a release. Also, for completeness you may need a property release in addition to a model release for animals, buildings, artwork, cars, and other property in the case where you need a release form for the subject depending quite a bit on what's in view and how important it is to the image.
There is one more thing. Some owners of private property may not want you to use your camera while you are inside their property. Example would be a department store or shopping mall. But even there you could shoot the outside of the property from a public location and use the image for your own use.
In general, it seems that though places like malls may ask you to stop taking pictures, they really can't do anything about the pictures you take before they ask you to stop.
Non-advertising commercial use is tricky without a release, because you can sell the picture, but you can't use the picture to advertise your photography, or perhaps even a book which includes the image.
Like all things legal, consult a licensed legal professional in the jurisdiction you're in before jumping in head first.