Oh Robin, you've gone and openned that can of worms haven't you? Haha. There are so many answers to your questions and I'm sure you'll see most of them posted here. Given your situation though, using a point and shoot, and probably you'll be using it mostly in P, or Auto or one of the creative modes (I could be wrong, feel free to correct me if i am), I'd say what you heard is definitely correct. The first few hours of the day and the last few before twilight are usually best. But other people are correct too when they say it depends on what your subject is. I would never say "don't take a photo at mid day", that's just silly, no one would ever take pictures if that were true. But there is a HUGE amount of contrast between light and dark in the mid day hours. Your camera simply can't cope with that much contrast. If your subject were a person in a field for instance, with no shade, the sun shining in on their face from above, you would get a picture of a white sky, and a subject with huge black shadows under their eyes and nose. Not very flattering, right? You could try to fill it in with flash, but your on-camera flash probably won't get the job done. So, yeah, this is a bad time of day to take that shot. It can be done, but you would need other equipment, which I'm sure isn't something you want. Now, if it were the same time of day and you were next to a building with a grey wall or a window that was reflecting light back at your subject, the picture would not have as much contrast in it and you'll be able to get that nice even exposure. But if you're looking for anything with texture (walls, rocks, rust, etc) morning and evening are probably your best times. Not that you can't do it otherwise, but with your equipment being you and a camera, it's just easier.
However, if it means the difference between getting a picture and not getting a picture, just take the picture and worry about it later. Like kids playing, or a pet or something. Those things happen quickly and obviously you can't always have the forethought of finding a shady place, or somewhere with even light. You can always try changing the picture to black and white as well afterwards. I find it can usually give the picture a new dimension since the mid day sun has probably washed the colour out of the photo anyway. Do an exercise. Find an interesting subject, take a picture of it in the morning (an hour or two after sunrise), take another picture in the afternoon (maybe 12-2) and then another around 7pm or 8pm. Look at the photos, and understand what the light has done in that photo to the subject. I can sit here all day and tell you about light, but unless you see what happens yourself and understand why it does what it does when it does, it won't make sense. Have fun with it, just keep practicing.