If you will be carrying a tripod/monopod or shooting under bright sunlight, non-IS version will work very well.
Many photographers (those without extremely steady hands) will need shutter speed equal to the focal length. If your Canon has APS-C image sensor, such as EOS 300D, 350D, 400D, 450D, 10D, 20D, 30D, and 40D, you need to further multiply the focal length by 1.6. To get steady image on these cameras (without tripod or steady hand or IS), you will need 1/320 seconds shutter speed.
That is why, despite your reluctance of admitting other lenses on this thread, IS is very useful to have, which for steady subjects, affords 2-4 stops of shutter speed (2 stops advantage at 320mm = 1/80, 4 stops = 1/20).