Starting with Lion Apple doesn't want to you have the startup partition on a RAID. I'm not aware of a workaround, but I last looked back when I tried to install Lion on an internal RAID.
Not interested in buying a raid card just for this old machine. I was trying to repurpose it as a file server. How should I set it up with SW RAID to make use of the 4 drive bays and 500gb hard drives?
Use one drive plain to hold your system, and the other three in a RAID0, but come up with a backup strategy in lieu of redundancy. RAID 5 isn't supported by Disk Utility or SoftRAID.
Not interested in buying a raid card just for this old machine. I was trying to repurpose it as a file server. How should I set it up with SW RAID to make use of the 4 drive bays and 500gb hard drives?
Use the OS drive with HFS and then use MacZFS to make a RAID5 with the other 3 drives.
You will not be able to run much of the system off the ZFS volume. This is a limitation of how OSX requires HFS for several features. But any of the "data" you are trying to serve will be plenty safe and useable. Use Time Machine to back up your OS drive.