I'm also having this exact same issue, and have been since Beta 4. I've done some Console checking and noticed that whenever the crash/freeze happens, I get kernel[0]: disk2: I/O error.
Now, normally, this would instantly make me jump to an impending drive failure, however I've verified the SMART status and everything seems to be alright, as well as verifying the disk itself using Disk Utility, which has again come back fine.
I'm not sure what's causing this (I guess it could be an oncoming drive failure, at a push), but it seems interesting that this same issue's being reported by a number of people on the forum and it's unlikely that, if it's the same cause for all of us, that all of our drives are on the verge of failure!
For reference, I'm running the latest developer build on a Late 2014 Retina iMac.
I'm considering making the jump back to Yosemite, just out of curiosity and to verify whether I'm getting the same issue on there, but I'm reluctant to think it's anything other than a strange bug in El Capitan.