Did more tests. Result shows that the Finder UI hang (and subsequently discovered iSight hang) ONLY happen when all CPU cores under high stress.
Since my cMP was encoding video almost 24/7 after upgrade to HS. That's why I get the hang. If I pause the process, Finder is perfectly fine, and iSight also never hang.
Since I only pause the process, but not stop of finish it. Therefore, the RAM usage clearly is not the reason of those hang. Couldn't quite understand why high CPU stress will cause the Finder UI hang (or iSight hang). It's clearly not "waiting for CPU resource", because when beach ball happen, even I pause the video encoding (CPU back to idle), Finder / iSight still hang.
May be HS make Finder (or some software) can either get the CPU resource now or never. If at that moment, CPU recourse is not available, then the UI hangs (until relaunch).
I also tested the "renice" command in terminal, make the video encoding priority to 20 doesn't help anything.