Does the software run while High Sierra is hosted in a virtual machine? You would probably has to assign the USB devices (presuming a USB capture dongle) to the Virtual Machine.
P.S. can also boot into Windows if primarily just using the device as video recorder.