Hi! I had the same problem when I started using HandBrake.
If you installed VLC to a subfolder in Applications, then HandBrake won't see it. HandBrake expects the application file itself to be in Applications, not in a subfolder. Copy the VLC application file from its folder to the main Applications folder.
Hope this helps.