BD players (software) suck. Not just on Mac but PC as well. There is so much copyright protection BD playing software needs to be one part player, two parts hacking/ripping software.
Honestly I find it less of a hassle to rip the BD using MakeMKV and watch the MKV file using VLC player, then delete it when done. Lol. Or instead of deleting it, transcode it with Handbrake and drop it in iTunes for future watching.
Blah blah legal disclaimer, don't steal movies blah blah blah.