Maybe I made a discovery.
Premise: does Photo Boot work in your Big Sur 11.2.3 with Opencore or Micropatcher?... I just found out that Photo Booth is not working properly in my << Late 2013 iMac i7 27 inch CPU and NVIDIA GeForce GTX 780M 4GB VRAM >>. (Note: my Big Sur installation, with Opencore Legacy Patcher is from scratch and not over Catalina or an old System).
My sudden doubt is that iMac 14,2 to 15,1 spoofing is not enough to activate all the original Big Sur features, contrary to what has been stated so far (!!!!). Maybe there are problems with Quartz and therefore it would be necessary to also patch the graphics of the iMac late 2013, inserting a special Kext or other?...
I explain the problem in detail and how I provisionally solved it, hoping to help @Kronokornel and
@Ausdauersportler solve it in an intelligent mode than me. Perhaps, behind the exclusion of late 2013 iMacs from official Big Sur support, there could also be an incompatibility of the graphics cards and not just the Fusion Drive (???).
ISSUE DESCRIPTION
1) Launch Photo Booth
2) The FaceTime HD camera (built-in) LED turns on, but the Photo Booth window appears black.
3) Taking a photo or starting a video, their creation works fine and the related preview icons appear, but the image of the HD FaceTime Camera (integrated) still does not appear on the screen.
4) ATTENTION, now! By clicking on "Effects" there are two effects that are active (I can see the previews with my image framed by the HD FaceTime Camera (integrated) and the two suggested effects.
5) So! If I select one of these effects now, my image taken by the HD FaceTime Camera (integrated) appears regularly in the video window of Photo Booth, obviously with the effect superimposed on me (birds or hearts surrounding my head).
HYPOTHESIS
All other Photo Booth's Quartz filters contained in /System/Library/Compositions/ are not accepted by Big Sur which perhaps sees a conflict with an officially unsupported graphics card.
FOLLOWS A MY PROVISIONAL SOLUTION which works in my case.
1) I make a copy of "Dizzy.qtz" effect contained in /System/Library/Compositions/ as it is one of only two working effects.
2) I have modified this effect, being in XML, using BBEdit and I experimented (casually) with cuts to the part concerning the birds that revolve around the head. I've been lucky!!!
3) So! I renamed "Dizzy.qtz" to "Normal.qtz"
4) I made a copy of Photo Booth which I called "Photo Booth my" and replaced the "Normal.qtz" file contained in Photo Boot me.app/Contents/Resources/Normal.qtz with the "Normal.qtz" file created by me.
5) Now my "Photo Booth my" opens showing normally my image taken normally and without overlay effects. Enjoy!
For those who want, I am attaching my "Normal.qtz" file. and if you don't want to create a copy of Photo Booth and use the file as explained above, put the file in the You_Home/Library/Compositions/ folder or in Macintosh HD/Library/Compositions/ (for all Mac users) and logout and log-in or restart your Mac.
In this case, starting the official Photo Booth app the black window will continue to appear; however, by clicking on Effects and scrolling through the pages, you will see the Normal filter with the image taken by the Facetime HD Camera (internal) without any effect on you. You can then select this normal effect and use it until a better solution than mine is found.
I hope there may be a more effective and radical solution that ALLOWS Photo Booth and therefore Big Sur to activate ALL other Quartz effects, as if it were an original iMac supported by Big Sur.
Kronokernel, Ausdauersportler and other talented friends, please help us, kindly