Ray tracing on Apple Silicon?

[sunny5]

I saw a lot of people including myself wishing to use Unreal Engine on Apple Silicon Mac natively but so far, they are not interested at all. Maybe because of the lawsuit between Epic and Apple? Hmmm... Someone mentioned that Apple Silicon's GPU sucks because it has no capability to run ray tracing features on Unreal engine and therefore, there won't be a native support for AS Macs. Well, not an official statement but lack of ray tracing and unreal engine support is quite disappointing so far. I dont know if Apple is gonna make a GPU for ray tracing since they weren't interested in gaming industry but for 3D, it can be useful.

Ray Tracing with Metal - WWDC19 - Videos - Apple Developer

Metal Performance Shaders (MPS) harness the massive parallelism of the GPU to dramatically accelerate calculations at the heart of modern...

developer.apple.com

Enhance your app with Metal ray tracing - WWDC21 - Videos - Apple Developer

Achieve photorealistic 3D scenes in your apps and games through ray tracing, a core part of the Metal graphics framework and Shading...

developer.apple.com

Explore hybrid rendering with Metal ray tracing - WWDC21 - Videos - Apple Developer

Discover how you can combine ray tracing with your rasterization engine to implement simplified graphics techniques and elevate visuals...

developer.apple.com

Discover ray tracing with Metal - WWDC20 - Videos - Apple Developer

Achieve photorealistic 3D scenes in your apps and games through ray tracing — a core part of the Metal graphics framework and Shading...

developer.apple.com

WWDC18 - Videos - Apple Developer

developer.apple.com

I know that Apple had been working on ray tracing since 2018 but they still didnt add any ray tracing features so far. Some people say Apple Silicon chip does not have a dedicated ray tracing chip or the GPU isn't powerful enough to use ray tracing. Since Both Nvidia and AMD have ray tracing technologies, it's a huge question that Apple need to show or answer. Not just for gaming but also for 3D works.

Do you think Apple will add and support ray tracing with Apple Silicon chip?

 

[crazy dave]

Yes? It’s not a question of “being powerful” enough. It’s just adding the necessary functional blocks. Apple recently penned a new deal with ImaginationTech (the company that used to make the iPhone GPU before Apple switched to an internal design) and people believe that it’s for their recently announced ray tracing tech. As to when we’ll see it being used … ?‍♂️.

 

[leman]

I know that Apple had been working on ray tracing since 2018 but they still didnt add any ray tracing features so far.

I am not sure what you mean by this. Metal includes state of the art raytracing API.

Some people say Apple Silicon chip does not have a dedicated ray tracing chip or the GPU isn't powerful enough to use ray tracing.

Apple GPUs currently do not have ray tracing acceleration hardware. So RT is slow. Not really usable for games (except some limited cases), but fine for certain pro use.

Since Both Nvidia and AMD have ray tracing technologies, it's a huge question that Apple need to show or answer. Not just for gaming but also for 3D works.

Do you think Apple will add and support ray tracing with Apple Silicon chip?

I think they will have hardware RT sooner or later. I was actually hoping for it in A15 but it didn’t seem to happen.

 

[JMacHack]

RT hardware is likely coming in future Apple Silicon.

iirc Unreal works on other iOS games, so I can’t imagine that an Apple Silicon version is technically impossible.

The Epic v Apple fight is dumb but I wouldn’t rule out Tim Sweeny blocking any Unreal releases for MacOS. It’s an easier market to ignore than iOS.

So native Unreal support is up in the air.

 

[tomO2013]

@sunny5 - ray tracing is supported via API on Apple Silicon. Apples implementation of a TBDR provides a natural advantage over traditionally computed immediate mode rendering and if you are using Metal, the performance will be a LOT faster than typical software based implementations of ray tracing.
Here is a nice developer track if you wish to learn more:

Accelerating ray tracing using Metal | Apple Developer Documentation

Implement ray-traced rendering using GPU-based parallel processing.

developer.apple.com

However as already noted there is no dedicated hardware on Apple Silicon - such as with nVidia RTX - at this juncture so performance on ray tracing (e.g. for games) won’t be as good as say a Windows / RTX equivalent.

That being said , I’d hazard a guess that Apple spent their silicon budget wisely NOT including ray tracing as today it is still a niche and incurs a significant energy and transistor budget in NVidias RTX graphics cards. Even in games with NVidias second generation RTX using DLSS 2 enabling RTX comes with a significant performance and energy consumption penalty of 350Watts TDP max.

Apple has been suspected to be licensing PowerVR Immagination Technologies IP for many years owing to the TBDR technologies used in A series chips for many years.
Although I cannot say definitively that the latest M1 GPU is utilizing their designs or not. I’m sure leman or Cmaier or somebody a lot smarter than me could possibly offer an opinion.
I’m personally hoping that Apple are intending to at least license or provide an equivalent to PowerVR CXT in their hardware GPU into the future. I say this because the power VR approach (if we believe their marketing) is a vastly more energy efficient approach to ray tracing than nVidias RTX hardware acceleration.

IMG CXT GPU https://www.imaginationtech.com/graphics-processors/architecture/powervr-photon-architecture/

PVR Ray Tracing Simulation Layer - Imagination Developers

developer.imaginationtech.com

If and when ray tracing does come to mac, I suspect it will be driven by creative app acceleration needs rather than gaming.
I say this because gaming on mac does not have the widespread adoption (chicken/egg on availability of first party AAA titles). Typically the mac gets a lousy port (if at all) a few years after initial PC release.
Considering the market place shift towards subscription cloud services - if you want to game on any AAA games with ray tracing, you don’t have to spend thousands of dollars on a high end RTX3090 anymore … you can simply sign up to GeForce Now , XBOX game pass and other streaming subscription services (Geforce now grants you access to RTX) right from your Mac!
The ‘local need’ for ray tracing (while it would be welcome by me to see some hardware acceleration) is simply not there for Apple and gaming right now.

With respect to M1’s design and further thoughts on why Apple didn’t include dedicated hardware - I believe that the perennial need to balance between transistor budget, energy efficiency targets, and a determination that greater value would be gotten for Apples target audience providing acceleration for ISP, encryption, neural learning, matrix co-processing, etc… Again, for the types of workload that creatives do - an M1 part with a neural engine would be of higher value than an M1 with ray tracing hardware but no Neural engine!!
Finally, from a ray tracing perspective, many 3d rendering houses, farm out their rendering tasks to dedicated render farms where you can easily avoid the local hardware energy and heat penalty of a high wattage local RTX part…..

This is not to say that it is a pity that there is NO dedicated ray trace hardware, but more to say that it’s not that big a deal when you actually sit back and look at the lay of the land right now! (In my humble opinion anyway).

 

[Kazgarth]

Ray tracing is definitely is coming to Apple's silicon, sooner or later.

If I would speculate, they'd be more comfortable to include it when they transition to 3nm node. As it will give them huge headroom to jam those ray tracing accelerators in the same silicon space.

And it will be even more important to them when they enter the VR space, where RT is necessary for realistic immersion in VR games & experiences.

 

[jmho]

Part of me thinks that this is the reason Apple got in on the Blender development fund.

If they come out with Raytracing hardware in a future release, it would be pretty sad if they didn't have any software to take advantage of it, which Apple don't given that they don't make any 3D creation software or games.

Seems like a decent plan to get something big like Blender ready and running on their own APIs before dropping the hardware so they can show the actual real world speed-up.

 

[hefeglass]

Does blender utilize the rtx hardware on nvidia cards currently? i was under the impression the raytracing hardware acceleration is used mainly for games.

 

[leman]

Part of me thinks that this is the reason Apple got in on the Blender development fund.

If they come out with Raytracing hardware in a future release, it would be pretty sad if they didn't have any software to take advantage of it, which Apple don't given that they don't make any 3D creation software or games.

Seems like a decent plan to get something big like Blender ready and running on their own APIs before dropping the hardware so they can show the actual real world speed-up.

Apples effort in Blender is so far about making their CUDA renderer compatible with Metal via targeted use of macros etc. Adding Metal RT would probably require a completely new tenderer backend which is not in scope from what I’ve seen.

 

[jmho]

Does blender utilize the rtx hardware on nvidia cards currently? i was under the impression the raytracing hardware acceleration is used mainly for games.

Yes, Blender has an entire OptiX backend (which is nVidia's proprietary API for ray tracing)

Apples effort in Blender is so far about making their CUDA renderer compatible with Metal via targeted use of macros etc. Adding Metal RT would probably require a completely new tenderer backend which is not in scope from what I’ve seen.

Yeah, it won't be coming any time soon, but hopefully Apple will keep contributing long-term.

 

[throAU]

One big thing the M1x has that will help massively with ray tracing (and high res 3d in general) are the AI/ML cores.

Not so much for RT directly, but more for an Apple equivalent to Nvidia's DLSS for intelligent upscaling of a lower resolution image for better performance.

 

[leman]

One big thing the M1x has that will help massively with ray tracing (and high res 3d in general) are the AI/ML cores.

Not so much for RT directly, but more for an Apple equivalent to Nvidia's DLSS for intelligent upscaling of a lower resolution image for better performance.

There is no evidence that Apples ML accelerators are capable of doing this kind of upscaling efficiently. The denoising filters Apple offers run on the GPU, not the NPU.

 

[throAU]

There is no evidence that Apples ML accelerators are capable of doing this kind of upscaling efficiently. The denoising filters Apple offers run on the GPU, not the NPU.

DLSS or the like is still an AI/ML problem. There's no evidence yet, but if they have ML/AI acceleration on the SOC, there's perhaps potential there to use it in the future.

Whether they do or not is another question entirely, but... they're already using AI/ML on the phones and iPads, etc. for image recognition (e.g., search photos for dogs, cats, cars, bikes, etc.).

 

[metapunk2077fail]

Ray tracing will come. It will be a standard feature set of all GPUs in the world. After that it will even come to all the iGPUs too.

Right now it is a new feature and very energy intensive. For heat management reasons it is not good to enable RT on a laptop or thin AIO desktop.

In a couple of generations computers will be powerful enough and efficient enough to do RT with a much smaller energy footprint.

 

[leman]

DLSS or the like is still an AI/ML problem. There's no evidence yet, but if they have ML/AI acceleration on the SOC, there's perhaps potential there to use it in the future.

Whether they do or not is another question entirely, but... they're already using AI/ML on the phones and iPads, etc. for image recognition (e.g., search photos for dogs, cats, cars, bikes, etc.).

Image upscaling is a fundamentally different problem than image recognition though, I am not sure the NPU was built for that (or that it even has the horsepower required for fast ML upscaling in the first place). But then again, it's not really Apple's problem — applications have access to the API and can implement whatever upscaling techniques they see fit. AMD's FidelityFX is open source and can be easily ported to Metal for example.

If you hope for Apple to release a game-optimized upscaling solution like Nvidia's DLSS, well, I don't think this will ever happen. Nvidia's has access to tremendous body of data they can use for training, Apple has nothing of sort.

 

[metapunk2077fail]

If you hope for Apple to release a game-optimized upscaling solution like Nvidia's DLSS, well, I don't think this will ever happen. Nvidia's has access to tremendous body of data they can use for training, Apple has nothing of sort.

They can buy that data from AWS.

 

[leman]

They can buy that data from AWS.

AWS has that kind of data? Since when?

 

[metapunk2077fail]

AWS has that kind of data? Since when?

I think they have just about every data type and training set imaginable. So many companies use AWS for training data.

 

[leman]

I think they have just about every data type and training set imaginable. So many companies use AWS for training data.

Many companies use AWS to rent compute time to train models, sure. But what you say suggests that Amazon itself has ben running a comprehensive network of gaming machines collecting high-res game frames with all the accompanying metadata. I've never heard about anything like that. So far the only company that has this kind of infrastructure that I know of is Nvidia.

 

[metapunk2077fail]

Many companies use AWS to rent compute time to train models, sure. But what you say suggests that Amazon itself has ben running a comprehensive network of gaming machines collecting high-res game frames with all the accompanying metadata. I've never heard about anything like that. So far the only company that has this kind of infrastructure that I know of is Nvidia.

What I was suggesting was that when third parties use AWS for training models then AWS might also be gathering and using those data sets. I know they are doing that on mturk.com so it might extend to everything else.

 

[leman]

What I was suggesting was that when third parties use AWS for training models then AWS might also be gathering and using those data sets. I know they are doing that on mturk.com so it might extend to everything else.

You are suggesting that Amazon is telling their paying customer's data? That would be... scary?

 

[throAU]

If you hope for Apple to release a game-optimized upscaling solution like Nvidia's DLSS, well, I don't think this will ever happen. Nvidia's has access to tremendous body of data they can use for training, Apple has nothing of sort.

Not what I'm saying at all.

They aren't gaming machines and I'm not using mine for gaming. I have a gaming PC and a house full of retro consoles for that.

My point is merely that they have dedicated AI processing cores that would be useful for the kinds of workloads that assist with getting ray tracing usable (mostly, upscaling and de-noising).

 

[jmho]

I believe Intel is working on an open ML based competitor to DLSS called XeSS that should work on all cards (much like AMD's FSR, except FSR isn't ML based)

 

[diamond.g]

This seems like a continuation of the TBDR/RT talk @leman started a while ago.

 

[metapunk2077fail]

You are suggesting that Amazon is telling their paying customer's data? That would be... scary?

Some of them work on open data sets that they freely share for 'the benefit' of the sector.