Observations & Discussion on Apple Silicon Graphics Performance with MetalFX Upscaling

[Malus120]

Looks like the Digital Foundry video about MetalFX is up

Well that is an interesting take. MFX Perf is spatial upscaling while quality is temporal. Not sure I am a fan of Capcom just going quarter resolution for the upscaling (in both modes).

The Digital Foundry review was illuminating as expected.

While their analysis confirms a lot of what I initially observed (quality mode is a temporal upscale achieving near native quality with a >50% performance uplift, performance mode is basically unusable), I found several things quite surprising:

1. Quality mode is achieved with a 4x upscale! That the quality is able to rival (and in some cases exceed) native 4K with a 1080P internal resolution is quite impressive indeed. I had assumed they were working with an internal res of at least around 1440P to get a result like this.

2. Disocclusion seems to be handled well (at least in this game), which is great news as this is something FSR 2.x can struggle with, but transparencies and hair need (a lot) of work vs the competition. Overall it seems comparable in quality to similar techniques (DLSS, FSR, XeSS) aside from the aforementioned weakness but as DF said, more datapoints (games) are needed.

3. The performance uplift for a 4x upscale is a little bit less impressive than one might expect. It seems like MetalFX Temporal Upscaling, at least in its current implantation has a significant frame time cost.

4. Performance mode is using the spatial upscaler (🤢). All the image breakup had me thinking it was just a really bad/low res temporal upscale but nope. This explains the poor quality (especially rendering from 1/4 res.) I honestly don't see the point of this when the Apple's temporal upscaler can achieve such great results at 4x. Even a 5-6x scale should give better results than this, and its already blown it out of the water by just quality mode at lower resolution.

6. The resolution issues people were speculating about (saying Mac 4K doesn't match PC 4K) didn't merit a mention in DF's video. While I'm not saying that's definitive proof it's a non issue, I think given that DF pixel counted a wide variety of scenes, I'm inclined to believe that if there is a difference in image quality it's likely related to the way the game is rendered on macOS (perhaps Metal is missing a feature, implements something in a way which impacts the final result) and not the actual number of pixels the Mac version is rendering. IMHO, it doesn't seem to be a big deal.

Overall a VERY strong first showing for MetalFX's temporal upscaler, and a real boon for Apple Silicon gaming.
The port itself is, as they note, less impressive (albeit serviceable). Performance isn't in line with what we'd expect given the GPU power available and shader compilation stutter is unfortunately still present (just like the PC version) despite Apple offering API's to avoid this :/

I think the main question now is how committed Apple is to continuing to develop and improve MetalFX. DLSS and FSR are constantly improving and have made huge strides in perceptual resolution, dealing with transparencies, disocclusion, fine detail (hair, fences, wires.) MetalFX Temporal Upscaling is an excellent first effort, but if left to rot will quickly become less impressive compared to the competition.

The future looks like it could be bright. I'm excited to see how No Man's Sky and RE Village Gold Edition DLC (Shadows of Rose, 3rd person mode, etc) hold up.

 

[mi7chy]

Too bad games that will have MetalFX support will unlikely ever have competing hardware agnostic open source upscaling like FSR and XeSS since Apple probably won't allow it. Otherwise, FSR 2.1 is looking pretty good in Gotham Knights at 4K max graphics with fast frame rates. It also has XeSS for comparison.

 

[leman]

Too bad games that will have MetalFX support will unlikely ever have competing hardware agnostic open source upscaling like FSR and XeSS since Apple probably won't allow it. Otherwise, FSR 2.1 is looking pretty good in Gotham Knights at 4K max graphics with fast frame rates. It also has XeSS for comparison.

What do you mean "Apple won't allow it"? FSR is under an open source license. Go port it to Metal and use it however you please.

 

[jmho]

Resident Evil Village already uses a bunch of AMD tech in it on the Mac like AMD FidelityFX CACAO for ambient occlusion

 

[diamond.g]

What do you mean "Apple won't allow it"? FSR is under an open source license. Go port it to Metal and use it however you please.

IIRC BG3 uses FSR on the Mac port. I don't remember if it is 1.0 or 2.0 though.

 

[diamond.g]

The Digital Foundry review was illuminating as expected.

While their analysis confirms a lot of what I initially observed (quality mode is a temporal upscale achieving near native quality with a >50% performance uplift, performance mode is basically unusable), I found several things quite surprising:

1. Quality mode is achieved with a 4x upscale! That the quality is able to rival (and in some cases exceed) native 4K with a 1080P internal resolution is quite impressive indeed. I had assumed they were working with an internal res of at least around 1440P to get a result like this.

2. Disocclusion seems to be handled well (at least in this game), which is great news as this is something FSR 2.x can struggle with, but transparencies and hair need (a lot) of work vs the competition. Overall it seems comparable in quality to similar techniques (DLSS, FSR, XeSS) aside from the aforementioned weakness but as DF said, more datapoints (games) are needed.

3. The performance uplift for a 4x upscale is a little bit less impressive than one might expect. It seems like MetalFX Temporal Upscaling, at least in its current implantation has a significant frame time cost.

4. Performance mode is using the spatial upscaler (🤢). All the image breakup had me thinking it was just a really bad/low res temporal upscale but nope. This explains the poor quality (especially rendering from 1/4 res.) I honestly don't see the point of this when the Apple's temporal upscaler can achieve such great results at 4x. Even a 5-6x scale should give better results than this, and its already blown it out of the water by just quality mode at lower resolution.

6. The resolution issues people were speculating about (saying Mac 4K doesn't match PC 4K) didn't merit a mention in DF's video. While I'm not saying that's definitive proof it's a non issue, I think given that DF pixel counted a wide variety of scenes, I'm inclined to believe that if there is a difference in image quality it's likely related to the way the game is rendered on macOS (perhaps Metal is missing a feature, implements something in a way which impacts the final result) and not the actual number of pixels the Mac version is rendering. IMHO, it doesn't seem to be a big deal.

Overall a VERY strong first showing for MetalFX's temporal upscaler, and a real boon for Apple Silicon gaming.
The port itself is, as they note, less impressive (albeit serviceable). Performance isn't in line with what we'd expect given the GPU power available and shader compilation stutter is unfortunately still present (just like the PC version) despite Apple offering API's to avoid this :/

I think the main question now is how committed Apple is to continuing to develop and improve MetalFX. DLSS and FSR are constantly improving and have made huge strides in perceptual resolution, dealing with transparencies, disocclusion, fine detail (hair, fences, wires.) MetalFX Temporal Upscaling is an excellent first effort, but if left to rot will quickly become less impressive compared to the competition.

The future looks like it could be bright. I'm excited to see how No Man's Sky and RE Village Gold Edition DLC (Shadows of Rose, 3rd person mode, etc) hold up.

Yeah it is a boon for sure, especially since MetalFX is on iOS and iPadOS as well.

I think the DF video insinuated that RE:Village isn't that GPU heavy without RT turned on, and some folks would argue that the RT the game does have is waaay lighter compared to others (like Metro Exodus).

I am excited to see No Man's Sky. I am going to have to think about if I want to buy it on MAS assuming it will be exclusive to that store.

 

[Malus120]

Yeah it is a boon for sure, especially since MetalFX is on iOS and iPadOS as well.

Indeed, the point about MetalFX being available on iOS and iPadOS (and presumably eventually tvOS) along with macOS is an important one that'd I'd actually forgotten.

Assuming MetalFX Temporal Upscaling can deliver a similar level of visual quality and performance uplift on iPhone class hardware it should quickly become a core piece of the iOS game development stack for developers around the world. This has important implications for the kinds of experiences that can be delivered on iOS devices, means there will be no shortage of people familiar with Metal/MetalFX when it comes to AAA ports for Mac, and also makes it much more likely that Apple will continue to invest in its development (whereas if it was Mac only I could see it languishing for years without updates.)

I think the DF video insinuated that RE:Village isn't that GPU heavy without RT turned on, and some folks would argue that the RT the game does have is waaay lighter compared to others (like Metro Exodus).

I didn't get that impression. It's true that RE:Village is fundamentally a cross generation release (PS4, PS5, Xbox One, Xbox Series S/X, PC, Mac) and thus can be run on a wide range of hardware (scales down well GPU wise,) but that doesn't mean that it doesn't take a good deal of GPU grunt to run the game at high resolutions and frame rates with high settings.
As far as the RT goes, I don't even think it's an argument to say that it's less demanding (and also less well optimized) than RT showcase titles like Metro Exodus Enhanced Edition, Control, Cyberpunk. It's definitely one of the simpler, more subtle implementations.

I am excited to see No Man's Sky. I am going to have to think about if I want to buy it on MAS assuming it will be exclusive to that store.

I don't really mind the exclusivity. I understand it turns a lot of people off (no one wants to pay more/buy twice), but I can't imagine the era of getting the Mac version of AAA titles for "free" with the PC version on Steam was very profitable for porting houses.
Companies are going to want to recoup development costs if they're going to take the time to port to Mac. Hopefully, the large number of Apple Silicon devices in circulation eventually makes this a non issue and we can see Mac games return to Steam/other platforms.

As for NMS itself, I haven't played it but I'm interested, probably won't get it at release as I have a large backlog ATM...

 

[hoo-man-b-ing]

I think the main question now is how committed Apple is to continuing to develop and improve MetalFX.

Two years ago I would have guessed that Apple would continue its relatively comfortable pace with improving Metal. However, with realityOS and related hardware expected to come to market in the next couple of years, rapidly iterating on Metal/MetalFX could prove wildly beneficial, especially since the devices are likely to be thermally constrained to passively cooled (e.g., M2) SOCs.

 

[Malus120]

Two years ago I would have guessed that Apple would continue its relatively comfortable pace with improving Metal. However, with realityOS and related hardware expected to come to market in the next couple of years, rapidly iterating on Metal/MetalFX could prove wildly beneficial, especially since the devices are likely to be thermally constrained to passively cooled (e.g., M2) SOCs.

That actually makes a lot of sense. While I was really happy with the reveal of MetalFX (and Metal 3 in general) I was also caught completely off guard. It really felt like it came totally out of left field.

That said color me a bit... skeptical...? of a lot of the rumors surrounding Apple's AR headset. Passively cooled M2s? That you're wearing on your heard? And they're supposed to deliver a convincing AR experience while remaining comfortably cool? In 2023!? It all just sounds waaaaaay too good to be true...

Then again, Apple generally avoids bringing things to the market until they're ready for mass adoption so maybe Apple really has done it (won't be cheap though, that's for sure), or people are just piecing together info based on what they're trying to do rather than what's actually being achieved in the lab / prototyping ATM...

 

[leman]

That actually makes a lot of sense. While I was really happy with the reveal of MetalFX (and Metal 3 in general) I was also caught completely off guard. It really felt like it came totally out of left field.

You think so? I can't say I was too surprised. They have been heavily investing in gaming APIs for a while. The scale of new offerings was quite impressive though, here I fully agree.

That said color me a bit... skeptical...? of a lot of the rumors surrounding Apple's AR headset. Passively cooled M2s? That you're wearing on your heard? And they're supposed to deliver a convincing AR experience while remaining comfortably cool? In 2023!? It all just sounds waaaaaay too good to be true...

Let's not forget that Apple also has another trick up their sleeve: variable rate rasterisation. They can apply a kind of a "fish eye" transform to the render target, increasing the resolution at a focal point and decreasing it elsewhere. Combine it with an eye tracker and you can deliver high quality rendering where it matters while maintaining much smaller render target that you would need otherwise. Add to it other bandwidth-saving techniques like automatic texture compression etc. and one could in principle extract quite a lot of performance from M2-class hardware for AR/VR applications.

 

[mi7chy]

Resident Evil Village already uses a bunch of AMD tech in it on the Mac like AMD FidelityFX CACAO for ambient occlusion

Time will tell if it continues to be split into Steam games that have FSR, XeSS, etc. vs Apple app store walled garden that's MetalFX only.

 

[jmho]

Time will tell if it continues to be split into Steam games that have FSR, XeSS, etc. vs Apple app store walled garden that's MetalFX only.

MetalFX is open for anyone to use. You can put it into a game and distribute it however you like, even on Steam. There are no limitations.

The main reason we aren't seeing FSR and XeSS in Metal games is because they're not ported to Metal.

Both FSR and XeSS are open source though, so if you wanted to port either to Metal (for whatever reason) you could.

 

[diamond.g]

Reviving this old thread in preparation of CDPR doing Frame Generation in CP2077. Is everyone assuming they are going to do it via a new Metal API, or are they going to just use FSR3 (which as far as I can tell no macOS game uses yet).

 

[diamond.g]

So Nvidia has announced multi frame generation. Displaying 33m pixels while only rendering 2m pixels.

 

[dmccloud]

So Nvidia has announced multi frame generation. Displaying 33m pixels while only rendering 2m pixels.

View attachment 2469830
View attachment 2469831

Nvidia got ripped by the computer and gaming press after the keynote because the 240 fps they cited is not a real framerate. What Nvidia is doing with DLSS4 is repeating the previous frame up to 3 times in order to "boost" the framerate. It's far easier for a GPU to push the same frame to the display 2-4 times in a row than it is to run the processing and generate 2-4 unique frames. This was also the number they used in the keynote to claim that the 5070 was achieving 4090 levels of performance despite only having a list price of $549. This latter claim has been dissected online several times, including over on Medium.

 

[diamond.g]

Nvidia got ripped by the computer and gaming press after the keynote because the 240 fps they cited is not a real framerate. What Nvidia is doing with DLSS4 is repeating the previous frame up to 3 times in order to "boost" the framerate. It's far easier for a GPU to push the same frame to the display 2-4 times in a row than it is to run the processing and generate 2-4 unique frames. This was also the number they used in the keynote to claim that the 5070 was achieving 4090 levels of performance despite only having a list price of $549. This latter claim has been dissected online several times, including over on Medium.

Is it really the same frame three times? Seems like that would introduce all kinds of stutter.

 

[diamond.g]

This still doesn't answer if Apple will switch to a Visual Transformer instead of a CNN (assumed this is what they are currently using for MetalFX) and if they are going to use straight AI for frame generation in Cyberpunk 2077.

 

[crazy dave]

Is it really the same frame three times? Seems like that would introduce all kinds of stutter.

It's not the same frame multiple times, but Nvidia's projection of what the frames will be. This may not be a problem for some games but it does introduce latency issues as well as potential for problems in games with key frames (e.g. think of games like Dark Souls where one is invulnerable during a specific action like a roll or where canceling an action needs to be immediate, timing that is crucial but now there is a disconnect between your input and what is happening on screen) - so certain types of games and gamers will not benefit.

 

[diamond.g]

It's not the same frame multiple times, but Nvidia's projection of what the frames will be. This may not be a problem for some games but it does introduce latency issues as well as potential for problems in games with key frames (e.g. think of games like Dark Souls where one is invulnerable during a specific action like a roll or where canceling an action needs to be immediate, timing that is crucial but now there is a disconnect between your input and what is happening on screen) - so certain types of games and gamers will not benefit.

I can't believe I am defending fake frames....

I'd assume in games where timing is critical frame generation won't be supported . Though nvidia does have Reflex2 which is supposed to take into account mouse movement when generating frames. I can't say if it is any good or not.

 

[crazy dave]

Though nvidia does have Reflex2 which is supposed to take into account mouse movement when generating frames. I can't say if it is any good or not.

It isn't just mouse movement. But yes the latency with the new system is better than the old frame generation, just not necessarily good enough

I can't believe I am defending fake frames....

I'd assume in games where timing is critical frame generation won't be supported .

And that's the problem, Nvidia's non-frame-gen uplift is seemingly not great this generation. So some people are being rather caustic towards Nvidia's marketing claims that rely on the frame-gen to achieve meaningful uplift. Whether or not that actually impacts Nvidia's sales of GPUs remains to be seen. The internet is not reality ... until it is.

 

[diamond.g]

@leman @crazy dave

Do you guys think it is possible for Apple to switch how MetalFX works on the backend like nvidia did for DLSS 4 CNN/Transformer models?

 

[dmccloud]

I can't believe I am defending fake frames....

I'd assume in games where timing is critical frame generation won't be supported . Though nvidia does have Reflex2 which is supposed to take into account mouse movement when generating frames. I can't say if it is any good or not.

It's a software-side implementation within the driver package itself, so it doesn't matter if the game officially supports it or not. nVidia doesn't care about the input latency, just the artificial framerates they can publicize to brag about their overpriced cards. Another part of the issue with these 50xx cards is that the generational performance uplifts are minor compared to prior transitions (e.g., 10xx to 20xx, 20xx to 30xx, 30xx to 40xx). In fact, the 5080 is the first 80-series card to have lower performance than the previous generation's xx80 and xx80 Ti series cards. I've seen several reviewers call the new frame generation routines terrible and even unplayable for some titles, which will not help Nvidia's case when they keep raising the price of their flagship 90-series GPUs without a corresponding performance boost. One potential advantage the 5080 has over the 5090 is headroom for overclocking. The 5090 runs at (and even slightly over) the 600W limit for the 12V connector, so there is no headroom available without undervolting the GPU. The 5080 has enough voltage headroom that performance may be boosted through overclocking, although I don't believe that they could squeeze enough performance out of the 5080 to even return to equal performance as the 4080

 

[novagamer]

It's a software-side implementation within the driver package itself, so it doesn't matter if the game officially supports it or not. nVidia doesn't care about the input latency, just the artificial framerates they can publicize to brag about their overpriced cards. Another part of the issue with these 50xx cards is that the generational performance uplifts are minor compared to prior transitions (e.g., 10xx to 20xx, 20xx to 30xx, 30xx to 40xx). In fact, the 5080 is the first 80-series card to have lower performance than the previous generation's xx80 and xx80 Ti series cards. I've seen several reviewers call the new frame generation routines terrible and even unplayable for some titles, which will not help Nvidia's case when they keep raising the price of their flagship 90-series GPUs without a corresponding performance boost. One potential advantage the 5080 has over the 5090 is headroom for overclocking. The 5090 runs at (and even slightly over) the 600W limit for the 12V connector, so there is no headroom available without undervolting the GPU. The 5080 has enough voltage headroom that performance may be boosted through overclocking, although I don't believe that they could squeeze enough performance out of the 5080 to even return to equal performance as the 4080

You can undervolt the 5090 and overclock the core to achieve ~25% performance increase over the 4090 at the same wattage (~380-440w).

Not defending nvidia but if you tweak the 5090 a little bit it gets a lot better than how they shipped it. It still sold out within seconds so this is all moot since nobody can get one because their manufacturing is focused on datacenters now and for the foreseeable future.

Using the same node for the 5090 was a mistake but maybe they had no other option, regardless it still sucks for consumers.


Unrelated aside not directed at you or anyone in paticular: Lord do I hate the phrase “uplift” that has spread like a virus because youtubers started using it a year or two ago. I wish people would say increase and use relative percentages. I want to uplift my foot into the ass of whoever first popularized it as the term du jour.

 

[crazy dave]

@leman @crazy dave

Do you guys think it is possible for Apple to switch how MetalFX works on the backend like nvidia did for DLSS 4 CNN/Transformer models?

I'm not sure. I know AMD did the same for its upscaling switch this generation, if the game supported FSR3 just swap the DLL file in the game and you'll get the new upscaling tech FSR4. FSR is open source and I believe Apple bases MetalFx on it, or can at least. So maybe?

You can undervolt the 5090 and overclock the core to achieve ~25% performance increase over the 4090 at the same wattage (~380-440w).

Not defending nvidia but if you tweak the 5090 a little bit it gets a lot better than how they shipped it. It still sold out within seconds so this is all moot since nobody can get one because their manufacturing is focused on datacenters now and for the foreseeable future.

Using the same node for the 5090 was a mistake but maybe they had no other option, regardless it still sucks for consumers.


Unrelated aside not directed at you or anyone in paticular: Lord do I hate the phrase “uplift” that has spread like a virus because youtubers started using it a year or two ago. I wish people would say increase and use relative percentages. I want to uplift my foot into the ass of whoever first popularized it as the term du jour.

I don't know who popularized recently but I did a quick search of these forums and apparently I've been using the term uplift on here to describe gen-on-gen improvements for at least 4 years ... so I guess, locally, that would be my posterior in need of an uplift 🙃

 

[novagamer]

I don't know who popularized recently but I did a quick search of these forums and apparently I've been using the term uplift on here to describe gen-on-gen improvements for at least 4 years ... so I guess, locally, that would be my posterior in need of an uplift 🙃

I’m sure I compressed time in my mind and it’s been 5 years so I think you are safe, these jargon phrases are sticky!

I watched a 50xx series video from Gamers Nexus (or similar channel) recently and they said it literally a dozen times within a minute or two and I’ve seen it pervasive on other parts of the internet, to the point that I think there is a younger generation that may just not use the phrase “increase” at all when talking about performance which is wild. I’m becoming the old person yelling at clouds.


So I’m still on-topic with the thread, apparently FSR2 works in BG3 if its run via crossover but even the latest native Mac patch still uses FSR1 (and introduced some stability issues). Unfortunate.