T2 security chip driver memory leak in Boot Camp (2018—2020 Intel Macs)

[tinygoblin]

I discovered a paged memory pool leak happening when any audio output of Apple Audio Device is active (this excludes Bluetooth devices because they use different driver) in Windows (in Boot Camp) and I request community help in collecting data regarding the issue.

The problem was proved to be present in 2019–2020 16-inch MacBook Pro equipped with T2 security chip (MacBookPro16,1 and MacBookPro16,4). I suspect all Apple computers equipped with T2 security chip (released in 2018 and later) are affected because they use the same Apple USB Virtual Host Controller Interface device in Windows (and experience same problems since very release of T2 security chip), here's official list, namely:

• MacBookPro15,2 (2018 & 2019 & 2020 13-inch MacBook Pro Four Thunderbolt 3 ports)
• MacBookPro15,4 (2019 & 2020 13-inch MacBook Pro Two Thunderbolt 3 ports)
• MacBookPro15,1 (2018 & 2019 15-inch MacBook Pro)
• MacBookPro15,3 (2018 & 2019 15-inch MacBook Pro)
• MacBookPro16,1 (2019 16-inch MacBook Pro with AMD Radeon Pro 5300M)
• MacBookPro16,1 (2019 16-inch MacBook Pro with AMD Radeon Pro 5500M)
• MacBookPro16,4 (2020 16-inch MacBook Pro with AMD Radeon Pro 5600M)
• iMac20,1 & iMac20,2 (2020 27-inch iMac)
• iMacPro1,1 (2019 iMac Pro)
• MaсPro7,1 (2019 Mac Pro)
• Macmini8,1 (2018 Mac mini)
• MacBookAir8,1 (2018 MacBook Air)
• MacBookAir8,2 (2019 MacBook Air)
• MacBookAir9,1 (2020 MacBook Air)

The problem was narrowed down to the activity of AppleUSBVHCI.sys driver of at least versions 6.1.7800.8 and 6.1.7800.9 (latest know versions extracted from Boot Cams Support Software for MacBookPro16,4 and iMac20,2 respectively). This driver provides access for other drivers (i.e. for AppleAudio.sys) to T2 security chip that consolidates various subsystem including audio chip. What happens is the audio driver asks AppleUSBVHCI.sys to allocate paged pool memory while AppleUSBVHCI.sys is active during playback (it's more or less normal for a driver) but after the playback stops the AppleUSBVHCI.sys does not free allocated paged pool memory it previously allocated (on the screenshot below the allocated paged pool memory blocks of 0.008 MB size have 9,223,372,036.854775807 seconds (292,47 years) free time meaning they never get freed while the normal time of freeing allocated blocks of 100 MB size is under 15 seconds):

It is important to address this issue because it may be not the driver leaking but the T2 security chip firmware (BridgeOS) has a bug. Only Apple can fix both things.

Since I do not own all the computers that may be affected, I ask everybody to contribute by submitting reports in the following manner:

1. Launch any media in any program to play for prolonged period of time (1 hour and more, the longer the better, and not via Bluetooth audio device) with a very low volume so it won't interrupt you, i.e.:
   – play 10 hours of absolute silence on YouTube
   – loop a MP3 file with VLC
2. Right-click Start Menu (Windows logo in the bottom left corner of screen) and launch Task Manager.
   Switch to Performance tab and to Memory row. Write down the Paged pool value. Close Task Manager
3. Use computer as usual but keep it lightly (without games or demanding software)
4. After 1+ hour of activity with a silence or any audio playing in the background (make sure computer is not in Sleep state!) check back to Task Manager > Performance > Memory and write down the Paged pool value
5. Post the result to this thread:
   – Mac model
   – Time elapsed
   – Paged pool size at start
   – Paged pool size at finish
   – Firmware version (use Command Prompt command systeminfo and look for BIOS field)
   – Apple USB Virtual Host Controller Interface version (optionally, look in Device Manager › System Devices)
   – Apple Audio Device driver version (optionally, look in Device Manager › Audio inputs & outputs)
   – Operating system version (use Command Prompt command systeminfo and look for OS version field)

Optionally you can also do the following in the very end of experiment:

1. Download PoolMon using this link
   PoolMon is a tiny (28 KB in size) command-line utility extracted from Microsoft Windows Driver Kit (WDK), it does not require installation. The link is not shady and is not a virus, but you can download entire WDK (which is 3.5 GB), install it and get poolmon.exe too.
2. Extract poolmon.exe to any folder
3. Hold Shift and right-click the folder with the extracted poolmon.exe and choose Open command window here
4. Enter the following command into the opened Command Prompt (there's no need to leave poolmon.exe running, it just reads current state of RAM poo usage):
   poolmon /p /p /b
5. Check if the first Tag is Pp
6. Report Pp size in bytes (as shown in PoolMon) to this thread

Are there any examples of what paged memory pool leak looks like?

• Capture at startup (no audio played yet, system fully booted), paged pool size 0,155 GB
  
  
  
  PoolMon
  RAMMap data
• Capture at 20 hours of audio playback, no other programs running, paged pool size 1.8 GB
  
  
  
  PoolMon
  RAMMap Data

 

[StellarVixen]

Did you submit this to the Apple (not that they care too much, probably, since Intel is done deal for them, but still...)?

 

[tinygoblin]

Thanks for the input, I appreciate! I have reported this issue via official feedback page — https://www.apple.com/feedback/ — the Boot Camp report form is inside macOS product section. However I still encourage everyone to double-check if I'm correct about the issue and report here. And also report to Apple via feedback form if possible (to gain traction).

Spoiler

The latter part is off-topic but I feel like commenting on it:

Did you submit this to the Apple (not that they care too much, probably, since Intel is done deal for them, but still...)?

Not sure why everyone here keeps repeating that Apple is not interested in Boot Camp. They still sell MacPro7,1 for $5999—$52'448.- which is an Intel computer and which is compatible with Boot Camp. All other 2019 Intel computers will continue to be supported and get software updates for at least 5 years after the last MacPro7,1 will be produced. Since no Apple Silicon machine is even close to handling decent amounts of RAM or continuous stress I believe there won't be an AS–based Mac Pro for at least 1 year. This means 2019 products and Boot Camp will be supported in some way will 2029. This holds nicely to the fact that usually Apple product becomes legacy after 10 years. Let alone Apple begged Microsoft to release an ARM-compatible Windows 10/11 like two years ago (Microsoft refused because of Qualcomm contract at the time).

 

[tinygoblin]

I have updated the initial claim to make it easier to read and since the cause of the leak has been discovered to be not the audio driver but the Apple USB Virtual Host Controller Interface device driver itself (which is a T2 security chip driver).

I hope someone finds time to participate. Leak was found in these configurations:

– Model: MacBookPro16,1, MacBookPro16,4
– Firmware: 1916.40.8.0.0 (iBridge 20.16.420.0.0,0), 1916.80.2.0.0 (iBridge 20.16.3045.0.0,0)
– OS: Windows 10, ver. 21H2 (10.0.19044), Windows 10, ver. 22H2
– AppleUSBVHCI.sys ver.: 6.1.7800.8, 6.1.7800.9 (latest)
– AppleAudio.sys ver.: 6.1.8000.2, 6.1.8000.3, 6.1.8100.4 (latest)

Previous versions of Windows OS will be tested shortly to check if it's Microsoft to blame.

 

[SoYoung]

Do you found a fix for this or have any cue from Apple about this?

 

[tinygoblin]

Do you found a fix for this or have any cue from Apple about this?

Hey there SoYoung! Sorry for delayed response.

Unfortunately I never got any response from Apple. I have submitted the issue dozen of times via Apple Feedback forms at official websites.

I have also contacted the author of Windows Internals book (basically a technical guide to Windows OS internals that explains how everything is organized under the hood and provided different troubleshooting methods). Unfortunately we couldn't look into it because of the lack of support from community. Basically nobody have ever supported this thread here or at Stack Exchange. So we suspect the issue is there but we've got only two Mac equipped with T2 chip.

Please let me know troubleshooting data (the one in the step 5 of opening message of this thread) and also please include information if your Boot Camp partition is on external or internal SSD of your Mac. You can do it via PM if you wish so!

P.S. I have also developed a fix for poor audio output in Boot Camp for 2019/2020 16-inch MacBook Pro and I'm looking for other owners to gather feedback before releasing it.

 

[Rimmsi]

Hi, according to your steps, I tested whether my MP 7.1 shows the memory leak you describe. I ran the test overnight to limit the possibility of being affected by the use of the device. After eight o'clock in the evening I started the Aver Media app to watch TV, I left it running all night with Core Temp and HWMonitor running in the background. I took the first Task Manager snapshot at 9:15 PM. Paging pool usage was 783MB. I did another check at 5:00 in the morning, paging pool usage was 783MB. Then I started internet radio in Chromium browser, now before 9:00 am the pool usage is 810MB. I attribute the increased usage to the browser itself rather than the memory leak you describe.

Model: Mac Pro 7.1 2019
Firmware: 1968.120.12.0.0 (iBridge: 20.16.5060.0.0,0)
OS: Microsoft Windows 10 Professional Workstation (x64) Build 19045.3208 (22H2)
AppleUSBVHCI.sys ver.: 6.1.7800.11
AppleAudio.sys ver.: 6.1.7100.0

 

[tinygoblin]

I attribute the increased usage to the browser itself rather than the memory leak you describe.

Hello Rimmsi, thank you so much for a meaningful report! I agree, your system works fine. Would you kindly also specify if you're running Boot Camp from internal SSD or from external SSD? And please let me know if you happen to remember the source of your AppleUSBVHCI.sys driver (the latest I could fetch using brigadier is 6.1.7800.8).

 

[Rimmsi]

BootCamp running from SSD Samsung 970 Pro on PCIe card Sonnet Fusion M.2 2x4, drivers are from Brigadier, but I don't upgrade drivers for Mac Pro because it usually gives me problems.

 

[tinygoblin]

Thanks so much for taking time to go into details Rimmsi, I appreciate! Since brigadier fetches AppleUSBVHCI 6.1.7800.5 for -m MacPro7,1 for me for some reason, it would be nice if you shared your AppleUSBVHCI.sys and AppleUSBVHCI.inf, AppleUSBVHCI.cat.

 

[Rimmsi]

No problem. I think this is because you have a MacBook Pro and there is a different driver package for that device.

 

[tinygoblin]

Hey there Rimmsi, unfortunately the drivers you attached are AppleAudio 6.1.7600.0 and AppleUSBVHCI 6.1.7500.5. The versions you reported earlier happen to be 6.1.7100.0 & 6.1.7800.11 respectively.

P.S. The least AppleAudio version that supports MacBookPro16,1 is 6.1.7700.3.

 

[Rimmsi]

It is strange. I extracted these files directly from the folder where I have the BootCamp files downloaded with Brigadier and installed on my Mac Pro.