I do a lot of stuff with Machine Learning and so I would love to have the 192GB of RAM which is shared with the GPU, no question.
But.. the GPU that Apple is providing isn't that great for machine learning and in my current models I actually load stuff into GPU memory from system memory. I have a system running Linux with 256GB of RAM and an EPYC CPU from AMD. It has just over 200GB/s memory bandwidth so quite a lot less than the M2 Ultra.
But if I was speccing my system today I would actually go with 512GB of RAM and not 256GB like I did because not only has the price more than halved since I purchased that RAM (8 x 32GB 3200MHz DDR4 RDIMM -> 8 x 64GB is half the price with the same spec now) but my needs to train larger models with more data necessitates it.
Right now I'm hobbling along with 4 x 980 Pro 2TB's in RAID0 on a PCIe card for 8TB of working data for my models and I really throw those things through the ringer, they can't keep up with the 200GB/s system memory or the RTX 3090's I use that have almost 1TB/s memory bandwidth each etc
I would have really liked to buy a Mac Pro for my work but I wasn't under any illusions, I did not expect Apple to give us expandable memory or dedicated GPU support. According to NVIDIA it has been Apple that refuses to sign their system driver for graphics which is why (again according to NVIDIA) we never saw RTX 3000 support for their cards under macOS.
Of course, you could buy a previous generation Intel Mac Pro and boot Windows or Linux and then use these cards just fine and I know some people who did just that for the 25-30% of the time that they needed to do something with machine learning or if they wanted to play a game now and then.
Apple is a company that seldom does what I want when it comes to my professional working life but they do suit me just fine for life stuff. I should note I'm not editing video which seems to be the number one key task these machines are designed for.
If Apple were to make the perfect Mac Pro it wouldn't need too many changes. Expandable RAM, single or maybe dual-socket processors and the ability to use NVIDIA GPU's for machine learning tasks. How likely is that to occur, I think 0% likely and I'm okay with that, I can just buy or build a rackmountable system and fill it with all the graphics cards I like and to be honest, do I need macOS on that system? what would it provide me that my current system doesn't which is headless and I mainly access it over a WebUI from my web browser where I'm just monitoring job results etc