What you describe may be one potential technical reason why Metal does not offer unified virtual memory, although I am not 100% convinced the cache flushing works exactly as you describe. Metal indeed requires you to tell it which memory objects are in use during a pass, but not the exact pointer range. There is nothing preventing you from allocating one huge Metal buffer and doing all your sub-allocations from there. In fact, that’s the preferred way since every buffer incurs additional overhead (and Metal gives you the heaps API to simplify these things). In summary, the system must already have a way to track which parts of the cache need flushing and so my guess is that the “use” APIs are there primarily to ensure that the resources are in memory and bound to the appropriate page tables. Vulkan doesn’t need the “use” API because all resources are explicitly assumed to be resident anyway, unlike Metal.
From the API consumer perspective the issue is that the CPU and the GPU use different virtual address spaces. And anyway, the buffer handle is just the GPU virtual address. It’s even called GPU address in the API and you can write valid GPU pointers from the CPU using these APIs. Real handles are used for textures and samplers and they indeed encode offsets into the per-process descriptor tables (same system Nvidia uses).
I don't know the API well, but I assume that you are referring to what is called "Private" memory?
As far as I can tell, the issue is not that that lives in a separate address space; it is that while that memory lives in the same physical address space, there is (by definition) no reason that the CPU should ever want to view it.
The consequences are that while I am using such address space, IF NECESSARY, it can overflow L1 to L2, to SLC, all the way to DRAM. But under normal circumstance ideally it will live purely in L1 and L2 and not even flow out to SLC. And then, when I am done using this memory, I can simply invalidate the cache lines; I have no obligation to flush them to SLC and DRAM because the reason to do that is to make them visible to the CPU.
......
Another issue where Apple and nVidia are different is that nVidia breaks up the 64 bit address space into regions that are mapped to different hardware types. Thus, simply by looking at the high bits of an address, you know where to route it (eg either device memory or which Scratchpad of which core).
Metal has typed pointers so that a pointer that accesses Scratchpad is a different type of thing from a pointer that accesses device memory (and uses different instructions, eg tile_load, device_load, stack_load, threadgroup_load). nVidia's scheme definitely seems more elegant and powerful, and maybe one day Apple Silicon will adopt it. But for now this seems an additional issue – you can't, for example, simply sling around these pointers and store them in generic data structures (eg trees or whatever), then extract them and know how to use them.
I know even less about Blender than about graphics APIs, but what may be meant by "Handle" here is simply Apple (or at least someone concerned about getting this working on Apple Silicon) suggesting that instead of storing raw pointers in various complex data structures, we wrap each pointer in a very minimal shell (call it a "handle", because that's the idea, regardless of what Metal means by Handle), store that handle in the various complex data structures, and use a few well-defined dereference macros (or properties if you want to go Object Oriented) to map from the handle to the pointer.
This would all be free (shell and dereference are no-op) for nvidia, while doing whatever Apple requires (eg maybe the dereference macro has enough info from the handle extra bits to generate the appropriate Scratchpad vs Device load instruction).
