I may be missing what you're saying here. Possibly because you are too terse, but also possibly because of a gap in my understanding.
Are you saying that you think that memory addresses (presumably hw addresses, but reflecting through into vm addresses) should be distributed page-by-page across both Max dies? This seems fairly fantastic to me - wouldn't that make it impossible to take advantage of memory locality to improve performance?
I am naively assuming that it is optimal for the working set of a process (thread) to be proximate to the core running it. If you spread adjacent pages across both chips, that's impossible (right?).
Thinking about this more, it's clear that if you have a set of threads that are large enough to require simultaneous use of both dies, in which individual threads frequently access the same pages, rather than mostly touching pages exclusive to themselves, this goes out the window. But that's a worst-case situation, and not (I thought, at least) all that common. Or am I wrong about that? And are there other considerations I'm missing?
Suppose you have eight DRAM chips, let's say each 1GB in size. (yes, yes, I'm simplifying tremendously to make a point)
How do you distribute addresses across them?
The naive way is to route all physical addresses from 0GB to 1GB to the first DRAM, then from 1GB to 2GB to the second DRAM, and so on. But is this optimal? It will tend to locate large blocks of contiguous physical memory (as in when an app or the OS asked for a large block of memory) in one DRAM chip, so we can't stream data from multiple DRAM chips at once.
And alternative is to rearrange the address bits ("hash" the address) so that say the first 64kB goes to the first DRAM, the next 64kB to the second DRAM, and so on. This will enhance the likelihood that various common use cases (like sequential streaming) are spread across multiple DRAMs.
Now how small should we make these units? The optimal size is probably the size of a DRAM page (not an OS page). A DRAM page is the smallest unit of DRAM that can be active. For LPDDR5 they're probably 1, 2 or 4kB depending on the exact DRAM details. So that's probably the unit we should interleave.
Note that this is all about how PHYSICAL (not virtual) addresses are routed to DRAM. It's invisible to any hardware above the memory controller.
Now take those same ideas and apply them to an Apple style design with an SLC. Suppose we have, say, two SLC blocks (SRAM plus memory controller) like on say an M1 Pro. [Again this is a simplification, work with me and just assume there are two, one on the left side, one on the right.]
Once again, how do we split data across those SLCs? The dumb alternative is to associate say the upper half of the physical address range with the left SLC, the lower half of the physical address range with the right SLC, for the same sort of reasons - in many cases you'll have most of the activity hitting one SLC but not the other.
Once again a much better choice is to interleave a smaller unit between the SLCs, so that, eg, the first 128B of physical address space routes to one SLC, then the second to the other SLC and back. Once again, now, common patterns like sequential streaming will alternate between SLCs.
Obviously we want to take advantage of both these ideas. This means we probably don't want to alternate SLC at the 128B level because that doesn't work for DRAM; we need to alternate addresses between SLCs at the DRAM page granularity, then the memory controller can further hash addresses to stripe over as many independent DRAM elements as it controls.
The same ideas then scale up to an M1 Max, and then an M1 Ultra.
The exact details will depend on capacities, timing, simulations, etc. The memory controllers and NoCs are programmable, so all this stuff can be varied. The obvious way to vary it is by load - if the load is small it's certainly possible in principle to freeze the system, flush SLCs, move data between DRAMs, reprogram the hash pattern, and restart. This would allow you to, for example, power down half your DRAMs, memory controller, and SLC. Apple have patents for doing this, but it's tricky getting the sequencing correct (and knowing when you should reduce or increase capacity) so I don't know the extent to which they actually do this.
I hope this explains the issues. At the virtual level all addresses are still contiguous. At the physical level, when streaming you'll ping pong between the two Max SLCs and sets of DRAM. Is this slower than using just data on the one Max? Define "slower"...
You will have higher latency to the SLC for half the requests, but also twice the SLC capacity, twice the number of open pages in DRAM, and twice the DRAM bandwidth.
I'm not sure what the optimal details are, but Apple systems have lots of telemetry tracking these things (along with simulations) so I'm sure they do a reasonable job of toggling between use cases.
This is the patent for dynamic address hashing:
https://patents.google.com/patent/US20220342806A1
(the actual patent describes a way of saving power by stripping bits off the hashed address as the data packets move through routing, but the background is address hashing)
My gut feeling is that most of the time either
- larger SLC (with half of it slightly higher latency, but still much better than DRAM) wins [small jobs] OR
- larger bandwidth wins [large jobs]
so in most cases it's a win.
BTW this business of address hashing has consequence people don't think of.
M3 Pro has THREE address controllers and SRAM block (as opposed to the four of M1 and M2 Pro).
OK, that's fine, but how do you spread addresses across these three (apart from the naive and dumb method of splitting the physical address range into three)???
You need address hashing circuits that are more sophisticated than just rearranging address lines, as described in http
https://patents.google.com/patent/US20240104017A1
