My suggestion would be to put the original four DIMMs in the desk drawer (in case they're needed to shlep the 7,1 into the Apple store for troubleshooting), and get six 64-GiB LRDIMMs for 384 GiB. Several reasons:
- Your "48 GiB for system and caches" seems like something pulled from thin air, not from a real analysis of system and caching performance. When I work with large files, or many medium files, it's no t unusual to see hundreds of GiB in the caches. Huge caches are good - don't starve the system for cache space.
- You'll have to "shelf" the originals if 288 GiB isn't enough, and buy more 32 GiB DIMMs - and still be stuck at 384 max. Going over 384 GiB will require "shelving" all of the new 32 GiB RDIMMs and getting all new 64 GiB LRDIMMs.
- It sounds like your planning on "just enough RAM for the current jobs", not "reasonable headroom if the jobs get bigger". (Hint: they never get smaller.)
Buying DIMMs that barely reach your current memory needs is pound foolish.
(Sorry about the typos in the list - I can't figure out how to get it to let me edit the list.)
Thanks for your insights. 👍
1) I can always remove my added RAM and ship the stock 32 GB to Apple Store if problems arise with the MP7,1, be they sw or hw.
2) Yes, the 48 GB for system, buffer cache and other user-level admin processes was a guess on my part, but not without some merit. My macOS should fit nicely most of the time in about 4 to 6GB. This leaves me around 40 GB for buffer caching, and other small memory Apps that are needed when running the big multi-core/thread Application. I will have a 4TB internal SSD, of which no more than 1 TB will be used for the macOS and all the other typical stuff that resides alongside it. This leaves me 3 TB of the very fast internal SSD. My multi-threaded application does require some access to data as it runs and this data will be initially staged to the remaining 3 TB of the SSD space and ready for the multi-threaded code when pushed into execution. I will be configuring my 4 TB internal SSD to be a single APFS Container with sharable Volumes; one for the OS, one for User-data, and one as Scratch space (OS, Data, Scratch). The Scratch Volume is where I'll stage data to for the multi-threaded application.
3) Will my problem size increase over time? Yes, no doubt about that. I normally address this by assuming growth could/would require my current problem size to multiply its requirements by threefold in order to future-proof myself. If I grew the 288GB to 384GB by employing 12 x 32GB R-DIMMs this then would allow each of my core threads access to around 14 GB of RAM (up from 10GB with 288GB). If I were to envision each thread needing say 20GB then I would need (24 * 20 ) + 48 = 528 GB. This would require I have at least 8 x 64GB R-DIMMs ($2,500) or 10 x 64GB R-DIMMs ($3,000) giving me 512GB or 640GB total. Cost for 512GB using 8 x 64GB LRDIMMs is currently around $2,600. Currently this $2500-$3000 is above my budget to support. Apple says R-DIMMs can be employed for a 12-core MP7,1 up to 768GB, and even for all other core count MP7,1 models above 12.
4) Your mention of hundreds of GB in the kernel buffer cache caught my attention. My multi-thread application periodically spits out a checkpoint file that can be up to 100 GB. Writing out these checkpoint files to disk takes an inordinate amount of time, but less time to my internal SSD Scratch space which can accommodate them. Writing these checkpoint files through the kernel buffer cache and then onwards to the SSD Scratch space would be even faster. I have at least two options for dealing with this. I could lower the RAM footprint that each execution thread requires to 7.5 GB (at the expense of lowering the problem size). This would be 180 GB for the execution threads, leaving some 108 GB for OS (6GB), Admin processes (2GB) and kernel buffer cache (100GB). Minimising the time for writing out checkpoint files is paramount as while this is being done, the multi-threaded application sits idle.
5) The only problem in relying on the kernel buffer cache in RAM for buffering my checkpoint files to the internal SSD is that if the system crashed during the transfer of the checkpoint file it's very likely the checkpoint file on the SSD is incomplete and consequently useless to me for recovering resuming the multi-threaded application after the system is rebooted. To deal with this I will need to round-robin two checkpoint files knowing at least one should be complete and useful for recovery/resumption of the multi-threaded application. This issue can occur regardless of how large the dynamic kernel buffer cache allocation is or can grow to at the time the checkpoint file has to be written out.
6) Drat... Oh dear.... compromises have to be made given a fixed budget.
