A new "Ramdrive" ???

[seveej]

Bear with me here, as I intend to provoke:

First, open up Activity monitor and check the amount of free RAM.

Most of you will have had a significant amount.

Moreover, now as RAM goes for around 10-20 €/$ per gig, the only real reason to not max out RAM is that there's no need or barely marginal utility...

Nevertheless, Every gig of unused ram is a form of (admittedly potential) waste.

And, even though SSD's have revolutionized hard drive speeds, RAM is still (by far) the fastest source of one's and zeroes...

Now just assume for a second that we had an Operating system, which would utilize the "spare time" of a fixed memory drive and empty RAM to pre-load assumedly useful data into a dynamic ramdrive, just in case it might be needed ...

Would that be useful?

RGDS,

 

[Stetrain]

Sounds basically like SuperFetch.

 

[Tesselator]

Now just assume for a second that we had an Operating system, which would utilize the "spare time" of a fixed memory drive and empty RAM to pre-load assumedly useful data into a dynamic ramdrive, just in case it might be needed ...

Would that be useful?

RGDS,

Yeah, that would be useful. And we pretty much already have it if I'm understanding your description correctly. I use it lots. It's called RDUP or Ram Disk Utility Pro and is found here: http://eshensh.net/rdu/網站/English.html

It's very stable and fast. The only albeit small, problem with it is that it seems to have a 16GB limit on my 32GB system.

Here it is benching in a few of SpeedTools modules on the MacPro 1,1 x5355 procs:

It's pretty easy to configure this thing to auto start on power up and copy some system libraries, project files, or whatever you like over to it and then use the RD files instead of the HDD ones. This is also a "dynamic" type RAM Drive too. Here's my system just after mounting a 16BG RD and running similar benchmarks to the above:

etc.

 

[wallysb01]

Yeah, there are a variety of solutions using a RAM disks. OSX makes pretty good use of RAM though as it is, at least for more normal amounts of RAM. And as with any Unix/Linux system, its very easier to manually create RAM disks for your own use. You can even boot to RAM disks with Linux, which probably means you hack some sort of thing to work with OSX too.

 

[derbothaus]

A starter:

diskutil erasevolume HFS+ "ram disk" `hdiutil attach -nomount ram://4629672`

This makes a 2.2GB RAM Disk. You can change the bits to make bigger or smaller. You can use launchctl or Login items to create at startup by making a executable shell command. Good if you always want the same size and don't want to rely on a 3rd party.
Tess has some good ideas as well for GUI based and dynamic.

 

[seveej]

Sounds basically like SuperFetch.

Yep, but then again, SuperFetch was introduced with Vista, which probably was bad marketing. Also, any "autocache" routine like SuperFetch is a hard sell, as it does not show up on synthetic benchmarks...
Also, there are quite conflicting reports regarding the usefulness of SuperFetch. Anyway - an implementation which was built for the time when 2 gigs of RAM was all the rage is not what we today are looking for...

Yeah, that would be useful. And we pretty much already have it if I'm understanding your description correctly. I use it lots. It's called RDUP or Ram Disk Utility Pro and is found here: http://eshensh.net/rdu/網站/English.html

SNIP

It's pretty easy to configure this thing to auto start on power up and copy some system libraries, project files, or whatever you like over to it and then use the RD files instead of the HDD ones. This is also a "dynamic" type RAM Drive too.

Nice. Apple should make that (or something like it) a standard feature for 10.10

OSX makes pretty good use of RAM though as it is, at least for more normal amounts of RAM.

Yeah...
Below is a quick listing of the max ram of every current Mac (courtesy everymac.com)
iMac 21,5" 16 GB
iMac 27" 32 GB
Mac Mini 16 GB
Mac Pro 48/128 GB
MacBook Air 8 GB
MBP 16 GB
rMBP 8/16 GB

Now, i bet that for all but the most extreme uses, all but the Air's and the 13" rMBP will have RAM to spare when maxed out - RAM, which with some nice OS engineering could be utilized to add real-world speed and responsiveness...

RGDS,

 

[subsonix]

Nice. Apple should make that (or something like it) a standard feature for 10.10

Something like it already is a standard feature of OS X. In regards to the question of the OS utilizing a RAM drive, I don't see the point frankly because if assets need to stay in ram they will, regardless of what disk you are using.

 

[seveej]

Something like it already is a standard feature of OS X. In regards to the question of the OS utilizing a RAM drive, I don't see the point frankly because if assets need to stay in ram they will, regardless of what disk you are using.

Sure, the "problem" here just is not related to memory retention, but was concerned with preloading (assumedly) useful files.

RGDS,

 

[subsonix]

Sure, the "problem" here just is not related to memory retention, but was concerned with preloading (assumedly) useful files.

RGDS,

Right, and my point was that if there are gains to having the file in memory, they will be preloaded and stored in memory regardless of what kind of drive you use, or if there is a ram drive present.

For files that needs to be updated it's useful to keep in mind that things are written to disk to be permanent, while a ram disk loses it's content on power failures and re-boots.

 

[snarfquest]

The zfs file system does a very good job IMO. It uses available ram to cache your hard drive. If an application needs more memory zfs will give up it. I use zfs on my OSX Server for the afp shares and timemachine volume. Works quite well.

 

[tomvos]

And as with any Unix/Linux system, its very easier to manually create RAM disks for your own use. You can even boot to RAM disks with Linux, which probably means you hack some sort of thing to work with OSX too.

I use the following method to handle my ramdisk:

1. I have a shell script which creates and mounts the ramdisk. I use something like this:

#!/bin/bash

if [ -n "$2" ]; then ARG_ERR=ERR; fi
if [ -z "$1" ]; then ARG_ERR=ERR; fi
if [ -n "$ARG_ERR" ];

then
   echo Please supply one argument: SIZE in MB
   exit
fi

MB_SIZE=$1
let "MB_SIZE *= 2048"

echo Creating ${MB_SIZE} 512-Blocks Ramdisk

CREATED_RAMDISK=`hdid -nomount ram://${MB_SIZE}`

echo New Device: ${CREATED_RAMDISK}

DISK_NAME=`basename ${CREATED_RAMDISK}`

echo Creating volume with the label: ${DISK_NAME}

newfs_hfs -v ${DISK_NAME} /dev/$CREATED_RAMDISK

echo Mounting to /Volumes/${DISK_NAME}

diskutil mount ${CREATED_RAMDISK}

This script is stored in a file called ramdisk.sh. It is given execute permissions.

2. I have a automator script which runs a shell-script. It has a simple setup which uses /bin/bash as the shell and sends the following text "to stdin".

/Users/myusername/ramdisk.sh 8192

This automator script is saved on my desktop. Whenever I need a ramdisk—in this case I set it to a 8 GB size—it's just one double click away. And when I'm done or need more free memory, I simply eject the ramdisk. It's really easy and a convenient method to handle adhoc ramdisks.

 

[Tesselator]

Right, and my point was that if there are gains to having the file in memory, they will be preloaded and stored in memory regardless of what kind of drive you use, or if there is a ram drive present.

For files that needs to be updated it's useful to keep in mind that things are written to disk to be permanent, while a ram disk loses it's content on power failures and re-boots.

The OS should do this - indeed. The trouble is OX S sucks nuggets at memory handling. Left to it's own it will kick out pages (hundreds of them) which I wanna keep in ram and which aren't bothering anything. Having OS X manage such ram retention is like hiring Daffy Duck to organize a legal office (when he's drunk!). Looking into that is part of what lead me to just say "no". and turn off one of the paging systems all together. I guess you've seen me post this around:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

followed by a reboot. You can turn it on again with

sudo launchctl load -wF /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

After installing 32GB I was trying to figure out why OS X 10.6.8 and 10.7.5 didn't behave more appropriately - instead of like Daffy Duck. I started looking into RAM drives which I hadn't used for a few years since writing application software on contract. That didn't do much. Then I tried just turing off the dynamic pager and that worked! I shot Daffy, feathered him, and had a nice lunch! Things have been much much faster and smoother since - I'm happy with it.

RAM drives are excellent BTW anytime you have a few hundred or thousand, small files that you need to access regularly. Application development is a good example here. I think I had something like 7-million code lines in 30 or 40 thousand files just off the top of my head. It didn't occupy much drive space but from an HDD compile times were in the hours. From an SSD wasn't much better. But from a RAM drive it was more like 15min. I guess the RAM in my MP1,1 is about 8 or 10 times faster (small file I/O) than a fast SSD. In something like the 2012 MP I guess it's 25 to 40 times faster and in the nMP maybe even more than that.

 

[subsonix]

The OS should do this - indeed. The trouble is OX S sucks nuggets at memory handling. Left to it's own it will kick out pages (hundreds of them) which I wanna keep in ram and which aren't bothering anything. Having OS X manage such ram retention is like hiring Daffy Duck to organize a legal office (when he's drunk!). Looking into that is part of what lead me to just say "no". and turn off one of the paging systems all together. I guess you've seen me post this around:

Hm, but the VM is a different matter from the OS utilizing a RAM drive to get fast access to commonly used files though. My point was that if a file needs to be accessed quickly by (one of the many programs that makes up) the OS, it's loaded into RAM and that is handled without a RAM drive. Secondly, even if you do use a RAM drive, even that RAM space may be paged out to disk.

 

[Tesselator]

Hm, but the VM is a different matter from the OS utilizing a RAM drive to get fast access to commonly used files though. My point was that if a file needs to be accessed quickly by (one of the many programs that makes up) the OS, it's loaded into RAM and that is handled without a RAM drive. Secondly, even if you do use a RAM drive, even that RAM space may be paged out to disk.

1st) Not sure what you mean. Loaded into RAM (from disk) is the problem RAM drives provide a solution to (partially at least).

2nd) Ya, I said it didn't do much.

 

[subsonix]

1st) Not sure what you mean.

It refers to the last paragraph in the original post. That the OS should utilize a RAM drive for commonly accessed files.

Yes, a RAM drive does that but if a program needs fast access to a file it probably loads the entire file into RAM, or the parts it needs access to. Operating on a file on a RAM disk, is probably slower than to load it into RAM directly compared to reloading it every time you need it. Operating on the memory region directly as opposed to, through the file operations that are available. Why bother with the whole concept of a file, if you can access it directly.

 

[Tesselator]

It refers to the last paragraph in the original post. That the OS should utilize a RAM drive for commonly accessed files.

Yes, a RAM drive does that but if a program needs fast access to a file it probably loads the entire file into RAM, or the parts it needs access to. Operating on a file on a RAM disk, is probably slower than to load it into RAM directly compared to reloading it every time you need it.

Well the two cases you're comparing are:

RAM Drive --> Local RAM (Between 60MB/s and 3GB/s on a MP 1/1 - depending [see charts above])
HDD Drive --> Local RAM (between 4KB/s and typically not more than 120MB/s - depending)

Which would you prefer?

Application I/O behavior doesn't change for either example. For example I download my DSLR camera files (photos) from the 8GB card into the 8GB RAM Drive and edit them in whatever CP1, LR, PS, etc. The I/O access between the app and the data is the same as if the files were on an HDD except MUCH MUCH faster. When done (and at the start) I drag & drop the images onto a large RAID which takes about 8 seconds each time. Yet if I were to edit from an HDD I would be a little frustrated by the slow speed during the 4 to 6 hours it took me to process all those RAW files.

I don't do that very often, it's mostly just and example for this discussion - that I've tried a few times. And then the application development example I gave at first - which is even more advantageous! About 5sec to load it all up onto the RAM Drive and editing/browsing is shockingly fast plus compiles take about 1/4 the time. Very useful. Maybe every developer uses a RAM Drive for this reason.

 

[subsonix]

Well the two cases you're comparing are:

RAM Drive --> Local RAM (Between 60MB/s and 3GB/s on a MP 1/1 - depending [see charts above])
HDD Drive --> Local RAM (between 4KB/s and typically not more than 120MB/s - depending)

Which would you prefer?

It's not possible to pick one or the other though, is it? I mean if the OS would use a RAM drive for commonly used files, it would need to start its life by loading the files from HDD once.

After that the file would live in RAM, regardless of method.

 

[Tesselator]

It's not possible to pick one or the other though, is it? I mean if the OS would use a RAM drive for commonly used files, it would need to start its life by loading the files from HDD once.

After that the file would live in RAM, regardless of method.

Well, you can put whatever you like in the RAM Drive. Whether or not it's faster depends on how often the files are accessed. But some RAM Drives, like the one i linked to, can use a backup file to populate the RAM drive - so it's pretty fast. Typically and of course you're only going to store a few gigs in a RAM drive anyway so it's not like it's slow to create or anything. It's typically a macro-key and then a momentary glance at your keyboard and it's done. After that access times are blazingly fast.

Of course it would be pretty silly to do something like DL that 8GB card onto the HDD and then place one photo at a time into the RAM Drive and edit that before saving it to HDD and then placing another in the RAM Drive.. (cont.). If that's what you're thinking then I can see your point.

For "commonly accessed OS files" like you mentioned and that's mentioned by the OP I think it would depend which files you place there and how big your RAM Drive was. If you had 256GB of RAM (YAY!) and created a 128GB RAM Drive and then during every startup you moved the entire OS into the RAM Drive and booted from it, Ya, I think it would be much faster overall. Startups would take 15min. if not stored in a continuous file structure on disk but once loaded the machine would hall arss compared to an HDD or even a single SSD. It would take some clever programming and/or some different hardware conventions to get it to be secure enough for something like that though.

 

[subsonix]

Of course it would be pretty silly to do something like DL that 8GB card onto the HDD and then place one photo at a time into the RAM Drive and edit that before saving it to HDD and then placing another in the RAM Drive.. (cont.). If that's what you're thinking then I can see your point.

I'm not advocating against using a RAM drive in general, I just don't see the benefits for a program that access a file often, because such a file would likely already be in RAM. And accessing the memory directly should be faster than going through file operations and a filesystem.

 

[Tesselator]

I'm not advocating against using a RAM drive in general, I just don't see the benefits for a program that access a file often, because such a file would likely already be in RAM. And accessing the memory directly should be faster than going through file operations and a filesystem.

Ya, if the app in question does that. But what apps do that? I dunno of any - at least not ones under user control. For example the app dev example I gave. Not only are those files browsed, edited, saved, closed, reopened later, and so on but the compiler itself then loads them all again at compile time. And it's a edit-compile-debug loop for most projects where that loop occurs 10 to 100 times a day.

For the photo example I gave I load them all up in LR and rate them for editing. Then I load all 400 of them into ACR. As i cursor to each file it's loaded and as I cursor off of it it's unloaded. When done it saves out all the XMP files. And as a user I may repeat that part of the process 10 or 20 times before I'm satisfied with the edits of the final 100 selected images I wanna keep. Then I load those up 3 to 5 at a time into PS which only keeps them in memory while they're open. As soon as their closed they are unloaded from RAM (saved or not as the case may be). For the 10 or 20 photos that came out exceptionally well I may load those around 5 different times in round-robin with the other 9 or 19 and in an app other than PS. etc. etc.

 

[subsonix]

Ya, if the app in question does that. But what apps do that? I dunno of any - at least not ones under user control. For example the app dev example I gave. Not only are those files browsed, edited, saved, closed, reopened later, and so on but the compiler itself then loads them all again at compile time. And it's a edit-compile-debug loop for most projects where that loop occurs 10 to 100 times a day.

For the photo example I gave I load them all up in LR and rate them for editing. Then I load all 400 of them into ACR. As i cursor to each file it's loaded and as I cursor off of it it's unloaded. When done it saves out all the XMP files. And as a user I may repeat that part of the process 10 or 20 times before I'm satisfied with the edits of the final 100 selected images I wanna keep. Then I load those up 3 to 5 at a time into PS which only keeps them in memory while they're open. As soon as their closed they are unloaded from RAM (saved or not as the case may be). For the 10 or 20 photos that came out exceptionally well I may load those around 5 different times in round-robin with the other 9 or 19 and in an app other than PS. etc. etc.

IME, most applications do that, at least for files that are not the users files. I mean if the data is loaded and actively used, then why free the memory and reload it later if it's continuously used. The applications in question would not be user applications but applications that are part of the OS. On top of this is also the fact that RAM is volatile, so any updates that needs to remain would be lost on power failure, for a lot of OS files that is probably not acceptable.

 

[VirtualRain]

This is a nugget on RAM disks that I got on another forum a few years ago from someone who claimed to be a former Apple employee...

That thing is snake oil, as are the vast majority of ramdisk products on OS X. Designing a ramdisk that works well on OS X is remarkably difficult because backing a block device into wired memory causes the contents to be double buffered above it in the Unified Buffer Cache.

For transient data on systems with no VM pressure, what will happen is the file is created, the object backing it sits in the UBC, it might get synched to disk, but unless you are running low on ram it stays in ram as well. That is why you see no speed increases, all you are doing is eliminating the background asynch writeout, and you are wasting a lot of ram to do it. More importantly, if you are actually creating a wired ram disk you are eating a ton of kernel address space which can be an issue if you have a lot of memory (large page tables) or several video cards.

Source: http://www.ocztechnologyforum.com/f...X-Speed-Tweaks&p=368792&viewfull=1#post368792

 

[Giuly]

Except that "a few years later", we have a max. RAM of at least 16GB instead of 4GB of RAM for all Macs, and Mavericks compresses unused RAM to turn 16GB into something between 16GB and 32GB. That's almost 1/8th of a 256GB SSD right there.

Oh and this, 5th picture.

 

[Tesselator]

IME, most applications do that, at least for files that are not the users files. I mean if the data is loaded and actively used, then why free the memory and reload it later if it's continuously used. The applications in question would not be user applications but applications that are part of the OS. On top of this is also the fact that RAM is volatile, so any updates that needs to remain would be lost on power failure, for a lot of OS files that is probably not acceptable.

Why? Because the conventional wisdom has always been that memory is a rare and expensive commodity. So for OS apps you mean something like Finder? Try it, go to a large folder of 1000 files all or most with custom icons. Scroll through to the bottom to get them all rendered and into the cache. Close that folder and load a few different similar ones. Now open Apple Mail assuming you haves as many emails as I and read for 20 to 40min. Now go back to that 1st 1000 file folder. Oops all the icons were dumped out of memory.

Try something similar with the Dashboard. Assuming your widgets are anything like mine they all but a small few unload after some magic peroid of time or as other apps are accessed. Most OS libraries are the same - Not in use, they're kicked out. Even with 32GB and my dynamic pager turned off (mostly only affecting anonymous data pages and cache accesses) my system doesn't use much more than about 3 to 4GB. But the OS data I use in even a few hours is much much larger than that - maybe 20GB. It all gets kicked out.




--
To the snake oil remark all one needs is a sense of speed or a stopwatch to prove or disprove that. And, use a proper RAM Drive "that works well on OS X".

 

[subsonix]

Why? Because the conventional wisdom has always been that memory is a rare and expensive commodity. So for OS apps you mean something like Finder? Try it, go to a large folder of 1000 files all or most with custom icons. Scroll through to the bottom to get them all rendered and into the cache. Close that folder and load a few different similar ones. Now open Apple Mail assuming you haves as many emails as I and read for 20 to 40min. Now go back to that 1st 1000 file folder. Oops all the icons were dumped out of memory.

If the file is accessed and used it's kept in memory, if it's continuously used it makes sense to just keep it in memory. Finder is accessing user files when you browse a folder, they are not continuously used, but temporarily. I was thinking of most of the applications you see if you hit top in the terminal, or view processes that doesn't belong to the user in ActivityMonitor.

Finder, Mail and Dashboard are all essentially user programs that happens to be part of the OS.

To the snake oil remark all one needs is a sense of speed or a stopwatch to prove or disprove that.

It's not a snake oil remark, if you load a file and access it through an array directly as opposed to access it through filesystem functions it should be faster, a RAM disk writes a filesystem on top of memory, an array provides random access. What I have seen (could not bother to look at current) memory bandwidth is somewhere around 50GB/s atm.