Scannerz Speeds 65 or 650 MB/s on nMP!

[k-hawinkler]

I have a new MacPro - 6 cores, 64 GB memory, 1 TB PCIe-based flash, D700s - and a number of hard drives and software RAIDS attached, spread across the 6 Thunderbolt2 connectors that are grouped into 3 different buses. The nMP is running Mavericks 10.9.2.

I know that I am supposed to run Scannerz by itself. But I noticed no apparent detrimental effect if I run Scannerz on one Thunderbolt2 bus and another I/O stream on another one.

So, on one Thunderbolt2 cable/bus I am copying from a 4-way software RAID to a single hard drive and sustain around 130-140 MB/s read and write speeds each.

On a different Thunderbolt2 cable/bus I am using Scannerz to do a Normal Scan of another 4-way software RAID. I am using the Activity Monitor to monitor data rates achieved.

When both the copying window and the Scannerz window are entirely visible I observe these typical data rates:

So, the copying process seems to get about 130-140 MB/s each for read and write operations. That leaves about 60-70 MB/s for Scannerz reading operation. Please, also note that in this case Scannerz uses about 6.1 %CPU activity.

However, when I cover up (almost/completely) the Scannerz window with another window - be it a Finder, BBEdit, or Safari window - then I measure these data rates:

Whereas as expected the write data rate for the copying process stayed roughly the same, the sustained combined read data rate has gone up to typically around 800 MB/s, varying between 600-900 MB/s. So, if we subtract the typical read data rate for the copying process we end up with an estimate of about 650 MB/s sustained for the Scannerz Normal Scan process, which happens to be the typical read data rate I measured for this particular software RAID.

So, that’s about a Tenfold increase in Scannerz scanning speed. A most welcome speed increase indeed!

Please, also note that now the Scannerz CPU utilization has gone up to 23.0 %CPU.

Question: What is the explanation for this? Answer: Could it be that in the covered up case no interrupt is necessary to display the running current scan numbers and the nMP with Mavericks takes advantage of this fact? Here is the Scannerz window:


BTW, the copying process has finished by now and I got these snapshots:

First, Scannerz window fully visible in display:


Second, Scannerz window completely covered up in display by another window:

This seems to confirm my previous estimates.

My conclusion - cover up!:

A scanning speed of 65 MB/s makes use of Scannerz impractical for very large RAIDs.
A scanning speed of 650 MB/s makes use of Scannerz practical for very large RAIDs.

Question: Does this behavior match your experiences?


Regards, KHW

 

[ZVH]

The problem, if we can call it that, has been around since PCs and Mac's stopped using character terminals for displays. It's really the same phenomena with games, where one is poky and slow on one system but on a new system it flies.

What you've done is stopped the graphics update from occurring by covering the GUI up. The application can then almost exclusively act like a disk tester. I would question your data rates because Activity Monitor is putting out overall system rates, not Scannerz specific rates. With the user interface hidden the rates Scannerz processes data will go up, but keep in mind the OS itself is doing things and assume Scannerz is running that fast is likely wrong. You would need to contact the developer and probably get the actual block sizes the read and then do an actual set of calculations and measurements to come up with a real figure.

MacOS is not a real time operating system and graphic updates more or less need to be sent, wait for the screen update, then return to the program. If a program didn't do that then you've have periods where the GUI would be empty one second, then showing info on another, then showing what amounts to jibberish on another because the source would be freely trying to ovewrite video display information, which is definitely not a good idea.

I do software development for a living. If you want' to see some real delays and nonsense, try running a Java based application on your system.

 

[k-hawinkler]

The problem, if we can call it that, has been around since PCs and Mac's stopped using character terminals for displays. It's really the same phenomena with games, where one is poky and slow on one system but on a new system it flies.

What you've done is stopped the graphics update from occurring by covering the GUI up. The application can then almost exclusively act like a disk tester.

Many thanks for your feedback. I have a similar understanding and invited the Scannerz developer to respond to this thread.

I would question your data rates because Activity Monitor is putting out overall system rates, not Scannerz specific rates. With the user interface hidden the rates Scannerz processes data will go up, but keep in mind the OS itself is doing things and assume Scannerz is running that fast is likely wrong. You would need to contact the developer and probably get the actual block sizes the read and then do an actual set of calculations and measurements to come up with a real figure.

Of course in principle you are correct what the Activity Monitor measures.
However, there were no other user processes running on the system.
Furthermore the system doesn't gratuitously exercise unnecessary I/O activity.
So I believe my numbers are actually in the right ball park.
The overall speed up I got for scanning a 16 TB RAID are consistent with those numbers.

MacOS is not a real time operating system and graphic updates more or less need to be sent, wait for the screen update, then return to the program. If a program didn't do that then you've have periods where the GUI would be empty one second, then showing info on another, then showing what amounts to jibberish on another because the source would be freely trying to ovewrite video display information, which is definitely not a good idea.

I do software development for a living. If you want' to see some real delays and nonsense, try running a Java based application on your system.

Thanks for your insights.


Here is a diagram of my nMP / FirmTek system:


• At the left four SeriTek/5PM Five-Bay Hot-Swap External SATA Port Multiplier (eSATA) Enclosures are attached to the nMP via a single SeriTek/Q6G 4-Port eSATA 6G Storage Adapter.

• At the right two SeriTek/2eEN4 Four-Bay Hot-Swap External eSATA Enclosures are attached with one Adapter each. These two enclosures don't have Port Multiplier functionality and require an eSATA connection for each of their drives.

In one of the enclosures to the right I have a 4-way software RAID 0 with Hitachi HGST 4TB Deskstar 3.5" SATA III Internal Desktop Hard Drives. Blackmagic Disk Speed Test gives these performance numbers:


In the other enclosure to the right I have a 4-way software RAID 0 with Crucial 256 GB SSDs. Blackmagic Disk Speed Test gives these performance numbers:


For the nMP internal PCIe-based flash storage Blackmagic Disk Speed Test gives these performance numbers:


This approach gave me the option to continue using my mostly already existing FirmTek gear with the nMP and obtain I/O performance numbers commensurate with my storage needs. The four 5PM enclosures to the left serve mainly as backups.

BTW, each of the SeriTek/Q6G adapters is attached to a different Thunderbolt bus. The four 5PM enclosures share a bus with my 30" ACD. The other two RAID 0 enclosures each have a bus for themselves.

Regards, KHW

 

[TheBSDGuy]

You're just posting this stuff to make everyone else jealous, aren't you?

 

[k-hawinkler]

The developer of Scannerz pointed out to me that minimizing the Scannerz window should have the same effect as covering up the window with another one. Thanks for the tip. Indeed it does.

As a test case I used Scannerz to scan the nMP internal 1 TB PCIe-based Flash storage. With the Scannerz progress window visible I measured a scanning speed of around 60 - 70 MB/s. When minimizing that window scanning speed went up to slightly over 1000 MB/s on average. So, scanning the entire 1 TB took roughly around 17 minutes instead of hours.

In conclusion, I have finally found a viable tool to periodically check the health of drives, especially older ones, but also brandnew ones before they get used in earnest.

Additionally, the eSATA enclosures - some of them already 7 or 8 years old - still deliver acceptable performance when properly integrated into the nMP. I may have to replace a cooling fan or two as time goes on. But that's about it. No need for me to buy Thunderbolt enclosures at this time.

Regards, KHW.

 

[TheBSDGuy]

I would guess the number of people on this forum with comparable systems could be counted on one hand, and I'm guessing there are tens of thousands or more here.

 

[k-hawinkler]

I would guess the number of people on this forum with comparable systems could be counted on one hand, and I'm guessing there are tens of thousands or more here.

You may be right on that count. I certainly don't know. But that's not what this thread was intended to be about.

The point of this thread is to show that one can get vastly different Scannerz scan performance if one eliminates UI I/O by either minimizing the Scannerz progress window or covering it up with another window.

I certainly hope that there are a few users of Scannerz on this forum that will take advantage of that insight.

Just a couple of weeks ago I first learned about the existence of Scannerz from a recent thread on this forum. When I began using it I was dismayed by the poor scanning speed I obtained on my nMP. That is until I discovered this feature when running Scannerz under Mavericks. As I didn't see this feature discussed anywhere I thought it could be useful to someone if I pointed out this feature.

I definitely believe many folks with more common Macintosh systems should be able to exploit this insight trying to ensure the health of their drives.

Regards, KHW

 

[ZVH]

You should contact their support and see if they can improve it. A long time ago I found a minor bug, they had it fixed within 3 days, and had update notices sent out to all users within 5 days. It can't hurt to ask. Take a look at how long other companies take to fix or improve things. It's usually months.


Technically, what you're reporting isn't a bug, but the hardware is now pushing the limits of existing technologies, and I'd be willing to bet it will show up on other applications as well.

 

[k-hawinkler]

You should contact their support and see if they can improve it. A long time ago I found a minor bug, they had it fixed within 3 days, and had update notices sent out to all users within 5 days. It can't hurt to ask. Take a look at how long other companies take to fix or improve things. It's usually months.


Technically, what you're reporting isn't a bug, but the hardware is now pushing the limits of existing technologies, and I'd be willing to bet it will show up on other applications as well.

Thanks for your feedback indeed. I have been in touch with Bill and they are already working on it. They should have an update shortly.

Also, you write, quote:"I'd be willing to bet it will show up on other applications as well." I suspect you are correct on this as well. I ran into some weird problems with the Finder, copying about 5 TB between software RAID 0 drives with one command, and Disk Utility, security zeroing out a drive. After awhile these processes seemed to have stalled, not crashed, when only one application was running at the time. However, the drives checked out fine when scanned. Also, I can copy all that data successfully without problems in 1 TB chunks in a more busy machine. I suspect Mavericks has still some teething problems.

BTW, my system seems very stable otherwise and SMARTReporter reporting all checks, namely S.M.A.R.T., I/O-Errors, R.A.I.D., and Disk-Space, with Status: O.K.

Thanks again, KHW

 

[TheBSDGuy]

There are a lot of people reporting performance problems with Mavericks, and it's not just Scannerz FWIW. People that seem to have the problem seem to have it en masse with their applications, and from my own judgement it looks like it's hardware specific.

Apple had better stop putting out buggy operating systems. I hate to say it but I've noticed that since Steve Jobs died the quality of their products has dropped significantly.