Would An iMac Pro Help Me?

[1957GoldTop]

I currently have the latest high end iMac (4.2 ghz Intel Core i7) with 32GB ram. I’m trying to decide if an upgrade to the 8-Core iMac Pro would be a significant enough upgrade to make the price worth it.

I am a photographer and I work with 80MP - 100MP raw files. Here are my typical use cases that causes me the most “wasted time” that I’m hoping to improve:
- Importing raw files into Capture 1 Pro
- Processing raw files from Capture 1 Pro
- Opening large tifs in Photoshop (over 1GB)
- Saving large tifs in Photoshop (over 1GB)

I never have any issues once inside the apps as far as making adjustments to the files, working with layers, etc.

Right now all my photos are stored on a 12TB G-Tech G-Raid (Raid 0, two drives) Thunderbolt 3.

I feel like when I’m doing the tasks above I hear my drive churning and churning. So my thought was, would I get a big increase in performance if I switched to something like a Promise Tech Pegasus3 R6 24T Thunderbolt 3 with 6 drives configured as Raid 5.

That way I get a big boost in performance without going all out with an iMac Pro.

Thoughts?

 

[Trebuin]

Honestly, if your software isn't holding to complete an effect, the CPU will not benefit you. As far as our last chunk in your message...I would look at any SSD to store your current work in progress. Raid is great for storage, but if you're working on the file, transfer it to a SSD first. I do this myself and have an internal SSD for that purpose. I start with projects in a "working" folder on my desktop, do what I need to, then transfer the completed job to the external raid. Some programs are good at loading the entire file in memory, in which this is a moot point. Also, jobs such as encoding doesn't thrash (what you hear in your drive as it continuously re-reads the file) as much and the performance gain is so minimal that it's not worth it.

In conclusion, keep your current computer until the CPU cannot do the job (probably 6-10 years from now) and get an external SSD if you don't have an internal one.

 

[jerwin]

How fast is your current raid setup? Can you test it with something along the lines of blackmagic?

 

[Bryan Bowler]

It sounds like you are working with medium-format cameras. Very nice!

Undoubtedly, moving to a 6-drive RAID 5 solution would greatly speed up the read/write of your files. As an approximation, a 2-drive spinning RAID 0 is probably clocking in around 260-300 Mbps. A 6-drive spinning RAID 5 should clock in around 700-800 Mbps. There are other factors at play that could effect the actual speed, but this is a good ballpark estimate.

When I move to an iMac Pro or the next release of the Mac Pro, I will get it with at least a 2 TB SSD. Since the speed is now approaching 3 Gbps, for large projects I will initially work off the SSD and then send the projects over to my RAID 5 after they are complete.

If I were in your shoes, I would stick with what you have for now and just look at upgrading your storage solution...if you can absorb the cost of the storage upgrade. Also, don't forget to give strong consideration to buying a cold-spare (hard drive) when you set up your next RAID. All it takes is one of the 6 drives to go wonky and you'll need it. So many people don't have a cold-spare laying around, but when my data is at risk, I want to mitigate the problem as quickly as possible versus waiting around for 2-3 days for another hard drive to arrive in the mail.

 

[joema2]

I currently have the latest high end iMac (4.2 ghz Intel Core i7) with 32GB ram. I’m trying to decide if an upgrade to the 8-Core iMac Pro would be a significant enough upgrade to make the price worth it....Importing raw files into Capture 1 Pro...Processing raw files from Capture 1 Pro....Opening large tifs in Photoshop (over 1GB)...Saving large tifs in Photoshop (over 1GB)....I feel like when I’m doing the tasks above I hear my drive churning and churning. So my thought was, would I get a big increase in performance if I switched to something like a Promise Tech Pegasus3 R6 24T Thunderbolt 3 with 6 drives configured as Raid 5...That way I get a big boost in performance without going all out with an iMac Pro....

If you observe your CPU levels during these time-consuming phases and they are consistently high -- say over 70%, then you are mostly CPU bound and additional I/O will probably not help much. This can be counter-intuitive especially if you viscerally hear your RAID-0 array shaking from activity. However that noise doesn't mean it is maxed out, only that it's being used.

If your CPU core activity levels are not consistently over 50%, then *maybe* more I/O might help. Or maybe the software is already bottlenecked on multi-thread synchronization issues. That can be hard to determine, and can vary widely within a single app. E.g, some Photoshop filters are well threaded and others are not.

That said, loading a 1GB TIFF will take at least three seconds on a 12TB two-drive G-RAID. So you have lots of I/O happening, but if using pre-generated 1:1 previews it is using those for a lot of the initial display. When skimming through 1GB TIFFs you aren't necessarily loading each 1GB file in succession.

Obtaining the full I/O potential of a device often requires issuing overlapped or multiple async I/O commands. This way a single thread can dispatch multiple outstanding I/Os, and the device works on fulfilling those and notifies or "calls back" via some mechanism to the issuing thread when each one completes. In POSIX the API is AIO and in Windows it is passing the flag FILE_FLAG_OVERLAPPED when the file is created with CreateFile(). If this isn't done there is a potential to underutilize the device bandwidth or else use up a lot of threads (hence overhead) on synchronous I/O calls.

The depth of the I/O queue is one indicator of how fully the device is utilized. In general you want a non-zero but modest I/O queue. So it is possible to better determine whether you are I/O limited or CPU limited.

From the OS X Terminal it is possible to examine the I/O queue histogram. The command is bitesize.d. E.g, "sudo iopending", and "sudo bitesize.d". Google for more details. Always exercise caution when using Terminal.

On Windows the disk queue length can be examined graphically for each device using Performance Monitor.

Note the Dtrace utilities cannot be used starting with El Capitan unless System Integrity Protection is first disabled: http://apple.stackexchange.com/ques...otless-is-there-any-way-to-get-dtrace-working

I have run them on Sierra so they can still be used.

 

[Bryan Bowler]

As always, anything joema2 writes is a joy to read! If we ever cross paths in the real world, beverages of his/her choice will be on me.

 

[1957GoldTop]

Thanks for the replies.

Bryan Bowler was dead on with the estimated speeds of my current 2 disk raid 0 setup: 290MB/s write 343MB/s reads.

If I knew what the hell joema2 was talking about, I'm sure I'd say he was dead on as well.

While I agree with Trebuin that working off an SSD would be ideal, and it's doable with the initial "intake" workflow, but once I move the files to the RAID system I often need to go back to them often. I've tried it in the past, but I know I'm too lazy to move things around, work on them, and then move the back to the RAID, so I end up working off the RAID anyway.

When processing out my 80MP files in Capture 1 (the biggest time suck of all my tasks), my CPU numbers are about: System 7%, User 52%

It sounds like upgrading my RAID is the best bet.

Thanks!

 

[Macintosh IIcx]

Thanks for the replies.

Bryan Bowler was dead on with the estimated speeds of my current 2 disk raid 0 setup: 290MB/s write 343MB/s reads.

If I knew what the hell joema2 was talking about, I'm sure I'd say he was dead on as well.

While I agree with Trebuin that working off an SSD would be ideal, and it's doable with the initial "intake" workflow, but once I move the files to the RAID system I often need to go back to them often. I've tried it in the past, but I know I'm too lazy to move things around, work on them, and then move the back to the RAID, so I end up working off the RAID anyway.

When processing out my 80MP files in Capture 1 (the biggest time suck of all my tasks), my CPU numbers are about: System 7%, User 52%

It sounds like upgrading my RAID is the best bet.

Thanks!

One thing to keep in mind with Capture One Pro is that it uses the GPU and OpenCL a lot for processing, so just looking at the CPU load doesn’t give you a full overview of bottlenecks. Capture One was actually one of the few apps that could utilize the second GPU in the 2012 Mac Pro. I think about 50% higher performance on 80MP files.

I don’t know whether the Vega GPUs in the iMac pro is substantially faster than the GPU in your newish iMac, so it might not make a massive performance difference.

But I sure as heck could use a more beefy GPU than the one in my MacBook Pro when working on big files in Capture One Pro while driving the LG 5K screen. The MacBook Pro gets HOT!
[doublepost=1513408026][/doublepost]I think the iMac Pro is just about perfect for working on big raw files in Capture One Pro, BTW. Would LOVE to have one. But you will need to work on the internal SDD drive to fully unlock its performance.

 

[danielwsmithee]

Almost all Adobe apps still benefit more from clock speed than core count.

I think you are right on with the suggestions here to upgrade your raid and storage. For the price of the difference between an iMac Pro and iMac, you could upgrade to a raid array of SSDs and I think your workflow would benefit more from it. An Akitio Thunder3 quad mini loaded with 4 SSDs in RAID0 would give you great performance and allow you to make your current raid a backup. This would give you completely silent data transfers @1500MB/s.

For workflows that benefit more from clock speed than core count I think the fully loaded iMac is the appropriate choice. The Vega GPU in the iMac Pro is better than the RX580 in the iMac but the 580 is no slouch.

 

[joema2]

....When processing out my 80MP files in Capture 1 (the biggest time suck of all my tasks), my CPU numbers are about: System 7%, User 52%....

I usually use Lightroom but I've used Capture 1 a while because ACR did not support the A7RIII raw format. Import to C1, preview generation and export as TIFF is really slow.

I tested importing and generating previews for 111 43MB 42 megapixel raw stills from a Sony A7RIII to Capture One 11.0.0.282 on a 2017 top-spec iMac 27.

I did two tests, one using a 16TB two-drive Thunderbolt G-RAID in RAID-0, and one using a four-drive Thunderbolt 2 8TB SSD Thunderbay 4 Mini in RAID-0. The G-RAID does about 350 MB/sec and the Thunderbay SSD does 1,230 MB/sec.

There was relatively little difference between the two: Import and preview generation with all media on the G-RAID took 52 sec, and 47 sec on the SSD array. This indicates the Capture 1 import and preview generation process is largely not I/O-bound but waiting on CPU, GPU or inter-thread synchronization. In general the CPU core levels were relatively high, say around 50% but not close to maxed out. This is consistent with an inefficient multi-thread code path in Capture 1.

Activity Monitor showed Capture One doing a very high number of small I/O calls -- 2,500 to 3,000 per second. The average I/O data rate was about 110 MB/sec, which implies an average I/O size of about 42,000 bytes. These were obviously cached by the macOS system buffer, otherwise the IOPS rate couldn't be that high on a G-RAID.

But there is system overhead in each I/O call and if multiple threads are doing those calls they must periodically pause and synchronize their activities. This caps overall multicore speedup due to Amdahl's Law: https://en.wikipedia.org/wiki/Amdahl's_law

It appears the Capture One ingest process is not efficiently parallelized and you can't fix that with an SSD or maybe not even more CPU cores. However it would be interesting to see this run on an iMac Pro.

 

[Macintosh IIcx]

I usually use Lightroom but I've used Capture 1 a while because ACR did not support the A7RIII raw format. Import to C1, preview generation and export as TIFF is really slow.

I tested importing and generating previews for 111 43MB 42 megapixel raw stills from a Sony A7RIII to Capture One 11.0.0.282 on a 2017 top-spec iMac 27.

How is that a7rIII treating you? I’m going to upgrade my a7rII (aka Mr. Turtle) in the Spring. Really looking forward to that!

I did two tests, one using a 16TB two-drive Thunderbolt G-RAID in RAID-0, and one using a four-drive Thunderbolt 2 8TB SSD Thunderbay 4 Mini in RAID-0. The G-RAID does about 350 MB/sec and the Thunderbay SSD does 1,230 MB/sec.

There was relatively little difference between the two: Import and preview generation with all media on the G-RAID took 52 sec, and 47 sec on the SSD array. This indicates the Capture 1 import and preview generation process is largely not I/O-bound but waiting on CPU, GPU or inter-thread synchronization. In general the CPU core levels were relatively high, say around 50% but not close to maxed out. This is consistent with an inefficient multi-thread code path in Capture 1.

Activity Monitor showed Capture One doing a very high number of small I/O calls -- 2,500 to 3,000 per second. The average I/O data rate was about 110 MB/sec, which implies an average I/O size of about 42,000 bytes. These were obviously cached by the macOS system buffer, otherwise the IOPS rate couldn't be that high on a G-RAID.

But there is system overhead in each I/O call and if multiple threads are doing those calls they must periodically pause and synchronize their activities. This caps overall multicore speedup due to Amdahl's Law: https://en.wikipedia.org/wiki/Amdahl's_law

It appears the Capture One ingest process is not efficiently parallelized and you can't fix that with an SSD or maybe not even more CPU cores. However it would be interesting to see this run on an iMac Pro.

Two small notes>

1) Keep in mind that Capture One Pro uses the GPU for processing which might explain why the CPU levels are not over 50%. I don’t know the work split between CPU and GPU, but it is entirely possible that the CPU’s is waiting for the GPU or there is a data transfer bottleneck between CPU <-> GPU.
2) I’m still thinking that an internal NVme SSD could be faster due to lower latency with all those small I/O calls, but I honestly don’t know enough about that stuff.

 

[joema2]

How is that a7rIII treating you?.... Really looking forward to that!

It is lightning fast and a vast improvement.

....Keep in mind that Capture One Pro uses the GPU for processing which might explain why the CPU levels are not over 50%. I don’t know the work split between CPU and GPU, but it is entirely possible that the CPU’s is waiting for the GPU or there is a data transfer bottleneck between CPU <-> GPU....

In the case I tested, the bottleneck wasn't I/O. I increased I/O bandwidth from 350 MB/sec to about 1,230 MB/sec (a four-drive RAID-0 SSD Thunderbolt 2 array) and it didn't make much difference. The bottleneck definitely wasn't CPU, so it was either GPU or a thread-level synchronization bottleneck. In multithreaded scenarios, synchronization issues are a common cause of slowdowns because they require great skill and diligence from the programmer to avoid.

However it was not doing very efficient I/O with all those small 40KB reads. But however inefficient that was, increasing I/O bandwidth did not help, which indicates the bottleneck was elsewhere.