Inconsistent transfer speeds

[Martinpa]

Hi! Hopefully this is the right place to post this.

I have been transferring data from a CF card to my iMac and the speeds have been really weird. So the first transfer was in the 60GB range and took about 30minutes. I put in another card (exact same brand and model (Sandisk Extreme Pro 128GB)) and the transfer, approx. 55GB to the same location, has been going for an hour, and it still says 2 hours left. The files being transfered are the same type as they were on the other card (mxf from a Canon C300). Why is the speed so different? No other app is running on the computer (iMac), same card reader (lexar Thunderbolt)... doesn't make much sense. Is there something I can do? I've had similar issues with REDmags taking forever, and when the crew on a set is waiting after a REDmag because it is taking so long to dump, thats not good.

Help!?
Thanks

 

[Weaselboy]

Does your iMac have a Fusion drive?

https://arstechnica.com/gadgets/201...ining-doc-ars-tears-open-apples-fusion-drive/

There is a long article here that explains there is some space set aside on the SSD part of the Fusion drive for writing new files, and once that space is used, the files start to get moved over the the hard drive portion of the Fusion drive.

So I am wondering if you first transfer is going to the SSD, then with the SSD now getting full, the second big transfer is being moved off to the hard drive, so much slower. Would that explain what you are seeing? Does it matter what order you use the cards? If you wait a while, then use the second card first, so you still see this slowdown?

 

[kschendel]

I wondered the same thing re fusion drive, but I'm not sure the numbers make sense unless the HDD part is failing. If the time estimate is anywhere near right he's getting something like 5 MB/sec written in the slow case. I've seen RP03's beat that.

(well maybe not quite.)

 

[Martinpa]

The computer I was on on Saturday, I am not sure if it has a fusion drive or not. My personal computer has it and is the one I've seen the phenomenon happen with REDMAGS. I don't necessarily know how the fusion drive distributes data, but a lot of the available space was already used, so I don't know if any space at all would've been free on te SSD part of it.

 

[Weaselboy]

The computer I was on on Saturday, I am not sure if it has a fusion drive or not. My personal computer has it and is the one I've seen the phenomenon happen with REDMAGS. I don't necessarily know how the fusion drive distributes data, but a lot of the available space was already used, so I don't know if any space at all would've been free on te SSD part of it.

Can you try reversing the order of the cards when you do this and see if that makes a difference?

 

[kschendel]

Another thing to try is to wait for a while before doing the second transfer to see if it goes faster. HFS+ does an auto defrag (hot file adaptive clustering). I could possibly imagine that if the first file transferred was fragmented, HFS+ might be busy trying to defrag it while you're writing the second one. I'd expect that it would put the HFC on hold when the system is otherwise busy, but who knows.

It's also possible that if free space is badly fragmented, and if you're writing to the HDD side of the fusion, you might well see terrible write speeds. I've seen this with for example the JFS filesystem for linux. The linux port of JFS was poorly designed / written and I'd expect HFS+ to be a lot better, but perhaps you're managing to hit a worst case.

 

[Martinpa]

I can try reversing the card order next time, but this was a very time sensitive project, and I got the cards at different times... I actually received the two cards at least 3 hours apart, so almost as much time between the first one finished transferring and I started the second one. As for the REDMAGS, it's not really something I could do, cause the card goes back to the camera department as soon as I am done with it as to not slow down the production. I rarely, if ever, get more than one card at a time, but it's happened once or twice that it would take so long to dump the footage that they'd fill up their other card and have to wait on the computer to be done with the first card...

I could see the free space being badly fragmented, but wouldn't it mean the two cards would be slow? Also is that something AFS will handle better.