Anyone using a USB-SSD for editing high bitrate video on M1 (e.g. DNxHR HQX, ProRes422 HQ)?

[Fomalhaut]

I've started experimenting my the M1 Mac Mini for video editing using Davinci Resolve.

Because the free version of Resolve doesn't support the 10-bit 422 H.264 video recorded by my camera, I've been transcoding the camera footage to DNxHR HQX (and have also experimented with ProRes422 HQ).

Both of these result in large files, but, in theory, they should be smoother to scrub because they are intra-frame codecs. Bitrates for UHD video at 24fps are about 700Mbps (although for some reason ffmpeg encodes ProRes422 HQ at 1700Mbps....which are huge files).

Even with my USB 3.1gen2 drive connecting at only 5Gbps (instead of 10Gbps, which my MBP supports), I would still have expected the external SSD to easily render 700Mbps DNxHR files . Note that this is 700 mega-bits, not bytes.

The external drive can read at >400MBps (mega-bytes), so I was surprised to see that it doesn't play these files smoothly, even at 24fps. At 30 and 60fps, it is unusable.

Black Magic Disk test confirms the drives are reading and writing at normal, if not spectacular, speeds (the SSDs in the enclosures are SATA-3, so I would expect 530-550MBps maximum).

These discs are not far removed from the Samsung T5, which many people seem to use for editing. Whether they are using high bit-rate codecs, I don't know.

Has anyone had any experience with M1 Macs using USB 3.1 external SSDs, and high bit-rate codecs in Davinci Resolve?

Does USB have a significant processing overhead when used for video playback?

The disks are formatted as NTFS and I'm using Paragon NTFS on the M1, but even with any overhead associated with the NTFS driver, the disk speed tests shows healthy read/write performance.

Getting on the TB3 disk bandwagon will be expensive, and it's too late to change my 512GB internal SSD, so I'm wondering if there are any things to try before spending any more money.

Am I just expecting too much from a USB SSD?

 

[Slartibart]

Did you check with a different file system on the external drive? I have no experience in working with high-bit codecs, but if there is no good reason - like transfer between different systems - I would suggest reformatting the drive to Apple File System. If you have to use the drive on a Windows machine too: ExFat.

One more thing: I suggest excluding the external drive from spotlight indexing.

 

[Dockland]

I edit 4K 60p 4:2:2 (with proxy) with FCP and with a External SSD. But I have a USB C enclosure. Works flawless

 

[Fomalhaut]

Did you check with a different file system on the external drive? I have no experience in working with high-bit codecs, but if there is no good reason - like transfer between different systems - I would suggest reformatting the drive to Apple File System. If you have to use the drive on a Windows machine too: ExFat.

One more thing: I suggest excluding the external drive from spotlight indexing.

Thanks. I was thinking of reformatting to APFS or HFS+ to see if it makes a difference, but will need to do a bit of data shuffling to make sure I have at least two copies of my clips. Good tip about spotlight indexing!

 

[Fomalhaut]

I edit 4K 60p 4:2:2 (with proxy) with FCP and with a External SSD. But I have a USB C enclosure. Works flawless

Presumably, you store your camera source on the external SSD and let FCP create ProRes proxies? Are the proxies created on the external disk or on the internal disk? The location depends on FCP project location I think.

What bitrate are your camera originals and the proxies (ProRes 422 LT)?

I have FCP as well, but have several unfinished projects that I started in Davinci Resolve, and I can't really justify buying the full Resolve Studio version just to allow me to edit the camera source (10-bit 422) directly (or with auto-generated proxies). The transcoding is inconvenient because it takes time (c. 1.5x real-time) and the files are huge, with high bitrates, which obviously requires really fast I/O - faster than I have on external disks in any case!

 

[ed.]

I suspect your bottleneck might be the codec and the bitrate of your transcoded files. It might be that the transcoding vector you picked is either not properly optimised for decoding or the bitrate is simply too high. What is the bitrate of the files coming out of the camera?
Maybe it's possible to find a better transcode medium.
For what it's worth, I happen to have a macbook pro m1 (16gb), and a t5 lying around. I did a quick test with a 4k red raw file and it played back smoothly in resolve from the t5 with no transcoding. I think the bitrate of that file is way lower than the numbers you put up here, though. I might have some more time to do some tests, but please do share the bitrates of your source files so I can simulate something similar. Maybe there are some samples online from the camera manufacturer?

 

[ed.]

I can't really justify buying the full Resolve Studio version just to allow me to edit the camera source

Also, I'm not sure what's your professional status and environment, but in any deadline based business time=money and if Resolve saves you hours of hassle for the transcoding... I would start making some calculations about how much every single saved hour of transcoding costs if I can make it disappear with 300$.

 

[Fomalhaut]

I suspect your bottleneck might be the codec and the bitrate of your transcoded files. It might be that the transcoding vector you picked is either not properly optimised for decoding or the bitrate is simply too high. What is the bitrate of the files coming out of the camera?

Maybe it's possible to find a better transcode medium.

The clips play back fine from the internal SSD, which is obviously a lot faster, but this does indicate that the M1 Mac can process these clips without trouble (with quite low CPU/GPU usage from what I have seen). However, the use of the external SSD does seem to slow things down much more than I would have expected. Are USB speeds affected by the type of data being transferred? Copying lots of small files is a lot slower than a few big files, so maybe the video files "present" as being lots of small files to copy? Sounds crazy, but I'm interested in understanding what is going on.

The camera files (Panasonic Lumix GH5) are 150Mbps (H.264 UHD 10-bit 422 @23.976fps). These play back fine from the external SSD.

I was wondering whether the M1 Mini might work better with ProRes instead of DNxHR, but I couldn't see much difference, other than the ProRes422 HQ files and bitrate being considerably greater than the equivalent DNxHR HQX files (c 1000Mbps for ProRes vs 700Mbps for DNxHR). [Update: quick test of CPU/GPU usage of playback with "mpv" showed DNxHR HQX using c. 200%CPU/ 4-5%GPU. ProRes422 HQ was xc. 325%CPU / 11% GPU. The disparity may be due to the almost 50% higher bitrate of the ProRes encoding ]

For what it's worth, I happen to have a macbook pro m1 (16gb), and a t5 lying around. I did a quick test with a 4k red raw file and it played back smoothly in resolve from the t5 with no transcoding. I think the bitrate of that file is way lower than the numbers you put up here, though. I might have some more time to do some tests, but please do share the bitrates of your source files so I can simulate something similar. Maybe there are some samples online from the camera manufacturer?

What are the bitrates of your RED raw clips? Even the smallest RED Raven 4.5K seems to be quite large (120MBps at 3:1 compression, or 960Mbps assuming 8bits per byte). The newer codecs used in the RED Komodo for example are over twice this, even in the "medium quality" setting.

Also, I'm not sure what's your professional status and environment, but in any deadline based business time=money and if Resolve saves you hours of hassle for the transcoding... I would start making some calculations about how much every single saved hour of transcoding costs if I can make it disappear with 300$.

I agree, but I am very far from professional! These are just personal videos of family & travel. If I were doing a lot of this, especially if earning money from it, the cost of the Studio edition would be easily recouped through improved productivity. I've already bought FCP with a view to using it for future projects, so the transcoding solution is just to allow me to finish projects that are already started in Davinci Resolve.

Fortunately, the M1 Mini seems to happily play back and scrub the UHD H.264 camera footage in FCP, so in the future I won't bother with transcoding at all, but I would let FCP create proxies if I end up more complex edits that stress the machine.

 

[ed.]

What are the bitrates of your RED raw clips? Even the smallest RED Raven 4.5K seems to be quite large (120MBps at 3:1 compression, or 960Mbps assuming 8bits per byte).

it's weird: according to a calculator on the RED website the 5K 8:1 clip I tried should be just 15 MB/s, but my file was 600 MB for 8 seconds, so it's more like 75MB/s.
Anyway I found that any Redcode clip with compression below 8:1 stutters from the T5 unless you lower the timeline resolution. But at least in this case the stuttering was really only for the first 3 seconds, and then the clip played smoothly. I'm not sure what's going on but it seems that there's more to bitrate and transfer speed involved...
But to come back to your problems... have you tried just lowering the timeline resolution to 1/2 or 1/4?

 

[Fomalhaut]

it's weird: according to a calculator on the RED website the 5K 8:1 clip I tried should be just 15 MB/s, but my file was 600 MB for 8 seconds, so it's more like 75MB/s.
Anyway I found that any Redcode clip with compression below 8:1 stutters from the T5 unless you lower the timeline resolution. But at least in this case the stuttering was really only for the first 3 seconds, and then the clip played smoothly. I'm not sure what's going on but it seems that there's more to bitrate and transfer speed involved...
But to come back to your problems... have you tried just lowering the timeline resolution to 1/2 or 1/4?

The published RED data rates are much lower than I would expect. From https://www.red.com/recording-time, 5K 8:1 @ 24fps uses 64MB/s (for the RED Scarlet) so much closer to your calculation.

This is still way lower than SSD sustained read speeds, and should be comfortably within USB 3 HDD speeds, so it’s interesting that you say your get stutters from the T5 at lower compression ratios.

As you say, there is more to performance than raw disk speed, but I wish I knew what it was!

My timeline resolution is already at 1980x1080, and I doubt this would affect the actual disk transfer because presumably the resolution drop is done in Davinci after it has received the full resolution file data. Th e problem isn’t processing power, it’s disk transfer speed (or the protocol, or USB controller ability)

 

[ed.]

My timeline resolution is already at 1980x1080, and I doubt this would affect the actual disk transfer

I meant to lower the resolution in the Playback > Timeline Proxy Mode menu. It only lowers the preview resolution and either 1/2 or 1/4 previews were running smoothly depending on the source material.

I suspect decoding (so processing power) is big chunk of what's going on, as there are probably codecs that are decode more easily than others or that are hardware accelerated.

 

[Fomalhaut]

I meant to lower the resolution in the Playback > Timeline Proxy Mode menu. It only lowers the preview resolution and either 1/2 or 1/4 previews were running smoothly depending on the source material.

I suspect decoding (so processing power) is big chunk of what's going on, as there are probably codecs that are decode more easily than others or that are hardware accelerated.

Good point on the decoding. It may be the case that the M1 chips actually decode H.264/H.265 faster and more smoothly than DNxHR or ProRes. If we avoid the need to transcode without sacrificing playback and editing performance, it can only make the workflow more efficient.

 

[ed.]

apparently the M1 chips have hardware acceleration for 10-bit HEVC, so it's clear that FCP uses that feature and Resolve makes you pay for it : )

 

[ed.]

Also, have you looked into exporting your existing project from Resolve to FCP? it looks like it might be possible, at least according to a very cursory google search.

 

[Fomalhaut]

Also, have you looked into exporting your existing project from Resolve to FCP? it looks like it might be possible, at least according to a very cursory google search.

Good question, and one that crossed my mind. I'd have to review the procedure and see if there are downsides. I only have rough edits so far, without any audio post-production, color-grading etc.