I've found two significant issues with FCPX 10.4 HEVC support on the iMac Pro:
(1) There's a bug with the persistence of the HEVC preset when applied to FCPX. If I add the preset it works, then if I simply restart FCPX I get the error "HEVC 4k 8-bit: The setting 'Apple Devices 4K (HEVC 8-bit)' uses software that is not installed or may no longer be supported." It's not unique to 4k 8 bit but also happens on 10 bit HEVC.
This is 100% reproducible and not random.
Background: Since FCPX 10.4 by itself doesn't support HEVC, the preset must be added from Compressor. The procedure is described in this article by Larry Jordan:
https://larryjordan.com/articles/configure-final-cut-pro-x-to-output-hevc-h-265/
My current workaround is remove and re-add the preset every time I use it. Fortunately I don't use HEVC except for testing.
(2) FCPX apparently only uses hardware acceleration for 8-bit encoding of 4k HEVC, not 10-bit (at least on the iMac Pro). A 60 sec 4k/29.97 ProRes clip takes about 43 sec to encode to 4k 8-bit HEVC, but it takes 22 *minutes* for 4k 10-bit HEVC, or 31 times slower than 8-bit encoding.
FCPX is apparently using AMD's VCE (Video Coding Engine) hardware acceleration for H264 and HEVC encode/decode. AMD's documentation says hardware acceleration for 4k 10-bit HEVC *decode* is available, but I cannot tell whether hardware acceleration is available for 4k 10-bit HEVC *encoding*. Whether it's not in the Vega hardware, or just doesn't work, or AMD's AMF framework doesn't expose it, or Apple isn't using it, this is a significant issue. It means on the iMac Pro 4k HEVC 10-bit encoding is almost unusable due to performance reasons.
See page 14 of the AMD white paper "Radeon's next generation Vega architecture:
https://radeon.com/_downloads/vega-whitepaper-11.6.17.pdf
I would be very interested if FCPX 4k HEVC 10-bit hardware accelerated encoding works on a 2017 iMac Pro which uses Quick Sync. Has anyone tested that?
System info:
FCPX 10.4, Compressor 4.4, Apple Pro Video Formats 2.0.6, macOS 10.13.3, 10-core Vega 64 iMac Pro