There is objective evidence that Silicon ports do not operate at their implied speeds.
Read this for some eye-opening analysis. So if OPs disappointment stems from what he expects from ports implied to work at typical speeds, perhaps it is as simple as the horses are either not actually there... or Apple needs to do some things in port management software to realize their full potential.
Due to other issues with "works fine on even pre-Big Sur Macs but not on Big Sur or newer Macs" situations, I've become near completely convinced that there are bugs in macOS port management software resulting in a very mixed bag of experiences ranging from "works perfectly" to "doesn't work at all" for U(niversal)SB devices.
Apple fans will- of course- shift blame to user, cables, third party enclosures, third party drives, user settings, etc, typically ignoring the reality that it works fine with PCs and even Macs running macOS before Big Sur using the same cable by the same user with the same enclosure and same drive. Why? Apparently, Apple can make no mistakes and/or they know there are problems but prefer to play the redirection game for presumably free for some unknown reason.
OP should certainly do the easy things like trying a different cable (just in case). However, if the same cable + enclosure + drive is plenty fast on other computers, it's not hard to grasp the unique variable(s) in the "too slow" scenario.
My advice: If OP wants near full speed, do what others are doing to try to work around the likely bugs in macOS: start trying a variety of enclosures until you find one that works faster than the one you have. This is fully hit or miss and not brand or age (of enclosure creation) dependent, meaning one product from a brand may work well and another won't... and assuming age is the culprit somewhat misses the point of the U in USB (verified in my own testing which included digging out some long retired USB enclosures to try to find a solution. For my issues, some of the oldies worked better than some brand new enclosures).
OR, accept it as it is and hope that Apple will eventually get around to debugging & refining their Silicon Port management code because it seems there are clearly some issues to improve there. Anyone with any doubts should do some searches for Silicon USB issues to find what seems like a near countless number of threads both here and elsewhere. If you really dig in, you are likely to find that most of the issues begin with Big Sur and are not a Silicon-only issue, implying macOS is the key player here. In many cases, all of the stuff "it could be" (that dodges pointing the finger at Apple) clash with it working fine on the same Mac BEFORE Big Sur (through same cable, by same user, with same user settings, etc), so the lone change was Big Sur or more recent macOS version upgrades.
