I'm writing this to all the poor souls out there that bought the Mac Mini M1 with the 10 GB adapter option:
Your Mac is probably not working at the correct network speed, and this is a hardware problem. It's not software (I think), as I have installed every possible patch, new macOS Ventura...