I have the same 185mm wrist I tried both size leather loops on in the store and decided the large was too long and the medium too short.
Technically, they both fit, but if I pulled the large real snug, it came up over the edge of the watch.
No problem like that with the medium leather loop, but I'd hoped the leather loop would slip on like the Milanese loop or link bracelet no need to thread the end of the strap into the slot every time. The medium size leather loop is way too short to go over my hand like that however, if you strap it on (as Apple no doubt intends), if fit just fine.
Unfortunately, I don't own either size leather loop I can only give up my initial impression from the store try on. I hope someone that's actually made the switch from large to medium will post.