Are there any downsides when doing this?
None. I have a 14" M1P MacBook Pro and it sits on a little shelf, connected to a Thunderbolt docking station, two external monitors, keyboard, mouse, external SSD, CD/DVD burner and ethernet. Most of the time it pretends to be an M1 Pro Mac mini. Lid closed. No issues.
A few people might say that if you're maxing out the CPU and GPU to 100% for continuous extended periods of time, the heat created could potentially impact the built-in display with the lid closed (since some heat will rise though the keyboard). It's rare that someone's workflow would do this, but something to consider. My machine barely rises above body temperature, so I have no concerns.