Yes, just logic. I don't use anything else. I've read Reaper may actually be better optimized for apple silicon, but without using it I can't comment.
It might be worth your time to try another DAW to see if that is a contributing factor. Yes, Reaper is supposedly better optimized for AS. You could also try Studio One as there is a free 'full' version available. Both are free to try. It may provide you with some addition information.
I'd have to re-read this thread but I don't recall you mentioning having a problem with any other apps other than Logic, that would then lend itself to point to Logic as a possibility.