Well, like The Fortunes sang in that song, "You've Got Your Troubles, I've Got Mine." (Heard that on the radio recently...)
My system's behavior is similar in that once the list is populated, it populates very quickly on subsequent views. The first time might take about two seconds for large folders.
What I noticed along the way was that after I used Onyx, the initial population of some folders that used to take 20 plus seconds came down to two or three. Smaller folders were almost instant. After some time, depending on actual computer use, not just time since boot, the listing time gradually stretched out. A redo with Onyx fixed that. As I said, I wasn't sure if it was whatever Onyx did or whether it was the restart. Now, I find that a restart cleans things up when Finder starts getting lazier. But, perhaps it was the combination action of Onyx cleaning up some cache file(s) and the restart that did it. I don't have any way to tell, unfortunately. Sorry.
Since Apple has either not recognized the problem to date or has chosen to ignore it for a couple years, I don't have much faith that it will be fixed soon. After all, at this point they'd be admitting that there is a bug. And, it seems like their culture is that there's no glory in fixing things. As
HDFan reported earlier, Apple will spend a long period deciding whether to even vote on looking into a bug after it's been reported by a lot of users. Plus, Apple's general customer facing approach seems to always be "We're moving ahead, we don't care about the past. Hey! Look at all the new emojis!"
Forklift might be the real solution for users. Waiting on Apple probably is not. It probably shouldn't be that way, but I'm finding more and more that Apple's supplied application software is getting less and less useful as they "improve" things.
UPDATE: Just did a little more research. It seems that I am really ignorant about Finder issues. I guess that's because its limitations haven't really annoyed me until now with this bug. But, a lot of people have been unhappy for quite some time. This is why there's a market for Finder alternatives.
This article has a good summary toward its end that explains the problems with Finder, in very polite terms. Methinks that looking at alternatives to Finder may be in order. Apple has zero incentive to improve or fix its product, and they've demonstrated that. When I look at the Apple applications that I actually use now, it's down to Finder, Preview, Messages, Mail, and Safari. The last two are workable since I've found suitable extensions that get around the original app's limitations. (I'm not counting utilities like Terminal, Activity Monitor, and the other applications located in the Utilities folder.) The problem with Finder alternatives is that apparently a number of other applications use Finder to list files in a folder when you go to access them. So, the alternative doesn't help there.