I paid the Apple tax to get 32gb memory in my m2 Mini Pro. It made a huge difference in my work compared to the 16gb m1 Mini it replaced and it still seems fast. I don't foresee updating to the m4 unless there's some magical new AI capability that I haven't heard about yet.
How did it make a difference? Just curious what your software is and where the memory helps.
Also, did you consider going with the Mac Studio? I see the base Mac Studio is US$1999 and the 32 GB Mac mini M2 Pro is $1699, so there is still a $300 difference. (I don't really care about the 10 GigE.)
Personally, I'd like to get a 24 / 512 GB M4 non-Pro for say US$1099, but sadly it will likely be short on ports. If the base M4 Pro came with 24 GB RAM for say $1399, that'd be an awesome option.
I'm currently running a 16 GB / 1 TB M1 mini, but I got the 1 TB model just because it happened to be relatively cheap, at CA$1099 / ~US$825 on eBay back in 2022.
Truthfully the 16 GB is fine for my actual work, but I wonder if going to 24 GB would help with my non-work stuff, specifically Apple Photos' speed and reliability.