Not at all, you won't notice a huge difference. I'm pushing my M1 Air at 8gb to the max, and it is running as fast or faster than my 16GB i7 Spectre.Now I am questioning whether I should I have gone with 8gb myself as I continue to wait 3-4 weeks for my 16gb/1TB Mini to arrive. I am very tempted to cancel it and just order the 8gb/256 base and have it arrive in a days time. I am coming from a 2013 iMac and just wanted to future proof myself, but I highly doubt I’ll keep this first gen silicon mini for another 7 years anyways. I don’t really do anything extreme. I run mostly all native apple apps. Safari (for my eBay business), Photos and Mail are open 100% of the time. Occasionally I use final cut with some pixelmator. Do I really need the 16gb config? would I notice a difference?
Now I may upgrade to the 16gb just because I may want to keep it for a minimum of five years, and 16gb will give me a peace of mind, but if you don't plan on keeping it for too long, I think 8GB will suffice.
I don't know how far Apple will take with this new unified memory RAM, but so far, it's way more efficient than Intel chips. I would recommend using it and really pushing it to its max potential to see if it fits your needs(do your regular workflow + throw in some extra things in the background to see if it can handle the load)Another thing to consider is that the M1 Mini seems to be the most efficient/powerful of the three new Macs.
What I plan on doing is using this until the 8th of January(last day to return), and see if I can swap it for a 16GB. I only opted for the 8 because it was the only model I could have gotten without waiting for a month. If the 16gb models aren't available, I'll trade my 8GB in for store credit and just wait til the delivery times are shorter.