4gb in the GPU should be fine.
What's more important is what kind of drive will be in it.
You DO NOT WANT a platter-based hard drive (only). You will regret this from the moment you first boot it up.
You DO NOT WANT a "1tb" fusion drive (SSD portion is only 32gb, too small).
You MIGHT WANT a 2tb fusion drive (SSD portion is 128gb, more workable).
You OUGHT TO consider a "straight SSD" inside -- BEST option for performance.
If you get the 27" iMac, get 8gb of RAM.
Buy more later and add it yourself -- easy and MUCH CHEAPER than buying it from Apple.