That'll be because the bit-rate of the movie has exceeded the Apple TV's limitation, not really anything to do with the size of the file. My Godfather 2 Blu-Ray rip is 6.5Gb in size, but the bitrate is within limits and looks and plays fine. The Apple TV preset in HandBrake is a little too rich (in terms of CQ percentage setting) for hi def content (IMHO). You can either control it with what you are doing now (setting the target size), or lower the CQ value for hi def content. Somewhere between 57 - 59% should give you acceptable quality and keep it within the ATV's limitations.
There are many arguments on here and over on the HandBrake forums about best perceived quality, but i think it is generally accepted that the target size option is the worst of the three options, with CQ being the best.
It's all about what looks good for you though.
Cheers