Well it is not a question of being annoyed, I was just trying to figure out what is causing the location to always being used. Also, besides that, the normal location icon would keep going on and off constantly, so something kept trying to use the location service. That has to be a minor drain on battery life.
So I just did a factory reset and reinstalled all my apps. The Green icon is gone and it is not checking location constantly now either. As fas as tapping the icon to see what was using it, it just said System Services, but I couldn't figure out what service was using it.
So now I will re-setup all my apps and see if any one of them set it off, but for now, factory reset got rid of it.