Location services are on an "as needed" basis. The location arrow appears only when that service is is use, nor just when it is turned on for an app.
But, did you try "restarting" your Mac after toggling that setting? Maybe that is all that is needed.
The initial prompt to turn location services on for a specific app is for your convenience, and is not a "once-only" setting change. You can always go back in to those location settings. Your app should be there in the list, and you can turn the service on (or off, if your want to turn it off) for that app. The box gets checked, and the service should be "on". The arrow in the menubar simply tells you that the service is being used, at that moment, by an app that needs to use it. Otherwise, the arrow should normally be hidden. Nothing wrong here. (I assume that MOST apps should make the arrow appear when the service is turned on for that app, but maybe that depends on how THAT app uses the service.
What specific app gives you these concerns, anyway?