The basic problem is that we do not perceive colors accurately, and the color we perceive is affected by the surrounding context.
If you put a teal color next to a blue, it will appear greenish.
If you put the same teal color next to a green it will look bluish.
If you look at a 5000K white surrounded by a 3000K white, it will look blue.
If you look at the same 5000K white surrounded by a 6500K white, it will look amber.
All of these whites are white, just different temperatures. There is no "correct" temperature for white.
True Tone attempts to compensate for our inaccurate color perception as the surrounding color temperature changes.
If you set True Tone off, then the only way to perceive the displayed colors accurately is to ensure your ambient surrounding light is at the same color temperature for which the screen is calibrated.
I suspect the phone screen is calibrated to a moderately high color temperature like 6500K. You would then need to change all your light bulbs to 6500K bulbs (!) in order to perceive the colors accurately (or view it outside on a cloudy day).
I do a lot of photo editing, and have changed all my light bulbs to 5000K. I should really go higher, so it is a compromise for livability.