If you want better colour accuracy, turn it off when editing and viewing images. True Tone is designed to make reading easier on the eyes by warming the background whites, it has nothing to do with colour accuracy when viewing images or movies.
I have seen it argued that it makes whites whiter in other threads, which is not actually the case, it warms the tones considerably and adapts the amount of warming according to the ambient light. It does not try and create accurate colours according to ambient light, it tries to make the whites of the backgrounds easier for the eye to deal with when reading.
Some people report that when they are in Photos it turns off, but I have never found that to be the case. Images are still considerably warmer when held next to my calibrated monitor, so both devices are under the same ambient light for comparison. As soon as True Tone is turned off, the images are much closer in appearance to the calibrated monitor.