I’ve been having this exact problem for a couple of months now on the 4th gen Apple TV, whatever’s causing it has made the Siri remote totally useless. When I do get it to work its function is very sluggish the touchpad doesn’t work at all, word to the wise don’t unpair it from the system. It won’t re-pair, I’ve just purchased the new remote in the hope that works. I can’t put my finger on when it started exactly, but to say it’s frustrating would be an understatement. When I contacted Apple about it they just ran through the usual steps, unplug it from the socket, change hdmi lead, do a system restore, hold the menu down and volume up button in for 5 seconds. If that failed (which it did) unpair it and re-pair it, only for it not to and when I explained this the person hung up the phone 😡