Just like that, we seem to have solved the issues with the Siri Remote on our primary AppleTV. I'm guessing that the problem lay with the AppleTV itself, and not the Siri Remote.
Creating a HomeKit home, bit by bit
Creating a HomeKit home, bit by bit