So I modified by web-remote Android app and my server so that they support UDP, along with the older HTTP API.
Results: Excellent. There’s no longer a 0.5-1.0 second pause when clicking buttons. I’m still not sure where the hell that came from, HTTP shouldn’t be this slow.