Don’t know if this would be your issue or not but sometimes network lag can cause the camera to seem to not respond immediately. Possibly it’s responding slightly after your last input?
Good suggestion but I have eliminated that by being very careful to press once or twice in the same direction and waiting at least 5 minutes between presses.
I figured it out. I had defined a motion tracking area. When you do that the camera always returns to the position the area was defined in after 15 seconds of no activity.