I mean none of these things are show-stoppers in the example above, but you can imagine how bad things would be over a link between the Sydney data centre and N. California if it was happening all the time.

There's probably almost certainly a correlation there that link speed increases with lower latency because of proximity? But having an SSH proxy accidentally pick a link that's say in Oregon instead of North California because at the time of sampling some local network issues were being had that made the local copy appear worse would be a reasonably bad situation for a few reasons:

Ah, link speed vs latency is interesting.

Is "use measured latency" probably always the correct rule in your (hypothetical) cluster setup? Or do you have any cases offhand where it isn't the desired result, and you really want to send traffic to a more distant node on the network?

Gut feeling is that network latency almost certainly isn't the only reasonable concern here.

I don't currently expect anyone to be deploying multi-region clusters in the first iteration of real clustering

I don't currently expect anyone to be deploying multi-region clusters in the first iteration of real clustering, so this isn't high on my list of things to fix in this round.

See also T10884 for related work: ranking similar nodes more intelligently.

See also T10883 for related work: making some nodes read-only.

After we receive a push, try very very very very hard to release the lock if we're the thread holding it, even if we lost the global lock