See PHI480. An install with a master/master repository cluster has seen feedback from users about this message on the SSH wire flow:

# Waiting up to 120 second(s) for a cluster write lock...

Specifically:

• This message is missing an English translation.
• This message isn't as clear as it could be that the lock will be acquired instantaneously when it's available (i.e., we aren't wasting time in sleep(...)).
• This message isn't as clear as it could be about the scope of the lock (global per repository).

It would be more helpful if waiting for a write lock gave you more information about who has the lock, so it's clear that the server is doing something and that you aren't just being made to wait for no reason or because of a bug:

Acquiring the repository write lock on rXYZ...
Waiting for epriestley to finish pushing abcdef1234 to master...
...

Beyond this, we could improve the write routing algorithm. The write process looks like this:

• Choose a random node.
• Acquire the repository write lock.
• Acquire the host read lock.
• Synchronize.
• Release the host read lock.
• Accept the write.
• Release the repository write lock.

This synchronize can be skipped if we get lucky, and randomly choose the same node that the writer ahead of us chose, since that node will already have the write we just waited for.

We can make clients exceptionally lucky by routing like this:

• If possible, choose the node with a writer holding the lock.
• If possible, choose a leader node.
• Choose a random node.

Since writes are always one-at-a-time, there's effectively no downside to this: we can't hit a "thundering herd" problem with write traffic.

See also T10884 for read routing, which is a little trickier, since a bad read routing algorithm can kill a cluster by focusing too much traffic on a small set of nodes.

I plan to make these changes:

• Modify the PushEvent table to contain more information about locks and waits (e.g., active queue status; how long the push waited for the write lock; how long the push spent synchronizing).
• Modify the "Waiting..." prompt to show more information by querying the PushEvent table.

Based on this data, we can make a decision about whether smarter write routing is worthwhile by seeing how much time pushes are spending synchronizing after acquiring the write lock.

(I suspect it will be, and the bar to implement this isn't very high, but it would still be nice to have some data suggesting that it will have a real effect: although I think the smarter routing is strictly better, it is more complex than routing with shuffle().)