Explicitly shuffle nodes before selecting one for cluster sync
- when performing a write, we can go to any node (D19734 should make our ranking good);
- when performing a read, we can go to any node (currently random, but T10884 discusses ideas to improve our ranking);
- when performing an internal synchronization before a read or a write, we must go to an up-to-date node.
Currently, case (3) is not-exactly-deterministic but not random, and we won't spread intracluster traffic acrosss the cluster evenly if, say, half of it is up to date and half of it is still synchronizing. For a given write, I believe all nodes will tend to synchronize from whichever node first received the write today.
Instead, shuffle the list and synchronize from any up-to-date node.
(I think we could improve upon this only by knowing which nodes actually have load and selecting the least-loaded -- doable, but not trivial.)
Test Plan: Poked at it locally, will deploy to secure. This is hard to measure/test terribly convincingly.
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D19735