I scratched at the surface of T2794, but @chad also bought me a larger monitor so I can put more SSH terminals on it which should buy us another few weeks.

@epriestley: Have you seen clustershell? It's a massively scaleable management shell with some pretty interesting features. Wikimedia operations are adopting it in place of a lot of dsh and saltstack stuff. From what I have seen it is a really solid framework to build upon.

(For anyone following along, some additional discussion of clustershell in T2794.)

I deployed the cluster with Phage yesterday with mostly good results. I hit a handful of minor bugs with Phage (some of which have diffs out now) and a few cases where planned improvements (signal handling, output control / web UI) would have been helpful, none of which was too surprising.

Phage was able to start deployments more quickly than I could manually type server name (shocking!), so we ran into a couple of cases of admin getting slammed by inefficient Conduit calls (T11506 was last related work on this, but focused on servers making inefficient calls during normal operation). I made a minor improvement which was immediately available (rCORE265aa1c5) but some of what we're doing is still way less efficient than it should be. This is currently the scalability and stability bottleneck for deployments, but should be relatively easy to fix.

I'm also likely to add "limit" (maximum parallelism) and "throttle" (artificial delay between starting commands) support to Phage mostly as generally reasonable features that move toward some other features like better retry support. They aren't the best solution here, but they're probably the simplest one.

This is currently the scalability and stability bottleneck for deployments, but should be relatively easy to fix.

I fixed a big chunk of this (rCORE0ba8e621bf47, rSAAScded669ca4f2) but there's still some sort of CPU bottleneck in instances.queryinstances. Unfortunately, it's also difficult to profile Conduit method calls on admin.phacility.com for a variety of reasons: the XHProf UI is not installed there, you can't use the DarkConsole button on the Conduit console page since it's a POST request, and uploading profiles into another XHProf UI is currently a pain. I'm not sure how far I want to run down that rabbit hole.

I did the last deploy (T12316) with Phage again and things went much more smoothly after the various bugfixes and improvements.

The biggest issue I hit was that occasionally commands (like apt-get) will hang for whatever reason, and there's currently no good way to figure out which nodes need to be retried (best available approach is to grep the logs). Helpful would be:

• Generic support for command timeouts.
• Maybe, generic support for "consider this command timed out if it doesn't produce any output for X seconds", although I'm less pumped about that.
• When we get ^C'd: handle the signal, shut down the agents, and print results (command failures/successes) so the operator can make better decisions about retrying failures.

For wikimedia's "scap" tool we built a feature called "checks" which are configurable commands that run on each target machine and confirm if the desired commands were successful.

Checks are essentially like unit tests for the deployment state and their exit code can fail the deploy. A failed check will prompt the operator whether to rollback or ignore.

Obviously this only works for things which can easily be tested. One straightforward example: Provide a URL in the application which will output the current version of the code. The check can confirm that the current version matches the expected version to validate that, not only was the code updated but the service restarted successfully and is able to serve a simple request without fatals.

I know this is nothing revolutionary but it's provided a lot of utility with minimal complexity in our tooling.