Add limit (maximum simultaneous commands) and throttle (delay between commands)…


Add limit (maximum simultaneous commands) and throttle (delay between commands) to Phage

Ref T12218. Ref T2794. This just provides a couple of general purpose tools to slow Phage down a bit if the speed is getting TOO EXTREME.

In particular, secure updates from itself so if all 4 nodes get hit simultaneously the deploy doesn't work. It deploys cleanly with --limit 1.

A separate change to the phage remote workflow in Instances exposes these options as flags:

  • --limit: Run only this many commands simultaneously.
  • --throttle: Wait this long between starting commands (for example, --throttle 0.25 will only start 4 commands per second).

Test Plan:

  • Ran phage remote status on secure with various --throttle and --limit arguments, and without any arguments.
  • Ran phage remote upgrade --limit 1 on secure, got a clean upgrade.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12218, T2794

Differential Revision: https://secure.phabricator.com/D17396