Phage (Phacility)

Updated 2,562 Days AgoPublic

Actions

Overview

Phage is a prototype parallel shell tool, like Hypershell. See T2794 and T12218 for some context.

Currently, it isn't super fancy, but it works well enough.

Phage is exposed as a wrapper around bin/remote, described in Phacility Cluster CLI Tools.

Setup

Phage is currently only available in the experimental branch of arcanist/.

arcanist/ $ git checkout experimental

After switching to experimental, put arcanist/bin/phage in your path if it isn't already.

Test Phage by running this command in core/:

core/ $ phage remote status --hosts web001

That should print out some status information and exit happily.

Deploy: secure

To deploy secure.phabricator.com, use this command from core/:

core/ $ phage remote deploy --hosts secure001-4 --limit 1

Notes:

• On the secure tier, use --limit 1. The tier is technically deploying from itself, and can end up in a bad state if you take the whole thing down (specifically: all readable repository hosts) at once.
• The remote deploy command is a full upgrade, use remote upgrade instead to just update Phacility software (Phabricator, etc). This mostly skips apt-get stuff and runs significantly faster.

Deploy: production

To deploy the production cluster, do something like this:

core/ $ phage remote deploy --hosts secure001-4 --limit 1

Verify that secure.phabricator.com is alive.

core/ $ phage remote deploy --hosts saux001,sbuild001,aux001
core/ $ phage remote deploy --hosts admin001

Verify that admin.phacility.com is alive.

core/ $ phage remote deploy --hosts db001-XXX --limit 8 --throttle 4 --timeout 600
core/ $ phage remote deploy --hosts repo001-YYY --limit 8 --throttle 4 --timeout 600
core/ $ phage remote deploy --hosts web001-ZZZ

Verify that instances (like meta.phacility.com) are alive. The throttling and timeouts are kind of arbitrary but we're soft-limited by API calls to admin at time of writing, see T12297.

core/ $ phage remote status --hosts db001-XXX,repo001-YYY

Verify that nothing's about to run out of storage. If anything is getting low, bin/host destroy (delete old instance data) and bin/host purge (delete old backups) can usually buy a bunch of headroom.

If storage adjustments are required, remote adjust these hosts: secure001, admin001, all dbXXX hosts. In some cases, these adjustments should happen immediately after deploy, although they're usually safe to perform at the end.