Phacility Cluster Devices

Updated 2,812 Days AgoPublic

Actions

This describes the devices we deploy in the Phacility Cluster, and some related things which aren't technically devices.

Device Classes

Obsolete Device Classes

Storage Classes

Service Classes

Device: admin

These servers host admin.phacility.com and run the Instances application.

Instances in the cluster also make calls here to retrieve configuration.

Currently, a maximum of one admin device is supported, because some details about database credentials are assumed. Minor software changes are required to expand the size of this pool.

Device: aux

These servers are used by internal processes. In particular, backups are staged here during export.

Device: alb

These are ELBs which sit in front of the admin pool and terminate SSL for public requests to the web UI.

Device: bastion

For details on bastion hosts, see Phacility Cluster Bastion.

Device: db

These database servers run MySQL.

Device: lb

These are ELBs which sit in front of the web pool and terminate SSL for instance requests.

Device: nlb

This does TCP forwarding of websocket requests.

Device: notify

Runs an Aphlict notification server.

Device: repo

These devices store and serve repositories, and run daemons.

Device: saux

These devices run auxiliary services (primarily, repository automation) for the upstream.

Device: sbuild

These devices run lower-trust build services for the upstream.

Device: secure

These devices serve the upstream, secure.phabricator.com. Because this service needs to be available to deploy the cluster, devices in this tier are only half-contained in the cluster. The cluster administration tools work, but the tier does not depend on cluster services. The goal is to prevent a cascading failure in the event of a cluster disaster.

Normally, you connect to these hosts with bin/remote ssh secure001 over the VPC. In a disaster scenario, keys with access to the bastion can access these hosts directly with ssh -p 2222 ubuntu@secure.phabricator.com.

Because this device is not entirely within the cluster, it runs some services via cron. You can review them with crontab -e.

Device: slb

Forwards requests to secure hosts.

Device: vault

These devices are SSH load balancers. They're implemented as normal hosts running HAProxy instead of as ELBs because ELBs can not forward inbound traffic on port 22.

Eventually, it would be nice to either merge these into the ELBs (if ELBs support forwarding port 22 in the future) or merge the ELBs into these (if we have some stronger reasoning for running our own load balancers).

The cost of having these machines in a separate class is twofold: the raw cost of more moving parts, and we have to serve SSH from vault.phacility.com (which we can direct here via DNS) instead of directly from instance domains. If we could merge the HTTP and SSH load balancing, we could accept both HTTP and SSH traffic on instance domains.

(We could serve SSH traffic on instance domains on a nonstandard port which the ELBs do support, but this seems worse to me than serving from a dedicated domain. We could also serve from both, so instance@vault.phacility.com and instance@instance.phacility.com:2345 would both work.)

Device: web

These are standard application web servers.