Phacility Cluster Devices
Updated 306 Days AgoPublic

This describes the devices we deploy in the Phacility Cluster, and some related things which aren't technically devices.

Device Classes

Device ClassPool SizeDescriptionNotes
admin1Admin Application ServerServes admin.phacility.com.
aux1Auxiliary ServicesInternal utility host.
alb1Admin Load BalancerELB for admin.phacility.com.
bastion1SSH Bastion HostBastion host for ops commands.
db24Database ServerRuns MySQL.
lb1Web Load BalancerELB for instances.
nlb1Notification Load BalancerELB for websockets.
notify1NotificationsAphlict notification server.
plb1Phurl LBServes phurl.io requests.
repo24Repos/DaemonsTerminates VCS HTTP/SSH traffic.
saux1Secure AuxiliaryIsolated auxiliary for secure.phabricator.com.
sbuild1Secure BuildIsolated builds for secure.phabricator.com.
secure4SecureServes the upstream and related services.
slb1Secure Load BalancerELB for secure.phacility.com.
vault1SSH Load BalancerLoad balances VCS SSH requests.
web4Application ServerTerminates normal HTTP traffic.

Obsolete Device Classes

Device ClassDescriptionNotes
clbCorp Site LBServed phacility.com before CORGI.
corpCorp SiteServeed phacility.com before CORGI.

Storage Classes

Storage ClassCountNotes
abak1Stores administrative backups.
adata1Administrative server working storage.
auxdata1Auxiliary tier working storage.
dbak24Stores database backups.
ddata24Database server working storage.
rbak24Stores repository backups.
rdata24Repository working storage.
rlog24Repository host logs.
sauxdata1Upstream auxiliary working storage.
sbak4Upstream backup.
sdata4Upstream database.
sbuilddata1Upstream build working storage.
slog4Upstream logs.
srepo4Upstream repos.
swap-Temporary swap volume.

Service Classes

Service ClassCountNotes
auxx1Auxiliary cluster service.
dbx24Database cluster service, may have multiple devices.
repox24Repository cluster service, may have multiple devices.
sauxx1Upstream auxiliary service.
sbuildx1Upstream build service.
srepox1Upstream repository service.

Device: admin

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalAccepts requests from alb and web devices.

These servers host admin.phacility.com and run the Instances application.

Instances in the cluster also make calls here to retrieve configuration.

Currently, a maximum of one admin device is supported, because some details about database credentials are assumed. Minor software changes are required to expand the size of this pool.

Device: aux

Listening PortInterfaceNotes
22InternalOperational SSH access.

These servers are used by internal processes. In particular, backups are staged here during export.

Device: alb

Listening PortInterfaceNotes
80Public InternetServes http://admin.phacility.com
443Public InternetServes https://admin.phacility.com

These are ELBs which sit in front of the admin pool and terminate SSL for public requests to the web UI.

Device: bastion

Listening PortInterfaceNotes
22Public InternetGatekeeper for operational access.

For details on bastion hosts, see Phacility Cluster Bastion.

Device: db

Listening PortInterfaceNotes
22InternalOperational SSH access.
3306InternalProvides MySQL services.

These database servers run MySQL.

Device: lb

Listening PortInterfaceNotes
80Public InternetServes phacility.com wildcard DNS.
443Public InternetServes phacility.com wildcard DNS.

These are ELBs which sit in front of the web pool and terminate SSL for instance requests.

Device: nlb

Listening PortInterfaceNotes
443Public InternetForwards websockets to notify hosts.

This does TCP forwarding of websocket requests.

Device: notify

Listening PortInterfaceNotes
22280InternalNotification client port.
22281InternalNotification server port.

Runs an Aphlict notification server.

Device: repo

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalServes Conduit requests from web hosts.
2223InternalServes SSH requests from web hosts.

These devices store and serve repositories, and run daemons.

Device: saux

Listening PortInterfaceNotes
22Public InternetServes auxiliary requests.

These devices run auxiliary services (primarily, repository automation) for the upstream.

Device: sbuild

Listening PortInterfaceNotes
22Public InternetServes build requests.

These devices run lower-trust build services for the upstream.

Device: secure

Listening PortInterfaceNotes
22InternalServes upstream repositories.
80InternalServes secure.phabricator.com, javelinjs.com, phabricator.org, blog.phacility.com.
2222InternalOperational SSH access.
22280InternalAphlict client server.
22281InternalAphlict admin server.

These devices serve the upstream, secure.phabricator.com. Because this service needs to be available to deploy the cluster, devices in this tier are only half-contained in the cluster. The cluster administration tools work, but the tier does not depend on cluster services. The goal is to prevent a cascading failure in the event of a cluster disaster.

Normally, you connect to these hosts with bin/remote ssh secure001 over the VPC. In a disaster scenario, keys with access to the bastion can access these hosts directly with ssh -p 2222 ubuntu@secure.phabricator.com.

Because this device is not entirely within the cluster, it runs some services via cron. You can review them with crontab -e.

Device: slb

Listening PortInterfaceNotes
22Public InternetBalances VCS SSH traffic.
80Public InternetBalances HTTP for below.
443Public InternetBalances secure.phabricator.com, javelinjs.com, phabricator.org, blog.phacility.com.
22280Public InternetBalances Aphlict client traffic.

Forwards requests to secure hosts.

Device: vault

Listening PortInterfaceNotes
22Public InternetServes vault.phacility.com over SSH, forwarding requests to the web tier.
2222InternalOperational SSH access.

These devices are SSH load balancers. They're implemented as normal hosts running HAProxy instead of as ELBs because ELBs can not forward inbound traffic on port 22.

Eventually, it would be nice to either merge these into the ELBs (if ELBs support forwarding port 22 in the future) or merge the ELBs into these (if we have some stronger reasoning for running our own load balancers).

The cost of having these machines in a separate class is twofold: the raw cost of more moving parts, and we have to serve SSH from vault.phacility.com (which we can direct here via DNS) instead of directly from instance domains. If we could merge the HTTP and SSH load balancing, we could accept both HTTP and SSH traffic on instance domains.

(We could serve SSH traffic on instance domains on a nonstandard port which the ELBs do support, but this seems worse to me than serving from a dedicated domain. We could also serve from both, so instance@vault.phacility.com and instance@instance.phacility.com:2345 would both work.)

Device: web

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalServes HTTP application traffic from lb balancers.
2223InternalServes SSH application traffic from vault balancers.

These are standard application web servers.

Last Author
epriestley
Projects
None
Subscribers
None