Page MenuHomePhabricator

Phacility Cluster Devices
Updated 1,592 Days AgoPublic

This describes the devices we deploy in the Phacility Cluster, and some related things which aren't technically devices.

Device Classes

Device ClassPool SizeDescriptionNotes
admin1Admin Application ServerServes
aux1Auxiliary ServicesInternal utility host.
alb1Admin Load BalancerELB for
bastion1SSH Bastion HostBastion host for ops commands.
db24Database ServerRuns MySQL.
lb1Web Load BalancerELB for instances.
nlb1Notification Load BalancerELB for websockets.
notify1NotificationsAphlict notification server.
plb1Phurl LBServes requests.
repo24Repos/DaemonsTerminates VCS HTTP/SSH traffic.
saux1Secure AuxiliaryIsolated auxiliary for
sbuild1Secure BuildIsolated builds for
secure4SecureServes the upstream and related services.
slb1Secure Load BalancerELB for
vault1SSH Load BalancerLoad balances VCS SSH requests.
web4Application ServerTerminates normal HTTP traffic.

Obsolete Device Classes

Device ClassDescriptionNotes
clbCorp Site LBServed before CORGI.
corpCorp SiteServeed before CORGI.

Storage Classes

Storage ClassCountNotes
abak1Stores administrative backups.
adata1Administrative server working storage.
auxdata1Auxiliary tier working storage.
dbak24Stores database backups.
ddata24Database server working storage.
rbak24Stores repository backups.
rdata24Repository working storage.
rlog24Repository host logs.
sauxdata1Upstream auxiliary working storage.
sbak4Upstream backup.
sdata4Upstream database.
sbuilddata1Upstream build working storage.
slog4Upstream logs.
srepo4Upstream repos.
swap-Temporary swap volume.

Service Classes

Service ClassCountNotes
auxx1Auxiliary cluster service.
dbx24Database cluster service, may have multiple devices.
repox24Repository cluster service, may have multiple devices.
sauxx1Upstream auxiliary service.
sbuildx1Upstream build service.
srepox1Upstream repository service.

Device: admin

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalAccepts requests from alb and web devices.

These servers host and run the Instances application.

Instances in the cluster also make calls here to retrieve configuration.

Currently, a maximum of one admin device is supported, because some details about database credentials are assumed. Minor software changes are required to expand the size of this pool.

Device: aux

Listening PortInterfaceNotes
22InternalOperational SSH access.

These servers are used by internal processes. In particular, backups are staged here during export.

Device: alb

Listening PortInterfaceNotes
80Public InternetServes
443Public InternetServes

These are ELBs which sit in front of the admin pool and terminate SSL for public requests to the web UI.

Device: bastion

Listening PortInterfaceNotes
22Public InternetGatekeeper for operational access.

For details on bastion hosts, see Phacility Cluster Bastion.

Device: db

Listening PortInterfaceNotes
22InternalOperational SSH access.
3306InternalProvides MySQL services.

These database servers run MySQL.

Device: lb

Listening PortInterfaceNotes
80Public InternetServes wildcard DNS.
443Public InternetServes wildcard DNS.

These are ELBs which sit in front of the web pool and terminate SSL for instance requests.

Device: nlb

Listening PortInterfaceNotes
443Public InternetForwards websockets to notify hosts.

This does TCP forwarding of websocket requests.

Device: notify

Listening PortInterfaceNotes
22280InternalNotification client port.
22281InternalNotification server port.

Runs an Aphlict notification server.

Device: repo

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalServes Conduit requests from web hosts.
2223InternalServes SSH requests from web hosts.

These devices store and serve repositories, and run daemons.

Device: saux

Listening PortInterfaceNotes
22Public InternetServes auxiliary requests.

These devices run auxiliary services (primarily, repository automation) for the upstream.

Device: sbuild

Listening PortInterfaceNotes
22Public InternetServes build requests.

These devices run lower-trust build services for the upstream.

Device: secure

Listening PortInterfaceNotes
22InternalServes upstream repositories.
2222InternalOperational SSH access.
22280InternalAphlict client server.
22281InternalAphlict admin server.

These devices serve the upstream, Because this service needs to be available to deploy the cluster, devices in this tier are only half-contained in the cluster. The cluster administration tools work, but the tier does not depend on cluster services. The goal is to prevent a cascading failure in the event of a cluster disaster.

Normally, you connect to these hosts with bin/remote ssh secure001 over the VPC. In a disaster scenario, keys with access to the bastion can access these hosts directly with ssh -p 2222

Because this device is not entirely within the cluster, it runs some services via cron. You can review them with crontab -e.

Device: slb

Listening PortInterfaceNotes
22Public InternetBalances VCS SSH traffic.
80Public InternetBalances HTTP for below.
443Public InternetBalances,,,
22280Public InternetBalances Aphlict client traffic.

Forwards requests to secure hosts.

Device: vault

Listening PortInterfaceNotes
22Public InternetServes over SSH, forwarding requests to the web tier.
2222InternalOperational SSH access.

These devices are SSH load balancers. They're implemented as normal hosts running HAProxy instead of as ELBs because ELBs can not forward inbound traffic on port 22.

Eventually, it would be nice to either merge these into the ELBs (if ELBs support forwarding port 22 in the future) or merge the ELBs into these (if we have some stronger reasoning for running our own load balancers).

The cost of having these machines in a separate class is twofold: the raw cost of more moving parts, and we have to serve SSH from (which we can direct here via DNS) instead of directly from instance domains. If we could merge the HTTP and SSH load balancing, we could accept both HTTP and SSH traffic on instance domains.

(We could serve SSH traffic on instance domains on a nonstandard port which the ELBs do support, but this seems worse to me than serving from a dedicated domain. We could also serve from both, so and would both work.)

Device: web

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalServes HTTP application traffic from lb balancers.
2223InternalServes SSH application traffic from vault balancers.

These are standard application web servers.

Last Author
Last Edited
Mar 17 2017, 12:41 PM

Event Timeline

epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)