Page MenuHomePhabricator

Phacility Cluster Devices
Updated 3,049 Days AgoPublic

Version 23 of 36: You are viewing an older version of this document, as it appeared on Nov 21 2015, 5:55 PM.

This describes the devices we deploy in the Phacility Cluster, and some related things which aren't technically devices.

Device Classes

Device ClassPool SizeDescriptionNotes
admin1Admin Application ServerServes admin.phacility.com.
aux1Auxiliary ServicesInternal utility host.
alb1Admin Load BalancerELB for admin.phacility.com.
bastion1SSH Bastion HostBastion host for ops commands.
clb1Corp Site LBLoad balances phacility.com requests.
corp1Corp SiteServes phacility.com requests.
db5Database ServerRuns MySQL.
lb1Web Load BalancerELB for instances.
nlb1Notification Load BalancerELB for websockets.
notify1NotificationsAphlict notification server.
repo5Repos/DaemonsTerminates VCS HTTP/SSH traffic.
saux1Secure AuxiliaryIsolated auxiliary for secure.phabricator.com.
secure1SecureServes the upstream and related services.
sbuild1Secure BuildIsolated builds for secure.phabricator.com.
vault1SSH Load BalancerLoad balances VCS SSH requests.
web2Application ServerTerminates normal HTTP traffic.

Storage Classes

Storage ClassCountNotes
abak1Stores administrative backups.
adata1Administrative server working storage.
auxdata1Auxiliary tier working storage.
dbak5Stores database backups.
ddata5Database server working storage.
rbak5Stores repository backups.
rdata5Repository working storage.
rlog5Repository host logs.
sauxdata1Upstream auxiliary working storage.
sbak1Upstream backup.
sdata1Upstream database.
sbuilddata1Upstream build working storage.
srepo1Upstream repos.
swap-Temporary swap volume.

Service Classes

Service ClassCountNotes
auxx1Auxiliary cluster service.
dbx5Database cluster service, may have multiple devices.
repox5Repository cluster service, may have multiple devices.
sauxx1Upstream auxiliary service.
sbuildx1Upstream build service.

Device: admin

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalAccepts requests from alb and web devices.

These servers host admin.phacility.com and run the Instances application.

Instances in the cluster also make calls here to retrieve configuration.

Currently, a maximum of one admin device is supported, because some details about database credentials are assumed. Minor software changes are required to expand the size of this pool.

Device: aux

Listening PortInterfaceNotes
22InternalOperational SSH access.

These servers are used by internal processes. In particular, backups are staged here during export.

Device: alb

Listening PortInterfaceNotes
80Public InternetServes http://admin.phacility.com
443Public InternetServes https://admin.phacility.com

These are ELBs which sit in front of the admin pool and terminate SSL for public requests to the web UI.

Device: bastion

Listening PortInterfaceNotes
22Public InternetGatekeeper for operational access.

For details on bastion hosts, see Phacility Cluster Bastion.

Device: clb

Listening PortInterfaceNotes
80Public InternetServes phacility.com corporate site.
443Public InternetServes phacility.com corporate site.

These are ELBs which sit in front of the corp pool and terminate SSL for corporate site requests.

Device: corp

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalServes phacility.com corporate site.

These are web application servers which serve corporate site requests.

These are currently launched into the block002 subnet because it's part of the us-west-1c availability zone, where we historically have reserved instances. It's desirable to spread across multiple availability zones in the long run, anyway.

Device: db

Listening PortInterfaceNotes
22InternalOperational SSH access.
3306InternalProvides MySQL services.

These database servers run MySQL.

Device: lb

Listening PortInterfaceNotes
80Public InternetServes phacility.com wildcard DNS.
443Public InternetServes phacility.com wildcard DNS.

These are ELBs which sit in front of the web pool and terminate SSL for instance requests.

Device: nlb

Listening PortInterfaceNotes
443Public InternetForwards websockets to notify hosts.

This does TCP forwarding of websocket requests.

Device: notify

Listening PortInterfaceNotes
22280InternalNotification client port.
22281InternalNotification server port.

Runs an Aphlict notification server.

Device: repo

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalServes Conduit requests from web hosts.
2223InternalServes SSH requests from web hosts.

These devices store and serve repositories, and run daemons.

Device: saux

Listening PortInterfaceNotes
22Public InternetServes auxiliary requests.

These devices run auxiliary services (primarily, repository automation) for the upstream.

Device: secure

Listening PortInterfaceNotes
22Public InternetServes upstream repositories.
80Public InternetServes javelinjs.com, phabricator.org, blog.phacility.com, redirects to HTTPS.
443Public InternetServes secure.phabricator.com.
2222Public InternetOperational SSH access.

These devices serve the upstream, secure.phabricator.com. Because this service needs to be available to deploy the cluster, devices in this tier are only half-contained in the cluster. The cluster administration tools work, but the tier does not depend on cluster services. The goal is to prevent a cascading failure in the event of a cluster disaster.

Normally, you connect to these hosts with bin/remote ssh secure001 over the VPC. In a disaster scenario, keys with access to the bastion can access these hosts directly with ssh -p 2222 ubuntu@secure.phabricator.com.

Because this device is not entirely within the cluster, it runs some services via cron. You can review them with crontab -e.

Device: sbuild

Listening PortInterfaceNotes
22Public InternetServes build requests.

These devices run lower-trust build services for the upstream.

Device: vault

Listening PortInterfaceNotes
22Public InternetServes vault.phacility.com over SSH, forwarding requests to the web tier.
2222InternalOperational SSH access.

These devices are SSH load balancers. They're implemented as normal hosts running HAProxy instead of as ELBs because ELBs can not forward inbound traffic on port 22.

Eventually, it would be nice to either merge these into the ELBs (if ELBs support forwarding port 22 in the future) or merge the ELBs into these (if we have some stronger reasoning for running our own load balancers).

The cost of having these machines in a separate class is twofold: the raw cost of more moving parts, and we have to serve SSH from vault.phacility.com (which we can direct here via DNS) instead of directly from instance domains. If we could merge the HTTP and SSH load balancing, we could accept both HTTP and SSH traffic on instance domains.

(We could serve SSH traffic on instance domains on a nonstandard port which the ELBs do support, but this seems worse to me than serving from a dedicated domain. We could also serve from both, so instance@vault.phacility.com and instance@instance.phacility.com:2345 would both work.)

Device: web

Listening PortInterfaceNotes
22InternalOperational SSH access.
80InternalServes HTTP application traffic from lb balancers.
2223InternalServes SSH application traffic from vault balancers.

These are standard application web servers.

Last Author
epriestley
Last Edited
Nov 21 2015, 5:55 PM

Event Timeline

epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)