Change Details

This describes the devices we deploy in the [[ phacility_cluster | Phacility Cluster]], and some related things which aren't technically devices. Device Classes ====== | Device Class | Pool Size | Description | Notes | |---|---|---|---| | `admin` | 1 | Admin Application Server | Serves `admin.phacility.com`. | | `aux` | 1 | Auxiliary Services | Internal utility host. | | `alb` | 1 | Admin Load Balancer | ELB for `admin.phacility.com`. | `bastion` | 1| SSH Bastion Host | Bastion host for ops commands. | | `db` | 6 | Database Server | Runs MySQL. | | `lb` | 1 | Web Load Balancer | ELB for instances. | | `nlb` | 1 | Notification Load Balancer | ELB for websockets. | | `notify` | 1 | Notifications | Aphlict notification server. | | `plb` | 1 | Phurl LB | Serves `phurl.io` requests. | | `repo` | 6 | Repos/Daemons | Terminates VCS HTTP/SSH traffic. | | `saux` | 1 | Secure Auxiliary | Isolated auxiliary for `secure.phabricator.com`. | | `sbuild` | 1 | Secure Build | Isolated builds for `secure.phabricator.com`. | | `secure` | 2 | Secure | Serves the upstream and related services. | | `slb` | 1 | Secure Load Balancer | ELB for `secure.phacility.com`. | `vault` | 1 | SSH Load Balancer | Load balances VCS SSH requests. | | `web` | 2 | Application Server | Terminates normal HTTP traffic. | //Obsolete Device Classes// | Device Class | Description | Notes | | `clb` | Corp Site LB | Served `phacility.com` before CORGI. | | `corp` | Corp Site | Serveed `phacility.com` before CORGI. | Storage Classes ======= | Storage Class | Count | Notes | |---|---|---| | `abak` | 1 | Stores administrative backups. | | `adata` | 1 | Administrative server working storage. | | `auxdata` | 1 | Auxiliary tier working storage. | | `dbak` | 6 | Stores database backups. | | `ddata` | 6 | Database server working storage. | | `rbak` | 6 | Stores repository backups. | | `rdata` | 6 | Repository working storage. | | `rlog` | 6 | Repository host logs. | | `sauxdata` | 1 | Upstream auxiliary working storage. | | `sbak` | 2 | Upstream backup. | | `sdata` | 2 | Upstream database. | | `sbuilddata` | 1 | Upstream build working storage. | | `slog` | 2 | Upstream logs. | | `srepo` | 2 | Upstream repos. | | //`swap`// | - | Temporary swap volume. | Service Classes ======= | Service Class | Count | Notes | |---|---|---| | `auxx` | 1 | Auxiliary cluster service. | | `dbx` | 5 | Database cluster service, may have multiple devices. | | `repox` | 5 | Repository cluster service, may have multiple devices. | | `sauxx` | 1 | Upstream auxiliary service. | | `sbuildx` | 1 | Upstream build service. | | `srepox` | 1 | Upstream repository service. Device: admin ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Accepts requests from `alb` and `web` devices. | These servers host `admin.phacility.com` and run the Instances application. Instances in the cluster also make calls here to retrieve configuration. Currently, a maximum of one `admin` device is supported, because some details about database credentials are assumed. Minor software changes are required to expand the size of this pool. Device: aux ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | These servers are used by internal processes. In particular, backups are staged here during export. Device: alb ===== | Listening Port | Interface | Notes | |---|---|---| | 80 | Public Internet | Serves `http://admin.phacility.com` | | 443 | Public Internet | Serves `https://admin.phacility.com` | These are ELBs which sit in front of the `admin` pool and terminate SSL for public requests to the web UI. Device: bastion ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Gatekeeper for operational access. | For details on bastion hosts, see [[ phacility_cluster/bastion | Phacility Cluster Bastion]]. Device: db ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 3306 | Internal | Provides MySQL services. | These database servers run MySQL. Device: lb ===== | Listening Port | Interface | Notes | |---|---|---| | 80 | Public Internet | Serves `phacility.com` wildcard DNS. | | 443 | Public Internet | Serves `phacility.com` wildcard DNS. | These are ELBs which sit in front of the `web` pool and terminate SSL for instance requests. Device: nlb ===== | Listening Port | Interface | Notes | |---|---|---| | 443 | Public Internet | Forwards websockets to `notify` hosts. | This does TCP forwarding of websocket requests. Device: notify ===== | Listening Port | Interface | Notes | |---|---|---| | 22280 | Internal | Notification client port. | | 22281 | Internal | Notification server port. | Runs an Aphlict notification server. Device: repo ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Serves Conduit requests from `web` hosts. | | 2223 | Internal | Serves SSH requests from `web` hosts. | These devices store and serve repositories, and run daemons. Device: saux ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves auxiliary requests. | These devices run auxiliary services (primarily, repository automation) for the upstream. Device: sbuild ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves build requests. | These devices run lower-trust build services for the upstream. Device: secure ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Serves upstream repositories. | | 80 | Internal | Serves `secure.phabricator.com`, `javelinjs.com`, `phabricator.org`, `blog.phacility.com`. | | 2222 | Internal | Operational SSH access. | | 22280 | Internal | Aphlict client server. | 22281 | Internal | Aphlict admin server. These devices serve the upstream, `secure.phabricator.com`. Because this service needs to be available to deploy the cluster, devices in this tier are only half-contained in the cluster. The cluster administration tools work, but the tier does not depend on cluster services. The goal is to prevent a cascading failure in the event of a cluster disaster. Normally, you connect to these hosts with `bin/remote ssh secure001` over the VPC. In a disaster scenario, keys with access to the bastion can access these hosts directly with `ssh -p 2222 ubuntu@secure.phabricator.com`. Because this device is not entirely within the cluster, it runs some services via `cron`. You can review them with `crontab -e`. Device: slb ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Balances VCS SSH traffic. | 80 | Public Internet | Balances HTTP for below. | 443 | Public Internet | Balances `secure.phabricator.com`, `javelinjs.com`, `phabricator.org`, `blog.phacility.com`. | 22280 | Public Internet | Balances Aphlict client traffic. Forwards requests to `secure` hosts. Device: vault ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves `vault.phacility.com` over SSH, forwarding requests to the `web` tier. | | 2222 | Internal | Operational SSH access. | These devices are SSH load balancers. They're implemented as normal hosts running HAProxy instead of as ELBs because ELBs can not forward inbound traffic on port 22. Eventually, it would be nice to either merge these into the ELBs (if ELBs support forwarding port 22 in the future) or merge the ELBs into these (if we have some stronger reasoning for running our own load balancers). The cost of having these machines in a separate class is twofold: the raw cost of more moving parts, and we have to serve SSH from `vault.phacility.com` (which we can direct here via DNS) instead of directly from instance domains. If we could merge the HTTP and SSH load balancing, we could accept both HTTP and SSH traffic on instance domains. (We could serve SSH traffic on instance domains on a nonstandard port which the ELBs do support, but this seems worse to me than serving from a dedicated domain. We could also serve from both, so `instance@vault.phacility.com` and `instance@instance.phacility.com:2345` would both work.) Device: web ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Serves HTTP application traffic from `lb` balancers. | | 2223 | Internal | Serves SSH application traffic from `vault` balancers. | These are standard application web servers.

This describes the devices we deploy in the [[ phacility_cluster | Phacility Cluster]], and some related things which aren't technically devices. Device Classes ====== | Device Class | Pool Size | Description | Notes | |---|---|---|---| | `admin` | 1 | Admin Application Server | Serves `admin.phacility.com`. | | `aux` | 1 | Auxiliary Services | Internal utility host. | | `alb` | 1 | Admin Load Balancer | ELB for `admin.phacility.com`. | `bastion` | 1| SSH Bastion Host | Bastion host for ops commands. | | `db` | 6 | Database Server | Runs MySQL. | | `lb` | 1 | Web Load Balancer | ELB for instances. | | `nlb` | 1 | Notification Load Balancer | ELB for websockets. | | `notify` | 1 | Notifications | Aphlict notification server. | | `plb` | 1 | Phurl LB | Serves `phurl.io` requests. | | `repo` | 6 | Repos/Daemons | Terminates VCS HTTP/SSH traffic. | | `saux` | 1 | Secure Auxiliary | Isolated auxiliary for `secure.phabricator.com`. | | `sbuild` | 1 | Secure Build | Isolated builds for `secure.phabricator.com`. | | `secure` | 2 | Secure | Serves the upstream and related services. | | `slb` | 1 | Secure Load Balancer | ELB for `secure.phacility.com`. | `vault` | 1 | SSH Load Balancer | Load balances VCS SSH requests. | | `web` | 2 | Application Server | Terminates normal HTTP traffic. | //Obsolete Device Classes// | Device Class | Description | Notes | | `clb` | Corp Site LB | Served `phacility.com` before CORGI. | | `corp` | Corp Site | Serveed `phacility.com` before CORGI. | Storage Classes ======= | Storage Class | Count | Notes | |---|---|---| | `abak` | 1 | Stores administrative backups. | | `adata` | 1 | Administrative server working storage. | | `auxdata` | 1 | Auxiliary tier working storage. | | `dbak` | 6 | Stores database backups. | | `ddata` | 6 | Database server working storage. | | `rbak` | 6 | Stores repository backups. | | `rdata` | 6 | Repository working storage. | | `rlog` | 6 | Repository host logs. | | `sauxdata` | 1 | Upstream auxiliary working storage. | | `sbak` | 2 | Upstream backup. | | `sdata` | 2 | Upstream database. | | `sbuilddata` | 1 | Upstream build working storage. | | `slog` | 2 | Upstream logs. | | `srepo` | 2 | Upstream repos. | | //`swap`// | - | Temporary swap volume. | Service Classes ======= | Service Class | Count | Notes | |---|---|---| | `auxx` | 1 | Auxiliary cluster service. | | `dbx` | 6 | Database cluster service, may have multiple devices. | | `repox` | 6 | Repository cluster service, may have multiple devices. | | `sauxx` | 1 | Upstream auxiliary service. | | `sbuildx` | 1 | Upstream build service. | | `srepox` | 1 | Upstream repository service. Device: admin ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Accepts requests from `alb` and `web` devices. | These servers host `admin.phacility.com` and run the Instances application. Instances in the cluster also make calls here to retrieve configuration. Currently, a maximum of one `admin` device is supported, because some details about database credentials are assumed. Minor software changes are required to expand the size of this pool. Device: aux ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | These servers are used by internal processes. In particular, backups are staged here during export. Device: alb ===== | Listening Port | Interface | Notes | |---|---|---| | 80 | Public Internet | Serves `http://admin.phacility.com` | | 443 | Public Internet | Serves `https://admin.phacility.com` | These are ELBs which sit in front of the `admin` pool and terminate SSL for public requests to the web UI. Device: bastion ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Gatekeeper for operational access. | For details on bastion hosts, see [[ phacility_cluster/bastion | Phacility Cluster Bastion]]. Device: db ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 3306 | Internal | Provides MySQL services. | These database servers run MySQL. Device: lb ===== | Listening Port | Interface | Notes | |---|---|---| | 80 | Public Internet | Serves `phacility.com` wildcard DNS. | | 443 | Public Internet | Serves `phacility.com` wildcard DNS. | These are ELBs which sit in front of the `web` pool and terminate SSL for instance requests. Device: nlb ===== | Listening Port | Interface | Notes | |---|---|---| | 443 | Public Internet | Forwards websockets to `notify` hosts. | This does TCP forwarding of websocket requests. Device: notify ===== | Listening Port | Interface | Notes | |---|---|---| | 22280 | Internal | Notification client port. | | 22281 | Internal | Notification server port. | Runs an Aphlict notification server. Device: repo ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Serves Conduit requests from `web` hosts. | | 2223 | Internal | Serves SSH requests from `web` hosts. | These devices store and serve repositories, and run daemons. Device: saux ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves auxiliary requests. | These devices run auxiliary services (primarily, repository automation) for the upstream. Device: sbuild ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves build requests. | These devices run lower-trust build services for the upstream. Device: secure ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Serves upstream repositories. | | 80 | Internal | Serves `secure.phabricator.com`, `javelinjs.com`, `phabricator.org`, `blog.phacility.com`. | | 2222 | Internal | Operational SSH access. | | 22280 | Internal | Aphlict client server. | 22281 | Internal | Aphlict admin server. These devices serve the upstream, `secure.phabricator.com`. Because this service needs to be available to deploy the cluster, devices in this tier are only half-contained in the cluster. The cluster administration tools work, but the tier does not depend on cluster services. The goal is to prevent a cascading failure in the event of a cluster disaster. Normally, you connect to these hosts with `bin/remote ssh secure001` over the VPC. In a disaster scenario, keys with access to the bastion can access these hosts directly with `ssh -p 2222 ubuntu@secure.phabricator.com`. Because this device is not entirely within the cluster, it runs some services via `cron`. You can review them with `crontab -e`. Device: slb ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Balances VCS SSH traffic. | 80 | Public Internet | Balances HTTP for below. | 443 | Public Internet | Balances `secure.phabricator.com`, `javelinjs.com`, `phabricator.org`, `blog.phacility.com`. | 22280 | Public Internet | Balances Aphlict client traffic. Forwards requests to `secure` hosts. Device: vault ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves `vault.phacility.com` over SSH, forwarding requests to the `web` tier. | | 2222 | Internal | Operational SSH access. | These devices are SSH load balancers. They're implemented as normal hosts running HAProxy instead of as ELBs because ELBs can not forward inbound traffic on port 22. Eventually, it would be nice to either merge these into the ELBs (if ELBs support forwarding port 22 in the future) or merge the ELBs into these (if we have some stronger reasoning for running our own load balancers). The cost of having these machines in a separate class is twofold: the raw cost of more moving parts, and we have to serve SSH from `vault.phacility.com` (which we can direct here via DNS) instead of directly from instance domains. If we could merge the HTTP and SSH load balancing, we could accept both HTTP and SSH traffic on instance domains. (We could serve SSH traffic on instance domains on a nonstandard port which the ELBs do support, but this seems worse to me than serving from a dedicated domain. We could also serve from both, so `instance@vault.phacility.com` and `instance@instance.phacility.com:2345` would both work.) Device: web ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Serves HTTP application traffic from `lb` balancers. | | 2223 | Internal | Serves SSH application traffic from `vault` balancers. | These are standard application web servers.

This describes the devices we deploy in the [[ phacility_cluster | Phacility Cluster]], and some related things which aren't technically devices. Device Classes ====== | Device Class | Pool Size | Description | Notes | |---|---|---|---| | `admin` | 1 | Admin Application Server | Serves `admin.phacility.com`. | | `aux` | 1 | Auxiliary Services | Internal utility host. | | `alb` | 1 | Admin Load Balancer | ELB for `admin.phacility.com`. | `bastion` | 1| SSH Bastion Host | Bastion host for ops commands. | | `db` | 6 | Database Server | Runs MySQL. | | `lb` | 1 | Web Load Balancer | ELB for instances. | | `nlb` | 1 | Notification Load Balancer | ELB for websockets. | | `notify` | 1 | Notifications | Aphlict notification server. | | `plb` | 1 | Phurl LB | Serves `phurl.io` requests. | | `repo` | 6 | Repos/Daemons | Terminates VCS HTTP/SSH traffic. | | `saux` | 1 | Secure Auxiliary | Isolated auxiliary for `secure.phabricator.com`. | | `sbuild` | 1 | Secure Build | Isolated builds for `secure.phabricator.com`. | | `secure` | 2 | Secure | Serves the upstream and related services. | | `slb` | 1 | Secure Load Balancer | ELB for `secure.phacility.com`. | `vault` | 1 | SSH Load Balancer | Load balances VCS SSH requests. | | `web` | 2 | Application Server | Terminates normal HTTP traffic. | //Obsolete Device Classes// | Device Class | Description | Notes | | `clb` | Corp Site LB | Served `phacility.com` before CORGI. | | `corp` | Corp Site | Serveed `phacility.com` before CORGI. | Storage Classes ======= | Storage Class | Count | Notes | |---|---|---| | `abak` | 1 | Stores administrative backups. | | `adata` | 1 | Administrative server working storage. | | `auxdata` | 1 | Auxiliary tier working storage. | | `dbak` | 6 | Stores database backups. | | `ddata` | 6 | Database server working storage. | | `rbak` | 6 | Stores repository backups. | | `rdata` | 6 | Repository working storage. | | `rlog` | 6 | Repository host logs. | | `sauxdata` | 1 | Upstream auxiliary working storage. | | `sbak` | 2 | Upstream backup. | | `sdata` | 2 | Upstream database. | | `sbuilddata` | 1 | Upstream build working storage. | | `slog` | 2 | Upstream logs. | | `srepo` | 2 | Upstream repos. | | //`swap`// | - | Temporary swap volume. | Service Classes ======= | Service Class | Count | Notes | |---|---|---| | `auxx` | 1 | Auxiliary cluster service. | | `dbx` | 56 | Database cluster service, may have multiple devices. | | `repox` | 56 | Repository cluster service, may have multiple devices. | | `sauxx` | 1 | Upstream auxiliary service. | | `sbuildx` | 1 | Upstream build service. | | `srepox` | 1 | Upstream repository service. Device: admin ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Accepts requests from `alb` and `web` devices. | These servers host `admin.phacility.com` and run the Instances application. Instances in the cluster also make calls here to retrieve configuration. Currently, a maximum of one `admin` device is supported, because some details about database credentials are assumed. Minor software changes are required to expand the size of this pool. Device: aux ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | These servers are used by internal processes. In particular, backups are staged here during export. Device: alb ===== | Listening Port | Interface | Notes | |---|---|---| | 80 | Public Internet | Serves `http://admin.phacility.com` | | 443 | Public Internet | Serves `https://admin.phacility.com` | These are ELBs which sit in front of the `admin` pool and terminate SSL for public requests to the web UI. Device: bastion ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Gatekeeper for operational access. | For details on bastion hosts, see [[ phacility_cluster/bastion | Phacility Cluster Bastion]]. Device: db ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 3306 | Internal | Provides MySQL services. | These database servers run MySQL. Device: lb ===== | Listening Port | Interface | Notes | |---|---|---| | 80 | Public Internet | Serves `phacility.com` wildcard DNS. | | 443 | Public Internet | Serves `phacility.com` wildcard DNS. | These are ELBs which sit in front of the `web` pool and terminate SSL for instance requests. Device: nlb ===== | Listening Port | Interface | Notes | |---|---|---| | 443 | Public Internet | Forwards websockets to `notify` hosts. | This does TCP forwarding of websocket requests. Device: notify ===== | Listening Port | Interface | Notes | |---|---|---| | 22280 | Internal | Notification client port. | | 22281 | Internal | Notification server port. | Runs an Aphlict notification server. Device: repo ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Serves Conduit requests from `web` hosts. | | 2223 | Internal | Serves SSH requests from `web` hosts. | These devices store and serve repositories, and run daemons. Device: saux ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves auxiliary requests. | These devices run auxiliary services (primarily, repository automation) for the upstream. Device: sbuild ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves build requests. | These devices run lower-trust build services for the upstream. Device: secure ====== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Serves upstream repositories. | | 80 | Internal | Serves `secure.phabricator.com`, `javelinjs.com`, `phabricator.org`, `blog.phacility.com`. | | 2222 | Internal | Operational SSH access. | | 22280 | Internal | Aphlict client server. | 22281 | Internal | Aphlict admin server. These devices serve the upstream, `secure.phabricator.com`. Because this service needs to be available to deploy the cluster, devices in this tier are only half-contained in the cluster. The cluster administration tools work, but the tier does not depend on cluster services. The goal is to prevent a cascading failure in the event of a cluster disaster. Normally, you connect to these hosts with `bin/remote ssh secure001` over the VPC. In a disaster scenario, keys with access to the bastion can access these hosts directly with `ssh -p 2222 ubuntu@secure.phabricator.com`. Because this device is not entirely within the cluster, it runs some services via `cron`. You can review them with `crontab -e`. Device: slb ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Balances VCS SSH traffic. | 80 | Public Internet | Balances HTTP for below. | 443 | Public Internet | Balances `secure.phabricator.com`, `javelinjs.com`, `phabricator.org`, `blog.phacility.com`. | 22280 | Public Internet | Balances Aphlict client traffic. Forwards requests to `secure` hosts. Device: vault ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Public Internet | Serves `vault.phacility.com` over SSH, forwarding requests to the `web` tier. | | 2222 | Internal | Operational SSH access. | These devices are SSH load balancers. They're implemented as normal hosts running HAProxy instead of as ELBs because ELBs can not forward inbound traffic on port 22. Eventually, it would be nice to either merge these into the ELBs (if ELBs support forwarding port 22 in the future) or merge the ELBs into these (if we have some stronger reasoning for running our own load balancers). The cost of having these machines in a separate class is twofold: the raw cost of more moving parts, and we have to serve SSH from `vault.phacility.com` (which we can direct here via DNS) instead of directly from instance domains. If we could merge the HTTP and SSH load balancing, we could accept both HTTP and SSH traffic on instance domains. (We could serve SSH traffic on instance domains on a nonstandard port which the ELBs do support, but this seems worse to me than serving from a dedicated domain. We could also serve from both, so `instance@vault.phacility.com` and `instance@instance.phacility.com:2345` would both work.) Device: web ===== | Listening Port | Interface | Notes | |---|---|---| | 22 | Internal | Operational SSH access. | | 80 | Internal | Serves HTTP application traffic from `lb` balancers. | | 2223 | Internal | Serves SSH application traffic from `vault` balancers. | These are standard application web servers.