diff --git a/src/docs/user/cluster/cluster.diviner b/src/docs/user/cluster/cluster.diviner index 0180c7ff42..d78c445c4a 100644 --- a/src/docs/user/cluster/cluster.diviner +++ b/src/docs/user/cluster/cluster.diviner @@ -1,289 +1,326 @@ @title Clustering Introduction @group cluster Guide to configuring Phabricator across multiple hosts for availability and performance. + Overview ======== -WARNING: This feature is a very early prototype; the features this document -describes are mostly speculative fantasy. +WARNING: This feature is a prototype. Installs should expect a challening +adventure when deploying clusters. In the best of times, configuring a +cluster is complex and requires significant operations experience. Phabricator can be configured to run on multiple hosts with redundant services to improve its availability and scalability, and make disaster recovery much easier. Clustering is more complex to setup and maintain than running everything on a single host, but greatly reduces the cost of recovering from hardware and network failures. Each Phabricator service has an array of clustering options that can be -configured independently. Configuring a cluster is inherently complex, and this -is an advanced feature aimed at installs with large userbases and experienced -operations personnel who need this high degree of flexibility. +configured somewhat independently. Configuring a cluster is inherently complex, +and this is an advanced feature aimed at installs with large userbases and +experienced operations personnel who need this high degree of flexibility. The remainder of this document summarizes how to add redundancy to each service and where your efforts are likely to have the greatest impact. For additional guidance on setting up a cluster, see "Overlaying Services" and "Cluster Recipes" at the bottom of this document. +Clusterable Services +==================== + +This table provides an overview of clusterable services, their setup +complexity, and the rough impact that converting them to run on multiple hosts +will have on availability, resistance to data loss, and scalability. + +| Service | Setup | Availability | Loss Resistance | Scalability +|---------|-------|--------------|-----------|------------ +| **Databases** | Moderate | **High** | **High** | Low +| **Repositories** | Complex | Moderate | **High** | Moderate +| **Daemons** | Minimal | Low | No Risk | Low +| **SSH Servers** | Minimal | Low | No Risk | Low +| **Web Servers** | Minimal | **High** | No Risk | Moderate +| **Notifications** | Minimal | Low | No Risk | Low + +See below for a walkthrough of these services in greater detail. + + Preparing for Clustering ======================== To begin deploying Phabricator in cluster mode, set up `cluster.addresses` in your configuration. This option should contain a list of network address blocks which are considered to be part of the cluster. Hosts in this list are allowed to bend (or even break) some of the security and policy rules when they make requests to other hosts in the cluster, so this list should be as small as possible. See "Cluster Whitelist Security" below for discussion. If you are deploying hardware in EC2, a reasonable approach is to launch a dedicated Phabricator VPC, whitelist the whole VPC as a Phabricator cluster, and then deploy only Phabricator services into that VPC. If you have additional auxiliary hosts which run builds and tests via Drydock, you should //not// include them in the cluster address definition. For more detailed discussion of the Drydock security model, see @{Drydock User Guide: Security}. Most other clustering features will not work until you define a cluster by configuring `cluster.addresses`. Cluster Whitelist Security ======================== When you configure `cluster.addresses`, you should keep the list of trusted cluster hosts as small as possible. Hosts on this list gain additional capabilities, including these: **Trusted HTTP Headers**: Normally, Phabricator distrusts the load balancer HTTP headers `X-Forwarded-For` and `X-Forwarded-Proto` because they may be client-controlled and can be set to arbitrary values by an attacker if no load balancer is deployed. In particular, clients can set `X-Forwarded-For` to any value and spoof traffic from arbitrary remotes. These headers are trusted when they are received from a host on the cluster address whitelist. This allows requests from cluster loadbalancers to be interpreted correctly by default without requiring additional custom code or configuration. **Intracluster HTTP**: Requests from cluster hosts are not required to use HTTPS, even if `security.require-https` is enabled, because it is common to terminate HTTPS on load balancers and use plain HTTP for requests within a cluster. **Special Authentication Mechanisms**: Cluster hosts are allowed to connect to other cluster hosts with "root credentials", and to impersonate any user account. The use of root credentials is required because the daemons must be able to bypass policies in order to function properly: they need to send mail about private conversations and import commits in private repositories. The ability to impersonate users is required because SSH nodes must receive, interpret, modify, and forward SSH traffic. They can not use the original credentials to do this because SSH authentication is asymmetric and they do not have the user's private key. Instead, they use root credentials and impersonate the user within the cluster. These mechanisms are still authenticated (and use asymmetric keys, like SSH does), so access to a host in the cluster address block does not mean that an attacker can immediately compromise the cluster. However, an over-broad cluster address whitelist may give an attacker who gains some access additional tools to escalate access. Note that if an attacker gains access to an actual cluster host, these extra powers are largely moot. Most cluster hosts must be able to connect to the master database to function properly, so the attacker will just do that and freely read or modify whatever data they want. Cluster: Databases ================= Configuring multiple database hosts is moderately complex, but normally has the highest impact on availability and resistance to data loss. This is usually the most important service to make redundant if your focus is on availability and disaster recovery. Configuring replicas allows Phabricator to run in read-only mode if you lose the master and to quickly promote the replica as a replacement. For details, see @{article:Cluster: Databases}. Cluster: Repositories ===================== Configuring multiple repository hosts is complex, but is required before you can add multiple daemon or web hosts. Repository replicas are important for availability if you host repositories on Phabricator, but less important if you host repositories elsewhere (instead, you should focus on making that service more available). The distributed nature of Git and Mercurial tend to mean that they are naturally somewhat resistant to data loss: every clone of a repository includes the entire history. Repositories may become a scalability bottleneck, although this is rare unless your install has an unusually heavy repository read volume. Slow clones/fetches may hint at a repository capacity problem. Adding more repository hosts will provide an approximately linear increase in capacity. For details, see @{article:Cluster: Repositories}. Cluster: Daemons ================ Configuring multiple daemon hosts is straightforward, but you must configure repositories first. -With daemons running on multiple hosts, you can transparently survive the loss +With daemons running on multiple hosts you can transparently survive the loss of any subset of hosts without an interruption to daemon services, as long as at least one host remains alive. Daemons are stateless, so spreading daemons across multiple hosts provides no resistance to data loss. Daemons can become a bottleneck, particularly if your install sees a large volume of write traffic to repositories. If the daemon task queue has a backlog, that hints at a capacity problem. If existing hosts have unused resources, increase `phd.taskmasters` until they are fully utilized. From there, adding more daemon hosts will provide an approximately linear increase in capacity. For details, see @{article:Cluster: Daemons}. +Cluster: SSH Servers +==================== + +Configuring multiple SSH hosts is straightforward, but you must configure +repositories first. + +With multiple SSH hosts you can transparently survive the loss of any subset +of hosts without interruption to repository services, as long as at last one +host remains alive. SSH services are stateless, so putting multiple hosts in +service provides no resistance to data loss because no data is at risk. + +SSH hosts are very rarely a scalability bottleneck. + +For details, see @{article:Cluster: SSH Servers}. + + Cluster: Web Servers ==================== Configuring multiple web hosts is straightforward, but you must configure repositories first. -With multiple web hosts, you can transparently survive the loss of any subset -of hosts as long as at least one host remains alive. Web hosts are stateless, +With multiple web hosts you can transparently survive the loss of any subset +of hosts as long as at least one host remains alive. Web services are stateless, so putting multiple hosts in service provides no resistance to data loss because no data is at risk. Web hosts can become a bottleneck, particularly if you have a workload that is heavily focused on reads from the web UI (like a public install with many anonymous users). Slow responses to web requests may hint at a web capacity problem. Adding more hosts will provide an approximately linear increase in capacity. For details, see @{article:Cluster: Web Servers}. Cluster: Notifications ====================== Configuring multiple notification hosts is simple and has no pre-requisites. With multiple notification hosts, you can survive the loss of any subset of hosts as long as at least one host remains alive. Service may be breifly disrupted directly after the incident which destroys the other hosts. Notifications are noncritical, so this normally has little practical impact on service availability. Notifications are also stateless, so clustering this service provides no resistance to data loss because no data is at risk. Notification delivery normally requires very few resources, so adding more hosts is unlikely to have much impact on scalability. For details, see @{article:Cluster: Notifications}. Overlaying Services =================== Although hosts can run a single dedicated service type, certain groups of services work well together. Phabricator clusters usually do not need to be very large, so deploying a small number of hosts with multiple services is a good place to start. In planning a cluster, consider these blended host types: **Everything**: Run HTTP, SSH, MySQL, notifications, repositories and daemons on a single host. This is the starting point for single-node setups, and usually also the best configuration when adding the second node. **Everything Except Databases**: Run HTTP, SSH, notifications, repositories and daemons on one host, and MySQL on a different host. MySQL uses many of the same resources that other services use. It's also simpler to separate than other services, and tends to benefit the most from dedicated hardware. **Repositories and Daemons**: Run repositories and daemons on the same host. Repository hosts //must// run daemons, and it normally makes sense to completely overlay repositories and daemons. These services tend to use different resources (repositories are heavier on I/O and lighter on CPU/RAM; daemons are heavier on CPU/RAM and lighter on I/O). Repositories and daemons are also both less latency sensitive than other service types, so there's a wider margin of error for under provisioning them before performance is noticeably affected. These nodes tend to use system resources in a balanced way. Individual nodes in this class do not need to be particularly powerful. **Frontend Servers**: Run HTTP and SSH on the same host. These are easy to set up, stateless, and you can scale the pool up or down easily to meet demand. Routing both types of ingress traffic through the same initial tier can simplify load balancing. These nodes tend to need relatively little RAM. Cluster Recipes =============== This section provides some guidance on reasonable ways to scale up a cluster. The smallest possible cluster is **two hosts**. Run everything (web, ssh, database, notifications, repositories, and daemons) on each host. One host will serve as the master; the other will serve as a replica. Ideally, you should physically separate these hosts to reduce the chance that a natural disaster or infrastructure disruption could disable or destroy both hosts at the same time. From here, you can choose how you expand the cluster. To improve **scalability and performance**, separate loaded services onto dedicated hosts and then add more hosts of that type to increase capacity. If you have a two-node cluster, the best way to improve scalability by adding one host is likely to separate the master database onto its own host. Note that increasing scale may //decrease// availability by leaving you with too little capacity after a failure. If you have three hosts handling traffic and one datacenter fails, too much traffic may be sent to the single remaining host in the surviving datacenter. You can hedge against this by mirroring new hosts in other datacenters (for example, also separate the replica database onto its own host). After separating databases, separating repository + daemon nodes is likely the next step to consider. To improve **availability**, add another copy of everything you run in one datacenter to a new datacenter. For example, if you have a two-node cluster, the best way to improve availability is to run everything on a third host in a third datacenter. If you have a 6-node cluster with a web node, a database node and a repo + daemon node in two datacenters, add 3 more nodes to create a copy of each node in a third datacenter. You can continue adding hosts until you run out of hosts. Next Steps ========== Continue by: - learning how Phacility configures and operates a large, multi-tenant production cluster in ((cluster)). diff --git a/src/docs/user/cluster/cluster_daemons.diviner b/src/docs/user/cluster/cluster_daemons.diviner index 61cedd464d..8cde3e7b7d 100644 --- a/src/docs/user/cluster/cluster_daemons.diviner +++ b/src/docs/user/cluster/cluster_daemons.diviner @@ -1,59 +1,57 @@ @title Cluster: Daemons @group cluster Configuring Phabricator to use multiple daemon hosts. Overview ======== -WARNING: This feature is a very early prototype; the features this document -describes are mostly speculative fantasy. - You can run daemons on multiple hosts. The advantages of doing this are: - you can completely survive the loss of multiple daemon hosts; and - worker queue throughput may improve. This configuration is simple, but you must configure repositories first. For details, see @{article:Cluster: Repositories}. Since repository hosts must run daemons anyway, you usually do not need to do -any additional work and can skip this entirely. +any additional work and can skip this entirely if you have already configured +multiple repository hosts. Adding Daemon Hosts =================== After configuring repositories for clustering, launch daemons on every repository host according to the documentation in @{article:Cluster: Repositories}. These daemons are necessary: repositories will not fetch, update, or synchronize properly without them. If your repository clustering is redundant (you have at least two repository hosts), these daemons are also likely to be sufficient in most cases. If you want to launch additional hosts anyway (for example, to increase queue capacity for unusual workloads), see "Dedicated Daemon Hosts" below. Dedicated Daemon Hosts ====================== You can launch additional daemon hosts without any special configuration. Daemon hosts must be able to reach other hosts on the network, but do not need to run any services (like HTTP or SSH). Simply deploy the Phabricator software and configuration and start the daemons. Normally, there is little reason to deploy dedicated daemon hosts. They can improve queue capacity, but generally do not improve availability or increase resistance to data loss on their own. Instead, consider deploying more repository hosts: repository hosts run daemons, so this will increase queue capacity but also improve repository availability and cluster resistance. Next Steps ========== Continue by: - returning to @{article:Clustering Introduction}; or - configuring repositories first with @{article:Cluster: Repositories}. diff --git a/src/docs/user/cluster/cluster_devices.diviner b/src/docs/user/cluster/cluster_devices.diviner index 197620e461..57729fdb86 100644 --- a/src/docs/user/cluster/cluster_devices.diviner +++ b/src/docs/user/cluster/cluster_devices.diviner @@ -1,247 +1,247 @@ @title Cluster: Devices @group cluster Guide to configuring hosts to act as cluster devices. Cluster Context =============== This document describes a step in configuring Phabricator to run on multiple hosts in a cluster configuration. This is an advanced feature. For more information on clustering, see @{article:Clustering Introduction}. In this context, device configuration is mostly relevant to configuring repository services in a cluster. You can find more details about this in @{article:Cluster: Repositories}. Overview ======== Some cluster services need to be able to authenticate themselves and interact with other services. For example, two repository hosts holding copies of the same repository must be able to fetch changes from one another, even if the repository is private. Within a cluster, devices authenticate using SSH keys. Some operations happen over SSH (using keys in a normal way, as you would when running `ssh` from the command line), while others happen over HTTP (using SSH keys to sign requests). Before hosts can authenticate to one another, you need to configure the credentials so other devices know the keys can be trusted. Beyond establishing trust, this configuration will establish //device identity//, so each host knows which device it is explicitly. Today, this is primarily necessary when configuring repository clusters. Using Almanac ============= The tool Phabricator uses to manage cluster devices is the **Almanac** application, and most configuration will occur through the application's web UI. If you are not familiar with it, see @{article:Almanac User Guide} first. This document assumes you are familiar with Almanac concepts. What Lies Ahead =============== Here's a brief overview of the steps required to register cluster devices. The remainder of this document walks through these points in more detail. - Create an Almanac device record for each device. - Generate, add, and trust SSH keys if necessary. - Install Phabricator on the host. - Use `bin/almanac register` from the host to register it as a device. See below for guidance on each of these steps. Individual vs Shared Keys ========================= Before getting started, you should choose how you plan to manage device SSH keys. Trust and device identity are handled separately, and there are two ways to set up SSH keys so that devices can authenticate with one another: - you can generate a unique SSH key for each device; or - you can generate one SSH key and share it across multiple devices. Using **unique keys** allows the tools to do some more sanity/safety checks and makes it a bit more difficult to misconfigure things, but you'll have to do more work managing the actual keys. This may be a better choice if you are setting up a small cluster (2-3 devices) for the first time. Using **shared keys** makes key management easier but safety checks won't be able to catch a few kinds of mistakes. This may be a better choice if you are setting up a larger cluster, plan to expand the cluster later, or have experience with Phabricator clustering. Because all cluster keys are all-powerful, there is no material difference between these methods from a security or trust viewpoint. Unique keys are just potentially easier to administrate at small scales, while shared keys are easier at larger scales. Create Almanac Device Records ============================= For each host you plan to make part of a Phabricator cluster, go to the {nav Almanac} application and create a **device** record. For guidance on this application, see @{article:Almanac User Guide}. Add **interfaces** to each device record so Phabricator can tell how to connect to these hosts. Normally, you'll add one HTTP interface (usually on port 80) and one SSH interface (by default, on port 2222) to each device: For example, if you are building a two-host repository cluster, you may end up with records that look like these: - Device: `repo001.mycompany.net` - Interface: `123.0.0.1:2222` - Interface: `123.0.0.1:80` - - Device: `repo002.mycopmany.net` + - Device: `repo002.mycompany.net` - Interface: `123.0.0.2:2222` - Interface: `123.0.0.2:80` Note that these hosts will normally run two `sshd` ports: the standard `sshd` which you connect to to operate and administrate the host, and the special Phabricator `sshd` that you connect to to clone and push repositories. You should specify the Phabricator `sshd` port, **not** the standard `sshd` port. If you're using **unique** SSH keys for each device, continue to the next step. If you're using **shared** SSH keys, create a third device with no interfaces, like `keywarden.mycompany.net`. This device will just be used as a container to hold the trusted SSH key and is not a real device. NOTE: Do **not** create a **service** record yet. Today, service records become active immediately once they are created, and you haven't set things up yet. Generate and Trust SSH Keys =========================== Next, you need to generate or upload SSH keys and mark them as trusted. Marking a key as trusted gives it tremendous power. If you're using **unique** SSH keys, upload or generate a key for each individual device from the device detail screen in the Almanac web UI. Save the private keys for the next step. If you're using a **shared** SSH key, upload or generate a single key for the keywarden device from the device detail screen in the Almanac web UI. Save the private key for the next step. Regardless of how many keys you generated, take the key IDs from the tables in the web UI and run this command from the command line for each key, to mark each key as trusted: ``` phabricator/ $ ./bin/almanac trust-key --id phabricator/ $ ./bin/almanac trust-key --id ... ``` The warnings this command emits are serious. The private keys are now trusted, and allow any user or device possessing them to sign requests that bypass policy checks without requiring additional credentials. Guard them carefully! If you need to revoke trust for a key later, use `untrust-key`: ``` phabricator/ $ ./bin/almanac untrust-key --id ``` Once the keys are trusted, continue to the next step. Install Phabricator =================== If you haven't already, install Phabricator on each device you plan to enroll in the cluster. Cluster repository devices must provide services over both HTTP and SSH, so you need to install and configure both a webserver and a Phabricator `sshd` on these hosts. Generally, you will follow whatever process you otherwise use when installing Phabricator. NOTE: Do not start the daemons on the new devices yet. They won't work properly until you've finished configuring things. Once Phabricator is installed, you can enroll the devices in the cluster by registering them. Register Devices ================ To register a host as an Almanac device, use `bin/almanac register`. If you are using **unique** keys, run it like this: ``` $ ./bin/almanac register \ --device \ --private-key ``` For example, you might run this command on `repo001` when using unique keys: ``` $ ./bin/almanac register \ --device repo001.mycompany.net \ --private-key /path/to/private.key ``` If you are using a **shared** key, this will be a little more complicated because you need to override some checks that are intended to prevent mistakes. Use the `--identify-as` flag to choose a device identity: ``` $ ./bin/almanac register \ --device \ --private-key \ --identify-as ``` For example, you might run this command on `repo001` when using a shared key: ``` $ ./bin/almanac register --device keywarden.mycompany.net \ --private-key /path/to/private-key \ --identify-as repo001.mycompany.net ``` In particular, note that `--device` is always the **trusted** device associated with the trusted key. The `--identify-as` flag allows several different hosts to share the same key but still identify as different devices. The overall effect of the `bin/almanac` command is to copy identity and key files into `phabricator/conf/keys/`. You can inspect the results by examining that directory. The helper script just catches potential mistakes and makes sure the process is completed correctly. Note that a copy of the active private key is stored in the `conf/keys/` directory permanently. When converting a host into a cluster host, you may need to revisit @{article:Diffusion User Guide: Repository Hosting} and double check the `sudo` permission for the host. In particular, cluster hosts need to be able to run `ssh` via `sudo` so they can read the device private key. Next Steps ========== Now that devices are registered, you can build cluster services from them. Return to the relevant cluster service documentation to continue: - build repository clusters with @{article:Cluster: Repositories}; - return to @{article:Clustering Introduction}; or - review the Almanac application with @{article:Almanac User Guide}. diff --git a/src/docs/user/cluster/cluster_notifications.diviner b/src/docs/user/cluster/cluster_notifications.diviner index 3dd4e9d903..3cdeec3c39 100644 --- a/src/docs/user/cluster/cluster_notifications.diviner +++ b/src/docs/user/cluster/cluster_notifications.diviner @@ -1,174 +1,171 @@ @title Cluster: Notifications @group cluster Configuring Phabricator to use multiple notification servers. Overview ======== -WARNING: This feature is a very early prototype; the features this document -describes are mostly speculative fantasy. - You can run multiple notification servers. The advantages of doing this are: - you can completely survive the loss of any subset so long as one remains standing; and - performance and capacity may improve. This configuration is relatively simple, but has a small impact on availability and does nothing to increase resitance to data loss. Clustering Design Goals ======================= Notification clustering aims to restore service automatically after the loss of some nodes. It does **not** attempt to guarantee that every message is delivered. Notification messages provide timely information about events, but they are never authoritative and never the only way for users to learn about events. For example, if a notification about a task update is not delivered, the next page you load will still show the notification in your notification menu. Generally, Phabricator works fine without notifications configured at all, so clustering assumes that losing some messages during a disruption is acceptable. How Clustering Works ==================== Notification clustering is very simple: notification servers relay every message they receive to a list of peers. When you configure clustering, you'll run multiple servers and tell them that the other servers exist. When any server receives a message, it retransmits it to all the severs it knows about. When a server is lost, clients will automatically reconnect after a brief delay. They may lose some notifications while their client is reconnecting, but normally this should only last for a few seconds. Configuring Aphlict =================== To configure clustering on the server side, add a `cluster` key to your Aphlict configuration file. For more details about configuring Aphlict, see @{article:Notifications User Guide: Setup and Configuration}. The `cluster` key should contain a list of `"admin"` server locations. Every message the server receives will be retransmitted to all nodes in the list. The server is smart enough to avoid sending messages in a cycle, and to avoid sending messages to itself. You can safely list every server you run in the configuration file, including the current server. You do not need to configure servers in an acyclic graph or only list //other// servers: just list everything on every server and Aphlict will figure things out from there. A simple example with two servers might look like this: ```lang=json, name="aphlict.json (Cluster)" { ... "cluster": [ { "host": "notify001.mycompany.com", "port": 22281, "protocol": "http" }, { "host": "notify002.mycompany.com", "port": 22281, "protocol": "http" } ] ... } ``` Configuring Phabricator ======================= To configure clustering on the client side, add every service you run to `notification.servers`. Generally, this will be twice as many entries as you run actual servers, since each server runs a `"client"` service and an `"admin"` service. A simple example with the two servers above (providing four total services) might look like this: ```lang=json, name="notification.servers (Cluster)" [ { "type": "client", "host": "notify001.mycompany.com", "port": 22280, "protocol": "https" }, { "type": "client", "host": "notify002.mycompany.com", "port": 22280, "protocol": "https" }, { "type": "admin", "host": "notify001.mycompany.com", "port": 22281, "protocol": "http" }, { "type": "admin", "host": "notify002.mycompany.com", "port": 22281, "protocol": "http" } ] ``` If you put all of the `"client"` servers behind a load balancer, you would just list the load balancer and let it handle pulling nodes in and out of service. ```lang=json, name="notification.servers (Cluster + Load Balancer)" [ { "type": "client", "host": "notify-lb.mycompany.com", "port": 22280, "protocol": "https" }, { "type": "admin", "host": "notify001.mycompany.com", "port": 22281, "protocol": "http" }, { "type": "admin", "host": "notify002.mycompany.com", "port": 22281, "protocol": "http" } ] ``` Notification hosts do not need to run any additional services, although they are free to do so. The notification server generally consumes few resources and is resistant to most other loads on the machine, so it's reasonable to overlay these on top of other services wherever it is convenient. Next Steps ========== Continue by: - reviewing notification configuration with @{article:Notifications User Guide: Setup and Configuration}; or - returning to @{article:Clustering Introduction}. diff --git a/src/docs/user/cluster/cluster_repositories.diviner b/src/docs/user/cluster/cluster_repositories.diviner index 35508b0813..07203f03eb 100644 --- a/src/docs/user/cluster/cluster_repositories.diviner +++ b/src/docs/user/cluster/cluster_repositories.diviner @@ -1,492 +1,487 @@ @title Cluster: Repositories @group cluster Configuring Phabricator to use multiple repository hosts. Overview ======== -WARNING: This feature is a very early prototype; the features this document -describes are mostly speculative fantasy. - If you use Git, you can deploy Phabricator with multiple repository hosts, configured so that each host is readable and writable. The advantages of doing this are: - you can completely survive the loss of repository hosts; - reads and writes can scale across multiple machines; and - read and write performance across multiple geographic regions may improve. This configuration is complex, and many installs do not need to pursue it. This configuration is not currently supported with Subversion or Mercurial. How Reads and Writes Work ========================= Phabricator repository replicas are multi-master: every node is readable and writable, and a cluster of nodes can (almost always) survive the loss of any arbitrary subset of nodes so long as at least one node is still alive. Phabricator maintains an internal version for each repository, and increments it when the repository is mutated. Before responding to a read, replicas make sure their version of the repository is up to date (no node in the cluster has a newer version of the repository). If it isn't, they block the read until they can complete a fetch. Before responding to a write, replicas obtain a global lock, perform the same version check and fetch if necessary, then allow the write to continue. Additionally, repositories passively check other nodes for updates and replicate changes in the background. After you push a change to a repositroy, it will usually spread passively to all other repository nodes within a few minutes. Even if passive replication is slow, the active replication makes acknowledged changes sequential to all observers: after a write is acknowledged, all subsequent reads are guaranteed to see it. The system does not permit stale reads, and you do not need to wait for a replication delay to see a consistent view of the repository no matter which node you ask. HTTP vs HTTPS ============= Intracluster requests (from the daemons to repository servers, or from webservers to repository servers) are permitted to use HTTP, even if you have set `security.require-https` in your configuration. It is common to terminate SSL at a load balancer and use plain HTTP beyond that, and the `security.require-https` feature is primarily focused on making client browser behavior more convenient for users, so it does not apply to intracluster traffic. Using HTTP within the cluster leaves you vulnerable to attackers who can observe traffic within a datacenter, or observe traffic between datacenters. This is normally very difficult, but within reach for state-level adversaries like the NSA. If you are concerned about these attackers, you can terminate HTTPS on repository hosts and bind to them with the "https" protocol. Just be aware that the `security.require-https` setting won't prevent you from making configuration mistakes, as it doesn't cover intracluster traffic. Other mitigations are possible, but securing a network against the NSA and similar agents of other rogue nations is beyond the scope of this document. Repository Hosts ================ Repository hosts must run a complete, fully configured copy of Phabricator, including a webserver. They must also run a properly configured `sshd`. If you are converting existing hosts into cluster hosts, you may need to revisit @{article:Diffusion User Guide: Repository Hosting} and make sure the system user accounts have all the necessary `sudo` permissions. In particular, cluster devices need `sudo` access to `ssh` so they can read device keys. Generally, these hosts will run the same set of services and configuration that web hosts run. If you prefer, you can overlay these services and put web and repository services on the same hosts. See @{article:Clustering Introduction} for some guidance on overlaying services. When a user requests information about a repository that can only be satisfied by examining a repository working copy, the webserver receiving the request will make an HTTP service call to a repository server which hosts the repository to retrieve the data it needs. It will use the result of this query to respond to the user. Setting up a Cluster Services ============================= To set up clustering, first register the devices that you want to use as part of the cluster with Almanac. For details, see @{article:Cluster: Devices}. NOTE: Once you create a service, new repositories will immediately allocate on it. You may want to disable repository creation during initial setup. Once the hosts are registered as devices, you can create a new service in Almanac: - First, register at least one device according to the device clustering instructions. - Create a new service of type **Phabricator Cluster: Repository** in Almanac. - Bind this service to all the interfaces on the device or devices. - For each binding, add a `protocol` key with one of these values: `ssh`, `http`, `https`. For example, a service might look like this: - Service: `repos001.mycompany.net` - Binding: `repo001.mycompany.net:80`, `protocol=http` - Binding: `repo001.mycompany.net:2222`, `protocol=ssh` The service itself has a `closed` property. You can set this to `true` to disable new repository allocations on this service (for example, if it is reaching capacity). Migrating to Clustered Services =============================== To convert existing repositories on an install into cluster repositories, you will generally perform these steps: - Register the existing host as a cluster device. - Configure a single host repository service using //only// that host. This puts you in a transitional state where repositories on the host can work as either on-host repositories or cluster repositories. You can move forward from here slowly and make sure services still work, with a quick path back to safety if you run into trouble. To move forward, migrate one repository to the service and make sure things work correctly. If you run into issues, you can back out by migrating the repository off the service. To migrate a repository onto a cluster service, use this command: ``` $ ./bin/repository clusterize --service ``` To migrate a repository back off a service, use this command: ``` $ ./bin/repoistory clusterize --remove-service ``` This command only changes how Phabricator connects to the repository; it does not move any data or make any complex structural changes. When Phabricator needs information about a non-clustered repository, it just runs a command like `git log` directly on disk. When Phabricator needs information about a clustered repository, it instead makes a service call to another server, asking that server to run `git log` instead. In a single-host cluster the server will make this service call to itself, so nothing will really change. But this //is// an effective test for most possible configuration mistakes. If your canary repository works well, you can migrate the rest of your repositories when ready (you can use `bin/repository list` to quickly get a list of all repository monograms). Once all repositories are migrated, you've reached a stable state and can remain here as long as you want. This state is sufficient to convert daemons, SSH, and web services into clustered versions and spread them across multiple machines if those goals are more interesting. Obviously, your single-device "cluster" will not be able to survive the loss of the single repository host, but you can take as long as you want to expand the cluster and add redundancy. After creating a service, you do not need to `clusterize` new repositories: they will automatically allocate onto an open service. When you're ready to expand the cluster, continue below. Expanding a Cluster =================== To expand an existing cluster, follow these general steps: - Register new devices in Almanac. - Add bindings to the new devices to the repository service, also in Almanac. - Start the daemons on the new devices. For instructions on configuring and registering devices, see @{article:Cluster: Devices}. As soon as you add active bindings to a service, Phabricator will begin synchronizing repositories and sending traffic to the new device. You do not need to copy any repository data to the device: Phabricator will automatically synchronize it. If you have a large amount of repository data, you may want to help this process along by copying the repository directory from an existing cluster device before bringing the new host online. This is optional, but can reduce the amount of time required to fully synchronize the cluster. You do not need to synchronize the most up-to-date data or stop writes during this process. For example, loading the most recent backup snapshot onto the new device will substantially reduce the amount of data that needs to be synchronized. Contracting a Cluster ===================== To reduce the size of an existing cluster, follow these general steps: - Disable the bindings from the service to the dead device in Almanac. If you are removing a device because it failed abruptly (or removing several devices at once) it is possible that some repositories will have lost all their leaders. See "Loss of Leaders" below to understand and resolve this. Monitoring Services =================== You can get an overview of repository cluster status from the {nav Config > Repository Servers} screen. This table shows a high-level overview of all active repository services. **Repos**: The number of repositories hosted on this service. **Sync**: Synchronization status of repositories on this service. This is an at-a-glance view of service health, and can show these values: - **Synchronized**: All nodes are fully synchronized and have the latest version of all repositories. - **Partial**: All repositories either have at least two leaders, or have a very recent write which is not expected to have propagated yet. - **Unsynchronized**: At least one repository has changes which are only available on one node and were not pushed very recently. Data may be at risk. - **No Repositories**: This service has no repositories. - **Ambiguous Leader**: At least one repository has an ambiguous leader. If this screen identifies problems, you can drill down into repository details to get more information about them. See the next section for details. Monitoring Repositories ======================= You can get a more detailed view the current status of a specific repository on cluster devices in {nav Diffusion > (Repository) > Manage Repository > Cluster Configuration}. This screen shows all the configured devices which are hosting the repository and the available version on that device. **Version**: When a repository is mutated by a push, Phabricator increases an internal version number for the repository. This column shows which version is on disk on the corresponding device. After a change is pushed, the device which received the change will have a larger version number than the other devices. The change should be passively replicated to the remaining devices after a brief period of time, although this can take a while if the change was large or the network connection between devices is slow or unreliable. You can click the version number to see the corresponding push logs for that change. The logs contain details about what was changed, and can help you identify if replication is slow because a change is large or for some other reason. **Writing**: This shows that the device is currently holding a write lock. This normally means that it is actively receiving a push, but can also mean that there was a write interruption. See "Write Interruptions" below for details. **Last Writer**: This column identifies the user who most recently pushed a change to this device. If the write lock is currently held, this user is the user whose change is holding the lock. **Last Write At**: When the most recent write started. If the write lock is currently held, this shows when the lock was acquired. - - Cluster Failure Modes ===================== There are three major cluster failure modes: - **Write Interruptions**: A write started but did not complete, leaving the disk state and cluster state out of sync. - **Loss of Leaders**: None of the devices with the most up-to-date data are reachable. - **Ambiguous Leaders**: The internal state of the repository is unclear. Phabricator can detect these issues, and responds by freezing the repository (usually preventing all reads and writes) until the issue is resolved. These conditions are normally rare and very little data is at risk, but Phabricator errs on the side of caution and requires decisions which may result in data loss to be confirmed by a human. The next sections cover these failure modes and appropriate responses in more detail. In general, you will respond to these issues by assessing the situation and then possibly choosing to discard some data. Write Interruptions =================== A repository cluster can be put into an inconsistent state by an interruption in a brief window during and immediately after a write. This looks like this: - A change is pushed to a server. - The server acquires a write lock and begins writing the change. - During or immediately after the write, lightning strikes the server and destroys it. Phabricator can not commit changes to a working copy (stored on disk) and to the global state (stored in a database) atomically, so there is necessarily a narrow window between committing these two different states when some tragedy can befall a server, leaving the global and local views of the repository state possibly divergent. In these cases, Phabricator fails into a frozen state where further writes are not permitted until the failure is investigated and resolved. When a repository is frozen in this way it remains readable. You can use the monitoring console to review the state of a frozen repository with a held write lock. The **Writing** column will show which device is holding the lock, and whoever is named in the **Last Writer** column may be able to help you figure out what happened by providing more information about what they were doing and what they observed. Because the push was not acknowledged, it is normally safe to resolve this issue by demoting the device. Demoting the device will undo any changes committed by the push, and they will be lost forever. However, the user should have received an error anyway, and should not expect their push to have worked. Still, data is technically at risk and you may want to investigate further and try to understand the issue in more detail before continuing. There is no way to explicitly keep the write, but if it was committed to disk you can recover it manually from the working copy on the device (for example, by using `git format-patch`) and then push it again after recovering. If you demote the device, the in-process write will be thrown away, even if it was complete on disk. To demote the device and release the write lock, run this command: ``` phabricator/ $ ./bin/repository thaw --demote ``` {icon exclamation-triangle, color="yellow"} Any committed but unacknowledged data on the device will be lost. Loss of Leaders =============== A more straightforward failure condition is the loss of all servers in a cluster which have the most up-to-date copy of a repository. This looks like this: - There is a cluster setup with two devices, X and Y. - A new change is pushed to server X. - Before the change can propagate to server Y, lightning strikes server X and destroys it. Here, all of the "leader" devices with the most up-to-date copy of the repository have been lost. Phabricator will freeze the repository refuse to serve requests because it can not serve it consistently, and can not accept new writes without data loss. The most straightforward way to resolve this issue is to restore any leader to service. The change will be able to replicate to other devices once a leader comes back online. If you are unable to restore a leader or unsure that you can restore one quickly, you can use the monitoring console to review which changes are present on the leaders but not present on the followers by examining the push logs. If you are comfortable discarding these changes, you can instruct Phabricator that it can forget about the leaders in two ways: disable the service bindings to all of the leader devices so they are no longer part of the cluster, or use `bin/repository thaw` to `--demote` the leaders explicitly. If you do this, **you will lose data**. Either action will discard any changes on the affected leaders which have not replicated to other devices in the cluster. To remove a device from the cluster, disable all of the bindings to it in Almanac, using the web UI. {icon exclamation-triangle, color="red"} Any data which is only present on the disabled device will be lost. To demote a device without removing it from the cluster, run this command: ``` phabricator/ $ ./bin/repository thaw rXYZ --demote repo002.corp.net ``` {icon exclamation-triangle, color="red"} Any data which is only present on **this** device will be lost. Ambiguous Leaders ================= Repository clusters can also freeze if the leader devices are ambiguous. This can happen if you replace an entire cluster with new devices suddenly, or make a mistake with the `--demote` flag. This generally arises from some kind of operator error, like this: - Someone accidentally uses `bin/repository thaw ... --demote` to demote every device in a cluster. - Someone accidentally deletes all the version information for a repository from the database by making a mistake with a `DELETE` or `UPDATE` query. - Someone accidentally disable all of the devices in a cluster, then add entirely new ones before repositories can propagate. When Phabricator can not tell which device in a cluster is a leader, it freezes the cluster because it is possible that some devices have less data and others have more, and if it choses a leader arbitrarily it may destroy some data which you would prefer to retain. To resolve this, you need to tell Phabricator which device has the most up-to-date data and promote that device to become a leader. If you know all devices have the same data, you are free to promote any device. If you promote a device, **you may lose data** if you promote the wrong device and some other device really had more up-to-date data. If you want to double check, you can examine the working copies on disk before promoting by connecting to the machines and using commands like `git log` to inspect state. Once you have identified a device which has data you're happy with, use `bin/repository thaw` to `--promote` the device. The data on the chosen device will become authoritative: ``` phabricator/ $ ./bin/repository thaw rXYZ --promote repo002.corp.net ``` {icon exclamation-triangle, color="red"} Any data which is only present on **other** devices will be lost. Backups ====== Even if you configure clustering, you should still consider retaining separate backup snapshots. Replicas protect you from data loss if you lose a host, but they do not let you rewind time to recover from data mutation mistakes. If something issues a `--force` push that destroys branch heads, the mutation will propagate to the replicas. You may be able to manually restore the branches by using tools like the Phabricator push log or the Git reflog so it is less important to retain repository snapshots than database snapshots, but it is still possible for data to be lost permanently, especially if you don't notice the problem for some time. Retaining separate backup snapshots will improve your ability to recover more data more easily in a wider range of disaster situations. Next Steps ========== Continue by: - returning to @{article:Clustering Introduction}. diff --git a/src/docs/user/cluster/cluster_ssh.diviner b/src/docs/user/cluster/cluster_ssh.diviner new file mode 100644 index 0000000000..940ef98b05 --- /dev/null +++ b/src/docs/user/cluster/cluster_ssh.diviner @@ -0,0 +1,47 @@ +@title Cluster: SSH Servers +@group cluster + +Configuring Phabricator to use multiple SSH servers. + +Overview +======== + +You can run Phabricator on multiple SSH servers. The advantages of doing this +are: + + - you can completely survive the loss of multiple SSH hosts. + +This configuration is simple, but you must configure repositories first. For +details, see @{article:Cluster: Repositories}. + +SSH servers accept SSH requests from commands like `git clone` and relay them +to hosts that can serve the requests. + + +Adding SSH Hosts +================ + +After configuring repositories in cluster mode, you can add more web hosts +at any time. + +First, deploy the Phabricator software and configuration to a host, then +register the host as a cluster device if it is not already registered (for +help, see @{article:Cluster: Devices}. + +Once the host is registered, start the SSH server, and then add the host to the +SSH load balancer pool. + +Phabricator SSH servers are stateless, so you can pull them in and out of +production freely. + +You may also want to run web services on these hosts, since the service is very +similar to SSH, also stateless, and it may be simpler to load balance the +services together. For details, see @{cluster: Web Servers}. + + +Next Steps +========== + +Continue by: + + - returning to @{article:Clustering Introduction}. diff --git a/src/docs/user/cluster/cluster_webservers.diviner b/src/docs/user/cluster/cluster_webservers.diviner index 95699a8291..744696af66 100644 --- a/src/docs/user/cluster/cluster_webservers.diviner +++ b/src/docs/user/cluster/cluster_webservers.diviner @@ -1,42 +1,45 @@ @title Cluster: Web Servers @group cluster Configuring Phabricator to use multiple web servers. Overview ======== -WARNING: This feature is a very early prototype; the features this document -describes are mostly speculative fantasy. - You can run Phabricator on multiple web servers. The advantages of doing this are: - you can completely survive the loss of multiple web hosts; and - performance and capacity may improve. This configuration is simple, but you must configure repositories first. For details, see @{article:Cluster: Repositories}. Adding Web Hosts ================ After configuring repositories in cluster mode, you can add more web hosts -at any time: simply deploy the Phabricator software and configuration to a -host, start the web server, and then add the host to the load balancer pool. +at any time. + +First, deploy the Phabricator software and configuration to a host, then +register the host as a cluster device if it is not already registered (for +help, see @{article:Cluster: Devices}. + +Once the host is registered, start the web server, and then add the host to the +load balancer pool. Phabricator web servers are stateless, so you can pull them in and out of production freely. You may also want to run SSH services on these hosts, since the service is very similar to HTTP, also stateless, and it may be simpler to load balance the -services together. +services together. For details, see @{cluster:SSH Servers}. Next Steps ========== Continue by: - returning to @{article:Clustering Introduction}.