Changeset View
Changeset View
Standalone View
Standalone View
src/docs/user/cluster/cluster.diviner
@title Clustering Introduction | @title Clustering Introduction | ||||
@group cluster | @group cluster | ||||
Guide to configuring Phabricator across multiple hosts for availability and | Guide to configuring Phabricator across multiple hosts for availability and | ||||
performance. | performance. | ||||
Overview | Overview | ||||
======== | ======== | ||||
WARNING: This feature is a very early prototype; the features this document | WARNING: This feature is a prototype. Installs should expect a challening | ||||
describes are mostly speculative fantasy. | adventure when deploying clusters. In the best of times, configuring a | ||||
cluster is complex and requires significant operations experience. | |||||
Phabricator can be configured to run on multiple hosts with redundant services | Phabricator can be configured to run on multiple hosts with redundant services | ||||
to improve its availability and scalability, and make disaster recovery much | to improve its availability and scalability, and make disaster recovery much | ||||
easier. | easier. | ||||
Clustering is more complex to setup and maintain than running everything on a | Clustering is more complex to setup and maintain than running everything on a | ||||
single host, but greatly reduces the cost of recovering from hardware and | single host, but greatly reduces the cost of recovering from hardware and | ||||
network failures. | network failures. | ||||
Each Phabricator service has an array of clustering options that can be | Each Phabricator service has an array of clustering options that can be | ||||
configured independently. Configuring a cluster is inherently complex, and this | configured somewhat independently. Configuring a cluster is inherently complex, | ||||
is an advanced feature aimed at installs with large userbases and experienced | and this is an advanced feature aimed at installs with large userbases and | ||||
operations personnel who need this high degree of flexibility. | experienced operations personnel who need this high degree of flexibility. | ||||
The remainder of this document summarizes how to add redundancy to each | The remainder of this document summarizes how to add redundancy to each | ||||
service and where your efforts are likely to have the greatest impact. | service and where your efforts are likely to have the greatest impact. | ||||
For additional guidance on setting up a cluster, see "Overlaying Services" | For additional guidance on setting up a cluster, see "Overlaying Services" | ||||
and "Cluster Recipes" at the bottom of this document. | and "Cluster Recipes" at the bottom of this document. | ||||
Clusterable Services | |||||
==================== | |||||
This table provides an overview of clusterable services, their setup | |||||
complexity, and the rough impact that converting them to run on multiple hosts | |||||
will have on availability, resistance to data loss, and scalability. | |||||
| Service | Setup | Availability | Loss Resistance | Scalability | |||||
|---------|-------|--------------|-----------|------------ | |||||
| **Databases** | Moderate | **High** | **High** | Low | |||||
| **Repositories** | Complex | Moderate | **High** | Moderate | |||||
| **Daemons** | Minimal | Low | No Risk | Low | |||||
| **SSH Servers** | Minimal | Low | No Risk | Low | |||||
| **Web Servers** | Minimal | **High** | No Risk | Moderate | |||||
| **Notifications** | Minimal | Low | No Risk | Low | |||||
See below for a walkthrough of these services in greater detail. | |||||
Preparing for Clustering | Preparing for Clustering | ||||
======================== | ======================== | ||||
To begin deploying Phabricator in cluster mode, set up `cluster.addresses` | To begin deploying Phabricator in cluster mode, set up `cluster.addresses` | ||||
in your configuration. | in your configuration. | ||||
This option should contain a list of network address blocks which are considered | This option should contain a list of network address blocks which are considered | ||||
to be part of the cluster. Hosts in this list are allowed to bend (or even | to be part of the cluster. Hosts in this list are allowed to bend (or even | ||||
▲ Show 20 Lines • Show All 100 Lines • ▼ Show 20 Lines | |||||
Cluster: Daemons | Cluster: Daemons | ||||
================ | ================ | ||||
Configuring multiple daemon hosts is straightforward, but you must configure | Configuring multiple daemon hosts is straightforward, but you must configure | ||||
repositories first. | repositories first. | ||||
With daemons running on multiple hosts, you can transparently survive the loss | With daemons running on multiple hosts you can transparently survive the loss | ||||
of any subset of hosts without an interruption to daemon services, as long as | of any subset of hosts without an interruption to daemon services, as long as | ||||
at least one host remains alive. Daemons are stateless, so spreading daemons | at least one host remains alive. Daemons are stateless, so spreading daemons | ||||
across multiple hosts provides no resistance to data loss. | across multiple hosts provides no resistance to data loss. | ||||
Daemons can become a bottleneck, particularly if your install sees a large | Daemons can become a bottleneck, particularly if your install sees a large | ||||
volume of write traffic to repositories. If the daemon task queue has a | volume of write traffic to repositories. If the daemon task queue has a | ||||
backlog, that hints at a capacity problem. If existing hosts have unused | backlog, that hints at a capacity problem. If existing hosts have unused | ||||
resources, increase `phd.taskmasters` until they are fully utilized. From | resources, increase `phd.taskmasters` until they are fully utilized. From | ||||
there, adding more daemon hosts will provide an approximately linear increase | there, adding more daemon hosts will provide an approximately linear increase | ||||
in capacity. | in capacity. | ||||
For details, see @{article:Cluster: Daemons}. | For details, see @{article:Cluster: Daemons}. | ||||
Cluster: SSH Servers | |||||
==================== | |||||
Configuring multiple SSH hosts is straightforward, but you must configure | |||||
repositories first. | |||||
With multiple SSH hosts you can transparently survive the loss of any subset | |||||
of hosts without interruption to repository services, as long as at last one | |||||
host remains alive. SSH services are stateless, so putting multiple hosts in | |||||
service provides no resistance to data loss because no data is at risk. | |||||
SSH hosts are very rarely a scalability bottleneck. | |||||
For details, see @{article:Cluster: SSH Servers}. | |||||
Cluster: Web Servers | Cluster: Web Servers | ||||
==================== | ==================== | ||||
Configuring multiple web hosts is straightforward, but you must configure | Configuring multiple web hosts is straightforward, but you must configure | ||||
repositories first. | repositories first. | ||||
With multiple web hosts, you can transparently survive the loss of any subset | With multiple web hosts you can transparently survive the loss of any subset | ||||
of hosts as long as at least one host remains alive. Web hosts are stateless, | of hosts as long as at least one host remains alive. Web services are stateless, | ||||
so putting multiple hosts in service provides no resistance to data loss | so putting multiple hosts in service provides no resistance to data loss | ||||
because no data is at risk. | because no data is at risk. | ||||
Web hosts can become a bottleneck, particularly if you have a workload that is | Web hosts can become a bottleneck, particularly if you have a workload that is | ||||
heavily focused on reads from the web UI (like a public install with many | heavily focused on reads from the web UI (like a public install with many | ||||
anonymous users). Slow responses to web requests may hint at a web capacity | anonymous users). Slow responses to web requests may hint at a web capacity | ||||
problem. Adding more hosts will provide an approximately linear increase in | problem. Adding more hosts will provide an approximately linear increase in | ||||
capacity. | capacity. | ||||
▲ Show 20 Lines • Show All 110 Lines • Show Last 20 Lines |