I'm probably done for the night, here's a quick update:

• Daemons got some planning work (collected in T10756). The immediate issue is that the PullLocal daemon needs to be way smarter about figuring out which repositories are expected to be on the current host, which runs into the repository stuff.
• Haven't touched the repository stuff yet.
• Databases seem like they may be the more straightforward of the two tracks and progress has been easy so far, so I'm focusing on that for now.

In practice, nothing very useful has come of this yet. You can:

• kind of inspect replication status; and
• manually put an install in read-only mode and sort of limp along.

I think there is no real-world reason to ever do either of these things yet.

Progress so far:

• Phabricator now survives the loss of the primary database by degrading into read-only mode and serving traffic using a replica (at least, in a controlled sandbox environment).
• Phabricator can now automatically degrade to read-only mode briefly if it can't connect to the master (temporary interruption).
• Phabricator can now automatically health-check databases and degrade into a more persistent read-only mode if they repeatedly fail checks (major interruption), and return to read/write mode if/when they recover.

Database availability is on fairly solid footing now (for a feature with zero production hours, at least -- I'm definitely not encouraging anyone to actually deploy this yet). I want to put this install (secure.phabricator.com) in real HA cluster mode before pursuing more database changes so I can get more confidence that the core works, since I believe most remaining features build on top of this and are primarily quality-of-life stuff (although some, like the ability to establish new sessions while in read-only mode, provide very important quality-of-life improvements).

Before doing that, I'm planning to pursue the other pathway (repositories) so I can also get that basically functional too and consolidate all the operations work to swap us into HA mode. There are two major pieces there:

• Changing repositories to understand that multiple copies may exist on different hosts (and whether a copy should exist on the current host or not, so PullLocal can get smarter).
• Implementing the logical clocks / synchronization / pull-before-clone stuff to make multiple masters work, described in T4292.

This may run into T10748 at some point but I need to plan the details out a bit first. We probably don't need any editable controls in the UI for a while and may be fine until then.

I've started moving these changes into production (T10784) so I can begin gathering confidence that they actually work. The repository stuff definitely still doesn't work since several core pieces are currently implemented as // TODO comments, but it has the synchronization bookkeeping in place for all the locking/versioning: if that's stable, it will give me confidence that we're on relatively firm ground and can move forward.

This has hit a few minor issues and will likely hit a few more before I'm done, but seems to be going alright so far. Two larger issues:

Way Too Much Header Magic: Most of the issues I've hit are related to header/configuration magic. Currently, we deploy a SiteSource in the Phacilty cluster and on this host which does a lot of header fiddling to make the production clustering (which is focused on multi-tenancy, not availability) work properly.

One example is that security.require-https needs to be disabled for cluster-originated requests if you plan to terminate SSL at the load balancer and run intracluster requests over plain HTTP, as we currently do.

Another example is that X-Forwarded-For is not trusted by default, but should be trusted when provided by a load balancer. Likewise, X-Forwarded-Proto should be trusted if the load balancer provides it.

I suspect Phabricator should try to get smarter about dealing with more of this stuff automatically, so we can reduce the need for weird custom header magic. T10784 discusses raising a better error about security.require-https, which is a start, but I think it will be more fruitful to be more ambitious about this. Particularly, we should be able to trust hosts in cluster.addresses by default (or, if necessary, add a separate cluster.nice-trustworthy-load-balancer-address-blocks option), and mostly get things right without additional work once that's configured.

Notification Server: The biggest blocker for putting multiple hosts behind the secure.phabricator.com ELB is that the ELB also routes notification traffic, and the notification server does not currently support any kind of clustering.

I could resolve this by running it through a different ELB (as we do in the production cluster), or by just forwarding 22280 on secure002 directly to 22280 on secure001 (as at least one other install does in T6915), but I suspect a more ambitious pathway through T6915 / T10697 may not be particularly long, and this is work we need to do eventually anyway.

I currently plan to give this some more thought since I'm not sure I have a clustering plan I'm completely happy with yet, but I'm leaning toward pursuing this in the short term.

Given current progress and velocity, I'm aiming for a timeline roughly like this:

• This week (April 16): make all core services on this host (secure) highly available.
• Next week (April 23): maybe ready for third-party use some time around here? This may be a little ambitious.

I slightly reduced the amount of header magic, and the notification stuff is now cluster-ready (in theory, at least). I'm planning to get notifications, web nodes, and databases redundant in production today, and maybe repositories/daemons if nothing else comes up. No issues with repo bookkeeping so far but even if it's smooth sailing it needs a fair bit of UI work, and same with the daemons.

A big chunk of this is finally in production and seems to be working, which I'm thrilled about.

Now In Production

Web Servers: This service (secure.phabricator.com) is now backed by two redundant web nodes in different AWS availability zones. You can visit the loadbalancer status page and reload the page a few times to hit the different hosts.

Databases: We now run a primary on secure001 and a replica on secure002:

I tested failover by shutting down the master; Phabricator failed over to read-only mode correctly, and recovered cleanly when I restored service.

Daemons: Both hosts are running daemons. The UI needs some updates to make this more clear and there's probably some remaining work here, but I killed the daemons on one host and the obvious things still worked properly.

Notifications: Both hosts are running read/write notification servers:

This appears to be working properly.

SSH: We got this one for free since I built it in Phacility a long time ago, but SSH is being proxied by both boxes to the underlying repository service.

Remaining Work

Repositories: These aren't redundant yet and we'd currently lose them if secure001 exploded.

Updated Daemon UI: Current UI has a bunch of issues and does not make it clear where daemons are running.

SSH: This works in the Phacility cluster, but is wholly unrealistic for third-parties to configure today and has zero documentation. We also run a modified sshd, but shouldn't require installs to.

Just some cleanup today. New daemon console now shows which hosts daemons are running on:

The setup for secure has been stable so far.

secure is now running repositories in fully redundant multi-master mode. This appears to be working, although it has been live for about 20 minutes and survived about three pushes so far so I'm not yet brimming with confidence about it.

If this holds, I think that's about the end of the "hard" stuff. Still plenty of usability / documentation / cleanup / support / performance work remaining (for example, multi-master repositories are currently almost impossible to configure).

Mixed progress. Repository clustering has been stable in production since the last update, and appears to actually work, which is good. We've hit a few minor snags (like a missing sync-before-read in diffusion.querycommits) but all pretty much expected stuff and nothing troubling.

However, progress on making repository clustering plausible to deploy, configure and administrate has been slow. There's a lot of ground to cover and Diffusion is already hard to configure and this makes everything harder. I've made some progress that I'm happy with, but I've also made some efforts to reduce and simplify the existing complexity (particularly the sudo/ssh situation, which is particularly complex), without much success with it so far.

Beyond configuration, there's an issue of how to choose which cluster service to host a repository on when creating it. This overlaps heavily with T10748 since EditEngine + custom forms seem like they're probably the best solution, although the current rule ("if there's exactly one, choose that") is probably fine in the short term since I don't expect installs to be deploying multiple clusters for some time.

We had an unrelated production incident this morning that I'm still cleaning up today, and I want to focus on tying up loose ends tomorrow before the release cut, so this probably won't make too much more headway this week.

Here's the general state of the world I expect for this release:

• Database Replication: Deploy freely. There are rough edges, but deploying this should be reasonable today and should strictly put installs in a more available state. The documentation is up to date (see Clustering: Databases).
• Aphlict: Deploy cautiously. I think this still has a couple of wrinkles that need to be worked out but it is straightforward to configure and seems to be working properly for the most part.
• Repositories: Do not deploy. They work but are impossible to configure.
• Daemons: Must wait for repositories.
• Web: Must wait for repositories.
• SSH: Must wait for repositories.

I'm still not really thrilled with where the documentation situation on repositories is, but it's getting a little more manageable.

Handling of potential link disruption during writes (discussed in T10860) should be better now, too. I think it should now be fairly hard to freeze repositories outside of an actual disaster which legitimately puts data at risk.

The ssh link got a lot more chatty:

$ git push
...
# Push received by "secure001.phacility.net", forwarding to cluster host.
# Waiting up to 120 second(s) for a cluster write lock...
# Acquired write lock immediately.
# Waiting up to 120 second(s) for a cluster read lock on "secure002.phacility.net"...
# Acquired read lock immediately.
# Device "secure002.phacility.net" is already a cluster leader and does not need to be synchronized.
# Ready to receive on cluster host "secure002.phacility.net".
Counting objects: 13, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (13/13), 4.77 KiB | 0 bytes/s, done.
Total 13 (delta 11), reused 0 (delta 0)
# Released cluster write lock.
...

This is probably over-tuned a bit, but should make the operation more obvious and hopefully help debug any issues which do arise. We can quiet it down later.

T10748, which provides the new cluster management UI for repositories, is nearing completion. This makes cluster setup more manageable and at least provides a pathway forward for Almanac stuff in the future, although it does not yet make Almanac / service management explicitly selectable when creating a repository (for now, new repositories allocate on a random open service).

Most of what remains there is UI/UX and documentation; I expect to complete that, then return here and finish the repository/web/SSH documentation, then consolidate where things are today and plan where it makes the most sense to go next.

The last round of repository changes have been stable in production here on secure, and deployed to the Phacility cluster last week. The cluster is not running HA repository services yet (repository services are still single-node clusters) but it is hitting about 95% of the code (all the locking/versioning/proxying/etc).

T10748 is wrapping up, and repositories now show cluster status in the new "Storage" panel of the new UI:

(The "Last Writer" column is a bug which I'll fix shortly.)

Per above, I'm expecting to complete the remaining documentation which was waiting on this next, then do some PM'ish stuff to consolidate the state of the world so there's a clearer picture of exactly where things stand and what's expected to work now vs not work yet.

I think this is the current state of the world. I'm planning to tackle the three major issues noted below (observed repository versioning, repository lock granularity, device enrollment behavior) and then try to wind down this phase, move all this followup work to a new "make clustering more powerful" sort of task, and then plan where to go from there based on which goals are most important (scalability vs multi-region vs read-only vs hg/svn vs drydock vs whatever crops up as installs actually deploy this stuff).

From this point, future work has few interconnections and can be prioritized and pursued mostly independently.

General

Today, cluster behavior has a greater emphasis on multi-tenancy and availability/loss-resistance than it does on scalability and multi-region deployments. A mixture of general changes and service-specific changes could improve its suiltability to these tasks in the future.

• (Future) T10884: Sort repository, database and notification services better (by network distance)
• (Future) T10768: Provide tools to drop severed nodes from load balancer pools by failing status checks

Databases

The database read-only failover mode still has a number of limitations, some of which may be fairly severe. Improving this mode could reduce the disruption associated with losing masters.

Database clusters only provide redundancy and availability today: no traffic is sent to the replica unless the master is unreachable. In the future, we could send some reads to the replica during normal operation. This likely has the greatest impact for open source installs with a large public userbase.

Database replicas must be promoted manually. I have no current plans to attempt automatic promotion because of the mortal danger that mistakes represent.

All service types other than databases failover automatically today.

• (Various) T10769: Read-Only Mode Errata
• (Future) It would be nice to be able to send read traffic to replicas even if the master is alive.

Repositories

Hosted Git repositories are in good shape, but other types of repositories still have limited or nonexistent support.

In the future, better administrative tools and smarter connection management could improve the behavior of multi-region clusters.

• (Major) Observed repositories (vs hosted) do not version correctly.
• (Future) No explicit management of which cluster service repositories allocate on.
• (Future) No Mercurial support.
• (Future) No Subversion support.
• (Future) T10883: Allow repository cluster nodes to be read-only

Daemons and CLI

Some daemon and administrative behaviors still have rough edges.

• (Major) Repository lock granularity for PullLocal daemon is too coarse.
• (Major) T10940: Enrolling an existing device in a repository cluster has surprising effects and inconsistent severity
• (Minor) PullLocal should prioritize by examining repository logical clocks.
• (Minor) T6768: Worker queue lease names are unwieldy and could be better implemented
• (Future) T10861: Provide a tool to rewind the push log for a repository

Notifications

Notification HTTP / HTTPS could be made easier to configure.

• (Minor) T10402: Add a client setup check to detect that the browser is using HTTPS but Phabricator does not recognize it

Drydock

Drydock could be significantly better at detecting and recovering from broken resources and losts hosts than it is today. In particular, if you lose connectivity to a datacenter you're probably looking at some degree of manually pulling bad hosts out of the pool.

• (Various) T8153: Improve detection and recovery when resources are mangled outside of Drydock's control

Two of the three issues discussed above (lock granularity, severity of device enrollment) got tackled. I think I have an attack on versioning hosted repositories, but I'm not going to try to squeeze it in before the release cut today.

Observed repositories should now version in a reasonable way.

I've created followups to track the other points above that didn't previously have dedicated tracking tasks:

• T11056: Send some database traffic to replicas even while the master is still alive
• T11057: Allow installs to manage which cluster service new repositories allocate on

This also wasn't explicitly covered above, but is effectively a followup:

• T11044: Support partitioning application databases across multiple database hosts

I'm going to pause this for feedback since we have no more planned direct action here.

(We are pursuing application partitioning (T11044) and improved drydock recovery (T8153) outside of this, but I'll track those separately since they're only tangentially related to this.)