Does anyone have an opinion on the best way to go about solving this?

...put all the masters in the same region? Why isn't that a possibility?

Basically because we want to keep the version control master near the dev servers, which are mostly going to land in oregon. The db masters landed in carolina because that's the center of what our web footprint will look like.

But there's a slave in Oregon (if not, reading from slaves won't help anything), right? Why can't the DB team just make the Oregon DB be the master and the Carolina DB be the slave?

Basically because they hate one-offs more than we do, and putting the Phabricator master in Oregon would make it the only master in Oregon.

Also, why is there a 5 minute replication lag on the SVN master/slave? Does it fax the changes? If not, can it be made to? It would probably be faster? Why is SVN so much slower cross-country? Is it a protocol issue (like the protocol inherently does a zillion round trips), or is the link a lossy mess?

I expect to pursue master/slave at some point which is why we've left the door open in the codebase (r and w flags on connections, e.g., and hopefully-proper internal handling of reads vs writes), but the motivation should be that we need to shed load on the master for installs with lots of public content, not that the servers are separated by 5 minute replication delays which have no technical reason to exist.

Everything here just reeks of Facebook being awful:

1. The replication delay should be a few seconds, not 5 minutes.
2. The overhead of cross-country svn commands should be ~1 round trip, not ~20.
3. Putting the frontends in a different region from the DB masters should increase query overhead by ~1 round trip, not cause timeouts because of slow queries.
4. The masters should clearly just be in the same location. This is obviously the right solution to the problem. Every other solution is crazy and basically why I quit in the first place.

If (3) is a problem with master-only, there's no reason to believe it won't remain a problem with master/slave, because we must send readers to the master after write. So you'll submit a comment in Differential, and then load the page again and get timeouts (if we sent you to the slave, you wouldn't see your comment). So even if we build this for the right reasons (preparation for future installs which need to load shed) it won't fix Facebook's problems.

I offer these solutions:

1. Put the frontend with the DB masters and read off the SVN slave. Tell engineers that they can complain to the DB team about the insane master/slave setup, complain to the SVN team about the insane replication lag, go join one of those teams to fix the problems, or use arc close if they aren't happy with this. You can also trigger arc close through local git pre-commit hooks, a commit hook on the master, or arc alias.
2. For a modest $20M per year, I will install Phabricator + SVN in EC2 and manage it for Facebook.

Just a side note: Our DB replication is quite fast. I read from slave on my devbox and write to master and commenting in Differential just works. Not that this is a solution to anything, just a note.

The svn replication is NFS snapmirror. We're working on speeding it up as well as pursuing this. The root cause of the rather highly variable lag that averages around 5 minutes seem to be giant-ass commits that take a while to replicate.

Cross-country svn commands go across ssh connections, which take a while to set up, hence why it's more than one round-trip-time delay. I have no idea why db queries are a problem.

Also, what we would really like to do is put the front-ends in every region, and then let you just hit the closest one.

Regarding the svn replication issues.

The svn repos are on NFS and use Netapp Snap-Mirror to do replication. The Netapp volume which hosts www also hosts about 15 other svn repos (svnhive, opsfiles, admin, etc).

The vast majority of the replication take 1-2 minutes when there is less than a few 100MB of changes. However, when I sat down with the storage team, I saw that there were multiple syncs each day that took between 10-45 minutes. These long syncs were due to 1-5GB changes. At this point, I don't know which repo was generating such large changes, but I'm doubting it was due to www (although I didn't check the commit logs to correlate).

One option I had was to move www to a separate netapp volume so that the other repos wouldn't impact its replication time.

I'm okay with pursuing these things in the upstream, since they aren't strictly unique to Facebook's perfect storm:

Awareness of master/slave DBs: This is already "supported" (r/w mode on connections) but not in use as far as I know (except by @vrana, apparently). Remaining pieces are probably:

• "Force reads to the master until time X" cookie after establishing a write connection, unless someone has a better idea for preventing stale reads.
• Ideally, identify reads-before-writes somehow and force them to the master. POST might be an approximation but probably not a perfect one. This isn't critical to get right since most writes are append-only and most objects have very low write contention.
• Add some "you're doing a write on a read connection" tests in the query layer (just looking for INSERT/UPDATE/etc is probably good enough).
• Expose read/write connections better in the UI (DarkConsole).
• Fix any bad 'r' / 'w' settings identified by this stuff.
• Always hit the master from scripts?
• ...unless you just want to assume that replication is always faster than an HTTP request, in which case you can just turn it on and see what happens I guess.

Tunneling SSH: If svn cat (and I guess svn diff) are slow because of setting up SSH, we could presumably tunnel it rather than building the link every time. This is already abstracted and necessarily complicated / special cased (because of the different auth methods, protocols, and VCSes) so we would do very little harm by adding configuration/awareness of tunneling.

Improve error handling in Diffusion: We probably explode more than necessary when failing to pull data out of Subversion. Any improvements here which would also benefit a more general install are obviously welcome. The obvious example is probably rendering file errors inline rather than in popups.

• One possible solution to read-after-write is to execute SHOW MASTER STATUS after write, store it locally (readable fast from all web frontends) and call MASTER_POS_WAIT() before reading (which would be usually very quick or no-op in our setup).
• We use write connection in transactions which should cover all significant read-before-writes.
• Detecting write-on-read-connection is better done in MySQL setup by read_only. We already do it and it throws when misused.

Could it just be configured to pull the entire SVN repo local to Phabricator/Diffusion? Possibly using such hacks as git-svn?