I'm partitioning secure.phabricator.com now. Things will drop into read-only mode for a bit.

• T11908 is a followup for executing queries for multiple applications on a single connection. I believe the pathway for that is straightforward and fairly short, but that no install would really reap substantial benefits from it today, so I don't expect to pursue it for some time.
• I believe everything else is now complete, so I'll put this in production as soon as everything here lands and we can see what catches on fire.

We have at least one tricky issue remaining: when applying storage upgrades, we currently apply them like this:

General state of the world here:

D16115 + D16847 + D16848 make these changes:

It's just very
important to
thoroughly explore
the applications

This went out a while ago and I confirmed the fix in production.

I've banged on this a reasonable amount locally without issues and the originating instance reports that this seems to have calmed things down in production with these patches, so it seems like this pretty much just worked.

From reviewing the code, it appears that this should be handled correctly already.

I believe this should be fixed in HEAD of master. It should promote to stable in about 24 hours.

A bad half-fix for this is here:

I added a note about this to T7338 for the longer term.

The particular issue is that that the change removed a file containing the bytes ... 0xE5 0x76 .... The 0xE5 is probably an å in Latin 1.

I extracted the response body prior to encoding as a serialized()'d object so I can narrow down what's causing the encoding to fail and it looks reasonable, looking like it's almost certainly a UTF8 issue now.

Not too surprisingly, the response is empty. AphrontResponse->encodeJSONForHTTPResponse() uses json_encode() and does not detect UTF8-related encoding failures, so the chance that this is a UTF8 issue is pretty high. We should also probably fix that so fail earlier and more explicitly (and don't emit HTTP 200).

The next step I want to take is figuring out what the actual problematic response body looks like, but that's slightly tricky right now because there's no way to make Conduit calls from the CLI.

From IRC, this might be a false positive / not the root cause since destroying the binding didn't fix the problem, but I can run through things locally.

Although this is cluster-related, it can be tested with a single database host and is probably not exceptionally difficult to handle better. Briefly:

@jmeador, this error should no longer be misleading in HEAD of master. Let us know if you run into more stuff, and thanks for the report!

Presuming resolved based on followup issues and stargazing.

After a little thought, I'm leaning strongly to putting every schema on every database. I think this makes a lot of the administration, maintenance and implementation easier.

Prior to D15987, the easiest fix for this was to add --force:

I was kind of expecting that 'cluster.read-only' would be a database configuration value, not a local configuration value. Otherwise you need to update all nodes in the cluster instead of just setting the flag in the DB. My expectation when the cluster DB is read-only is that only the following operations work:

I think I didn't actually do these things, I did a different and more horrible set of things and then wrote these great instructions to avoid all the msitakes I made. I'll attempt to let dump pierce through read-only, or find some other solution if that isn't straightforward.

Should be fixed in HEAD of master. Thanks for the report!

Going any farther than this risks some inconsistency / data loss and we haven't managed to break this yet, so I'm going to consider this sufficiently robust until we see evidence otherwise.

Yeah, measuring latency and using it as a signal is tricky. If there's a long disruption and Oregon measures faster than N. California for "a while", we probably legitimately want to use it. If N. California completely goes down, we almost certainly want to keep things alive and send writes to Mars, even if Mars is at a huge network distance, since "slow write" is probably almost always better than "complete failure" (although maybe Mars bandwith is so much more expensive that we don't want to fail over to it, or the size of the link is small enough that it simply can't accommodate the failover traffic).