Some additional response steps:

From here on, this analysis discusses some private instance data. I'm generally very hesitant to disclose information like this, but the data is non-identifying and the nature of the data is important to understanding the nature of the failure, so I think disclosure is reasonable in this case. I've limited the disclosure to only include information important in understanding the technical aspects of the issue.

As a very mild silver lining, I'm pretty sure this was the actual root cause of the webserver hang, and it's relatively tractable. Here's what appears to have happened:

I think the issue is initially resolved now, is anyone with an affected instance still seeing an issue?

Pretty sure I've figured this out but resolving it will take some finesse.

Looking up service metadata for some subset of instances is failing, I'm not sure why yet or how large that subset is.

Looking into that. There are some additional issues with queued daemon tasks on admin as well that I'm looking into, unsure if those are symptoms of the web issue or not yet.

Events so far:

This is my Aphlict configuration:

(It's quite possible that I am doing something wrong here)

very nice

I think that story strength thing is non-retroactive? This may not really be fixed if new stories aren't better.

There's some level of magical heuristic, so yell if you still see issues.

When we make progress, we update tasks. We don't have a secret hidden version of Phabricator full of progress updates that we're hiding from the world.

I'm going to merge this into T6018, which I think describes the same bug.

I think it's OK for the UI to show different stuff from the actual query fields -- it's OK to query by "Invited" but see "Attending" in the list. It's just confusing right now because it says "Attending: ..." but includes not-RSVP'd users.

test

(I don't like actually un-cancelling the instances because that makes cancel + reinstate not get you back to the original state, which is inconsistent with how those buttons work for other objects.)

Or the button could just say "Cancel All" and "Reinstate All" maybe? I think it's probably clear enough as long as it has different wording than the instance version, and has the word "all" or "entire" or "whole" or "series" or something on there: the dialog can clarify exactly what it means.

Maybe it's fine to just use "Reinstate All Recurrences", and then explain what that means in the dialog? I suspect we aren't going to find a succinct, clear wording for "Reinstate All Currently Un-Cancelled Recurrences".

Technically I suppose you could have cancelled an instance after using "Cancel All Recurrences", which might make the word "previously" misleading -- "prior to now" is correct but "prior to the time when you cancelled all recurrences" might be the interpretation. So maybe "Reinstate All Currently Un-Cancelled Instances" is perhaps even more clear?

This is probably ultimately the same as T6878 -- T6878#167273 has the current plans here.

Sorry, we don't have the resources to help users debug setup and environmental issues which we can not reproduce. The documentation is extremely clear about this. See Support Resources. Bug reports must have clear, complete, self-contained reproduction steps. See Providing Reproduction Steps.

I followed your instructions on a new, clean Phacility test instance, but the issue did not reproduce for me. Here's a screenshot:

This report doesn't have enough information for us to help you solve the problem. See Providing Reproduction Steps for help with providing reproduction steps.

You can update the phabricator_user.user table and remove the isMailingList flag from the account, something like this:

Some of the changes associated with T4292/T10748 -- principally, D15742 -- have pushed this forward, although it doesn't really do anything yet:

Every service on secure is now supported by a redundant version in a separate AWS availability zone. Failover wouldn't be completely clean/automatic, but we could lose either node with zero data loss, and would have a very short path to promoting the remaining node as a full master.

secure is now running repositories in fully redundant multi-master mode. This appears to be working, although it has been live for about 20 minutes and survived about three pushes so far so I'm not yet brimming with confidence about it.

This got some progress over the weekend -- it dovetailed a bit with T4292, and I wanted to let the Phacility cluster sit on the new availability stuff for a little while. Upshot:

Yes, Ubuntu is very popular. We use Ubuntu ourselves. This very server is running Ubuntu.

@CodeMouse92, I'm specifically curious about the claim I quoted: that Ubuntu 16.04 is a major platform for Phabricator hosts.

Why not collapsing the diff in this case?

I'm merging this into T10691 under the assumption that the real root problem is the one described there (your users want to treat git push as "save a copy", not "publish").

There's an inline comment, visible in the left margin of the left-hand diff. You previously collapsed it.

Maybe see also T3277. But it sounds like there's no actual problem here that isn't solved by running git branch -D slightly more often, and the root problem is a minor convenience issue, not a workflow breakdown? Everything works fine, you just don't want extra branches because you have to spend a few extra seconds cleaning them up later?

Why does the workflow break down?

Broadly, I lean toward this policy going forward:

At least for now, maybe I'll remove the setup issue to avoid confusion and just note this in the changelog instead. This is much more likely to be overlooked, but vendors other than Apple seem to have generally taken care of this and using git --version to test for the presence of the vulnerability isn't really meaningful.

safe to ignore if you've updated to the latest version even though it's a lower version number

In other cases (like with Shellshock) we can just test for the vulnerability to see if a binary is vulnerable. Ideally, we'd just perform this test to figure out if you need to upgrade, but I don't think a reasonable test case exists here because we need to create a 2GB pathname and probably can not do that in a reasonable amount of time.

what do you think about supporting a mode that always uses the client (browser) timezone by default without any manual action taken by the user?