Any test we execute must be distributed with phabricator/ and run on the first page load after an upgrade. That generally means it needs to be small, very fast, and completely conclusive. Compare to the test for Shellshock here, which runs in a few milliseconds, requires 10 lines of code, and is totally conclusive:

We'll fix this eventually, but probably not until the next time we do major work on Phriction (T9379) since the impact is very minor.

Thanks! The intended instruction was just "don't specify ssl.key, ssl.cert, etc.". D15787 should fix it.

I don't have to click "Cluster", it's just a subheader. Are you seeing something different?

A few things that might motivate writing an API panel (which pulls from external sources) instead:

(I agree that this is a bad solution for notifying users of downtime/maintenance on the current instance, and that rooster/broadcast is better.)

Maybe T8145 / T5231, but I think the driving use case there was "notify users of Phabricator downtime" specifically. I'm imagining the driving use cases here not affecting all users and not directly impacting Phabricator, just having wide enough impact that there's interest in surfacing them more broadly. e.g., "We're having a production incident affecting video upload, ops is investigating" is important, but not "every Phabricator user must be notified" important.

Doing a Phriction panel, an iframe panel, or a custom API panel would also be reasonable approaches.

Here's an example I could imagine us doing: we have some button on admin.phacility.com that says "Start a deploy". When that's pressed it causes these actions to occur:

This one seems OKish to me -- I think those use cases are pretty reasonable and root-problem-ish? I'm assuming "waht's currently broken" and "can we do deploys" are information known by some other system or tool.

I think D15784 should fix the issue. It should land in master later today and be available in stable in about 24 hours. Let me know if you're still seeing issues once you update past that.

I think I know what's going on, although I'm unsure why the <2MB files would work, unless you happened to push those to a public ("Visible To: Public (No Login Required)") repository and pushed the large files to a private repository?

• Is security.alternate-file-domain configured on your install?
• When you say "redirect to the correct file", do you mean you get taken to the file detail page /F1234?
• Also, do you have rP37b93f4 locally (April 7)? (You can grab the hashes in Config → Versions if you aren't immediately sure and I can figure it out from there.)

You shouldn't need to configure diffusion.allow-http-auth -- the auth token is basically the first [shortened] in the URI, so that appears to be working properly.

I think we generate this warning long before attaching the diff to a revision, so it isn't trivial to know where to look for a "you already confirmed this prompt once for the current revision" flag.

We appear to be stable here now:

This is also maybe a good time to fix all the bad old mysql vs mysqli stuff, mostly:

I can't reproduce this. For example, this host does not reproduce the issue:

D15781 is promising and it looks like we can do this purely on the client if it has relatively old PHP, without requiring relatively new MySQL and special configuration on the server.

Mixed progress. Repository clustering has been stable in production since the last update, and appears to actually work, which is good. We've hit a few minor snags (like a missing sync-before-read in diffusion.querycommits) but all pretty much expected stuff and nothing troubling.

Like just don't show it at all? That seems reasonable to me, although 0.00000000001 is very funny and we would be losing a truly great joke at such a young age.

That's because it was deliberate self-subscribe which is so completely boring that we normally wouldn't even show it. I think it's arguably correct to show the Herald story? Maybe?

Thanks, I think that should be fixed now. Not clear if it was related to everything else and I can't immediately come up with a way that it would have been -- I'll keep an eye on it. We've made some changes to the Aphlict server recently and it might have some issues that still need to be worked out, so it could be purely coincidental.

The diffusion.querycommits method needs to sync-before-read (at least, if bypassCache is provided?) but currently does not. This can lead to tasks failing on the daemon on an un-synchronized node. Things self-heal, but it would be nice to prevent this.

Some additional response steps:

From here on, this analysis discusses some private instance data. I'm generally very hesitant to disclose information like this, but the data is non-identifying and the nature of the data is important to understanding the nature of the failure, so I think disclosure is reasonable in this case. I've limited the disclosure to only include information important in understanding the technical aspects of the issue.

As a very mild silver lining, I'm pretty sure this was the actual root cause of the webserver hang, and it's relatively tractable. Here's what appears to have happened:

I think the issue is initially resolved now, is anyone with an affected instance still seeing an issue?

Pretty sure I've figured this out but resolving it will take some finesse.

Looking up service metadata for some subset of instances is failing, I'm not sure why yet or how large that subset is.

Looking into that. There are some additional issues with queued daemon tasks on admin as well that I'm looking into, unsure if those are symptoms of the web issue or not yet.

Events so far:

very nice

I think that story strength thing is non-retroactive? This may not really be fixed if new stories aren't better.

There's some level of magical heuristic, so yell if you still see issues.

When we make progress, we update tasks. We don't have a secret hidden version of Phabricator full of progress updates that we're hiding from the world.

I'm going to merge this into T6018, which I think describes the same bug.

I think it's OK for the UI to show different stuff from the actual query fields -- it's OK to query by "Invited" but see "Attending" in the list. It's just confusing right now because it says "Attending: ..." but includes not-RSVP'd users.

test

(I don't like actually un-cancelling the instances because that makes cancel + reinstate not get you back to the original state, which is inconsistent with how those buttons work for other objects.)

Or the button could just say "Cancel All" and "Reinstate All" maybe? I think it's probably clear enough as long as it has different wording than the instance version, and has the word "all" or "entire" or "whole" or "series" or something on there: the dialog can clarify exactly what it means.

Maybe it's fine to just use "Reinstate All Recurrences", and then explain what that means in the dialog? I suspect we aren't going to find a succinct, clear wording for "Reinstate All Currently Un-Cancelled Recurrences".

Technically I suppose you could have cancelled an instance after using "Cancel All Recurrences", which might make the word "previously" misleading -- "prior to now" is correct but "prior to the time when you cancelled all recurrences" might be the interpretation. So maybe "Reinstate All Currently Un-Cancelled Instances" is perhaps even more clear?