There's nothing particularly useful or actionable here now, so closing it out. (I believe this was the most severe incident Phacility ever experienced while actively maintained.)

This hasn't caused any more problems in like 4 years, so I guess it's kind of whatever.

This is moot given the Phacility wind-down.

This is almost certainly "PhabricatorMetaMTAReceivedMail incorrectly assumes email is valid UTF8", and the solution begins by making all the associated storage binary, but it's hard to imagine that ever happening now.

Difficult to imagine this ever getting fixed in the upstream since I don't have any Android devices.

This only impacts the "Support" application in Phacility and it's hard to imagine building a Doorkeeper bridge for it now.

Discourse got nuked, so just throwing this away until another report somehow surfaces.

This hasn't cropped up again and is presumably resolved.

This isn't really resolved, but almost certainly does not make sense to pursue given the Phacility wind-down.

I believe the above changes resolved this, and it hasn't cropped up again.

I believe the above changes covered everything, and there hasn't been any more action on the motivating customer issues, although this isn't terribly meaningful.

I believe D21466 resolved this, since it hasn't cropped up again even though daemon runtimes between restarts have been very long at various periods in the last year.

It's likely that D21451 didn't catch exhaustively all of these, but the motivating use case is now moot. If further need arises somehow, this strategy can be applied to those cases.

This is resolved and/or mooted.

I believe these were all hunted down.

I think this was resolved. I nuked Discourse anyway so it's effectively resolved until another report shows up.

Almost every host currently in production was provisioned with Piledriver and things have been stable for quite a while, so I'm calling this resolved. See elsewhere for issues with Ubuntu20, mail, etc.

I made this change and deployed it to secure, and this "critical security vulnerability" is now a proper Phabricator 404 page:

• Fix some more "strlen(null)" behavior.

Calling this resolved, since it has been in production in the Phacility cluster for some time and worked correctly through relevant hardware changes.

Moved the rest of this to T13640.

See also T13646: some very early service calls during initialization currently occur before --trace becomes active. These calls should be visible, possibly by buffering them and then dumping the buffer once --trace comes online.

Browsing Revision

Ah! lgtm

Use a simpler "is new?" test, by just testing if the changeset ID is in the new list.

• Also fix some PHP 8.1 "strlen(null)" behavior.

PHP 8.1 disallows strlen(null).

• Also fix some unit test behaviors.

This was resolved by D21452. PHI1880 has some unrelated additional plans.

I deployed this and it seems to be working properly.

Hey, it worked once. Good enough for me!

No dice. We need bin/upgrade to run before mysql because it has to mount the data volume. So now I'm trying this:

... service ... start rather than service ... restart ...

...probably tested...

I deployed this everywhere in the Phacility cluster yesterday and things have been quiet, so I'm assuming it worked until evidence arises to the contrary.

With max_connections at 1024, the server above still looks fine after ~14 days:

D21756 effectively makes all Git pathways call setSudoAsDaemon(true).

Just for visibility, the error messages you'll see if you're affected by this issue look something like this:

...maybe this is an actual bug in Phabricator where some pathways are just missing the "sudo" wrapper?

Elsewhere, I compacted Phacility free instances onto a small number of database servers. I attempted to "fill up" these servers, and succeeded, sort of, but maybe too much -- they eventually fall over and need to be rebooted.