Just for visibility, the error messages you'll see if you're affected by this issue look something like this:

...maybe this is an actual bug in Phabricator where some pathways are just missing the "sudo" wrapper?

Elsewhere, I compacted Phacility free instances onto a small number of database servers. I attempted to "fill up" these servers, and succeeded, sort of, but maybe too much -- they eventually fall over and need to be rebooted.

This has some rough edges that I'm not going to deal with for now:

Will this work will with Phabricator?

We came up with a flow in Phorge we think would work well, to progress this task (https://we.phorge.it/E1):

• We'll have a bunch of diffs, on a per-application basis, along the style established here and in D21712.
• We'll review each diff on the Phorge
• When it matures, we'll copy it here for for your review and landing in Phabricator.

Ah thank you!

GitHub was refusing mirror pushes with a credential issue (see also T12896, vaguely). I'm not entirely sure which side of things changed, but the config was set up in a previous-generation way anyway (through a "bot" account instead of a "Deploy Key") so I updated things. Looks like mirroring is back online now and this change has propagated.

@epriestley - the fix for this landed in the master branch but it doesn't look like that has carried over yet into the GitHub mirror which is used as the install location for most users, and is the current recommendation on https://secure.phabricator.com/book/phabricator/article/arcanist_quick_start/. Is there a delay before the mirror observes the changes or does something need poked for it to propagate?

See T12847. All the technical parts of this are now solved except for billing, but since Phacility is winding down I no longer plan to pursue it.

I resolved this in rCORE320b2854.

After T13630:

Only one instance was impacted by this and I just credited them until 2099. I don't currently expect to pursue this.

I no longer expect to pursue this.

• Hosts in the repo class are now build by Piledriver (see T13630), which automatically creates the rbak device entries, so this error isn't likely to occur again.
• I also don't expect to launch any more hosts.

I compacted secure onto new hardware (T13671) and shut down saux001 ("Land Revision") and sbuild001 (Harbormaster remote builds). I think all the remaining work is covered under T13630 (largely, just a handful of large database migrations remain).

I just swapped configs over without merging the LBs, since it wasn't immediately obvious to me what the Application vs Classic state of the world is and swapping was good enough.

The aphlict/notify stuff still needs to be tweaked. I think the snlb + slb setup can be merged into a single slb with "TCP (Secure)" forwarding now.

Databases are moved and secure is out of read-only mode. I'm going to adjust repository configuration, then I should be able to tear down secure001.

I'm going to put secure back into read-only mode now and move the databases to the new host.

I brought up the new host and pointed the slb001 load balancer at it. The database is still on the old host, and the new host doesn't have repositories yet, but the basics seem to be working.

Merging 003 into 001 worked fine with a few expected tricks (e.g., when secure is in read-only mode, you can't push a change to take it out of read-only mode, since pushing is a write). Next up is launching a modern m4.large secure-pool host and then migrating the data.

I'm putting secure into read-only mode now, with the intent of completing steps 1-5 above.

PHP 8.1 changed semantics of static variables in methods. See:

See also T13588. Using static::class to work around changes to the scope of static in PHP 8.1 is also possible now that PHP 5.5 is the minimum version, and some uses of get_class($this) might be replaceable. There's no strong motivator for this, it's just a little cleaner.

See also T13588.

It would still be nice to have this from a completeness/correctness perspective, but other changes have made it less valuable:

...possibly, but not conclusively, a result of an SSH version upgrade...

I put all the database migration stuff everywhere and it appears stable. I'm hooking up Postmark as an outbound pathway now. If I get that working, I'll let it sit for a while and start migrating databases.

Finally, there are other some MySQL version issues which can be avoided with:

The new core/ support for the API is partially deployed; the new services/ support isn't anywhere yet.

See also T13232. Here's a possible build strategy for PHP 8.1 under macOS Mojave. Anyone reasonable should probably give up and use brew instead.

The latest version of Phabricator itself is everywhere.

I'm going to hold it until the weekend and try deploying then if things look calm on my end.

I'm satisfied that we aren't violating our commitment to our customers by continuing to use Mailgun as a service provider...

Purely venting, but the advanced version of "click here to schedule a mysterious meeting" is to outright lie -- pretending that you deeply respect the recipient's achievements -- before asking them to schedule a mysterious meeting.

While waiting to deploy db stuff, I was planning to look at pruning dead data out of S3 -- but, on closer examination, the total S3 bill is something like $1/day, so no priority on that whatsoever.

Piledriver also needs to be able to provision database hosts, but these are more-or-less a trivial subset of repository hosts.

• Make InstancesStateQuery use a dictionary when building the database ref information internally.

Piledriver also needs to be able to provision database hosts, but these are more-or-less a trivial subset of repository hosts.

I completed all the repository migrations over the weekend and seemingly haven't run into any issues.

This appears resolved: the workflow now tests that /proc/meminfo reports an appropriate value for TotalSwap.

Just for completeness, vault used to be an HAProxy host serving as an SSH load balancer, but this responsibility moved to lb001 once ELBs became able to listen on inbound port 22 and TCP forward, so there is no longer a vault class of machines.

The new provisioning process for repository shards is:

Piledriver was built before the FutureGraph stuff settled in T11968; it runs into the same general set of sequencing problems and yield would likely be a good approach.

I can't figure out how to delete...

I got rid of everything I could, and nothing appears to be affected.

We have a lot of leftover VPC cruft that I'm going to nuke, notably meta and admin VPCs that (as far as I can tell) have nothing in them, and then a bunch of subnets (meta.private-a, meta.private-b, block-public-222, admin.public-a, admin.public-b, meta.public-a, meta.public-b, block-private-3) and some NGWs etc. I'm like 99% sure this stuff is all leftover from testing years ago and nothing depends on it, but I guess we'll see what happens when I delete all of it.

Here's the last known state of the world from T12816:

See also NAT carryover from T12816, via T13542.

Closing this in favor of T13630, which covers the same ground.

I'm planning to simply delete the Discourse forum without preserving any content.