Piledriver sequence is roughly:

Found one, creating a new project returns:

Found one, creating a new project returns:

I suspect a lot of behavior is still broken, but at least some things should be working now. I'm working on restoring SSH access to repositories (which I suspect is just an ops/config issue) and then I'll fix whatever shows up in the error log.

We are live! Thank you Evan for your relentless work. 🙏

• Restoring MySQL from backup took a very long time (partly because I didn't realize the volume was an ancient gp2 volume with a tight limit on iops) but succeeded.
• admin is back up and cluster directory services are mostly working.
• I'm working on redeploying web now; this tier is stateless and doesn't need to load backups.
• The most recent available backup was slightly out of date (Oct 16th IIRC). If this caused issues (e.g., double-billing or deleting recently-invited user accounts) I'll sort everything out once services are back up.

One issue I'm struggling with is that some daemon parsing is failing...

One issue I'm struggling with is that some daemon parsing is failing:

Actually, Phabricator uses static::class since D21744 and it works in older PHP per 3v4l.org, so I think I'm incorrect that this requires PHP 8.

See T13637. See T13588. In PHP 8.1, the behavior of static variables in methods changed, per this snippet from the PHP Documentation:

I do want $array[null] to kill the server in the general case so all the high-quality Phabricator code I write in the future can't do it, but I realize I may need to concede and silence the error. I am going to give up on this one:

also maybe copy https://we.phorge.it/D25387, which makes it so deprecation
messages don't kill the server.

• AWS now has over 800 instance types???
• I believe I've launched a new admin with a patched PHP that is surviving a least some of the deployment process. MySQL can't upgrade databases across major versions, so I'm restoring from backup now.
• @jmeador oh hi there
• @avivey Yeah, like half the pages are working for me while logged in but things are still pretty rough.

Hi Evan,

A wild epriestley appears!

Typical state-of-the-art software upgrade process:

General state of affairs:

Recent versions of Ubuntu default to a sudo implemented in Rust which doesn't support -E to pass the entire environment. A quick fix for this is to select sudo.ws as an alternative:

(Of course, it'll probably just work the first time now...)

The export process is already robust at a coarse level: the dump is retained on disk and the process can be retried at the "upload the whole file again" level, then picked up with bin/host export using the --database or --database-file flags (probably with --keep-file).

The (anonymized) error the process encountered occurred while transferring the dump to central storage was: