Found one, creating a new project returns:

Found one, creating a new project returns:

I suspect a lot of behavior is still broken, but at least some things should be working now. I'm working on restoring SSH access to repositories (which I suspect is just an ops/config issue) and then I'll fix whatever shows up in the error log.

We are live! Thank you Evan for your relentless work. 🙏

• Restoring MySQL from backup took a very long time (partly because I didn't realize the volume was an ancient gp2 volume with a tight limit on iops) but succeeded.
• admin is back up and cluster directory services are mostly working.
• I'm working on redeploying web now; this tier is stateless and doesn't need to load backups.
• The most recent available backup was slightly out of date (Oct 16th IIRC). If this caused issues (e.g., double-billing or deleting recently-invited user accounts) I'll sort everything out once services are back up.

One issue I'm struggling with is that some daemon parsing is failing...

One issue I'm struggling with is that some daemon parsing is failing:

Actually, Phabricator uses static::class since D21744 and it works in older PHP per 3v4l.org, so I think I'm incorrect that this requires PHP 8.

See T13637. See T13588. In PHP 8.1, the behavior of static variables in methods changed, per this snippet from the PHP Documentation:

I do want $array[null] to kill the server in the general case so all the high-quality Phabricator code I write in the future can't do it, but I realize I may need to concede and silence the error. I am going to give up on this one:

also maybe copy https://we.phorge.it/D25387, which makes it so deprecation
messages don't kill the server.

• AWS now has over 800 instance types???
• I believe I've launched a new admin with a patched PHP that is surviving a least some of the deployment process. MySQL can't upgrade databases across major versions, so I'm restoring from backup now.
• @jmeador oh hi there
• @avivey Yeah, like half the pages are working for me while logged in but things are still pretty rough.

Hi Evan,

A wild epriestley appears!

Typical state-of-the-art software upgrade process:

General state of affairs:

Recent versions of Ubuntu default to a sudo implemented in Rust which doesn't support -E to pass the entire environment. A quick fix for this is to select sudo.ws as an alternative:

There are some remaining non-security bugs with this that I'll follow up on in T13682. I believe the security side of this is now resolved.

The details of this attack will be disclosed at a later date, once installs have had some sort of plausible chance to upgrade.

I believe D21811 covers this completely.

Just for visibility, this is I believe the change that broke Diffusion (which was fixed in rP52df4ff515b7), where the error message is something like

I believe these were all hunted down.

I deployed this everywhere in the Phacility cluster yesterday and things have been quiet, so I'm assuming it worked until evidence arises to the contrary.

D21756 effectively makes all Git pathways call setSudoAsDaemon(true).

Just for visibility, the error messages you'll see if you're affected by this issue look something like this:

...maybe this is an actual bug in Phabricator where some pathways are just missing the "sudo" wrapper?

I'm satisfied that we aren't violating our commitment to our customers by continuing to use Mailgun as a service provider...

Yes. I closed down registration on this install (secure.phabricator.com) several years ago because the overwhelming majority of users who registered accounts here didn't read or follow the rules. Access to secure.phabricator.com is now invite-only.

In T13589#254320, @epriestley wrote:

Please use Discourse to report bugs.

Please use Discourse to report bugs. See https://discourse.phabricator-community.org/t/repository-view-git-command-failed-error/4510/.

It works with Git 2.1.4 (shipped with Debian Wheezy), but not with Git 2.20.1 (shipped with Debian Buster), or Git 2.30.0 (latest version).

My apologies if this is not the right place to post about this, but seems like due to ea9cb0b625fb6922c45aecbfdebacc60788ed92d we now get following error message when visiting diffusion repository page, i.e. URL /diffusion/$REPOID/:

This could be made slightly cleaner with a setSummary() to set a shorter summary:

hmmmmmm