Recent versions of Ubuntu default to a sudo implemented in Rust which doesn't support -E to pass the entire environment. A quick fix for this is to select sudo.ws as an alternative:

$ sudo update-alternatives --config sudo

Enumerating the entire environment explicitly is probably challenging (we often pass a bunch of per-VCS stuff which changes across versions and protocols, and I vaguely recall that SVN and Mercurial both have a lot of Env stuff going on at times). Enumerating it automatically by passing every key is possible, although kind of messy.

General state of affairs:

When this was disclosed, all Phabricator/Phacility hosts were running on old versions of Ubuntu that are out of LTS, and PHP 5.4, which I believe there is no official patch for. Broadly, I perform essentially zero maintenance on any of these hosts and probably haven't deployed anything to the Phacility cluster in a couple of years.

I'm electing to fix this forward by upgrading to Ubuntu 26 LTS and bringing Phabricator to PHP 8.5. This is causing some short-term pain with version compatibility issues, since the entire ecosystem is moving forward by five years in one leap. A possible alternative is to backport the patch, but since we're in the brave new world of AI security research I'm not sure that's the wisest course.

The largest issue so far is that PHP 8.5 sensibly forbids use of $array[null]. Phabricator doesn't do this intentionally, but has code paths which do this by accident. There's no way to find these statically and the Phabricator codebase is large.

I have secure up with Ubuntu 26, PHP 8.5, MySQL 8.4, etc. I have functioning deployment scripts, have ported the ForceUser patch to sshd 10.6p1, can push and pull, daemons are running, etc. I believe this is the bulk of the work and that I'm mostly looking at hunting down use of null as an array key now, but I also ran into a lot of surprises yesterday and don't have a clear timeline.

If this drags on too long there are alternatives available that might restore access sooner, including:

• silence and ignore the use of null as an array key; or
• backport the RCE patch to ancient PHP; or
• whitelist customer access to Phacility by IP.

...but these have tradeoffs and kick the can down the road. My desired plan is:

• fix enough of the [null] stuff, and whatever else I stumble on, that Phabricator roughly functions;
• deploy it to the Phacility cluster;
• whack-a-mole whatever I missed.

I believe I'm close to having a "roughly functional" version (e.g., secure seems to be working 90% of the way). Once I run out of obviously-broken stuff to fix I'll land all the support infrastructure changes and begin deploying the Phacility cluster.

Typical state-of-the-art software upgrade process:

Since secure seems to be roughly working, I attempted to upgrade admin and the three web hosts to pick up patched PHP 8.5 so I could bring them back up using the same process I successfully applied to secure yesterday. However, do-release-upgrade seems to have bricked all of them; they come up with 100% CPU utilization and don't respond to attempts to SSH into them.

The web hosts are stateless, and all the admin state is on network volumes, so I can rebuild everything, but it's going to take quite a while -- unless the hosts magically recover, this is a fairly substantial setback.

A wild epriestley appears!

Hi Evan,

just FYI, I can't visit any page in secure while logged in - I get the
"Using null as an array offset" exception.
I can see stuff while being logged-out, so it might be something specific
to my account.

• AWS now has over 800 instance types???
• I believe I've launched a new admin with a patched PHP that is surviving a least some of the deployment process. MySQL can't upgrade databases across major versions, so I'm restoring from backup now.
• @jmeador oh hi there
• @avivey Yeah, like half the pages are working for me while logged in but things are still pretty rough.

also maybe copy https://we.phorge.it/D25387, which makes it so deprecation
messages don't kill the server.

I do want $array[null] to kill the server in the general case so all the high-quality Phabricator code I write in the future can't do it, but I realize I may need to concede and silence the error. I am going to give up on this one:

+    if (preg_match('/^Implicitly marking parameter (\w+) as nullable is deprecated/', $str)) {
+      return;
+    }

..since I don't think there's a way to write code which will run in PHP 5 and PHP 8 and not raise that error.

See T13637. See T13588. In PHP 8.1, the behavior of static variables in methods changed, per this snippet from the PHP Documentation:

<?php
class Foo {
    public static function counter() {
        static $counter = 0;
        $counter++;
        return $counter;
    }
}
class Bar extends Foo {}
var_dump(Foo::counter()); // int(1)
var_dump(Foo::counter()); // int(2)
var_dump(Bar::counter()); // int(3), prior to PHP 8.1.0 int(1)
var_dump(Bar::counter()); // int(4), prior to PHP 8.1.0 int(2)

Phorge fixed one of these in LiskDAO by using static::class, but this requires PHP 8 and I'm hesitant to break compatibility prior to producing a stable release. I believe get_class($this) is generally equivalent and compatible.

Actually, Phabricator uses static::class since D21744 and it works in older PHP per 3v4l.org, so I think I'm incorrect that this requires PHP 8.

One issue I'm struggling with is that some daemon parsing is failing:

[2026-10-22 22:03:48] EXCEPTION: (ConduitClientException) ERR-CONDUIT-CORE: <internal.commit.search> Command failed with error #1!
COMMAND
/usr/bin/sudo -E -n -u ubuntu -- git log -n 1 -n 1 '--encoding=UTF-8' '--format=%e%x00%cn%x00%ce%x00%an%x00%ae%x00%T%x00%at%x00%B%x00~' 3e70d292a51abd0575abfb4723a0aaa087b62837 --

STDOUT
(empty)

STDERR
sudo: a password is required

This worked before, it's all configured the same way, sudoers.d seems fine, and sudo -E -n -u www-data -- sudo -E -n -u ubuntu -- git log works fine.

One issue I'm struggling with is that some daemon parsing is failing...

Recent Ubuntu has a master service override which silently prevents apache from reading sudoers rules. It can be removed via sudo systemctl edit apache2.service.

• Restoring MySQL from backup took a very long time (partly because I didn't realize the volume was an ancient gp2 volume with a tight limit on iops) but succeeded.
• admin is back up and cluster directory services are mostly working.
• I'm working on redeploying web now; this tier is stateless and doesn't need to load backups.
• The most recent available backup was slightly out of date (Oct 16th IIRC). If this caused issues (e.g., double-billing or deleting recently-invited user accounts) I'll sort everything out once services are back up.

We are live! Thank you Evan for your relentless work. 🙏

I suspect a lot of behavior is still broken, but at least some things should be working now. I'm working on restoring SSH access to repositories (which I suspect is just an ops/config issue) and then I'll fix whatever shows up in the error log.

Found one, creating a new project returns:

Found one, creating a new project returns:

Thanks! I think this is fixed, let me know what else you run into.

SSH access to repositories is also working properly for me now.