Although this problem is severe, the absolute size of the leak appears to be small (my task title is hyperbolic). From the Cloudflare disclosure:

Those 770 unique URIs covered 161 unique domains.
https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

Although this probably underestimates the number of potentially impacted URIs and domains, the chance that any Phabricator install was affected whatsoever is still very small given how little total data appears to have leaked.

If you have configured your entire install behind Cloudflare (we do not encourage this in the documentation), you are at greater risk, as anything served by Phabricator could have been leaked. If you believe you are affected and would like help rotating sessions and invalidating credentials, let us know.

My whole instance is behind CloudFlare and I feel 'safe better than sorry', so is there any way to invalidate all passwords and forcing the password reset?

Only our websocket host is behind CF proxies, so I invalidated all auth sessions by changing the security.hmac-key. I wanted to also invalidate all Conduit tokens just as a fire drill but couldn't found a method of doing this.

PASSWORDS

To cycle passwords, you can do this:

mysql> UPDATE phabricator_user.user SET passwordHash = NULL;

You can test this for one user first like this:

mysql> UPDATE phabricator_user.user SET passwordHash = NULL WHERE username = "alice";

You may also want to invalidate sessions (below) to force users who are currently logged in to log out.

Users will need to use "Forgot your password?" to manually reset passwords by email.

VCS PASSWORDS

VCS passwords are used ONLY if you configure diffusion.allow-http-auth, and are used to clone and push repositories over HTTP.

To cycle VCS passwords, do this:

mysql> TRUNCATE TABLE phabricator_repository.repository_vcspassword;

You can test this first for one user like this:

mysql> DELETE FROM phabricator_repository.repository_vcspassword WHERE userPHID = "<user-phid>";

Users will need to go set a new VCS password in Settings.

SESSIONS

You can invalidate all sessions like this:

mysql> TRUNCATE TABLE phabricator_user.phabricator_session;

You can test this for one user first like this:

mysql> DELETE FROM phabricator_user.phabricator_session WHERE userPHID = "<user-phid>";

(You could also cycle security.hmac-key, but I would tend to discourage this because it cycles many other hashes as well.)

CONDUIT TOKENS

You can invalidate all Conduit tokens like this:

mysql> TRUNCATE TABLE phabricator_conduit.conduit_token;

You can test this for one user first like this:

mysql> DELETE FROM phabricator_conduit.conduit_token WHERE objectPIHD = "<user-phid>";

Users will receive a reasonably clear error message instructing them that their token is invalid and that they need to arc install-certificate ... to get a new one.

Only our websocket host is behind CF proxies

By design, no (meaningfully) sensitive information passes through the aplict notification server. All messages it transmits look like this:

Object PHID-XYZQ-... was updated, go ask Phabricator what happened.

These messages only identify objects by PHID or other global identifiers, and contain no information like task titles or human-readable notification body text.

The client then makes an AJAX call to Phabricator using a normal user session to retrieve all details.

An attacker can still theoretically make some observations about when activity occurs, but these are very unlikely to be of any use or interest. Broadly, it would theoretically be safe to hand your aphlict server to an adversary to run on your behalf. It's architected strictly as a dumb message bus which doesn't need authentication because it doesn't carry sensitive information.

You can still reset things, but only lists of PHIDs ever reach the aphlict server, so it is not possible for responses from it to include anything legitimately sensitive.

Has cloudflare explicitly said that only responses were leaked, and never
requests? I recall "POST data" being explicitly called out as showing up in
the leaks, but I can't remember if this was in an official write up or
media coverage.

I ask because you focus on things Phabricator responds with but I'm equally
concerned about things users send to Phabricator being leaked (any
credentials used to access web or api)

Ah, you're right. It looks like request data may also have leaked. I'll edit my earlier posts to reflect that. Briefly, the change is: passwords and VCS passwords are potentially at risk.

We could also provide some kind of bin/auth burn-everything-to-the-ground to automate these steps, but I worry that it would almost never be tested (I think it's very difficult to test comprehensively in an automated way) or run, and there's a good chance it might be forgotten about when authentication changes occur.

Since these kinds of incidents are very rare, we might be worse off the next time this happens in 2022 if we wrote this script, since it would look like it would work but might not actually work (or, worse, might hurt things).

I'd be more comfortable writing and maintaining this if there were routine use cases (e.g. "albert lost his laptop so we need to purge all his credentials") so we wouldn't be jumping off a cliff and trusting a script no one has really run in four years the next time SSL stops working, but we haven't seen any requests like this that I can recall.

Broadly, there is nothing Phabricator could have done differently to anticipate or prevent this issue.

This is also a little unfair: there are at least two reasonable features which could at least mitigate the impact of this issue:

• binding sessions to remote addresses (this has been discussed in various places, although I don't think there's an open task for it anywhere) to limit the ability for attackers to reuse a session token;
• doing Conduit over SSH instead of HTTP (T550).

These mitigations would be relatively small, and I think this attack scenario on its own does very little to motivate either feature, but they would reasonably have reduced exposure here, if only slightly.