Overview

Beginning with 2017 Week 14, Phabricator computes and stores integrity hashes for files. This integrity check can detect corruption or tampering in stored file data.

Additionally, we have moved away from internal uses of SHA1 in Files now that a SHA1 collision is widely known.

These attacks are not particularly disastrous and require a high level of access to execute (the most likely attacker in both cases is an employee), but installs are encouraged to upgrade.

After upgrading, installs should consider backfilling integrity hashes to protect existing file data. See below for instructions.

Threat Description

By default, files are stored in the main database, and an attacker who can overwrite file data can also perform more powerful attacks (like writing new credentials or sessions, or overwriting policies to give themselves access). However, if files are stored in an external service like Amazon S3, attackers could potentially gain access to the S3 bucket and overwrite file data even if they could not access Phabricator.

For example, an attacker could replace the data for trustworthy.exe with data for a malicious untrustworthy.exe, prompting Phabricator to serve a dangerous executable when the file was next downloaded.

If the attacker also had access to a Phabricator account, they could also previously conduct a padding oracle attack to tamper with data even if encryption was enabled. This attack has some limits, and required both write access to the underlying storage engine (usually S3) and access to a Phabricator account that could read the file (to serve as a padding oracle), so it is unlikely that a state actor storming an AWS datacenter could conduct this attack. However, a compromised employee potentially could.

The padding oracle attack is discussed in more detail in T12470, and was reported to us via HackerOne. See the HackerOne issue for a detailed technical description. This attack is now defused by integrity checking, which stores a digest of file data when files are written and refuses to serve it if the data doesn't match the stored hash when it is read from the underlying storage engine.

Additionally, an internal use of SHA1 potentially allowed attackers with permission to read and write files using the primary interfaces in Files to conduct a similar attack (replacing a trustworthy.exe with an untrustworthy.exe) in certain cases.

The SHA1 collision attack is discussed in more detail in T12464. This attack is now defused by shifting from SHA1 to SHA256.

Integrity checking can also detect mundane corruption caused by cosmic rays or evil spirits.

Upgrading

After upgrading, these changes will take effect automatically for new files.

To protect existing files, backfill hashes for them (see below).

Older versions of arc will still try to use SHA1 hashes in some cases. This may cause more data transfer than necessary, but workflows should succeed without issue (the client will fall back to transferring data normally). Upgrading arc will switch the client to SHA256 and enable it to skip transfer for data blocks known to the server.

Backfilling Old Integrity Hashes

To backfill integrity hashes for old files, run this command after upgrading:

phabricator/ $ ./bin/files integrity --compute --all

This command can be run while Phabricator is online and serving requests to users.

This command will iterate over all existing files and compute integrity hashes for them. Note that if an adversary has already infiltrated your storage engine and poisoned file data it will not be detected, but you'll be no worse off than you were before.

The new bin/files integrity workflow supports some other maintenance operations, which can be reviewed with: bin/files help integrity

Cutting Edge Installs

If you put a commit between rP63828f5806 (April 5, around 9:00 AM PDT) and rPd450a08890 (April 6, around 8:30 AM PDT) into production, your install may have computed some HMAC+SHA1 integrity hashes instead of HMAC+SHA256 integrity hashes. These changes were never promoted to stable, so you are only possibly affected if you run master and stay very close to HEAD. If you run into issues with this, you can forcibly recompute hashes like this:

phabricator/ $ ./bin/files integrity --overwrite --all

This is potentially dangerous (it could mark a file that has been tampered with as safe) so you should only run this command if you are impacted.

HMAC+SHA1

Although there is no immediate threat to use of SHA1 as an HMAC digest algorithm, we are in the process of moving away from HMAC+SHA1 more generally, and some usage sites have been transitioned to HMAC+SHA256. See T12509 for discussion.