HomePhabricator

Support storage of Differential hunk data in Files

Description

Support storage of Differential hunk data in Files

Summary:
Ref T12932. For long-lived installs, one of the largest tables tends to be the hunk data table. Although it doesn't grow tremendously fast, it's also well suited to storage in Files instead of the database (infrequent access, relatively large blobs of data, mostly one-at-a-time access), and earlier work anticipated eventually adding support for Files storage.

Make Files storage work, and provide bin/differential migrate-hunk to manually test/migrate hunks. This is currently the only way hunks get moved to file storage, but I expect to add a GC step which moves them to File storage after 30 days shortly.

The immediate motivation for this is to relieve storage pressure on db001/db002 so we have more headroom for deploying the Ferret engine and its larger indexes (see also T12819).

Test Plan:

  • Used bin/differential migrate-hunk to move a hunk to and from file storage, verified it survived intact.
  • Downloaded the actual stored file, sanity-checked it. Verified permissions.
  • Destroyed a diff with bin/remove destroy, saw the hunk and file storage destroyed.
  • Verified that going from file -> text destroys the old file properly with migrate-hunk --trace ....

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12932

Differential Revision: https://secure.phabricator.com/D18584