Page MenuHomePhabricator

Add `bin/files compact` for sharing file data storage
ClosedPublic

Authored by epriestley on Aug 21 2014, 5:58 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Dec 15, 3:02 AM
Unknown Object (File)
Wed, Dec 11, 2:22 PM
Unknown Object (File)
Thu, Dec 5, 2:16 AM
Unknown Object (File)
Nov 19 2024, 10:07 AM
Unknown Object (File)
Nov 5 2024, 5:31 AM
Unknown Object (File)
Nov 5 2024, 5:31 AM
Unknown Object (File)
Oct 25 2024, 12:37 PM
Unknown Object (File)
Oct 24 2024, 10:00 AM
Subscribers

Details

Summary

Fixes T5912. When we write files, we attempt to share storage if two files have the same content.

In some cases, we may not share storage. Examples include:

  • Files migrated with bin/files migrate (it's simpler not to try to dedupe them).
  • Old files, from before storage was sharable (the mechanism did not exist).
  • Files broken by the bug fixed in T5912.

Add a script to compact files by pointing files with the same content hash at the same file contnet.

In the particular case of files broken by the bug in T5912, we know the hash of the file's content and will only point them at a file that we can load the data for, so this fixes them.

Compaction is not hugely useful in general, but this script isn't too complex and the ability to fix damage from the bug in T5912 is desirable. We could remove this capability eventually.

Test Plan
  • Ran files compact --all --dry-run and sanity checked a bunch of the duplicates for actually being duplicates.
  • Migrated individual files with files compact Fnnn --trace and verified the storage compacted and all files survived the process.
  • Verified unused storage was correctly destroyed after removing the last reference to it.

Diff Detail

Repository
rP Phabricator
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

epriestley retitled this revision from to Add `bin/files compact` for sharing file data storage.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: btrahan.
btrahan edited edge metadata.
This revision is now accepted and ready to land.Aug 21 2014, 6:36 PM
epriestley updated this revision to Diff 24868.

Closed by commit rPf43355855cb8 (authored by @epriestley).