Page MenuHomePhabricator

Phacility export process removes data from S3 which may still be referenced
Closed, ResolvedPublic

Description

The export process uses bin/files migrate to migrate file data to blob storage on the aux tier. As a side effect, unreferenced files have their source data destroyed.

For files in S3, this can destroy the live copy of the file data. If the export subsequently fails, this can destroy the only copy of the data.

To fix this, I plan to give bin/files migrate some sort of --idempotent flag to avoid removing data.

Revisions and Commits

Restricted Differential Revision
Restricted Differential Revision

Event Timeline

epriestley added a revision: Restricted Differential Revision.Sep 6 2016, 2:37 PM
epriestley added a commit: Restricted Diffusion Commit.Sep 6 2016, 8:54 PM
epriestley added a commit: Restricted Diffusion Commit.

I cherry-picked those fixes to stable, created a test instance (test-on6dfrmgkt7t), uploaded a 100KB file which went to S3, exported the instance, and got a clean export with the file data retained on the instance.

I repeated the export process without issues, and verified the file still functioned correctly on the instance after multiple exports.