Page MenuHomePhabricator

Migrate files from MySQL to Local Disk fails with "unable to load MySQL blob file"
Closed, ResolvedPublic

Description

Tried to migrate from mysql blob storage to local disk and got a little over 1200 failures. On investigating this was the trace for one example:

./bin/files migrate --engine local-disk F137 --trace
>>> [2] <connect> phabricator_file
<<< [2] <connect> 2,626 us
>>> [3] <query> SELECT f.* FROM `file` f  WHERE (f.id IN (137)) ORDER BY f.id DESC
<<< [3] <query> 577 us
>>> [4] <query> SELECT edge.* FROM `edge` edge WHERE (edge.src IN ('PHID-FILE-vj7icvfermee5m7qie5a')) AND (edge.type IN ('26')) ORDER BY edge.dateCreated DESC, edge.seq DESC
<<< [4] <query> 579 us
F137: Migrating from 'blob' to 'local-disk'...>>> [5] <query> SELECT * FROM `file_storageblob` WHERE `id` = 7
<<< [5] <query> 656 us
failed!
exception 'Exception' with message 'Unable to load MySQL blob file '7'!' in /home/phabricator/phabricator/src/applications/files/engine/PhabricatorMySQLFileStorageEngine.php:75
Stack trace:
#0 /home/phabricator/phabricator/src/applications/files/engine/PhabricatorMySQLFileStorageEngine.php(49): PhabricatorMySQLFileStorageEngine->loadFromMySQLFileStorage('7')
#1 /home/phabricator/phabricator/src/applications/files/storage/PhabricatorFile.php(479): PhabricatorMySQLFileStorageEngine->readFile('7')
#2 /home/phabricator/phabricator/src/applications/files/storage/PhabricatorFile.php(317): PhabricatorFile->loadFileData()
#3 /home/phabricator/phabricator/src/applications/files/management/PhabricatorFilesManagementMigrateWorkflow.php(82): PhabricatorFile->migrateToEngine(Object(PhabricatorLocalDiskFileStorageEngine))
#4 /home/phabricator/libphutil/src/parser/argument/PhutilArgumentParser.php(394): PhabricatorFilesManagementMigrateWorkflow->execute(Object(PhutilArgumentParser))
#5 /home/phabricator/libphutil/src/parser/argument/PhutilArgumentParser.php(290): PhutilArgumentParser->parseWorkflowsFull(Array)
#6 /home/phabricator/phabricator/scripts/files/manage_files.php(21): PhutilArgumentParser->parseWorkflows(Array)
#7 {main}
Failures!
F137

Runing the queries shown individually shows that the id exists in the file table, but an empty set is returned from the edge table.

I'm not sure how the relationship in question was lost, or perhaps never existed, but am curious if it's possible to migrate these remaining files.

Revisions and Commits

Restricted Differential Revision
Restricted Differential Revision

Event Timeline

webframp raised the priority of this task from to Needs Triage.
webframp updated the task description. (Show Details)
webframp added a project: Files.
webframp added a subscriber: webframp.
epriestley added a revision: Restricted Differential Revision.Aug 20 2014, 10:01 PM
epriestley added a commit: Restricted Diffusion Commit.

Closed by commit rP66fa59d04d7c.

I'm going to write a bin/files compact which will repair this damage and potentially save some storage space.

epriestley added a revision: Restricted Differential Revision.Aug 21 2014, 5:58 PM

If you were affected by this, you should be able to repair the damage after D10327 with:

phabricator/ $ ./bin/files compact --all

You can run this in --dry-run mode first to see what it will do:

phabricator/ $ ./bin/files compact --all --dry-run

If you want to test on a specific file to make sure it works, you can specify a file:

phabricator/ $ ./bin/files compact Fnnn [--dry-run]
epriestley added a commit: Restricted Diffusion Commit.

Closed by commit rPf43355855cb8.

I ran the migration on this install, and the results seems sensible from inspection.

looks like this will help, test ran with ./bin/files compact --all --dry-run --trace

I do get a decent amount of "Unable to load file data; declining to compact." errrors

https://secure.phabricator.com/P1245

That's expected if you were affected by the bug earlier. Those will be repaired (at least, most of them) when you run it without --dry-run