Page MenuHomePhabricator

When the search table is marked as crashed, bin/storage should automatically repair it
Closed, WontfixPublic

Description

As far as I know, the search database / table is the only table that can crash because it uses a different table type (InnoDB instead of MyISAM or something?) When MySQL marks it as crashed for whatever reason, bin/storage should automatically run REPAIR TABLE phabricator_search.search_documentfield.

I occasionally see this when restarting my Phabricator server, and it causes the launch to fail because bin/storage upgrade fails with:

Apr 11 03:20:51 vps docker[6571]: [2016-04-11 13:20:51] EXCEPTION: (AphrontQueryException) #145: Table './phabricator_search/search_documentfield' is marked as crashed and should be repaired at
Apr 11 03:20:51 vps docker[6571]: arcanist(head=hachquemaster, ref.master=737f5c0df976, ref.hachquemaster=9b8c9d280ea4), phabricator(head=hachquemaster, ref.master=146fb646f92b, ref.hachquemast
Apr 11 03:20:51 vps docker[6571]: #0 AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDataba
Apr 11 03:20:51 vps docker[6571]: #1 AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.
Apr 11 03:20:51 vps docker[6571]: #2 AphrontBaseMySQLDatabaseConnection::executeRawQuery(string) called at [<phutil>/src/xsprintf/queryfx.php:6]
Apr 11 03:20:51 vps docker[6571]: #3 queryfx(AphrontMySQLiDatabaseConnection, string, string, string)
Apr 11 03:20:51 vps docker[6571]: #4 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:11]
Apr 11 03:20:51 vps docker[6571]: #5 queryfx_all(AphrontMySQLiDatabaseConnection, string, string, string) called at [<phabricator>/src/applications/config/schema/PhabricatorConfigSchemaQuery.ph
Apr 11 03:20:51 vps docker[6571]: #6 PhabricatorConfigSchemaQuery::loadActualSchema() called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflo
Apr 11 03:20:51 vps docker[6571]: #7 PhabricatorStorageManagementWorkflow::loadSchemata() called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWor
Apr 11 03:20:51 vps docker[6571]: #8 PhabricatorStorageManagementWorkflow::findAdjustments() called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagement
Apr 11 03:20:51 vps docker[6571]: #9 PhabricatorStorageManagementWorkflow::doAdjustSchemata(boolean) called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageMa
Apr 11 03:20:51 vps docker[6571]: #10 PhabricatorStorageManagementWorkflow::adjustSchemata(boolean) called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageMan
Apr 11 03:20:51 vps docker[6571]: #11 PhabricatorStorageManagementUpgradeWorkflow::didExecute(PhutilArgumentParser) called at [<phabricator>/src/infrastructure/storage/management/workflow/Phabr
Apr 11 03:20:51 vps docker[6571]: #12 PhabricatorStorageManagementWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:410]
Apr 11 03:20:51 vps docker[6571]: #13 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:303]
Apr 11 03:20:51 vps docker[6571]: #14 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/sql/manage_storage.php:178]

For now to workaround it I've modified my launch scripts to run mysqlcheck ... --all-databases before the migration happens, but really I think this should be handled by bin/storage itself because it can run it only against the tables which can require repairs (rather than me running it against everything).

Event Timeline

epriestley claimed this task.
epriestley added a subscriber: epriestley.

We generally stopped seeing this after moving to InnoDB FULLTEXT, which seems less prone to table crashes than MyISAM FULLTEXT was. See also T12819 for the fate of InnoDB FULLTEXT.