Page MenuHomePhabricator

Can't stop daemons if MySQL is down
Closed, ResolvedPublic

Description

I have a VM that I am using for dev purposes and it has very little RAM at the moment. As a result, sometimes MySQL just dies completely. What's worse is that (when this happens) I can't seem to be able to stop the daemons with ./bin/phd stop. They can be killed with the kill command, but this is slightly more difficult.

> ./bin/phd stop
[2015-01-01 14:51:11] EXCEPTION: (AphrontConnectionQueryException) Attempt to connect to root@localhost failed with error #2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2). at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:72]
  #0 AphrontMySQLiDatabaseConnection::connect() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:97]
  #1 AphrontBaseMySQLDatabaseConnection::establishConnection() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:120]
  #2 AphrontBaseMySQLDatabaseConnection::requireConnection() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:15]
  #3 AphrontMySQLiDatabaseConnection::escapeBinaryString(string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:11]
  #4 AphrontMySQLiDatabaseConnection::escapeUTF8String(string) called at [<phutil>/src/xsprintf/qsprintf.php:170]
  #5 xsprintf_query(AphrontMySQLiDatabaseConnection, string, integer, string, integer) called at [<phutil>/src/xsprintf/xsprintf.php:63]
  #6 xsprintf(string, AphrontMySQLiDatabaseConnection, array) called at [<phutil>/src/xsprintf/qsprintf.php:64]
  #7 qsprintf(AphrontMySQLiDatabaseConnection, string, string, string, integer, integer, string)
  #8 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:5]
  #9 queryfx(AphrontMySQLiDatabaseConnection, string, string, string, integer, integer, string)
  #10 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:16]
  #11 queryfx_all(AphrontMySQLiDatabaseConnection, string, string, string, integer, integer, string)
  #12 call_user_func_array(string, array) called at [<phutil>/src/aphront/storage/connection/AphrontDatabaseConnection.php:20]
  #13 AphrontDatabaseConnection::queryData(string, string, string, integer, integer, string)
  #14 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:509]
  #15 LiskDAO::loadRawDataWhere(string, string, integer, integer)
  #16 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:473]
  #17 LiskDAO::loadOneWhere(string, string, integer, integer) called at [<phabricator>/src/infrastructure/daemon/control/PhabricatorDaemonReference.php:39]
  #18 PhabricatorDaemonReference::newFromDictionary(array) called at [<phabricator>/src/infrastructure/daemon/control/PhabricatorDaemonReference.php:22]
  #19 PhabricatorDaemonReference::newFromFile(string) called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php:48]
  #20 PhabricatorDaemonManagementWorkflow::loadRunningDaemons() called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php:368]
  #21 PhabricatorDaemonManagementWorkflow::executeStopCommand(array, integer, boolean) called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementStopWorkflow.php:40]
  #22 PhabricatorDaemonManagementStopWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:394]
  #23 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:290]
  #24 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/daemon/manage_daemons.php:23]
root@phabricator:/usr/src/phabricator# ./bin/phd status
^[[6;7~[2015-01-01 14:51:20] EXCEPTION: (AphrontConnectionQueryException) Attempt to connect to root@localhost failed with error #2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2). at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:72]
  #0 AphrontMySQLiDatabaseConnection::connect() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:97]
  #1 AphrontBaseMySQLDatabaseConnection::establishConnection() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:120]
  #2 AphrontBaseMySQLDatabaseConnection::requireConnection() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:15]
  #3 AphrontMySQLiDatabaseConnection::escapeBinaryString(string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:11]
  #4 AphrontMySQLiDatabaseConnection::escapeUTF8String(string) called at [<phutil>/src/xsprintf/qsprintf.php:170]
  #5 xsprintf_query(AphrontMySQLiDatabaseConnection, string, integer, string, integer) called at [<phutil>/src/xsprintf/xsprintf.php:63]
  #6 xsprintf(string, AphrontMySQLiDatabaseConnection, array) called at [<phutil>/src/xsprintf/qsprintf.php:64]
  #7 qsprintf(AphrontMySQLiDatabaseConnection, string, string, string, integer, integer, string)
  #8 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:5]
  #9 queryfx(AphrontMySQLiDatabaseConnection, string, string, string, integer, integer, string)
  #10 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:16]
  #11 queryfx_all(AphrontMySQLiDatabaseConnection, string, string, string, integer, integer, string)
  #12 call_user_func_array(string, array) called at [<phutil>/src/aphront/storage/connection/AphrontDatabaseConnection.php:20]
  #13 AphrontDatabaseConnection::queryData(string, string, string, integer, integer, string)
  #14 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:509]
  #15 LiskDAO::loadRawDataWhere(string, string, integer, integer)
  #16 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:473]
  #17 LiskDAO::loadOneWhere(string, string, integer, integer) called at [<phabricator>/src/infrastructure/daemon/control/PhabricatorDaemonReference.php:39]
  #18 PhabricatorDaemonReference::newFromDictionary(array) called at [<phabricator>/src/infrastructure/daemon/control/PhabricatorDaemonReference.php:22]
  #19 PhabricatorDaemonReference::newFromFile(string) called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php:48]
  #20 PhabricatorDaemonManagementWorkflow::loadRunningDaemons() called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php:55]
  #21 PhabricatorDaemonManagementWorkflow::loadAllRunningDaemons() called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementStatusWorkflow.php:25]
  #22 PhabricatorDaemonManagementStatusWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:394]
  #23 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:290]
  #24 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/daemon/manage_daemons.php:23]

Event Timeline

joshuaspence raised the priority of this task from to Needs Triage.
joshuaspence updated the task description. (Show Details)
joshuaspence added a project: Daemons.
joshuaspence updated the task description. (Show Details)
joshuaspence added a subscriber: joshuaspence.

I think it should loudly complain MySQL is down. Optionally / maybe not a good idea, it can include instructions on how to kill the daemons otherwise and then clean up the MySQL entries that will (likely) be there when MySQL restarts.

I think its reasonable to require MySQL to be up? I mean, we don't think the website should also work when MySQL is down, right?

The site doesn't work at all with mysql down (and it doesn't look pretty
either)

We could add a --force mode which would kill the daemons regardless of the status of MySQL. That seems like a reasonable compromise.