Page MenuHomePhabricator

Daemon crashing due to uncaught exception during database transactions
Closed, ResolvedPublic

Description

PhabricatorTaskmasterDaemon is crashing when doing a PhabricatorRepositoryGitCommitMessageParserWorker job. This job is continuously retried and fails forever with:

[2015-10-27 17:59:26] EXCEPTION: (PhutilProxyException) Error while executing Task ID 809104. {>} (AphrontQueryException) #1153: Got a packet bigger than 'max_allowed_packet' bytes at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:311]
arcanist(head=master, ref.master=3308da5f8fbe), phabricator(head=master, ref.master=d92f7a14739a), phutil(head=master, ref.master=59f5a8d2bb82)
  #0 <#2> AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:275]
  #1 <#2> AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:181]
  #2 <#2> AphrontBaseMySQLDatabaseConnection::executeRawQuery(string) called at [<phutil>/src/xsprintf/queryfx.php:6]
  #3 <#2> queryfx(AphrontMySQLiDatabaseConnection, string, string, string, array, string)
  #4 <#2> call_user_func_array(string, array) called at [<phutil>/src/aphront/storage/connection/AphrontDatabaseConnection.php:28]
  #5 <#2> AphrontDatabaseConnection::query(string, string, string, array, string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1261]
  #6 <#2> LiskDAO::insertRecordIntoDatabase(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1106]
  #7 <#2> LiskDAO::insert() called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1075]
  #8 <#2> LiskDAO::save() called at [<phabricator>/src/applications/differential/storage/DifferentialModernHunk.php:87]
  #9 <#2> DifferentialModernHunk::save() called at [<phabricator>/src/applications/differential/storage/DifferentialChangeset.php:92]
  #10 <#2> DifferentialChangeset::save() called at [<phabricator>/src/applications/differential/storage/DifferentialDiff.php:120]
  #11 <#2> DifferentialDiff::save() called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:338]
  #12 <#2> PhabricatorRepositoryCommitMessageParserWorker::generateFinalDiff(DifferentialRevision, string) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:219]
  #13 <#2> PhabricatorRepositoryCommitMessageParserWorker::updateCommitData(DiffusionCommitRef) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryGitCommitMessageParserWorker.php:11]
  #14 <#2> PhabricatorRepositoryGitCommitMessageParserWorker::parseCommitWithRef(PhabricatorRepository, PhabricatorRepositoryCommit, DiffusionCommitRef) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:40]
  #15 <#2> PhabricatorRepositoryCommitMessageParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:44]
  #16 <#2> PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
  #17 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
  #18 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
  #19 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:183]
  #20 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:125]

Also in the daemon logs:

FAIL	Tue, Oct 27	5:59 PM	Process exited with error 255
STDE	Tue, Oct 27	5:59 PM	[2015-10-27 17:59:26] EXCEPTION: (PhutilProxyException) Error while executing Task ID 809104. {>} (AphrontQueryException) #1153: Got a packet bigger than 'max_allowed_packet' bytes at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:311]
arcanist(head=master, ref.master=3308da5f8fbe), phabricator(head=master, ref.master=d92f7a14739a), phutil(head=master, ref.master=59f5a8d2bb82)
  #0 <#2> AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:275]
  #1 <#2> AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:181]
  #2 <#2> AphrontBaseMySQLDatabaseConnection::executeRawQuery(string) called at [<phutil>/src/xsprintf/queryfx.php:6]
  #3 <#2> queryfx(AphrontMySQLiDatabaseConnection, string, string, string, array, string)
  #4 <#2> call_user_func_array(string, array) called at [<phutil>/src/aphront/storage/connection/AphrontDatabaseConnection.php:28]
  #5 <#2> AphrontDatabaseConnection::query(string, string, string, array, string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1261]
  #6 <#2> LiskDAO::insertRecordIntoDatabase(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1106]
  #7 <#2> LiskDAO::insert() called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1075]
  #8 <#2> LiskDAO::save() called at [<phabricator>/src/applications/differential/storage/DifferentialModernHunk.php:87]
  #9 <#2> DifferentialModernHunk::save() called at [<phabricator>/src/applications/differential/storage/DifferentialChangeset.php:92]
...
Show 11 more line(s)...
STDO	Tue, Oct 27	5:59 PM	<Malformed>   thrown in /Users/osystem/Sites/phabricator/libphutil/src/aphront/storage/connection/AphrontDatabaseTransactionState.php on line 78
STDO	Tue, Oct 27	5:59 PM	<Malformed> #1 {main}
STDO	Tue, Oct 27	5:59 PM	<Malformed> #0 [internal function]: AphrontDatabaseTransactionState->__destruct()
STDO	Tue, Oct 27	5:59 PM	<Malformed> Stack trace:
STDO	Tue, Oct 27	5:59 PM	<Malformed> Fatal error: Uncaught exception 'Exception' with message 'Process exited with an open transaction! The transaction will be implicitly rolled back. Calls to openTransaction() must always be paired with a call to saveTransaction() or killTransaction().' in /Users/osystem/Sites/phabricator/libphutil/src/aphront/storage/connection/AphrontDatabaseTransactionState.php:78
STDO	Tue, Oct 27	5:59 PM	<Malformed> 
INIT	Tue, Oct 27	5:54 PM	Starting process.

Event Timeline

jcarrillo7 updated the task description. (Show Details)
jcarrillo7 added projects: Daemons, Database.
jcarrillo7 updated the task description. (Show Details)
jcarrillo7 added subscribers: jcarrillo7, epriestley.

What is your max_allowed_packet set to in MySQL?

Ah that was it. I increased that and also had to increase the innodb_log_file_size.