Page MenuHomePhabricator

Search daemon fails when document title >255 chars
Open, Needs TriagePublic

Description

Recently our daemons crashed due to this:

1[2014-07-18 11:39:06] PHLOG: Unable to index document PHID-TASK-tethsf3ggl7b527jd2uu with engine PhabricatorSearchEngineMySQL. at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchDocumentIndexer.php:60]
2 #0 phlog(Unable to index document PHID-TASK-tethsf3ggl7b527jd2uu with engine PhabricatorSearchEngineMySQL.) called at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchDocumentIndexer.php:60]
3 #1 PhabricatorSearchDocumentIndexer::indexDocumentByPHID(PHID-TASK-tethsf3ggl7b527jd2uu) called at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchIndexer.php:20]
4 #2 PhabricatorSearchIndexer::indexDocumentByPHID(PHID-TASK-tethsf3ggl7b527jd2uu) called at [/var/lib/phabricator/phabricator/src/applications/search/worker/PhabricatorSearchWorker.php:10]
5 #3 PhabricatorSearchWorker::doWork() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorWorker.php:85]
6 #4 PhabricatorWorker::executeTask() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:123]
7 #5 PhabricatorWorkerActiveTask::executeTask() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:19]
8 #6 PhabricatorTaskmasterDaemon::run() called at [/var/lib/phabricator/libphutil/src/daemon/PhutilDaemon.php:85]
9 #7 PhutilDaemon::execute() called at [/var/lib/phabricator/libphutil/scripts/daemon/exec/exec_daemon.php:112]
10[2014-07-18 11:39:06] EXCEPTION: (AphrontQueryException) #1406: Data too long for column 'documentTitle' at row 1 at [/var/lib/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:311]
11 #0 phlog(Object AphrontQueryException) called at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchDocumentIndexer.php:61]
12 #1 PhabricatorSearchDocumentIndexer::indexDocumentByPHID(PHID-TASK-tethsf3ggl7b527jd2uu) called at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchIndexer.php:20]
13 #2 PhabricatorSearchIndexer::indexDocumentByPHID(PHID-TASK-tethsf3ggl7b527jd2uu) called at [/var/lib/phabricator/phabricator/src/applications/search/worker/PhabricatorSearchWorker.php:10]
14 #3 PhabricatorSearchWorker::doWork() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorWorker.php:85]
15 #4 PhabricatorWorker::executeTask() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:123]
16 #5 PhabricatorWorkerActiveTask::executeTask() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:19]
17 #6 PhabricatorTaskmasterDaemon::run() called at [/var/lib/phabricator/libphutil/src/daemon/PhutilDaemon.php:85]
18 #7 PhutilDaemon::execute() called at [/var/lib/phabricator/libphutil/scripts/daemon/exec/exec_daemon.php:112]

The task PHID-TASK-tethsf3ggl7b527jd2uu had a very long title - 324 chars long. I haven't looked into the code, but quick glance at db schema showed that maniphest_task.originalTitle field is of type TEXT, while search_document.documentTitle is a VARCHAR(255). I guess this might be the culprit.

Now, I haven't worked with Phabricator migrations and the search infrastructure, so I didn't fix it myself, but I hope this helps if someone else encounters similar problem or knows more about relevant code.

It seems that changing Maniphest task name to a short one and restarting daemon remedied the problem.

Overall, I'd say we hit a quite rare edge case here.

Related Objects

Event Timeline

seporaitis raised the priority of this task from to Needs Triage.
seporaitis updated the task description. (Show Details)
seporaitis added a project: Maniphest.
seporaitis added subscribers: seporaitis, epriestley.

Merged https://github.com/phacility/phabricator/issues/678 here.

I think we can just make the documentTitle column a LONGTEXT.

This is probably just a one-line SQL statement to make the documentTitle column on the search index table LONGTEXT. A reasonable test plan would be:

  • Create a task with >255 characters in the title.
  • Use bin/search index to explicitly reindex it and make sure there are no warnings/errors.
  • Search for a word appearing more than 255 characters into the title and make sure we get a hit.

Specifically:

  • Create a task with >255 characters in the title.
  • Run bin/search index Tnnn and you should reproduce the original error.
  • Add a SQL patch to resources/sql/autopatches/ to make search_document.documentTitle a LONGTEXT.
  • Run bin/storage upgrade to apply it.
  • Follow the test plan above to verify it fixes the issue.

I think we can actually just delete documentTitle outright -- it is only used in one place, as far as I can tell.