Page MenuHomePhabricator
Create Task
Maniphest T5679

Search daemon fails when document title >255 chars
Open, Needs TriagePublic
Actions

Description

Recently our daemons crashed due to this:

P1207 daemon.log
1[2014-07-18 11:39:06] PHLOG: Unable to index document PHID-TASK-tethsf3ggl7b527jd2uu with engine PhabricatorSearchEngineMySQL. at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchDocumentIndexer.php:60]
2 #0 phlog(Unable to index document PHID-TASK-tethsf3ggl7b527jd2uu with engine PhabricatorSearchEngineMySQL.) called at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchDocumentIndexer.php:60]
3 #1 PhabricatorSearchDocumentIndexer::indexDocumentByPHID(PHID-TASK-tethsf3ggl7b527jd2uu) called at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchIndexer.php:20]
4 #2 PhabricatorSearchIndexer::indexDocumentByPHID(PHID-TASK-tethsf3ggl7b527jd2uu) called at [/var/lib/phabricator/phabricator/src/applications/search/worker/PhabricatorSearchWorker.php:10]
5 #3 PhabricatorSearchWorker::doWork() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorWorker.php:85]
6 #4 PhabricatorWorker::executeTask() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:123]
7 #5 PhabricatorWorkerActiveTask::executeTask() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:19]
8 #6 PhabricatorTaskmasterDaemon::run() called at [/var/lib/phabricator/libphutil/src/daemon/PhutilDaemon.php:85]
9 #7 PhutilDaemon::execute() called at [/var/lib/phabricator/libphutil/scripts/daemon/exec/exec_daemon.php:112]
10[2014-07-18 11:39:06] EXCEPTION: (AphrontQueryException) #1406: Data too long for column 'documentTitle' at row 1 at [/var/lib/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:311]
11 #0 phlog(Object AphrontQueryException) called at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchDocumentIndexer.php:61]
12 #1 PhabricatorSearchDocumentIndexer::indexDocumentByPHID(PHID-TASK-tethsf3ggl7b527jd2uu) called at [/var/lib/phabricator/phabricator/src/applications/search/index/PhabricatorSearchIndexer.php:20]
13 #2 PhabricatorSearchIndexer::indexDocumentByPHID(PHID-TASK-tethsf3ggl7b527jd2uu) called at [/var/lib/phabricator/phabricator/src/applications/search/worker/PhabricatorSearchWorker.php:10]
14 #3 PhabricatorSearchWorker::doWork() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorWorker.php:85]
15 #4 PhabricatorWorker::executeTask() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:123]
16 #5 PhabricatorWorkerActiveTask::executeTask() called at [/var/lib/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:19]
17 #6 PhabricatorTaskmasterDaemon::run() called at [/var/lib/phabricator/libphutil/src/daemon/PhutilDaemon.php:85]
18 #7 PhutilDaemon::execute() called at [/var/lib/phabricator/libphutil/scripts/daemon/exec/exec_daemon.php:112]

The task PHID-TASK-tethsf3ggl7b527jd2uu had a very long title - 324 chars long. I haven't looked into the code, but quick glance at db schema showed that maniphest_task.originalTitle field is of type TEXT, while search_document.documentTitle is a VARCHAR(255). I guess this might be the culprit.

Now, I haven't worked with Phabricator migrations and the search infrastructure, so I didn't fix it myself, but I hope this helps if someone else encounters similar problem or knows more about relevant code.

It seems that changing Maniphest task name to a short one and restarting daemon remedied the problem.

Overall, I'd say we hit a quite rare edge case here.

Related Objects

Mentioned Here
P1207 daemon.log

Event Timeline

seporaitis created this task.Jul 22 2014, 12:59 PM
seporaitis raised the priority of this task from to Needs Triage.
seporaitis updated the task description. (Show Details)
seporaitis added a project: Maniphest.
seporaitis added subscribers: seporaitis, epriestley.
joshuaspence added a subscriber: joshuaspence.Jul 22 2014, 2:15 PM
epriestley added a comment.Jul 22 2014, 3:26 PM

Merged https://github.com/phacility/phabricator/issues/678 here.

I think we can just make the documentTitle column a LONGTEXT.

epriestley added a subscriber: btrahan.Aug 6 2014, 8:19 PM

This is probably just a one-line SQL statement to make the documentTitle column on the search index table LONGTEXT. A reasonable test plan would be:

  • Create a task with >255 characters in the title.
  • Use bin/search index to explicitly reindex it and make sure there are no warnings/errors.
  • Search for a word appearing more than 255 characters into the title and make sure we get a hit.
epriestley added a project: Badge Awarded.Aug 6 2014, 8:19 PM
epriestley assigned this task to lpriestley.Aug 12 2014, 3:03 PM

Specifically:

  • Create a task with >255 characters in the title.
  • Run bin/search index Tnnn and you should reproduce the original error.
  • Add a SQL patch to resources/sql/autopatches/ to make search_document.documentTitle a LONGTEXT.
  • Run bin/storage upgrade to apply it.
  • Follow the test plan above to verify it fixes the issue.
epriestley removed lpriestley as the assignee of this task.Nov 8 2014, 12:46 AM

I think we can actually just delete documentTitle outright -- it is only used in one place, as far as I can tell.

epriestley removed a project: Badge Awarded.Nov 12 2014, 12:49 AM
epriestley merged a task: T12513: Failed to index task with too long title.Apr 6 2017, 1:48 PM
epriestley added a subscriber: eFrane.
Herald added a subscriber: eadler. · View Herald TranscriptApr 6 2017, 1:48 PM
epriestley moved this task from Backlog to vNext on the Maniphest board.Feb 20 2019, 11:28 PM
Phabricator · Built by Phacility