Page MenuHomePhabricator

Custom Field indexes are rebuilt on every ticket change even if the fields did not
Closed, DuplicatePublic

Description

PhabricatorApplicationTransactionEditor rebuilds the index even if it's a PhabricatorCustomFieldInterface without checking if any of the fields actually canged.

if ($object instanceof PhabricatorCustomFieldInterface) {
  // Maybe this makes more sense to move into the search index itself? For
  // now I'm putting it here since I think we might end up with things that
  // need it to be up to date once the next page loads, but if we don't go
  // there we we could move it into search once search moves to the daemons.

  // It now happens in the search indexer as well, but the search indexer is
  // always daemonized, so the logic above still potentially holds. We could
  // possibly get rid of this. The major motivation for putting it in the
  // indexer was to enable reindexing to work.

  $fields = PhabricatorCustomField::getObjectFields(
    $object,
    PhabricatorCustomField::ROLE_APPLICATIONSEARCH);
  $fields->readFieldsFromStorage($object);
  $fields->rebuildIndexes($object);
}

Query trace from adding a comment on a ticket with 3 custom fields:

		2092550 Query	DELETE FROM `maniphest_customfieldstringindex` WHERE objectPHID = 'PHID-TASK-d6ejmcyax6v3jt6dd7oc' AND indexKey IN ('Z3AXbb.E6AlM', 'gdYAoZw063Tb', 'lZg_rQC.o1qk')
		2092550 Query	INSERT INTO `maniphest_customfieldstringindex` (objectPHID, indexKey, indexValue) VALUES ('PHID-TASK-d6ejmcyax6v3jt6dd7oc', 'Z3AXbb.E6AlM', 'FOO!'), ('PHID-TASK-d6ejmcyax6v3jt6dd7oc', 'gdYAoZw063Tb', 'BAR'), ('PHID-TASK-d6ejmcyax6v3jt6dd7oc', 'lZg_rQC.o1qk', 'BAX')

This:

  • Seems obviously inefficient.
  • I think the potential for multiple identical queries if a ticket is rapidly edited tends to induce a mysql deadlock. But I've only seen that when editing tickets with a script so I don't expect that to be weighted with any particular urgency. D10669 helped with debugging this I was convinced it was just a 'VMs are slow' problem before. I've gone back and forth on if a web ui could trigger this and have settled on "I don't know".
------------------------
LATEST DETECTED DEADLOCK
------------------------
2014-10-09 17:57:42 7fc271e88700
*** (1) TRANSACTION:
TRANSACTION 23087490, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
MySQL thread id 1702130, OS thread handle 0x7fc271b7c700, query id 14670284 localhost root update
INSERT INTO `maniphest_customfieldstringindex` (objectPHID, indexKey, indexValue) VALUES ('PHID-TASK-lolv2ssldmcauaz4hxqb', 'Z3AXbb.E6AlM', 'Web'), ('PHID-TASK-lolv2ssldmcauaz4hxqb', 'gdYAoZw063Tb', 'Milestone Q4 2007'), ('PHID-TASK-lolv2ssldmcauaz4hxqb', 'lZg_rQC.o1qk', 'task')
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 132 page no 464 n bits 232 index `key_join` of table `phabricator_maniphest`.`maniphest_customfieldstringindex` trx id 23087490 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 88 PHYSICAL RECORD: n_fields 4; compact format; info bits 0
 0: len 30; hex 504849442d5441534b2d6c6f706932676a6f73773733713437756f727573; asc PHID-TASK-lopi2gjosw73q47uorus;;
 1: len 12; hex 5a33415862622e4536416c4d; asc Z3AXbb.E6AlM;;
 2: len 6; hex 6f7264657273; asc orders;;
 3: len 4; hex 00008256; asc    V;;

*** (2) TRANSACTION:
TRANSACTION 23087489, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
MySQL thread id 689024, OS thread handle 0x7fc271e88700, query id 14670285 localhost root update
INSERT INTO `maniphest_customfieldstringindex` (objectPHID, indexKey, indexValue) VALUES ('PHID-TASK-lolv2ssldmcauaz4hxqb', 'Z3AXbb.E6AlM', 'Web'), ('PHID-TASK-lolv2ssldmcauaz4hxqb', 'gdYAoZw063Tb', 'Milestone Q4 2007'), ('PHID-TASK-lolv2ssldmcauaz4hxqb', 'lZg_rQC.o1qk', 'task')
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 132 page no 464 n bits 232 index `key_join` of table `phabricator_maniphest`.`maniphest_customfieldstringindex` trx id 23087489 lock_mode X locks gap before rec
Record lock, heap no 88 PHYSICAL RECORD: n_fields 4; compact format; info bits 0
 0: len 30; hex 504849442d5441534b2d6c6f706932676a6f73773733713437756f727573; asc PHID-TASK-lopi2gjosw73q47uorus;;
 1: len 12; hex 5a33415862622e4536416c4d; asc Z3AXbb.E6AlM;;
 2: len 6; hex 6f7264657273; asc orders;;
 3: len 4; hex 00008256; asc    V;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 132 page no 464 n bits 232 index `key_join` of table `phabricator_maniphest`.`maniphest_customfieldstringindex` trx id 23087489 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 88 PHYSICAL RECORD: n_fields 4; compact format; info bits 0
 0: len 30; hex 504849442d5441534b2d6c6f706932676a6f73773733713437756f727573; asc PHID-TASK-lopi2gjosw73q47uorus;;
 1: len 12; hex 5a33415862622e4536416c4d; asc Z3AXbb.E6AlM;;
 2: len 6; hex 6f7264657273; asc orders;;
 3: len 4; hex 00008256; asc    V;;

*** WE ROLL BACK TRANSACTION (2)

Event Timeline

cburroughs raised the priority of this task from to Needs Triage.
cburroughs updated the task description. (Show Details)
cburroughs added a project: Custom Fields.
cburroughs added a subscriber: cburroughs.
chad renamed this task from task custom field indexes are rebuilt on every ticket change even if the fields did not to Custom Field indexes are rebuilt on every ticket change even if the fields did not.Oct 9 2014, 8:44 PM
chad triaged this task as Normal priority.

Another user (ki9a) hit this in IRC too, off API calls, with this trace:

2015/02/24 00:20:12 [error] 9614#0: *53655 FastCGI sent in stderr: "PHP message: [2015-02-24 01:20:11] EXCEPTION: (AphrontDeadlockQueryException) #1213: Deadlock found when trying to get lock; try restarting transaction at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:290]
PHP message:   #0 AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:275]
PHP message:   #1 AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:181]
PHP message:   #2 AphrontBaseMySQLDatabaseConnection::executeRawQuery(string) called at [<phutil>/src/xsprintf/queryfx.php:6]
PHP message:   #3 queryfx(AphrontMySQLiDatabaseConnection, string, string, string, array) called at [<phabricator>/src/infrastructure/customfield/field/PhabricatorCustomFieldList.php:314]
PHP message:   #4 PhabricatorCustomFieldList::rebuildIndexes(ManiphestTask) called at [<phabricator>/src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php:847]
PHP message:   #5 PhabricatorApplicationTransactionEditor::applyTransactions(ManiphestTask, array) called at [<phabricator>/src/applications/maniphest/conduit/ManiphestConduitAPIMethod.php:222]
PHP message:   #6 ManiphestConduitAPIMethod::applyRequest(ManiphestTask, ConduitAPIRequest, boolean) called at [<phabricator>/src/applications/maniphest/conduit/ManiphestUpdateConduitAPIMethod.php:60]
PHP message:   #7 ManiphestUpdateConduitAPIMethod::execute(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/method/ConduitAPIMethod.php:65]
PHP message:   #8 ConduitAPIMethod::executeMethod(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:134]
PHP message:   #9 ConduitCall::executeMethod() called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:84]
PHP message:   #10 ConduitCall:

Broadly, I'm not too concerned about the efficiency of pushing too excessive updates: the big advantage to always writing the entire correct state is that for a large class of bugs, we just fix it and move on and the dataset self-heals without an invasive migration. But we obviously shouldn't be deadlocking.