Page MenuHomePhabricator

Queueing search indexers is very slow
Closed, ResolvedPublic


I am currently rebuilding our search index as we are moving to ElasticSearch. One thing I noticed is that the reindexing is very slow... it seems that operations are queued only slightly faster than they are being processed. The command has been running for roughly 36 minutes and has processed 14.2% of all indexes, which puts it at a rate of around 12 objects per second.

> ./bin/search index --all --background
Indexing 12 object of type CONP.
Indexing 20791 object of type DREV.
Indexing 24566 object of type TASK.
Indexing 4 object of type CDTL.
Indexing 132 object of type PROJ.
Indexing 146602 object of type CMIT.
Indexing 290 object of type USER.
Indexing 1 object of type MOCK.
Indexing 17 object of type WIKI.
[=========-                                                        ]  14.2%

After this 36 minutes, there are less than 10 tasks sitting in "Next in Queue".

Event Timeline

joshuaspence raised the priority of this task from to Needs Triage.
joshuaspence updated the task description. (Show Details)
joshuaspence added a project: Search.
joshuaspence added a subscriber: joshuaspence.

It might be worth noting that I am running the ./bin/search index command on the same machine that is running our daemons.

Is this different you think than T8306

Sorta... My issue isn't that indexing ElasticSearch is slow, rather my issue is that queueing tasks into the daemons seems to be slow (this shouldn't be touching ElasticSearch, also I believe this issue existed when we were using MySQL indexes as well).

Here's a profile:

Nothing particularly interesting... most of the time is spent in MySQL. This was from running ./bin/search index --all --background --type PROJ.

joshuaspence claimed this task.

Pretty sure this was fixed by D13519.