Page MenuHomePhabricator

Manual Activity: Rebuild Search Index
Open, NormalPublic

Description

After upgrading, Phabricator may direct you to rebuild the search index. This task discusses rebuilding the index in greater depth.

Phabricator uses a fulltext search index to find results that match text queries. Users most commonly issue queries against this index by searching using the global search input in the main menu bar, although some other searches and applications also use the index. After some types of changes to Phabricator (usually aimed at improving the quality of search results) the search index must be rebuilt. Until the index is rebuilt, fulltext searches will not work properly (usually, they will find too few results).

Changes Which Required an Index Rebuild

You can check this table to figure out why you need to do a rebuild:

DateChangeDetails
Nov 25 2016Added Stemming to MySQLSee T11922 for discussion.
Sep 12 2017Ferret fulltext engineSee T12974 for discussion.

Rebuilding the Index

To rebuild the index, run this command:

phabricator/ $ ./bin/search index --force --all --background

This command can be run while Phabricator is running. The --background flag queues the actual indexing for the daemons. After all objects are queued for reindex, you can monitor progress at https://your.phabricator.com/daemon/ in the web UI. As the index is rebuilt, global searches should gradually improve.

This process can take a long time to run if you have a large number of searchable objects (tasks, commits, revisions, etc). Circa November 2016, this install (secure.phabricator.com) required about an hour to rebuild (at the time, we had about 180,000 indexable objects).

Advanced Reindexing

You can omit the --background flag to rebuild the index in the foreground. However, this will generally take longer than using --background with no real benefit.

If your install is large and specific indexes are more important, you can use --type instead of --all to rebuild those indexes first. For example, you could do this to reindex tasks first:

phabricator/  $ ./bin/search index --force --type task --background
phabricator/  $ ./bin/search index --force --all --background

This will do more total work (task indexes will be rebuilt twice) but search results for tasks will improve before results for other object types.

Analyzing Tables

Once the indexing completes, analyzing the database table may help MySQL arrive at better cardinality estimates for some indexes, which can improve performance significantly. Do this with:

phabricator/ $ ./bin/storage analyze

This analysis step is run automatically during bin/storage upgrade, so you do not normally need to run it on its own. However, search indexes are particularly susceptible to key cardinality inaccuracies and running it explicitly after rebuilding indexes may improve things until the next time upgrade is run.

Completing Reindexing

After you've completed reindexing, clear the setup notice by running this command:

phabricator/ $ ./bin/config done reindex

(You have to manually mark this step as complete because the search index can be rebuilt in different ways, so we can not be sure that executing any particular command completes the rebuild.)

Event Timeline

epriestley triaged this task as Normal priority.Nov 30 2016, 7:13 PM
epriestley updated the task description. (Show Details)
epriestley added a project: Search.

Should the description here include the section from T11922 about minimizing downtime? The release notes link to T11922 and not here, but someone from IRC had been going based on the information in this task without seeing that note which may have saved them some indexing time.

The section in T11922 only applies to the storage engine portion of that change, not to reindexing in general, and dropping the index is likely to hurt things overall if we need to do this more in the future (no performance improvement but totally nukes the search index until the reindex occurs).