Put some kind of stemmer on the MySQL search index
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	epriestley
	Dec 12 2014, 5:47 PM

Description

Concretely, I just searched for "tokens" and missed a hit on "token", which is trivially resolvable with stemming. We can dump Porter or whatever the state of the art is in pretty easily.

D10955 is pursuing proper support for a real index under ElasticSearch, but I don't really want to deal with ElasticSearch on this install or in the Phacility cluster until much later: the cost of adding a new type of service tier is likely far higher than the cost of dropping a stemmer into the indexer.

Revisions and Commits

rPHU libphutil
	D16943	rPHU42d7ebd477f9 Support using the Porter stemmer when compiling search queries
rP Phabricator
	D16947	rP7c5b5327c8cc Use stemming in the MySQL fulltext search engine
	D16945	rP2b7ec1deea31 Boost search result title matches
	D16944	rP54470a12d430 Execute fulltext queries using a subquery instead of by ordering the entire…

Related Objects

Mentioned In: T11922: Upgrading: Changes to Fulltext Indexing (InnoDB Fulltext)
T11905: Search shouldn't use word boundaries
T11675: "Text" type custom field in Maniphest not displayed in Maniphest Advanced Search form
T11133: Allow non-ASCII usernames
T10642: Default search boolean logic is mildly inconvenient and not configurable in Amazon RDS
T9979: Build support for ngram indexes for substring searches (e.g., file, paste, package, task titles)
T9893: Support ElasticSearch 2.0 - 5.1
T6552: Implement partial / wildcard searching (Elasticsearch)
T3165: Provide Search in Conpherence
T6780: Can't search an exact phrase in Phabricator
Mentioned Here: D10955: Properly create Elasticsearch index

Event Timeline

epriestley created this task.Dec 12 2014, 5:47 PM

epriestley raised the priority of this task from to Low.

epriestley updated the task description. (Show Details)

epriestley added a project: Search.

epriestley added subscribers: epriestley, chad.

I this.

The stemmer should probably make an effort to preserve exact phrase search. Likely, this means special handling of quoted terms when tokenizing and stemming the query.

epriestley mentioned this in T3165: Provide Search in Conpherence.Jan 6 2015, 2:57 AM

aklapper added a subscriber: aklapper.Feb 9 2015, 12:38 PM

qgil mentioned this in T6552: Implement partial / wildcard searching (Elasticsearch).Mar 13 2015, 5:00 PM

rfergu added a subscriber: rfergu.Mar 31 2015, 2:42 AM

epriestley mentioned this in T9893: Support ElasticSearch 2.0 - 5.1.Dec 3 2015, 2:16 PM

epriestley mentioned this in T9979: Build support for ngram indexes for substring searches (e.g., file, paste, package, task titles).Dec 14 2015, 11:50 AM

epriestley mentioned this in T10642: Default search boolean logic is mildly inconvenient and not configurable in Amazon RDS.Mar 22 2016, 1:14 AM

epriestley mentioned this in T11133: Allow non-ASCII usernames.Jun 12 2016, 4:07 AM

avivey merged a task: T11463: Search should probably do some minimal stemming.Aug 11 2016, 5:16 PM

avivey added a subscriber: venky.

Herald added a subscriber: eadler. · View Herald TranscriptAug 11 2016, 5:16 PM

chad merged a task: T11447: Make related object search more flexible.Aug 16 2016, 1:38 PM