Page MenuHomePhabricator
Differential D21128

Combine the two different ngram-splitting algorithms into a single engine
ClosedPublic
Actions

Authored by epriestley on Apr 16 2020, 4:38 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Feb 9, 8:59 AM
Unknown Object (File)
Sun, Feb 9, 8:59 AM
Unknown Object (File)
Sun, Feb 9, 8:59 AM
Unknown Object (File)
Sun, Feb 9, 8:59 AM
Unknown Object (File)
Tue, Jan 28, 2:46 AM
Unknown Object (File)
Sun, Jan 26, 10:02 PM
Unknown Object (File)
Sat, Jan 25, 7:08 AM
Unknown Object (File)
Sat, Jan 25, 7:08 AM
View All Files
Subscribers
None

Details

Reviewers
None
Maniphest Tasks
T13501: Improve search index normalization of "é" and other characters with variants or multiple representations
Commits
rP9bdf477f2fef: Combine the two different ngram-splitting algorithms into a single engine
Summary

Ref T13501. Depends on D21127. With the "prefix" behavior removed in D21127, we now have two virtually identical copies of the same code.

The newer one in Ferret is better: it slices utf8 correctly and is slightly more efficient on large inputs. Pull it out and make all callers call into it.

Test Plan
  • Grepped for all affected symbols.
  • Ran bin/search index --force ... to reindex various objects (tasks, files).
  • Searched for things in the UI.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

epriestley created this revision.Apr 16 2020, 4:38 PM
Harbormaster completed remote builds in B24131: Diff 50313.Apr 16 2020, 4:39 PM
epriestley requested review of this revision.Apr 16 2020, 4:39 PM
This revision was not accepted when it landed; it landed in state Needs Review.Apr 16 2020, 4:45 PM
Closed by commit rP9bdf477f2fef: Combine the two different ngram-splitting algorithms into a single engine (authored by epriestley). · Explain Why
This revision was automatically updated to reflect the committed changes.
epriestley added a commit: rP9bdf477f2fef: Combine the two different ngram-splitting algorithms into a single engine.

Revision Contents
Changeset List

Diff 50317

View Options

src/__phutil_library_map__.php

Loading...
View Options

src/applications/search/engine/PhabricatorSearchNgramEngine.php

Loading...
View Options

src/applications/search/engineextension/PhabricatorFerretFulltextEngineExtension.php

Loading...
View Options

src/applications/search/ferret/PhabricatorFerretEngine.php

Loading...
View Options

src/applications/search/ferret/__tests__/PhabricatorFerretEngineTestCase.php

Loading...
View Options

src/applications/search/ngrams/PhabricatorSearchNgrams.php

Loading...
View Options

src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php

Loading...
Phabricator · Built by Phacility