Improve search stemmer performance for large inputs
ClosedPublic
Actions

Authored by epriestley on Sep 26 2017, 2:18 AM.

Details

Reviewers

amckinley

Maniphest Tasks

T12974: Upgrading: "Ferret" Fulltext Engine

Commits

rPHU5764bd8cafb8: (stable) Improve search stemmer performance for large inputs
rPHU388d16d298a3: Improve search stemmer performance for large inputs

Summary

Ref T12974. See PHI87. As in D18647, we can improve the performance of some UTF8 operations here.

Instead of calling phutil_utf8_strtolower() on each token separately, call it once on the entire input up front. This has the same effect.

Test Plan

See D18647; indexed a 4MB task description.
Before, a bit over 9s. After, a little under 6s, with about 3s spent in 8 calls to mb_convert_case() that we probably can't improve easily.
Before: https://secure.phabricator.com/xhprof/profile/PHID-FILE-efxv56q2hulr6kjrxbx6/
After: https://secure.phabricator.com/xhprof/profile/PHID-FILE-pj4uv2pkdfoujxq44piq/

Diff Detail

Repository

rPHU libphutil

Branch

stemmer

Lint

Lint Passed

Unit

Tests Passed

Build Status

Buildable 18545
Build 24984: Run Core Tests
Build 24983: arc lint + arc unit

Event Timeline

epriestley created this revision.Sep 26 2017, 2:18 AM

Harbormaster completed remote builds in B18545: Diff 44767.Sep 26 2017, 2:18 AM

Harbormaster completed remote builds in B18545: Diff 44767.

amckinley accepted this revision.Sep 27 2017, 5:15 PM

This revision is now accepted and ready to land.Sep 27 2017, 5:15 PM

Closed by commit rPHU388d16d298a3: Improve search stemmer performance for large inputs (authored by epriestley). · Explain WhySep 27 2017, 5:24 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Path

Size

src/

search/

PhutilSearchStemmer.php

12 lines

Diff 44767

View Options

src/search/PhutilSearchStemmer.php

Improve search stemmer performance for large inputsClosedPublicActions