Page MenuHomePhabricator

Improve Ferret engine indexing performance for large blocks of text
ClosedPublic

Authored by epriestley on Sep 26 2017, 2:15 AM.
Tags
None
Referenced Files
F18762410: D18647.id44766.diff
Mon, Oct 6, 6:50 PM
F18754765: D18647.diff
Sun, Oct 5, 1:12 AM
F18754507: D18647.id44772.diff
Sat, Oct 4, 11:20 PM
F18739099: D18647.diff
Wed, Oct 1, 7:46 PM
F18652276: D18647.id44772.diff
Sep 21 2025, 7:28 AM
F18625390: D18647.diff
Sep 15 2025, 10:42 PM
F18460007: D18647.id.diff
Sep 1 2025, 6:28 PM
F18442238: D18647.diff
Aug 31 2025, 4:53 PM
Subscribers
None

Details

Summary

See PHI87. Ref T12974. Currently, we do a lot more work here than we need to: we call phutil_utf8_strtolower() on each token, but can do it once at the beginning on the whole block.

Additionally, since ngrams don't care about order, we only need to convert unique tokens into ngrams. This saves us some phutil_utf8v(). These calls can be slow for large inputs.

Test Plan

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable