Differential D18649
Improve performance of Ferret engine ngram extraction, particularly for large input strings Authored by epriestley on Sep 26 2017, 4:21 PM. Tags None Referenced Files
Subscribers None
Details See PHI87. Ref T12974. The array_slice() method of splitting the string apart can perform poorly for large input strings. I think this is mostly just the large number of calls plus building and returning an array being not entirely trivial. We can just use substr() instead, as long as we're a little bit careful about keeping track of where we're slicing the string if it has UTF8 characters.
Diff Detail
Event TimelineComment Actions I've spent a lot of time staring at this and I'm pretty convinced it works. Maybe add a few more unit tests for strings of length {0,1,2}? |