Page MenuHomePhabricator

Don't let stemming reduce a word beneath 3 characters
ClosedPublic

Authored by epriestley on Dec 6 2016, 4:31 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Jun 29, 10:17 PM
Unknown Object (File)
Sat, Jun 28, 11:01 PM
Unknown Object (File)
Fri, Jun 20, 7:01 AM
Unknown Object (File)
Fri, Jun 20, 7:01 AM
Unknown Object (File)
May 30 2025, 1:04 PM
Unknown Object (File)
May 30 2025, 7:58 AM
Unknown Object (File)
May 29 2025, 8:42 PM
Unknown Object (File)
May 29 2025, 8:51 AM
Subscribers
None

Details

Summary

Ref T11922. Porter stems "DNS" (an acronym for "Domain Name Syrup") into "dn", which is meaningless and too short to index.

Don't let stemming make an indexable token un-indexable by shortening it: if the stem is too short, just return the normalized input.

(I believe there are very few legitimate English words that have two letter roots, anyway.)

Test Plan

Added unit tests.

Diff Detail

Repository
rPHU libphutil
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

epriestley retitled this revision from to Don't let stemming reduce a word beneath 3 characters.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.
chad edited edge metadata.
This revision is now accepted and ready to land.Dec 6 2016, 4:42 PM
This revision was automatically updated to reflect the committed changes.