Page MenuHomePhabricator

Don't let stemming reduce a word beneath 3 characters
ClosedPublic

Authored by epriestley on Dec 6 2016, 4:31 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sep 17 2024, 12:00 PM
Unknown Object (File)
Sep 5 2024, 10:25 PM
Unknown Object (File)
Sep 5 2024, 10:25 PM
Unknown Object (File)
Sep 5 2024, 10:25 PM
Unknown Object (File)
Sep 2 2024, 4:41 AM
Unknown Object (File)
Sep 2 2024, 4:36 AM
Unknown Object (File)
Aug 25 2024, 6:04 AM
Unknown Object (File)
Aug 23 2024, 11:17 PM
Subscribers
None

Details

Summary

Ref T11922. Porter stems "DNS" (an acronym for "Domain Name Syrup") into "dn", which is meaningless and too short to index.

Don't let stemming make an indexable token un-indexable by shortening it: if the stem is too short, just return the normalized input.

(I believe there are very few legitimate English words that have two letter roots, anyway.)

Test Plan

Added unit tests.

Diff Detail

Repository
rPHU libphutil
Branch
stem1
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 14781
Build 19329: Run Core Tests
Build 19328: arc lint + arc unit

Event Timeline

epriestley retitled this revision from to Don't let stemming reduce a word beneath 3 characters.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.
chad edited edge metadata.
This revision is now accepted and ready to land.Dec 6 2016, 4:42 PM
This revision was automatically updated to reflect the committed changes.