Show users how fulltext search queries are parsed and executed; don't query stopwords or short tokens
ClosedPublic
Actions

Authored by epriestley on Apr 13 2017, 12:04 AM.

Details

Reviewers

Maniphest Tasks

T2632: MyISAM fulltext does not support non-latin languages and we don't warn you about it
T12003: Explain to users how fulltext queries are parsed and executed
T12137: MySQL fulltext search returns no results in boolean mode when a query includes +"x", where "x" is a stopword or short token

Commits

rP3245e74f16bb: Show users how fulltext search queries are parsed and executed; don't query…

Summary

Depends on D17670. Fixes T12137. Fixes T12003. Ref T2632.

This shows users a readout of which terms were actually searched for.

This also drops those terms from the query we submit to the backend, dodging the weird behaviors / search engine bugs in T12137.

This might need some design tweaking.

Test Plan

Diff Detail

Repository

rP Phabricator

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

epriestley created this revision.Apr 13 2017, 12:04 AM

Harbormaster completed remote builds in B16428: Diff 42497.Apr 13 2017, 12:06 AM

Use phutil_utf8_strlen() for identifying short tokens.

Harbormaster completed remote builds in B16429: Diff 42498.Apr 13 2017, 12:10 AM

hmmmmmmmm

how is then not a stopword

InnoDB list is pretty conservative:

https://dev.mysql.com/doc/refman/5.7/en/fulltext-stopwords.html

| a     |
| about |
| an    |
| are   |
| as    |
| at    |
| be    |
| by    |
| com   |
| de    |
| en    |
| for   |
| from  |
| how   |
| i     |
| in    |
| is    |
| it    |
| la    |
| of    |
| on    |
| or    |
| that  |
| the   |
| this  |
| to    |
| was   |
| what  |
| when  |
| where |
| who   |
| will  |
| with  |
| und   |
| the   |
| www   |

chad accepted this revision.Apr 13 2017, 1:59 AM

This revision is now accepted and ready to land.Apr 13 2017, 1:59 AM

Closed by commit rP3245e74f16bb: Show users how fulltext search queries are parsed and executed; don't query… (authored by epriestley). · Explain WhyApr 13 2017, 2:08 AM

This revision was automatically updated to reflect the committed changes.

epriestley mentioned this in T12596: Fulltext stopword filtering is applying before stemming and failing to filter forms of words which stem to stopwords.Apr 19 2017, 3:33 PM