Nothing new has arisen for a while, so presuming this is resolved.

Seems like it works:

I think we can be slightly more general about this, and assume any token beginning with _ is substring search. This covers __FILE__, __construct, etc. Users almost certainly intend these to be substring searches.

I've deployed these changes to secure, so hopefully any issues will present themselves.

If something goes wrong with this, the patch which fixes the problem can now change the indexer version and then all mis-indexed documents can be reindexed with:

The existing SearchIndexVersion table (which stores document versions) may reasonably be able to store index versions too. This limits the need to apply changes to fdocument.

This has stalled for a while because it's moderately expensive to recover from if the updated index logic has a bug: rebuilding all document indexes is expensive, and it's difficult to identify the set of documents that need to be reindexed if a bug is present.

This is now possible.

This is now supported.

These field functions have somewhat-weird scopes/context.

Getting through the ngram index alone isn't good enough, because LIKE operators against utf8mb4_unicode_ci treat combining accents as separate characters:

Normalizer requires intl which I'm hesitant to add a dependency on.

This appears to be working properly, now.

Query parsing of certain unusual or ambiguous inputs has changed slightly.

We also have two separate pieces of ngram extraction code:

For now, I'm going to change the ngram slicing to be character-oriented. This should never be worse than the current behavior, and moves us closer to effective normalization.

This appears to be the unicode normalization chart:

I thought this was some kind of complicated mess with the regex on line 420, but it's actually an issue with this:

Also, what is "By Relevance" ?

A related issue here is exemplified in https://discourse.phabricator-community.org/t/importing-libphutil-repository-on-fresh-phabricator-triggers-an-error/2391/, which basically amounts to:

Presumably resolved elsewhere by D19503.

@epriestley: I can help with testing and validating on elasticsearch, if that helps at all.

Yes. The diff is a relatively small amount of the work I must do to respond to this issue: install ElasticSearch, configure it in the way you describe, reproduce the issue, write or apply a change, verify it actually fixes the problem, go read through the documentation enough that I'm confident the fix is the best available fix and that we've understood the issue reasonably well.

Even if I'd provide the diff?

(Just to set expectations, anything which isn't coming in through Support Pacts may take a very long time for us to get to.)

Trying to reproduce this locally just hits the ft_doc.epochModified issue. I'm not immediately sure why the behavior differs between my local install and secure, but that issue probably needs to be fixed first.

Actually, I'm not entirely right in merging that task -- T13163 isn't quite the same as the other two issues here. I think they're similar, but the query text is relevant in the case of T13163. Notably:

In T8510#200946, @aklapper wrote:

In https://phabricator.wikimedia.org/T76732#2803813 ksmith brings up that "In the search bar in the toolbar at the top of the screen, searching for "Maps" brings up Maps as the third option. Searching for "Discovery" brings up Discovery as the second option."

See also T13102/PHI442. The MySQL setting optimizer_search_depth=0 may fix the weird explosive complexity here.

I think everything we know about has been resolved.

@epriestley
After i upgraded my application to the newest version，which swap the search engine to ‘Ferret’，I reindexed several times，but I can not get any result.These is my version information：
phabricator33756bcf1d70ea5579dff1ab276bbe660d10494c (Tue, Oct 3) (branched from f9110b87abf337dd1e7714d755775e53cffd4db9 on origin)arcanist0a7f403333fe9082b39bd007b9d5f9e765c8b9ce (Tue, Oct 3) (branched from c804c5026011f27614a7bbdb2bb32cab590d68ca on origin)phutilb400c6b04bb247a3e0f1941390bc450f36ac2ccd (Tue, Oct 3) (branched from 9f9c33797a3ebbf1c4dcaa474a0c4e0b32d5396a on origin)diff2.8.1 at /usr/bin/diffgit1.7.1 at /usr/bin/githgNot Availablepygmentize2.0.2 at /usr/bin/pygmentizesvn1.6.11 at /usr/bin/svn

Looks to be caused by this commit rPb75a4151c8996c5dfb1c8c14378fe3259666eac2

https://discourse.phabricator-community.org/t/daemons-tasks-crashing-in-a-loop-during-reindex/506

Running the reindex led to a couple crash looping tasks on my end: