I've updated {D17564} to address the tokenization and word stemming issues.

Yep, thanks!

It's possible T10872 is also related/is a duplicate.

so it's tokenizing maniphest.query as two separate tokens

Maniphest advanced search is somewhat buried, indeed. I think one easy solution to this would be to add "Task search" to the main phab menu (using the new custom menus feature)... In fact, I think I will do that now at https://phabricator.wikimedia.org

In T12450#216943, @epriestley wrote:

---

• Searching for f*a*c*t*o*r*y*s*u*r*p*l*u*s*z*z*q*q*z*z*q*q produces nonsenical results (many results, when I would expect no results: the results do not contain that sequence of letters in order).
• Searching or user fails to find task Grant users tokens when a mention is created, suggesting that stemming is not working.
• Searching for users finds that task, but fails to find a task containing "per user per month" in a comment, also suggesting that stemming is not working.
• Searching for maniphest fails to find task maniphest.query elephant, suggesting that tokenization is ElasticSearch is not as good as the MySQL tokenization for these words (see D17330).

f*a*c*t*o*r*y*s*u*r*p*l*u*s*z*z*q*q*z*z*q*q returns the same results as
f a c t o r y s u r p l u s z z q q z z q q so it appears to be treating those as individual single-letter tokens. strange.

I think it would make a lot of sense to construct the two queries separately (and in parallel) with a short timeout, then handle the timeout gracefully allowing the user to refine their query further. This would avoid the denial of service situation which happened to Wikimedia more than once due to users repeatedly executing really expensive searches until mysql fell over from the load.

The high level need here actually just came up as recently as yesterday with our new senior product manager; he had no idea that there was a Maniphest-specific search and had instead been searching for maniphest tasks using global search (and was convinced Phab search was very limited due to this). Only upon me watching him do this search and advising him did he discover it. I've seen this issue several times now in our organization and I believe it contributes to the perception from some people internally that they have problems with Phab search.

FWIW I don't see this as something people would express interest in so it doesn't surprise me you haven't heard it from people, but if a user discovered it (e.g. there were an indicator below/during a search query as a power feature), I imagine they would use it. Just my 2 cents.

In T12450#216943, @epriestley wrote:

I ran into a lot of confusion because the versioned object indexes are not namespaced per-service. Basically, if you insert version 95 of a document into Elastic, the indexer thinks that version 95 doesn't need to go into MySQL, even though it does. So when you run bin/search index ..., you may get only a subset of the updates you actually need. The object index versions need to change to become engine-aware so they are stored per-service, not globally, and/or the whole mechanism needs to include a hash of cluster.search or just be turned off. Until this is fixed, it can be worked around with using --force everywhere.

bin/search index might reasonably provide summary output about this ("392 documents were not indexed because they haven't changed, use --force to update them.").

In T12450#216943, @epriestley wrote:

---

• Searching for f*a*c*t*o*r*y*s*u*r*p*l*u*s*z*z*q*q*z*z*q*q produces nonsenical results (many results, when I would expect no results: the results do not contain that sequence of letters in order).
• Searching or user fails to find task Grant users tokens when a mention is created, suggesting that stemming is not working.
• Searching for users finds that task, but fails to find a task containing "per user per month" in a comment, also suggesting that stemming is not working.
• Searching for maniphest fails to find task maniphest.query elephant, suggesting that tokenization is ElasticSearch is not as good as the MySQL tokenization for these words (see D17330).
• Searching for users -blue returns a huge number of results: significantly more than users. Expected behavior: fewer results, omitting those results matching blue.
• Searching for users blue returns more results than users or blue. Expected behavior: fewer results, because only results which match "users" AND "blue" are returned. The result set includes completely irrelevant results.

@epriestley: Thanks for the detailed feedback... I'll get to work ;)

Personal notes:

In T12450#216917, @20after4 wrote:

I can not reproduce it on wikimedia's install.

Presuming this is resolved since @20after4 couldn't reproduce it after T12450, feel free to follow up there or in a new task if you're still seeing issues.

• Has T8602 been resolved?

I can not reproduce it on wikimedia's install.

I don't plan to implement the original feature as described.

We haven't seen significant additional interest in this in more than a year. I expect to implement T7860 but, not a query microlanguage.

• Write an "Upgrading: ..." guidance task with narrow instructions for installs that are upgrading.

TODO

• Do we need to add an indexing activity (T11932) for installs with ElasticSearch?

Yes, I think so

• We should more clearly detail exactly which versions of ElasticSearch are supported (for example, is ElasticSearch <2 no longer supported)? From >T9893 it seems like we may only have supported ElasticSearch <2 before, so are the two regions of support totally nonoverlapping and all ElasticSearch users will need to upgrade?

Presuming this is resolved by D17384, see T12450 for followups.

Presuming this is resolved by new indexes in D17384, see T12450 for followups.

I haven't been testing with elasticsearch < 2.0 so this might break backwards compatibility. It wouldn't be difficult to fix any compatibility issues though, with a tiny bit of testing.

With the elasticsearch 'simple_query_string' query parser it only works if you use *pricot, for example, outside of quoted phrases.

As a general product decision, I do not expect search to be substring search by default -- searching for pricot on Google does not match documents containing apricot. But we can sort this out in the long run.

This should work just fine with the index mapping and query generation in {rPe41c25de5050}

With subprojects, #project changed to mean "that project, and all its descendant projects", but any(...) and not(...) still mean "just the parameter, exactly". I think viewerprojects() and projects(user) are also "just the parameter".

I can confirm that In Any: does not seem to include subprojects. I tried to make some sense of the way the project search functions work but it's pretty complicated.

Elasticsearch has much better support for non-latin language analysis. See https://www.elastic.co/guide/en/elasticsearch/guide/current/icu-tokenizer.html discusses their ability to properly tokenize Thai, Chinese and Japanese text.

T12003 is also related to this (providing more explanatory, contextual help for users executing searches which may not be doing what they want for technical reasons).

There's some additional stuff in D17384, where Elastic has an internal hard limit of 10K results.

Hmm. Ok. Deleted.

Specifically, this task is about how the javascript in typeaheads rank and match choices.

Not similar.

I agree that results should almost always be on the first page and ideally near the top of the first page. I'm also tempted to say that 10 results per page might be closer to the right number than 100.

I believe that's unavoidable with offset-based paging (the solution is to switch to cursor-based paging, but I think we can't do that for fulltext).

I'll investigate further into the slowness (I wasn't involved in the debugging so far so I don't know more than what's said in the task)

(That is, the expected failure mode of this query when it fails to scale is "silently return an incomplete result set", not "be really slow".)

To start with here, let's identify exactly what's slow? It's not clear to me from the downstream task unless I'm just misreading, and not evident to me from reasoning about the behavior of this query. This query strategy is fundamentally not scalable, but I would not expect it to be slow unless the ElasticSearch query is slow.

The results are ordered by "match strength", but this is computed at runtime and isn't unique to each result: we can't start with the 100th result and use it to build any sort of clause like WHERE rank > 12345 that starts on the 101st result.

I guess this one can live for now since that's reproducible/actionable.

I think you don't need to rebuild the search index for that to take full effect. Although rebuilding it will change the content in the index in some cases, the new, more-tailored queries should match either version.

We transform x y into +"x" +"y" in the general case, but currently apply stemming to "a_b" and "c.d". This is because it's generally correct to stem "x-y", which is usually a hyphenated English phrase rather than a "constant" or "token", and correct to stem "z.", which is usually something copy-pasted with a stray period, not an intent to search for a token ending with a period, and the cases of "_" and "." got lumped in with those.

Interestingly, phabricator_maniphest maniphest_transaction gives zero results but "phabricator_maniphest" "maniphest_transaction" works. Seems like the underscores are a problem as much as the period.

Maybe phabricator should automatically try adding quotes to a search if the initial query returns zero results?

Hey, one of our teams is experiencing this issue as well - many of their tasks include the name of a flag with a lot of punctuation, such as subgrowth_biz_buy_order_summary. At some point in the not-terribly-distant past, searching for these flags started to return 0 results.

Completely fine to make the engine aware of the viewer running the query, I think we just didn't have use cases for that when this stuff was written.

@epriestley: This all sounds excellent. I'll start by fixing any instances of

I have this working downstream in Wikimedia's instance of phabricator, however, our fork is significantly modified. I intend to refactor this into something that is either a) mergeable upstream or b) installable as an extension but for now the code is on phabricator.wikimedia.org PhabricatorElasticFulltextStorageEngine.php

Broadly, I'm supportive.

So, @epriestley: How do you feel about me taking over elasticsearch backend and moving it out to an extension rather than keeping it in phabricator core? The other possibility is I could just fork the elastic backend and name it something different, as it seems phabricator already supports loading the engine extension dynamically.

@ramm: I'm already on it: https://phabricator.wikimedia.org/T155299

I can find this task by searching for "あああああ" so the CJK stuff may not really be much of an issue anymore? Maybe just an issue with Chinese because the index doesn't know how to tokenize words?