@epriestley: I can help with testing and validating on elasticsearch, if that helps at all.

Yes. The diff is a relatively small amount of the work I must do to respond to this issue: install ElasticSearch, configure it in the way you describe, reproduce the issue, write or apply a change, verify it actually fixes the problem, go read through the documentation enough that I'm confident the fix is the best available fix and that we've understood the issue reasonably well.

Even if I'd provide the diff?

(Just to set expectations, anything which isn't coming in through Support Pacts may take a very long time for us to get to.)

Slightest of issues with the repository indexing mentioned above:

luca:~/phabricator$ ./bin/search index --type repository --force
Usage Exception: Type "repository" matches multiple indexable objects. Use a more specific string. Matching object types are: PhabricatorRepository, PhabricatorRepositoryCommit.
luca:~/phabricator$

This has been live for nearly two weeks without anything apparently exploding so we don't have any further acton planned upstream (although see T12450 for some followups). If you do run into issues, feel free to file new tasks describing them.

I'm just going to merge this into T8646 which broadly discusses richer search results.

Presuming this is resolved since @20after4 couldn't reproduce it after T12450, feel free to follow up there or in a new task if you're still seeing issues.

Presuming this is resolved by D17384, see T12450 for followups.

Presuming this is resolved by new indexes in D17384, see T12450 for followups.

This was tagged ElasticSearch and may now be resolved (see T12450) but also lacks information we require in a modern bug report (specifically: precise reproduction instructions). If this is still an issue at HEAD of master, feel free to follow up on T12450 or file a new report with reproduction steps.

Presuming this is resolved by D17384 (but maybe @20after4 can correct me).

Presuming this is either resolved by T9893/D17384 or no longer relevant. Follow up on T12450 or file a new task if you're still seeing issues.

With the elasticsearch 'simple_query_string' query parser it only works if you use *pricot, for example, outside of quoted phrases.

I'm just going to close this in favor of T12450.

As a general product decision, I do not expect search to be substring search by default -- searching for pricot on Google does not match documents containing apricot. But we can sort this out in the long run.

Note there will finally be a little bit of documentation once this install rebuilds diviner docs: The url should be https://secure.phabricator.com/book/phabricator/article/cluster_search/ (eventually)

This should work just fine with the index mapping and query generation in {rPe41c25de5050}

I agree that results should almost always be on the first page and ideally near the top of the first page. I'm also tempted to say that 10 results per page might be closer to the right number than 100.

I believe that's unavoidable with offset-based paging (the solution is to switch to cursor-based paging, but I think we can't do that for fulltext).

I'll investigate further into the slowness (I wasn't involved in the debugging so far so I don't know more than what's said in the task)

(That is, the expected failure mode of this query when it fails to scale is "silently return an incomplete result set", not "be really slow".)

To start with here, let's identify exactly what's slow? It's not clear to me from the downstream task unless I'm just misreading, and not evident to me from reasoning about the behavior of this query. This query strategy is fundamentally not scalable, but I would not expect it to be slow unless the ElasticSearch query is slow.

The results are ordered by "match strength", but this is computed at runtime and isn't unique to each result: we can't start with the 100th result and use it to build any sort of clause like WHERE rank > 12345 that starts on the 101st result.

Completely fine to make the engine aware of the viewer running the query, I think we just didn't have use cases for that when this stuff was written.

@epriestley: This all sounds excellent. I'll start by fixing any instances of

I have this working downstream in Wikimedia's instance of phabricator, however, our fork is significantly modified. I intend to refactor this into something that is either a) mergeable upstream or b) installable as an extension but for now the code is on phabricator.wikimedia.org PhabricatorElasticFulltextStorageEngine.php

Broadly, I'm supportive.

So, @epriestley: How do you feel about me taking over elasticsearch backend and moving it out to an extension rather than keeping it in phabricator core? The other possibility is I could just fork the elastic backend and name it something different, as it seems phabricator already supports loading the engine extension dynamically.

@ramm: I'm already on it: https://phabricator.wikimedia.org/T155299