@epriestley: I can help with testing and validating on elasticsearch, if that helps at all.

Yes. The diff is a relatively small amount of the work I must do to respond to this issue: install ElasticSearch, configure it in the way you describe, reproduce the issue, write or apply a change, verify it actually fixes the problem, go read through the documentation enough that I'm confident the fix is the best available fix and that we've understood the issue reasonably well.

Even if I'd provide the diff?

(Just to set expectations, anything which isn't coming in through Support Pacts may take a very long time for us to get to.)

Slightest of issues with the repository indexing mentioned above:

luca:~/phabricator$ ./bin/search index --type repository --force
Usage Exception: Type "repository" matches multiple indexable objects. Use a more specific string. Matching object types are: PhabricatorRepository, PhabricatorRepositoryCommit.
luca:~/phabricator$

This has been live for nearly two weeks without anything apparently exploding so we don't have any further acton planned upstream (although see T12450 for some followups). If you do run into issues, feel free to file new tasks describing them.

I'm just going to merge this into T8646 which broadly discusses richer search results.

Presuming this is resolved since @20after4 couldn't reproduce it after T12450, feel free to follow up there or in a new task if you're still seeing issues.

Presuming this is resolved by D17384, see T12450 for followups.

Presuming this is resolved by new indexes in D17384, see T12450 for followups.

This was tagged ElasticSearch and may now be resolved (see T12450) but also lacks information we require in a modern bug report (specifically: precise reproduction instructions). If this is still an issue at HEAD of master, feel free to follow up on T12450 or file a new report with reproduction steps.

Presuming this is resolved by D17384 (but maybe @20after4 can correct me).

Presuming this is either resolved by T9893/D17384 or no longer relevant. Follow up on T12450 or file a new task if you're still seeing issues.

With the elasticsearch 'simple_query_string' query parser it only works if you use *pricot, for example, outside of quoted phrases.

I'm just going to close this in favor of T12450.

As a general product decision, I do not expect search to be substring search by default -- searching for pricot on Google does not match documents containing apricot. But we can sort this out in the long run.

Note there will finally be a little bit of documentation once this install rebuilds diviner docs: The url should be https://secure.phabricator.com/book/phabricator/article/cluster_search/ (eventually)

This should work just fine with the index mapping and query generation in {rPe41c25de5050}

I agree that results should almost always be on the first page and ideally near the top of the first page. I'm also tempted to say that 10 results per page might be closer to the right number than 100.

I believe that's unavoidable with offset-based paging (the solution is to switch to cursor-based paging, but I think we can't do that for fulltext).

I'll investigate further into the slowness (I wasn't involved in the debugging so far so I don't know more than what's said in the task)

(That is, the expected failure mode of this query when it fails to scale is "silently return an incomplete result set", not "be really slow".)

To start with here, let's identify exactly what's slow? It's not clear to me from the downstream task unless I'm just misreading, and not evident to me from reasoning about the behavior of this query. This query strategy is fundamentally not scalable, but I would not expect it to be slow unless the ElasticSearch query is slow.

The results are ordered by "match strength", but this is computed at runtime and isn't unique to each result: we can't start with the 100th result and use it to build any sort of clause like WHERE rank > 12345 that starts on the 101st result.

Completely fine to make the engine aware of the viewer running the query, I think we just didn't have use cases for that when this stuff was written.

@epriestley: This all sounds excellent. I'll start by fixing any instances of

I have this working downstream in Wikimedia's instance of phabricator, however, our fork is significantly modified. I intend to refactor this into something that is either a) mergeable upstream or b) installable as an extension but for now the code is on phabricator.wikimedia.org PhabricatorElasticFulltextStorageEngine.php

Broadly, I'm supportive.

So, @epriestley: How do you feel about me taking over elasticsearch backend and moving it out to an extension rather than keeping it in phabricator core? The other possibility is I could just fork the elastic backend and name it something different, as it seems phabricator already supports loading the engine extension dynamically.

@ramm: I'm already on it: https://phabricator.wikimedia.org/T155299

BTW its probably time to make it 5.0, not 2.0

I doubt that upstream is interested in adding yet another configuration option, however, in case anyone runs across this issue, the following patch will make phabricator work with newer elasticsearch versions (2+)

@epriestley:
So far innodb is working. I can't say if it's worse than myisam really, seems pretty similar. Stemming would help a lot but the main complaint I have is the ranking algorithm seems really bad. It doesn't return the most relevant results first. myisam had the same problem though.

We're planning to put application-level stemming on the MySQL index (see T6740). Are there problems with the InnoDB search that you're aware of that aren't fixable with application adjustments?

With InnoDB now mostly finished indexing, search seems to be working and the service isn't getting killed by locked tables. More info at the link I pasted above ^

Wikimedia is considering another stab at elasticsearch because we just hit a fairly serious scalability issue with myisam (https://phabricator.wikimedia.org/T146673) and it looks like innodb isn't a lot better. In addition to that, the mysql fulltext engines don't seem to handle stemming or even simple prefix matching.

We actually have some problems with our phabricator install.
We use it with mysql, but in a Galera (PXC) cluster and it does not work well with myisam tables.

If there aren't docs yet, anyone willing to speak quickly on what elasticsearch integration entails?

T10161 is a case of another install that selected ElasticSearch without a specific reason, based on a general belief that it is better in some intangible way.

Just closing this since it's a huge fishing expedition that we can't help with, see Contributing Bug Reports and Support Resources.

I'm going to merge this into T5155, since the root issue here is philosophical opposition to the use of IAM Roles to manage credentials in the upstream. Two possible pathways forward outside of upstream support are:

Can you just use an IP-based access policy instead?

If you're using ElasticSearch on your install, can you let us know why you're choosing to configure it over the builtin MySQL search?

Integrating Phabricator into an external (unsupported) tool is considered by us to be custom application development. We only can take upstreamable feature requests that state a core problem when using Phabricator itself. I don't know what these apps are nor why you want to use them. It could be there are already upstream plans to provide similar features.

Sorry to say, but this has nothing to do with custom development. As long as you want to feed elasticsearch with phabricator events, my feature request/suggestion was to do it in a way that is usable by elasticsearch-friendly tools without resorting to hacks and customizations on the elasticsearch side ( which is ok to do at some extent, but right now leaves phabricator in a state requesting to fix index mappings on the elasticsearch side - at least).

our goal is to visualize phabricator events using Kibana and/or Grafana using elasticsearch as a data source.

Hi epriestley,
just added all relevant details. Everything is on the task now.

When submitting a feature request, please describe the root problem you are trying to solve. See Contributing Feature Requests.

Hi epriestley,
I switched this to Feature Request and added Elasticsearch on the projects lists.

A quick update here, as per our analysis we came to know that the current issue was arised due to the elastic search related issue and we have fixed it. Now everything is working fine. :-)

Upgraded the phabricator search to elastic search using the following command.

Unfortunately, given the lack of reproducible instructions and no other reports, there isn't anything currently for the upstream to do on this ticket except to ask whoever is having it, to provide us with more details. If we can't reproduce something, there isn't really much more we can do. The onus is on you to help with that effort.

Upgraded the phabricator search features to elasticsearch.

I have updated the phabricator tio recent version on Dec 2.

Specifically, we need full steps to reproduce the issue. These need to be, from start to finish, full steps any engineer either one day or one year from now can follow and achieve the same error:
https://secure.phabricator.com/book/phabcontrib/article/bug_reports/#include-reproduction-ste

This ticket doesn't provide steps to reproduce the issue. Is Contributing Bug Reports unclear on this matter? Can we provide better documentation somehow?

Thanks for the reply. I have downgraded Elasticsearch to 1.7.2 still it not working. Now it shows as "Loading..." rather than getting "No Data".

Thanks for the reply. I have downgraded Elasticsearch to 1.7.2 still it not working. Now it shows as "Loading..." rather than getting "No Data".