Address epriestley's feedback about tooltip and string concatenation

• Fixed up the cli workflows for search init and search index
• Misc other cleanup

Fix elasticsearch setup checks

Completely fine to make the engine aware of the viewer running the query, I think we just didn't have use cases for that when this stuff was written.

@epriestley: This all sounds excellent. I'll start by fixing any instances of

I have this working downstream in Wikimedia's instance of phabricator, however, our fork is significantly modified. I intend to refactor this into something that is either a) mergeable upstream or b) installable as an extension but for now the code is on phabricator.wikimedia.org PhabricatorElasticFulltextStorageEngine.php

Broadly, I'm supportive.

So, @epriestley: How do you feel about me taking over elasticsearch backend and moving it out to an extension rather than keeping it in phabricator core? The other possibility is I could just fork the elastic backend and name it something different, as it seems phabricator already supports loading the engine extension dynamically.

@ramm: I'm already on it: https://phabricator.wikimedia.org/T155299

BTW its probably time to make it 5.0, not 2.0

I doubt that upstream is interested in adding yet another configuration option, however, in case anyone runs across this issue, the following patch will make phabricator work with newer elasticsearch versions (2+)

@epriestley:
So far innodb is working. I can't say if it's worse than myisam really, seems pretty similar. Stemming would help a lot but the main complaint I have is the ranking algorithm seems really bad. It doesn't return the most relevant results first. myisam had the same problem though.

We're planning to put application-level stemming on the MySQL index (see T6740). Are there problems with the InnoDB search that you're aware of that aren't fixable with application adjustments?

With InnoDB now mostly finished indexing, search seems to be working and the service isn't getting killed by locked tables. More info at the link I pasted above ^

Wikimedia is considering another stab at elasticsearch because we just hit a fairly serious scalability issue with myisam (https://phabricator.wikimedia.org/T146673) and it looks like innodb isn't a lot better. In addition to that, the mysql fulltext engines don't seem to handle stemming or even simple prefix matching.

We actually have some problems with our phabricator install.
We use it with mysql, but in a Galera (PXC) cluster and it does not work well with myisam tables.

If there aren't docs yet, anyone willing to speak quickly on what elasticsearch integration entails?

T10161 is a case of another install that selected ElasticSearch without a specific reason, based on a general belief that it is better in some intangible way.

Just closing this since it's a huge fishing expedition that we can't help with, see Contributing Bug Reports and Support Resources.

I'm going to merge this into T5155, since the root issue here is philosophical opposition to the use of IAM Roles to manage credentials in the upstream. Two possible pathways forward outside of upstream support are:

Can you just use an IP-based access policy instead?

If you're using ElasticSearch on your install, can you let us know why you're choosing to configure it over the builtin MySQL search?

Integrating Phabricator into an external (unsupported) tool is considered by us to be custom application development. We only can take upstreamable feature requests that state a core problem when using Phabricator itself. I don't know what these apps are nor why you want to use them. It could be there are already upstream plans to provide similar features.

Sorry to say, but this has nothing to do with custom development. As long as you want to feed elasticsearch with phabricator events, my feature request/suggestion was to do it in a way that is usable by elasticsearch-friendly tools without resorting to hacks and customizations on the elasticsearch side ( which is ok to do at some extent, but right now leaves phabricator in a state requesting to fix index mappings on the elasticsearch side - at least).

our goal is to visualize phabricator events using Kibana and/or Grafana using elasticsearch as a data source.

Hi epriestley,
just added all relevant details. Everything is on the task now.

When submitting a feature request, please describe the root problem you are trying to solve. See Contributing Feature Requests.

Hi epriestley,
I switched this to Feature Request and added Elasticsearch on the projects lists.

A quick update here, as per our analysis we came to know that the current issue was arised due to the elastic search related issue and we have fixed it. Now everything is working fine. :-)

Upgraded the phabricator search to elastic search using the following command.

Unfortunately, given the lack of reproducible instructions and no other reports, there isn't anything currently for the upstream to do on this ticket except to ask whoever is having it, to provide us with more details. If we can't reproduce something, there isn't really much more we can do. The onus is on you to help with that effort.

Upgraded the phabricator search features to elasticsearch.

I have updated the phabricator tio recent version on Dec 2.

Specifically, we need full steps to reproduce the issue. These need to be, from start to finish, full steps any engineer either one day or one year from now can follow and achieve the same error:
https://secure.phabricator.com/book/phabcontrib/article/bug_reports/#include-reproduction-ste

This ticket doesn't provide steps to reproduce the issue. Is Contributing Bug Reports unclear on this matter? Can we provide better documentation somehow?

Thanks for the reply. I have downgraded Elasticsearch to 1.7.2 still it not working. Now it shows as "Loading..." rather than getting "No Data".

Thanks for the reply. I have downgraded Elasticsearch to 1.7.2 still it not working. Now it shows as "Loading..." rather than getting "No Data".

We don't currently support ElasticSearch 2.0. I'm collecting issues we've seen with it in T9893.

@ru31337, that issue isn't related to this one and isn't "wontfix". Please see Contributing Bug Reports for instructions on how to report bugs. Choosing an existing similar task, changing it to "wontfix" and dumping your issue into it won't get it fixed.

I just upgraded Phabricator, now the version is,

Your install is fairly out of date. See https://secure.phabricator.com/book/phabcontrib/article/bug_reports/#common-fixes

Phabricator Version d878f215617bb5a97bd5ea4145566308cb68d7e3

What version of Phabricator are you running.. See Contributing Bug Reports

Index _status in 2.0 replaced _stats https://www.elastic.co/guide/en/elasticsearch/reference/2.0/indices-status.html

In T9670#142765, @epriestley wrote:

Thanks for testing the patch, should be in HEAD now. Let us know if you run into anything else.

Not sure what @tycho.tatitscheff ran into.

@epriestley i think I didn't delete index first.
Just bin/search index --all.

Thanks for testing the patch, should be in HEAD now. Let us know if you run into anything else.

@epriestley/@tycho.tatitscheff: We managed to test D14370 now, and it totally fixes our problem!

I patched with D14370 and still get an issue. I can retry but I pretty sure I did think corectly.

In T9670#142547, @tycho.tatitscheff wrote:

I tested and i does not fix !

Well, I didn't move to elastic 2.0 but restarting elasticsearch, reindexing rerestarting and reruning plugin doesn't leave me the same error.

In T9670#142544, @epriestley wrote:

D14370 likely fixes this, although I don't have ElasticSearch 2.0 locally and haven't tested it. I'd guess this will give you a clean bill of health:

• Apply the patch;
• run bin/search index --all to rebuild all indexes;
• run the ElasticSearch tool again.

Can you let me know if that works?

I tested and i does not fix !

Oh, I think I misremembered and it does foreground by default. You can use --background to improve performance, and monitor from /daemon/.

Note that index --all queues stuff into the background, so you may need to wait a while before it finishes. You can use --foreground (slower) or monitor progress from /daemon/ in the web UI.

D14370 likely fixes this, although I don't have ElasticSearch 2.0 locally and haven't tested it. I'd guess this will give you a clean bill of health:

After a closer look there seem to be quite a lot of inconsistencies for the different .when fields.

Uhm I tested the same last month when 2.0 was still beta then conclude that phabricator only supported 1.x.
I will follow the resolution of this.