@epriestley: I can help with testing and validating on elasticsearch, if that helps at all.

Yes. The diff is a relatively small amount of the work I must do to respond to this issue: install ElasticSearch, configure it in the way you describe, reproduce the issue, write or apply a change, verify it actually fixes the problem, go read through the documentation enough that I'm confident the fix is the best available fix and that we've understood the issue reasonably well.

Even if I'd provide the diff?

(Just to set expectations, anything which isn't coming in through Support Pacts may take a very long time for us to get to.)

Slightest of issues with the repository indexing mentioned above:

luca:~/phabricator$ ./bin/search index --type repository --force
Usage Exception: Type "repository" matches multiple indexable objects. Use a more specific string. Matching object types are: PhabricatorRepository, PhabricatorRepositoryCommit.
luca:~/phabricator$

This has been live for nearly two weeks without anything apparently exploding so we don't have any further acton planned upstream (although see T12450 for some followups). If you do run into issues, feel free to file new tasks describing them.

I'm just going to merge this into T8646 which broadly discusses richer search results.

Presuming this is resolved since @20after4 couldn't reproduce it after T12450, feel free to follow up there or in a new task if you're still seeing issues.

Presuming this is resolved by D17384, see T12450 for followups.

Presuming this is resolved by new indexes in D17384, see T12450 for followups.

This was tagged ElasticSearch and may now be resolved (see T12450) but also lacks information we require in a modern bug report (specifically: precise reproduction instructions). If this is still an issue at HEAD of master, feel free to follow up on T12450 or file a new report with reproduction steps.

Presuming this is resolved by D17384 (but maybe @20after4 can correct me).

Presuming this is either resolved by T9893/D17384 or no longer relevant. Follow up on T12450 or file a new task if you're still seeing issues.

With the elasticsearch 'simple_query_string' query parser it only works if you use *pricot, for example, outside of quoted phrases.

I'm just going to close this in favor of T12450.

As a general product decision, I do not expect search to be substring search by default -- searching for pricot on Google does not match documents containing apricot. But we can sort this out in the long run.

Note there will finally be a little bit of documentation once this install rebuilds diviner docs: The url should be https://secure.phabricator.com/book/phabricator/article/cluster_search/ (eventually)

This should work just fine with the index mapping and query generation in rPe41c25de5050: Support multiple fulltext search clusters with 'cluster.search' config

try to get harbormaster to build (push to staging?)

• actually, acutally utilize the health monitoring...

• Improved the status monitoring UI in config/cluster/search/
• Actually utilize the health monitoring cache to avoid connecting to downed servers.

@epriestley sweet, I'll land this as soon as I see that you've merged to stable.

I don't see any noteworthy issues here.

Haha, thanks. Let me finish testing one other change and then I'll give this another look.

@epriestley: I think this is ready to land but I want to give you one more chance to change your mind.

• Created diviner documentation: Cluster: Search
• removed stray phlog

• Fix searching relationships which I had inadvertantly broken.
• Better elasticsearch 2.x and 5.x support
• more optimized query

Fix method signature un-final PhabricatorElasticFulltextStorageEngine

Ok I think I've eliminated the problematic parts like indexing project slugs.

Get rid of static.

address review feedback that I hadn't gotten to yet.

Note: I'm not sure why harbormaster is failing?

• Cleaned up the elastic query and added comments describing the purpose of the clauses
• a couple of bugfixes found by further testing

Ok I've reworked this quite a bit and I may have messed up somewhere in the process.

In D17384#209987, @epriestley wrote:

Just to make sure I haven't missed anything:

• We currently write health checks but never read them, right? So there's no effect (other than the UI "Status" changing) when a service fails health checks? That seems fine for now, I just want to make sure I didn't miss a health check read somewhere.

Just to make sure I haven't missed anything:

rebased on top of D17509: Updated PhabricatorElasticFulltextStorageEngine for elasticsearch 5

• Move the stats definitions into the engine so the status UI remains engine agnostic.
• Fix a bug where role => false was being treated like role => true in the UI

I'm pleased to report that this has been live on wikimedia's phabricator for about a week without any incidents whatsoever. Additionally, we are in the process of migrating from elasticsearch 2.x to 5.x and the ability to write to multiple clusters is really working out nicely for transition.

• Added index stas to status ui
• Separate mysql status from elasticsearch status and show different set of columns appropriate to each cluster type.

• Remove unused healthrecord stuff from PhabricatorSearchCluster class
• Add back getDisplayName to the PhabricatorSearchCluster class because it's needed.

Addressed latest round of feedback.

Few more minor things.

$limit = 10000 - $offset

• updated PhabricatorExtraConfigSetupCheck
• 10,000 results
• remove unused methods

This is now live on https://phab-01.wmflabs.org for testing. Everything seems to be working well, including the health monitoring.

In D17384#209367, @epriestley wrote:

This is shaping up nicely, couple of other minor inlines.

This is shaping up nicely, couple of other minor inlines.

Fix unit test case.

Getting closer...

I agree that results should almost always be on the first page and ideally near the top of the first page. I'm also tempted to say that 10 results per page might be closer to the right number than 100.

I believe that's unavoidable with offset-based paging (the solution is to switch to cursor-based paging, but I think we can't do that for fulltext).

I'll investigate further into the slowness (I wasn't involved in the debugging so far so I don't know more than what's said in the task)

In D17384#208882, @epriestley wrote:

• Using the same objects as both Host and Service feels confusing to me. I think this would probably be clearer as separate Service and Host classes? Like PhabricatorMySQLSearchClusterService extends PhabricatorSearchClusterService and PhabricatorMySQLSearchClusterHost extends PhabricatorSearchClusterHost or similar. Particularly because setHostRefs() seems like it's getting called with a raw dictionary in one case and a list of objects in another? And then there's weird magic around getHostRefs() for the MySQL case?

I'll split out the changes to the engine if I can figure out how to do that... Update coming soon.

(That is, the expected failure mode of this query when it fails to scale is "silently return an incomplete result set", not "be really slow".)

To start with here, let's identify exactly what's slow? It's not clear to me from the downstream task unless I'm just misreading, and not evident to me from reasoning about the behavior of this query. This query strategy is fundamentally not scalable, but I would not expect it to be slow unless the ElasticSearch query is slow.

The results are ordered by "match strength", but this is computed at runtime and isn't unique to each result: we can't start with the 100th result and use it to build any sort of clause like WHERE rank > 12345 that starts on the 101st result.

General stuff:

@epriestley: Ok I believe this addresses all of your feedback and other than documentation it should be very close to finished.