@epriestley: I can help with testing and validating on elasticsearch, if that helps at all.
Mar 11 2021
Sep 20 2018
Sep 5 2018
Yes. The diff is a relatively small amount of the work I must do to respond to this issue: install ElasticSearch, configure it in the way you describe, reproduce the issue, write or apply a change, verify it actually fixes the problem, go read through the documentation enough that I'm confident the fix is the best available fix and that we've understood the issue reasonably well.
Even if I'd provide the diff?
(Just to set expectations, anything which isn't coming in through Support Pacts may take a very long time for us to get to.)
Jul 9 2017
Jun 3 2017
Slightest of issues with the repository indexing mentioned above:
luca:~/phabricator$ ./bin/search index --type repository --force Usage Exception: Type "repository" matches multiple indexable objects. Use a more specific string. Matching object types are: PhabricatorRepository, PhabricatorRepositoryCommit. luca:~/phabricator$
Apr 12 2017
This has been live for nearly two weeks without anything apparently exploding so we don't have any further acton planned upstream (although see T12450 for some followups). If you do run into issues, feel free to file new tasks describing them.
I'm just going to merge this into T8646 which broadly discusses richer search results.
Apr 7 2017
Apr 4 2017
Apr 3 2017
Apr 2 2017
Mar 26 2017
This was tagged ElasticSearch and may now be resolved (see T12450) but also lacks information we require in a modern bug report (specifically: precise reproduction instructions). If this is still an issue at HEAD of master, feel free to follow up on T12450 or file a new report with reproduction steps.
With the elasticsearch 'simple_query_string' query parser it only works if you use *pricot, for example, outside of quoted phrases.
I'm just going to close this in favor of T12450.
As a general product decision, I do not expect search to be substring search by default -- searching for pricot on Google does not match documents containing apricot. But we can sort this out in the long run.
Note there will finally be a little bit of documentation once this install rebuilds diviner docs: The url should be https://secure.phabricator.com/book/phabricator/article/cluster_search/ (eventually)
This should work just fine with the index mapping and query generation in rPe41c25de5050: Support multiple fulltext search clusters with 'cluster.search' config
try to get harbormaster to build (push to staging?)
Mar 25 2017
- actually, acutally utilize the health monitoring...
- Improved the status monitoring UI in config/cluster/search/
- Actually utilize the health monitoring cache to avoid connecting to downed servers.
Mar 23 2017
@epriestley sweet, I'll land this as soon as I see that you've merged to stable.
I don't see any noteworthy issues here.
Haha, thanks. Let me finish testing one other change and then I'll give this another look.
@epriestley: I think this is ready to land but I want to give you one more chance to change your mind.
- Created diviner documentation: Cluster: Search
- removed stray phlog
- Fix searching relationships which I had inadvertantly broken.
- Better elasticsearch 2.x and 5.x support
- more optimized query
Mar 22 2017
Fix method signature un-final PhabricatorElasticFulltextStorageEngine
Ok I think I've eliminated the problematic parts like indexing project slugs.
Get rid of static.
address review feedback that I hadn't gotten to yet.
Note: I'm not sure why harbormaster is failing?
- Cleaned up the elastic query and added comments describing the purpose of the clauses
- a couple of bugfixes found by further testing
Ok I've reworked this quite a bit and I may have messed up somewhere in the process.
Mar 21 2017
Just to make sure I haven't missed anything:
- We currently write health checks but never read them, right? So there's no effect (other than the UI "Status" changing) when a service fails health checks? That seems fine for now, I just want to make sure I didn't miss a health check read somewhere.
Mar 20 2017
Just to make sure I haven't missed anything:
Mar 16 2017
- Move the stats definitions into the engine so the status UI remains engine agnostic.
- Fix a bug where role => false was being treated like role => true in the UI
I'm pleased to report that this has been live on wikimedia's phabricator for about a week without any incidents whatsoever. Additionally, we are in the process of migrating from elasticsearch 2.x to 5.x and the ability to write to multiple clusters is really working out nicely for transition.
Mar 10 2017
- Added index stas to status ui
- Separate mysql status from elasticsearch status and show different set of columns appropriate to each cluster type.
Mar 9 2017
- Remove unused healthrecord stuff from PhabricatorSearchCluster class
- Add back getDisplayName to the PhabricatorSearchCluster class because it's needed.
Addressed latest round of feedback.
Few more minor things.
Mar 8 2017
$limit = 10000 - $offset
- updated PhabricatorExtraConfigSetupCheck
- 10,000 results
- remove unused methods
This is now live on https://phab-01.wmflabs.org for testing. Everything seems to be working well, including the health monitoring.
This is shaping up nicely, couple of other minor inlines.
Mar 7 2017
Fix unit test case.
Mar 6 2017
I agree that results should almost always be on the first page and ideally near the top of the first page. I'm also tempted to say that 10 results per page might be closer to the right number than 100.
Mar 5 2017
I believe that's unavoidable with offset-based paging (the solution is to switch to cursor-based paging, but I think we can't do that for fulltext).
I'll investigate further into the slowness (I wasn't involved in the debugging so far so I don't know more than what's said in the task)
Mar 4 2017
- Using the same objects as both Host and Service feels confusing to me. I think this would probably be clearer as separate Service and Host classes? Like PhabricatorMySQLSearchClusterService extends PhabricatorSearchClusterService and PhabricatorMySQLSearchClusterHost extends PhabricatorSearchClusterHost or similar. Particularly because setHostRefs() seems like it's getting called with a raw dictionary in one case and a list of objects in another? And then there's weird magic around getHostRefs() for the MySQL case?
I'll split out the changes to the engine if I can figure out how to do that... Update coming soon.
(That is, the expected failure mode of this query when it fails to scale is "silently return an incomplete result set", not "be really slow".)
To start with here, let's identify exactly what's slow? It's not clear to me from the downstream task unless I'm just misreading, and not evident to me from reasoning about the behavior of this query. This query strategy is fundamentally not scalable, but I would not expect it to be slow unless the ElasticSearch query is slow.
The results are ordered by "match strength", but this is computed at runtime and isn't unique to each result: we can't start with the 100th result and use it to build any sort of clause like WHERE rank > 12345 that starts on the 101st result.
Mar 3 2017