Recent security issues in GitHub, GitLab, etc., with markdown:

Then please fork Phabricator to provide this option on your local install. One interested party every few years isn't enough interest for us to add an option, and it's probably 1-2 lines of code for you to provide it for yourself.

I disagree. This option is a lot like email notification options which are already present in Phabricator. It expresses very personal preferences of how user prefers to read the same content. And for me it is more important than ability to set board backgrounds.

See T8227: Why not just add an option/setting/preference?

Hello, here is my use case.

Per the latest update in D17502, this remains involved.

try to get harbormaster to build (push to staging?)

• actually, acutally utilize the health monitoring...

• Improved the status monitoring UI in config/cluster/search/
• Actually utilize the health monitoring cache to avoid connecting to downed servers.

@epriestley sweet, I'll land this as soon as I see that you've merged to stable.

I don't see any noteworthy issues here.

Haha, thanks. Let me finish testing one other change and then I'll give this another look.

@epriestley: I think this is ready to land but I want to give you one more chance to change your mind.

• Created diviner documentation: Cluster: Search
• removed stray phlog

• Fix searching relationships which I had inadvertantly broken.
• Better elasticsearch 2.x and 5.x support
• more optimized query

Fix method signature un-final PhabricatorElasticFulltextStorageEngine

Hmm. Ok. Deleted.

I'm going to probably re-design this page a little.

Specifically, this task is about how the javascript in typeaheads rank and match choices.

Not similar.

Ok I think I've eliminated the problematic parts like indexing project slugs.

Get rid of static.

address review feedback that I hadn't gotten to yet.

Note: I'm not sure why harbormaster is failing?

• Cleaned up the elastic query and added comments describing the purpose of the clauses
• a couple of bugfixes found by further testing

Ok I've reworked this quite a bit and I may have messed up somewhere in the process.

I think we could even default it to "fancy mode" without a setting, at least to start with. I'm not sure anyone will want to access the plain text mode with enough frequency that we need it to be sticky.

@epriestley this doesn't seem hard to build for a fledgling designer/intern engineer.

In D17384#209987, @epriestley wrote:

Just to make sure I haven't missed anything:

• We currently write health checks but never read them, right? So there's no effect (other than the UI "Status" changing) when a service fails health checks? That seems fine for now, I just want to make sure I didn't miss a health check read somewhere.

Just to make sure I haven't missed anything:

rebased on top of D17509: Updated PhabricatorElasticFulltextStorageEngine for elasticsearch 5

• Move the stats definitions into the engine so the status UI remains engine agnostic.
• Fix a bug where role => false was being treated like role => true in the UI

I'm pleased to report that this has been live on wikimedia's phabricator for about a week without any incidents whatsoever. Additionally, we are in the process of migrating from elasticsearch 2.x to 5.x and the ability to write to multiple clusters is really working out nicely for transition.

T9789 has stabilized (the majority of major transaction types now run through ModularTransactions) although it didn't end up providing much clarity here. This is no longer blocked on anything, I just need to develop a specific technical plan to resolve the "adjust" vs "expand" sequence inside transaction application so the "expand" code has access to the old text by the time it runs.

That sounds reasonable to me. We remain open to more feedback here if it does turn up, but I think we don't have a clear set of next steps otherwise. Thanks for following up!

Here are some additional use cases and a sample implementation that is a Burn-up chart, not a Burndown chart, plus rationale.

In T11034#201013, @epriestley wrote:

If there's some remaining use case here, I'd also like to step back and understand it better since I suspect the best solution we can pursue is not any kind of UI change which makes it easier to find things by randomly typing guesses into the typeahead.

• Added index stas to status ui
• Separate mysql status from elasticsearch status and show different set of columns appropriate to each cluster type.

• Remove unused healthrecord stuff from PhabricatorSearchCluster class
• Add back getDisplayName to the PhabricatorSearchCluster class because it's needed.

Addressed latest round of feedback.

Few more minor things.

$limit = 10000 - $offset

• updated PhabricatorExtraConfigSetupCheck
• 10,000 results
• remove unused methods

This is now live on https://phab-01.wmflabs.org for testing. Everything seems to be working well, including the health monitoring.

In D17384#209367, @epriestley wrote:

This is shaping up nicely, couple of other minor inlines.

This is shaping up nicely, couple of other minor inlines.

Fix unit test case.

Getting closer...

I agree that results should almost always be on the first page and ideally near the top of the first page. I'm also tempted to say that 10 results per page might be closer to the right number than 100.

I believe that's unavoidable with offset-based paging (the solution is to switch to cursor-based paging, but I think we can't do that for fulltext).

I'll investigate further into the slowness (I wasn't involved in the debugging so far so I don't know more than what's said in the task)

In D17384#208882, @epriestley wrote:

• Using the same objects as both Host and Service feels confusing to me. I think this would probably be clearer as separate Service and Host classes? Like PhabricatorMySQLSearchClusterService extends PhabricatorSearchClusterService and PhabricatorMySQLSearchClusterHost extends PhabricatorSearchClusterHost or similar. Particularly because setHostRefs() seems like it's getting called with a raw dictionary in one case and a list of objects in another? And then there's weird magic around getHostRefs() for the MySQL case?

I'll split out the changes to the engine if I can figure out how to do that... Update coming soon.

(That is, the expected failure mode of this query when it fails to scale is "silently return an incomplete result set", not "be really slow".)

To start with here, let's identify exactly what's slow? It's not clear to me from the downstream task unless I'm just misreading, and not evident to me from reasoning about the behavior of this query. This query strategy is fundamentally not scalable, but I would not expect it to be slow unless the ElasticSearch query is slow.

The results are ordered by "match strength", but this is computed at runtime and isn't unique to each result: we can't start with the 100th result and use it to build any sort of clause like WHERE rank > 12345 that starts on the 101st result.

General stuff:

@epriestley: Ok I believe this addresses all of your feedback and other than documentation it should be very close to finished.

Address epriestley's feedback about tooltip and string concatenation

• Fixed up the cli workflows for search init and search index
• Misc other cleanup

Fix elasticsearch setup checks