Changing the type is going to run into the issue of what to do about the fields which differ between the two types. Fields which are present in the old type will continue to be displayed unless you clear them when changing types.

I implemented this recently but then pulled it out because ... project slug lookup is a gigantic mess right now.

@dreadlord2203 indeed, hiding those tags would be useful. I found one problem with using a separate repository for change handoff, and that is access control. If you maintain several repositories with different access controls, then you would have to duplicate each of them and maintain a second set of access controls for the corresponding staging repositories. This is a lot of extra work and it would be easy to make a mistake and leak the staging repo to people who shouldn't be able to see it.

Maybe let bin/search commands target a specific service.

It would be nice to build a practical test suite instead, where we put specific documents into the index and then search for them.

So I ran into one problem when deploying the latest code to WMF production. We now throw a setup error if we have a cluster with no readable hosts. It actually makes sense to have a cluster that is write-only so that setup error is bogus.

@cos: probably should not have a default value.

I've updated {D17564} to address the tokenization and word stemming issues.

Maniphest advanced search is somewhat buried, indeed. I think one easy solution to this would be to add "Task search" to the main phab menu (using the new custom menus feature)... In fact, I think I will do that now at https://phabricator.wikimedia.org

In T12450#216943, @epriestley wrote:

---

• Searching for f*a*c*t*o*r*y*s*u*r*p*l*u*s*z*z*q*q*z*z*q*q produces nonsenical results (many results, when I would expect no results: the results do not contain that sequence of letters in order).
• Searching or user fails to find task Grant users tokens when a mention is created, suggesting that stemming is not working.
• Searching for users finds that task, but fails to find a task containing "per user per month" in a comment, also suggesting that stemming is not working.
• Searching for maniphest fails to find task maniphest.query elephant, suggesting that tokenization is ElasticSearch is not as good as the MySQL tokenization for these words (see D17330).

f*a*c*t*o*r*y*s*u*r*p*l*u*s*z*z*q*q*z*z*q*q returns the same results as
f a c t o r y s u r p l u s z z q q z z q q so it appears to be treating those as individual single-letter tokens. strange.

I think it would make a lot of sense to construct the two queries separately (and in parallel) with a short timeout, then handle the timeout gracefully allowing the user to refine their query further. This would avoid the denial of service situation which happened to Wikimedia more than once due to users repeatedly executing really expensive searches until mysql fell over from the load.

In T12450#216943, @epriestley wrote:

I ran into a lot of confusion because the versioned object indexes are not namespaced per-service. Basically, if you insert version 95 of a document into Elastic, the indexer thinks that version 95 doesn't need to go into MySQL, even though it does. So when you run bin/search index ..., you may get only a subset of the updates you actually need. The object index versions need to change to become engine-aware so they are stored per-service, not globally, and/or the whole mechanism needs to include a hash of cluster.search or just be turned off. Until this is fixed, it can be worked around with using --force everywhere.

bin/search index might reasonably provide summary output about this ("392 documents were not indexed because they haven't changed, use --force to update them.").

In T12450#216943, @epriestley wrote:

---

• Searching for f*a*c*t*o*r*y*s*u*r*p*l*u*s*z*z*q*q*z*z*q*q produces nonsenical results (many results, when I would expect no results: the results do not contain that sequence of letters in order).
• Searching or user fails to find task Grant users tokens when a mention is created, suggesting that stemming is not working.
• Searching for users finds that task, but fails to find a task containing "per user per month" in a comment, also suggesting that stemming is not working.
• Searching for maniphest fails to find task maniphest.query elephant, suggesting that tokenization is ElasticSearch is not as good as the MySQL tokenization for these words (see D17330).
• Searching for users -blue returns a huge number of results: significantly more than users. Expected behavior: fewer results, omitting those results matching blue.
• Searching for users blue returns more results than users or blue. Expected behavior: fewer results, because only results which match "users" AND "blue" are returned. The result set includes completely irrelevant results.

@epriestley: Thanks for the detailed feedback... I'll get to work ;)

• Has T8602 been resolved?

I can not reproduce it on wikimedia's install.

• Write an "Upgrading: ..." guidance task with narrow instructions for installs that are upgrading.

TODO

• Do we need to add an indexing activity (T11932) for installs with ElasticSearch?

Yes, I think so

• We should more clearly detail exactly which versions of ElasticSearch are supported (for example, is ElasticSearch <2 no longer supported)? From >T9893 it seems like we may only have supported ElasticSearch <2 before, so are the two regions of support totally nonoverlapping and all ElasticSearch users will need to upgrade?

I haven't been testing with elasticsearch < 2.0 so this might break backwards compatibility. It wouldn't be difficult to fix any compatibility issues though, with a tiny bit of testing.

With the elasticsearch 'simple_query_string' query parser it only works if you use *pricot, for example, outside of quoted phrases.

Note there will finally be a little bit of documentation once this install rebuilds diviner docs: The url should be https://secure.phabricator.com/book/phabricator/article/cluster_search/ (eventually)

This should work just fine with the index mapping and query generation in {rPe41c25de5050}

I can confirm that In Any: does not seem to include subprojects. I tried to make some sense of the way the project search functions work but it's pretty complicated.

In T12441#216428, @epriestley wrote:

If there's any good content in this feature at all, why do I never see it reposted to Reddit or Facebook or Twitter? Are Reddit and Twitter just for old people now?

Elasticsearch has much better support for non-latin language analysis. See https://www.elastic.co/guide/en/elasticsearch/guide/current/icu-tokenizer.html discusses their ability to properly tokenize Thai, Chinese and Japanese text.

In T12296#216132, @epriestley wrote:

While the git ls-remote change isn't really motivated as a performance improvement, it does seem to have reduced CPU usage a measurable amount (deployed on the morning of 3/18), maybe 15%:

@avivey: I'm somewhat interested in this if you have any tips for getting it working locally I would like to try it out and see if I can contribute anything towards a finished extension.

@stettberger: If there is anything missing from the WMF policy API extension, please feel free to file tasks under https://phabricator.wikimedia.org/tag/wikimedia_phabricator_extensions/ or open a differential revision against https://phabricator.wikimedia.org/source/phab-extensions/. I will be glad to review patches, I can't make any promises about feature requests but reasonable changes will be considered.

What I would find especially useful would be a way to create named policies and then reference them when creating objects. That might warrant a separate feature-request task though.

I don't think this is related to wikimedia since we already have the functionality live in the WMF production install.

@ksmith: Thanks to the new 'favorites' menu, each user can customize their menu, there is no longer a global 'create new' menu, as it's been replaced.

In T12314#213245, @epriestley wrote:

Weakness: Subtyping is Probably Only Useful in Maniphest

Subtyping is likely to involve a large number of changes in shared infrastructure (EditEngine), but they will probably only ever be useful in Maniphest. Although most applications now use EditEngine, I can't really come up with any good use cases for subtyping in other applications. Perhaps Calendar could use subtyping on events, but this feels like a solution searching for a problem.

I don't suppose users count as objects in this context? I'd love to have a way to be notified of new @mentions for my username. A client app that queries conduit occasionally and sends me desktop notifications, well, that would just be super cool.

I agree that results should almost always be on the first page and ideally near the top of the first page. I'm also tempted to say that 10 results per page might be closer to the right number than 100.

I'll investigate further into the slowness (I wasn't involved in the debugging so far so I don't know more than what's said in the task)

This actually seems a little difficult to fix...maybe just make it truncate with an ellipsis?

heh nice. And I didn't even have to cheat by using U+00A0.

For wikimedia's "scap" tool we built a feature called "checks" which are configurable commands that run on each target machine and confirm if the desired commands were successful.

This is something I've wished for a few times. Along with just searching for repositories in general.

The commit cache should be a fairly large performance win.

My impression is that there is a lot of room for optimization in the RepositoryPullLocalDaemon. I've wanted to tackle that for a while because it causes a fair delay between when a commit goes into git and when it actually shows up in Diffusion.

FWIW I think this is pretty genius. Especially the repro-or-it-didn't-happen aspect.

I totally get the motivation to have a strong collection of highly integrated tools which all work together seamlessly and provide a value that is larger than the sum of it's parts.

Cool stuff. I didn't see you already had made so much progress before I commented on T12218. You can safely disregard my clustershell suggestion.

@epriestley: Have you seen clustershell? It's a massively scaleable management shell with some pretty interesting features. Wikimedia operations are adopting it in place of a lot of dsh and saltstack stuff. From what I have seen it is a really solid framework to build upon.

Interestingly, phabricator_maniphest maniphest_transaction gives zero results but "phabricator_maniphest" "maniphest_transaction" works. Seems like the underscores are a problem as much as the period.

Maybe phabricator should automatically try adding quotes to a search if the initial query returns zero results?

@epriestley: This all sounds excellent. I'll start by fixing any instances of

I have this working downstream in Wikimedia's instance of phabricator, however, our fork is significantly modified. I intend to refactor this into something that is either a) mergeable upstream or b) installable as an extension but for now the code is on phabricator.wikimedia.org PhabricatorElasticFulltextStorageEngine.php

So, @epriestley: How do you feel about me taking over elasticsearch backend and moving it out to an extension rather than keeping it in phabricator core? The other possibility is I could just fork the elastic backend and name it something different, as it seems phabricator already supports loading the engine extension dynamically.

@ramm: I'm already on it: https://phabricator.wikimedia.org/T155299

In T9530#177353, @hach-que wrote:

So I was thinking about software components recently and one of the issues I've had with both Jenkins and Harbormaster is that builds of one repository aren't aware of the builds from another repository. This is a common scenario in the software I build:

Our DBAs are currently in the process of testing the a patched mariadb package. I will try to keep you posted on that progress but it looks like the fix is now backported to 10.0

In T5000#176440, @epriestley wrote:

Another possibility is that we just build this behavior (intercept and react to ref changes) as an extension point prior to the Herald, and then you can implement whatever magic you want. I suspect very few installs want the default behavior of git push origin mybranch for arbitrarily named branches to be anything except "create a branch".

This all seems reasonable, I'm actually not 100% sure what happened so I'll try to gather a better understanding of the situation ;)

FWIW: I believe @paladox means well, he's just young/inexperienced and overly enthusiastic. The decision to ban him is entirely reasonable for upstream though, given the circumstances.

In T11034#199653, @epriestley wrote:

---

Now that this has been deployed for a while, I'm curious if there's any feedback about it? Specifically:

• Does this approach generally seem like it's a reasonable solution to the problem (if not in its current form, in some similar form which handles these issues more gracefully)? Or are we not really in the right ballpark?