- User Since
- Nov 28 2011, 9:35 AM (278 w, 2 d)
push to staging for harbormaster
Better formatting of setup warning messages.
this fixes the stemmer and tokenizer to do a better job of matching words.separated.by.punctuation as well as other issues found by @epriestley.
I've updated D17564: Address some New Search Configuration Errata to address the tokenization and word stemming issues.
- Fixed the stemmer. user matches users and vise-versa.
- Added a different tokenizer so that this.is.a.test tokenizes to the following:
Mon, Mar 27
trying once more...
Try to make harbormaster happy by setting repository.callsign globally in ~/.arcrc
Maniphest advanced search is somewhat buried, indeed. I think one easy solution to this would be to add "Task search" to the main phab menu (using the new custom menus feature)... In fact, I think I will do that now at https://phabricator.wikimedia.org
f*a*c*t*o*r*y*s*u*r*p*l*u*s*z*z*q*q*z*z*q*q returns the same results as
f a c t o r y s u r p l u s z z q q z z q q so it appears to be treating those as individual single-letter tokens. strange.
I think it would make a lot of sense to construct the two queries separately (and in parallel) with a short timeout, then handle the timeout gracefully allowing the user to refine their query further. This would avoid the denial of service situation which happened to Wikimedia more than once due to users repeatedly executing really expensive searches until mysql fell over from the load.
Sun, Mar 26
@epriestley: Thanks for the detailed feedback... I'll get to work ;)
- Has T8602 been resolved?
I can not reproduce it on wikimedia's install.
- Write an "Upgrading: ..." guidance task with narrow instructions for installs that are upgrading.
- Do we need to add an indexing activity (T11932) for installs with ElasticSearch?
Yes, I think so
- We should more clearly detail exactly which versions of ElasticSearch are supported (for example, is ElasticSearch <2 no longer supported)? From >T9893 it seems like we may only have supported ElasticSearch <2 before, so are the two regions of support totally nonoverlapping and all ElasticSearch users will need to upgrade?
I haven't been testing with elasticsearch < 2.0 so this might break backwards compatibility. It wouldn't be difficult to fix any compatibility issues though, with a tiny bit of testing.
With the elasticsearch 'simple_query_string' query parser it only works if you use *pricot, for example, outside of quoted phrases.
Note there will finally be a little bit of documentation once this install rebuilds diviner docs: The url should be https://secure.phabricator.com/book/phabricator/article/cluster_search/ (eventually)
This should work just fine with the index mapping and query generation in rPe41c25de5050: Support multiple fulltext search clusters with 'cluster.search' config
resubmit with arc diff --config repository.callsign=P
Addressed epriestley's feedback.
try to get harbormaster to build (push to staging?)
Sat, Mar 25
- actually, acutally utilize the health monitoring...
- Improved the status monitoring UI in config/cluster/search/
- Actually utilize the health monitoring cache to avoid connecting to downed servers.
Thu, Mar 23
@epriestley sweet, I'll land this as soon as I see that you've merged to stable.
I can confirm that In Any: does not seem to include subprojects. I tried to make some sense of the way the project search functions work but it's pretty complicated.
@epriestley: I think this is ready to land but I want to give you one more chance to change your mind.
- Created diviner documentation: Cluster: Search
- removed stray phlog
- Fix searching relationships which I had inadvertantly broken.
- Better elasticsearch 2.x and 5.x support
- more optimized query
Elasticsearch has much better support for non-latin language analysis. See https://www.elastic.co/guide/en/elasticsearch/guide/current/icu-tokenizer.html discusses their ability to properly tokenize Thai, Chinese and Japanese text.
Wed, Mar 22
Fix method signature un-final PhabricatorElasticFulltextStorageEngine
Ok I think I've eliminated the problematic parts like indexing project slugs.
Get rid of static.
address review feedback that I hadn't gotten to yet.
Note: I'm not sure why harbormaster is failing?
- Cleaned up the elastic query and added comments describing the purpose of the
- a couple of bugfixes found by further testing
Ok I've reworked this quite a bit and I may have messed up somewhere in the process.
Tue, Mar 21
So I've done a bit more thinking about how to implement the changes to the engine class, especially with regards to any bits that are not wanted in the upstream but are desirable for wikimedia's implementation.
Mon, Mar 20
@avivey: I'm somewhat interested in this if you have any tips for getting it working locally I would like to try it out and see if I can contribute anything towards a finished extension.