Page MenuHomePhabricator
Create Task
Maniphest T9670

Conflicting field mappings for phabricator index when upgrading to Elasticsearch 2.0
Closed, WontfixPublic
Actions

Description

Elasticsearch 2.0 is now released, but when using the migration plugin to check if we can upgrade smoothly there is an index mapping conflict (more info about the mapping conflicts here) for the phabricator index.

The problem seems to be that the .when field is sometimes pushed as a number field containing a unix timestamp, and sometimes a string field containing a unix timestamp. So with the new strict mapping regime in ES 2.0, this will not be allowed.

Have you been seeing this yourself?

Here is the output from the migration plugin:

phabricator_index_mapping_conflict.png (118×869 px, 17 KB)

Revisions and Commits

rP Phabricator
D14370rP4e112537b255 Probable fix for ElasticSearch 2.0 type strictness

Related Objects
Search...

Event Timeline

kozejonaz created this task.Oct 29 2015, 2:58 PM
kozejonaz updated the task description. (Show Details)
kozejonaz added a project: Elasticsearch.
kozejonaz added a subscriber: kozejonaz.
tycho.tatitscheff added a subscriber: tycho.tatitscheff.Oct 29 2015, 3:04 PM

Uhm I tested the same last month when 2.0 was still beta then conclude that phabricator only supported 1.x.
I will follow the resolution of this.

kozejonaz updated the task description. (Show Details)Oct 29 2015, 3:07 PM
kozejonaz added a comment.Oct 29 2015, 3:16 PM

After a closer look there seem to be quite a lot of inconsistencies for the different .when fields.

Examples:

FieldType
relationship.subs.whenstring
relationship.auth.whenstring
relationship.open.whennumber
relationship.ownr.whenconflict
relationship.unow.whenstring
relationship.proj.whenstring
relationship.clos.whennumber
relationship.repo.whenstring

As you can see, relationship.ownr.when is the one we have problems with right now.
I guess all the other ones are currently being consistently pushed as either number or string

joshuaspence added a subscriber: joshuaspence.Oct 29 2015, 8:14 PM
epriestley added a revision: D14370: Probable fix for ElasticSearch 2.0 type strictness.Oct 30 2015, 5:00 PM
epriestley added a subscriber: epriestley.Oct 30 2015, 5:02 PM

D14370 likely fixes this, although I don't have ElasticSearch 2.0 locally and haven't tested it. I'd guess this will give you a clean bill of health:

  • Apply the patch;
  • run bin/search index --all to rebuild all indexes;
  • run the ElasticSearch tool again.

Can you let me know if that works?

epriestley added a comment.Oct 30 2015, 5:03 PM

Note that index --all queues stuff into the background, so you may need to wait a while before it finishes. You can use --foreground (slower) or monitor progress from /daemon/ in the web UI.

epriestley added a comment.Oct 30 2015, 5:03 PM

Oh, I think I misremembered and it does foreground by default. You can use --background to improve performance, and monitor from /daemon/.

tycho.tatitscheff added a comment.EditedOct 30 2015, 5:16 PM

I tested and i does not fix !

Well, I didn't move to elastic 2.0 but restarting elasticsearch, reindexing rerestarting and reruning plugin doesn't leave me the same error.

kozejonaz added a comment.Oct 31 2015, 12:28 PM
In T9670#142544, @epriestley wrote:

D14370 likely fixes this, although I don't have ElasticSearch 2.0 locally and haven't tested it. I'd guess this will give you a clean bill of health:

  • Apply the patch;
  • run bin/search index --all to rebuild all indexes;
  • run the ElasticSearch tool again.

Can you let me know if that works?

Thanks for the quick response @epriestley, we'll try it out first thing Monday just to see if we have the same results as @tycho.tatitscheff.

kozejonaz added a comment.Nov 2 2015, 10:26 AM
In T9670#142547, @tycho.tatitscheff wrote:

I tested and i does not fix !

Well, I didn't move to elastic 2.0 but restarting elasticsearch, reindexing rerestarting and reruning plugin doesn't leave me the same error.

@tycho.tatitscheff: Hey, just so I'm not misunderstanding anything. Did D14370 fix the issue for you? It seems we won't be able to test this until later this week, but if you're saying it worked than I have no reason to believe it should fail for us.

tycho.tatitscheff added a comment.Nov 2 2015, 11:41 AM

I patched with D14370 and still get an issue. I can retry but I pretty sure I did think corectly.

kozejonaz added a comment.EditedNov 2 2015, 4:12 PM

@epriestley/@tycho.tatitscheff: We managed to test D14370 now, and it totally fixes our problem!

Steps to test:

  1. Apply the patch
  2. Delete the phabricator index
  3. bin/search init to create the index
  4. bin/search index --all to rebuild the index

Results from the migration plugin after rebuilding the index can be seen in

phabricator_migration_success.png (736×880 px, 82 KB)
.

I don't know what other places might cause similar issues, but for our use-case this is pretty sweet. Thanks @epriestley!

@tycho.tatitscheff: I assume your problem is that you didn't delete the Phabricator index first?

epriestley closed this task as Resolved by committing rP4e112537b255: Probable fix for ElasticSearch 2.0 type strictness.Nov 2 2015, 4:21 PM
epriestley added a commit: rP4e112537b255: Probable fix for ElasticSearch 2.0 type strictness.
epriestley added a comment.Nov 2 2015, 4:22 PM

Thanks for testing the patch, should be in HEAD now. Let us know if you run into anything else.

Not sure what @tycho.tatitscheff ran into.

tycho.tatitscheff added a comment.Nov 2 2015, 4:42 PM

@epriestley i think I didn't delete index first.
Just bin/search index --all.

I will apply head, delete index and phabricator template (custom one i made to improve phabricator handling french) and retest.

kozejonaz added a comment.Nov 2 2015, 5:08 PM
In T9670#142765, @epriestley wrote:

Thanks for testing the patch, should be in HEAD now. Let us know if you run into anything else.

Not sure what @tycho.tatitscheff ran into.

Awesome, appreciated! Will do!

ru31337 changed the task status from Resolved to Wontfix.EditedNov 9 2015, 12:54 AM
ru31337 claimed this task.
ru31337 added a subscriber: ru31337.

Index _status in 2.0 replaced _stats https://www.elastic.co/guide/en/elasticsearch/reference/2.0/indices-status.html

./bin/search init
[2015-11-09 03:51:33] EXCEPTION: (HTTPFutureHTTPResponseStatus) [HTTP/400] 
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No feature for name [_status]"}],"type":"illegal_argument_exception","reason":"No feature for name [_status]"},"status":400} at [<phutil>/src/future/http/BaseHTTPFuture.php:339]
arcanist(head=master, ref.master=2db40f995337), phabricator(head=master, ref.master=a2f909f0bd0a), phutil(head=master, ref.master=610ca9eba906)
  #0 BaseHTTPFuture::parseRawHTTPResponse(string) called at [<phutil>/src/future/http/HTTPSFuture.php:415]
  #1 HTTPSFuture::isReady() called at [<phutil>/src/future/Future.php:37]
  #2 Future::resolve() called at [<phutil>/src/future/http/BaseHTTPFuture.php:279]
  #3 BaseHTTPFuture::resolvex() called at [<phabricator>/src/applications/search/engine/PhabricatorElasticSearchEngine.php:425]
  #4 PhabricatorElasticSearchEngine::executeRequest(string, array) called at [<phabricator>/src/applications/search/engine/PhabricatorElasticSearchEngine.php:286]
  #5 PhabricatorElasticSearchEngine::indexExists() called at [<phabricator>/src/applications/search/management/PhabricatorSearchManagementInitWorkflow.php:19]
  #6 PhabricatorSearchManagementInitWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:406]
  #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:301]
  #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/search/manage_search.php:21]

In phabricator issue "Elasticsearch Misconfigured"

[HTTP/400] 

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No feature for name [_status]"}],"type":"illegal_argument_exception","reason":"No feature for name [_status]"},"status":400}
epriestley added a parent task: T9893: Support ElasticSearch 2.0 - 5.1.Dec 3 2015, 2:08 PM

@ru31337, that issue isn't related to this one and isn't "wontfix". Please see Contributing Bug Reports for instructions on how to report bugs. Choosing an existing similar task, changing it to "wontfix" and dumping your issue into it won't get it fixed.

Phabricator · Built by Phacility