Change Details

D17384 has just landed, making search configuration much more powerful. Just filing this to keep track of followups. [ ] No readable hosts throws setup error? [ ] `indexExists()` may not work correctly for ElasticSearch? [ ] @epriestley is locally seeing `a -b` not work. [ ] Missing "protocol" in ElasticSearch doesn't hit a warning? [ ] If `bin/search index` fails to write to one engine, it fails all engines? (Bad Elastic config, misses write to good MySQL config?) **Guidance** [ ] Write an "Upgrading: ..." guidance task with narrow instructions for installs that are upgrading. [ ] Do we need to add an indexing activity (T11932) for installs with ElasticSearch? [ ] Documentation should provide stronger guidance toward MySQL and away from Elastic for the vast majority of installs, because we've historically seen users choosing Elastic when they aren't actually trying to solve any specific problem. [ ] Some minor upgrade/release guidance on indexing D17300 (`bin/search index --type repository` or whatever) would be nice in the changelog if we don't end up issuing something more grand. [ ] We should more clearly detail exactly which versions of ElasticSearch are supported (for example, is ElasticSearch <2 no longer supported)? From T9893 it seems like we may //only// have supported ElasticSearch <2 before, so are the two regions of support totally nonoverlapping and all ElasticSearch users will need to upgrade? [ ] `bin/search index` should maybe guide you to `bin/search init`? **Future** [ ] When users search for fulltext results in Maniphest and hit too many documents, the current behavior is approximately silent failure (see T12443). D17384 has also lowered the ceiling for ElasticSearch, although previous changes lowered it for MySQL search. We should not fail silently, and ideally should build toward T12003. [ ] When a search cluster is down, we probably don't degrade with much grace (unhandled exception)? [ ] It would be nice to build a practical test suite instead, where we put specific documents into the index and then search for them. The upstream test could run against MySQL, and some `bin/search test` could run against a configured engine like ElasticSearch. This would make it easier to make sure that behavior was as uniform as possible across engine implementations. [ ] Maybe let `bin/search` commands target a specific service. --- [x] Does every assigned task now match "user" in ElasticSearch? [x] `PhabricatorElasticFulltextStorageEngine` has a `json_encode()` which should be `phutil_json_encode()`. [X] Has T8602 been resolved? (Presuming yes.) [X] `PhabricatorSearchEngineTestCase` is now pointless and only detects local misconfigurations. [x] `PhabricatorSearchService` throws an untranslated exception. [X] D17384 added a new "keywords" field, but MySQL does not search it (I think?). The behavior should be as consistent across MySQL and Elastic as we can make it. Likely cleaner is giving "Project" objects a body, with "slugs" and "description" separated by newlines? [X] `bin/search index` should have summary output. [X] The internal index version should not cause us to skip inserts into new engines.

D17384 has just landed, making search configuration much more powerful. Just filing this to keep track of followups. [ ] No readable hosts throws setup error? [ ] `indexExists()` may not work correctly for ElasticSearch? [ ] @epriestley is locally seeing `a -b` not work. [ ] If `bin/search index` fails to write to one engine, it fails all engines? (Bad Elastic config, misses write to good MySQL config?) **Guidance** [ ] Write an "Upgrading: ..." guidance task with narrow instructions for installs that are upgrading. [ ] Do we need to add an indexing activity (T11932) for installs with ElasticSearch? [ ] Documentation should provide stronger guidance toward MySQL and away from Elastic for the vast majority of installs, because we've historically seen users choosing Elastic when they aren't actually trying to solve any specific problem. [ ] Some minor upgrade/release guidance on indexing D17300 (`bin/search index --type repository` or whatever) would be nice in the changelog if we don't end up issuing something more grand. [ ] We should more clearly detail exactly which versions of ElasticSearch are supported (for example, is ElasticSearch <2 no longer supported)? From T9893 it seems like we may //only// have supported ElasticSearch <2 before, so are the two regions of support totally nonoverlapping and all ElasticSearch users will need to upgrade? [ ] `bin/search index` should maybe guide you to `bin/search init`? **Future** [ ] When users search for fulltext results in Maniphest and hit too many documents, the current behavior is approximately silent failure (see T12443). D17384 has also lowered the ceiling for ElasticSearch, although previous changes lowered it for MySQL search. We should not fail silently, and ideally should build toward T12003. [ ] When a search cluster is down, we probably don't degrade with much grace (unhandled exception)? [ ] It would be nice to build a practical test suite instead, where we put specific documents into the index and then search for them. The upstream test could run against MySQL, and some `bin/search test` could run against a configured engine like ElasticSearch. This would make it easier to make sure that behavior was as uniform as possible across engine implementations. [ ] Maybe let `bin/search` commands target a specific service. --- [x] Does every assigned task now match "user" in ElasticSearch? [x] `PhabricatorElasticFulltextStorageEngine` has a `json_encode()` which should be `phutil_json_encode()`. [X] Has T8602 been resolved? (Presuming yes.) [X] `PhabricatorSearchEngineTestCase` is now pointless and only detects local misconfigurations. [x] `PhabricatorSearchService` throws an untranslated exception. [X] D17384 added a new "keywords" field, but MySQL does not search it (I think?). The behavior should be as consistent across MySQL and Elastic as we can make it. Likely cleaner is giving "Project" objects a body, with "slugs" and "description" separated by newlines? [X] `bin/search index` should have summary output. [X] The internal index version should not cause us to skip inserts into new engines. [X] Missing "protocol" in ElasticSearch doesn't hit a warning? (Resolved elsewhere, it seems.)

D17384 has just landed, making search configuration much more powerful. Just filing this to keep track of followups. [ ] No readable hosts throws setup error? [ ] `indexExists()` may not work correctly for ElasticSearch? [ ] @epriestley is locally seeing `a -b` not work. [ ] Missing "protocol" in ElasticSearch doesn't hit a warning? [ ] If `bin/search index` fails to write to one engine, it fails all engines? (Bad Elastic config, misses write to good MySQL config?) **Guidance** [ ] Write an "Upgrading: ..." guidance task with narrow instructions for installs that are upgrading. [ ] Do we need to add an indexing activity (T11932) for installs with ElasticSearch? [ ] Documentation should provide stronger guidance toward MySQL and away from Elastic for the vast majority of installs, because we've historically seen users choosing Elastic when they aren't actually trying to solve any specific problem. [ ] Some minor upgrade/release guidance on indexing D17300 (`bin/search index --type repository` or whatever) would be nice in the changelog if we don't end up issuing something more grand. [ ] We should more clearly detail exactly which versions of ElasticSearch are supported (for example, is ElasticSearch <2 no longer supported)? From T9893 it seems like we may //only// have supported ElasticSearch <2 before, so are the two regions of support totally nonoverlapping and all ElasticSearch users will need to upgrade? [ ] `bin/search index` should maybe guide you to `bin/search init`? **Future** [ ] When users search for fulltext results in Maniphest and hit too many documents, the current behavior is approximately silent failure (see T12443). D17384 has also lowered the ceiling for ElasticSearch, although previous changes lowered it for MySQL search. We should not fail silently, and ideally should build toward T12003. [ ] When a search cluster is down, we probably don't degrade with much grace (unhandled exception)? [ ] It would be nice to build a practical test suite instead, where we put specific documents into the index and then search for them. The upstream test could run against MySQL, and some `bin/search test` could run against a configured engine like ElasticSearch. This would make it easier to make sure that behavior was as uniform as possible across engine implementations. [ ] Maybe let `bin/search` commands target a specific service. --- [x] Does every assigned task now match "user" in ElasticSearch? [x] `PhabricatorElasticFulltextStorageEngine` has a `json_encode()` which should be `phutil_json_encode()`. [X] Has T8602 been resolved? (Presuming yes.) [X] `PhabricatorSearchEngineTestCase` is now pointless and only detects local misconfigurations. [x] `PhabricatorSearchService` throws an untranslated exception. [X] D17384 added a new "keywords" field, but MySQL does not search it (I think?). The behavior should be as consistent across MySQL and Elastic as we can make it. Likely cleaner is giving "Project" objects a body, with "slugs" and "description" separated by newlines? [X] `bin/search index` should have summary output. [X] The internal index version should not cause us to skip inserts into new engines. [X] Missing "protocol" in ElasticSearch doesn't hit a warning? (Resolved elsewhere, it seems.)