Page MenuHomePhabricator

Elasticsearch search backend requires auto_create_index turned on
Closed, ResolvedPublic

Description

What it says on the tin. The ES integration basically just starts injecting documents into the index. This works unless you've got auto_create_index set to false in which case ES will spit back at you that it couldn't create the index when trying to run bin/search index

Event Timeline

WikiChad created this task.Aug 28 2014, 9:48 PM
WikiChad raised the priority of this task from to Needs Triage.
WikiChad updated the task description. (Show Details)
WikiChad added a project: Search.
WikiChad added a subscriber: WikiChad.

(for a bit of background: we've got this set to false at WMF because ES has a bad habit of not creating indexes exactly as we'd like them so we don't trust it to and have an index-creation script ourselves)

auto_create_index is on by default, right? Users would have had to turn it off explicitly when configuring ElasticSearch, and we'd compensate for this by checking for the value of the setting and then printing a warning reminding them about the configuration change they made? Or is the way we're inserting documents fundamentally incorrect?

No, nothing incorrect about your usage, just need to detect the scenario and error out properly. There's two workarounds:

  • Disable the setting completely, create your index, turn the setting back to how you had it
  • Configure the setting to whitelist the phabricator* index and blacklist the others.

We'll be doing the latter probably.

Cool, thanks!

Actually, we can work around this quite nicely after talking with folks. If auto_create_index is set to false, we just need to create an empty index and it'll figure out the mapping when the first document is inserted. In ES speak, it's just issuing a simple PUT to the bare index that doesn't exist yet: curl -s -XPUT <es-host>:9200/<index-name>

Two ways to do this I can see:

  • Either adding some sort of initialization method to PhabricatorSearchEngine
  • Catching the failure, trying to create the index, then bailing if we can't (or if it happens a second time).
aklapper added a subscriber: aklapper.
scfc added a subscriber: scfc.Sep 3 2014, 6:52 PM
qgil moved this task from Backlog to Details on the Wikimedia board.Sep 30 2014, 7:22 PM
epriestley closed this task as Resolved.Dec 22 2014, 11:55 PM
epriestley claimed this task.

Presuming resolved by D10955, yell if I missed something or there's some followup.