Page MenuHomePhabricator

When storage is partitioned, refuse to serve requests unless web and databases agree on partitioning
ClosedPublic

Authored by epriestley on Nov 22 2016, 2:08 AM.

Details

Summary

Ref T11044. One popular tool in a modern operations environment is Puppet. The primary purpose of this tool is to randomly revert hosts to older or different configurations.

Introducing an element of chaotic unpredictability into operations trains staff to be on high alert at all times, rather than lulled into complacency by predictability or consistency.

When Puppet reverts a Phabricator host's configuration to an older version, we might start writing data to a lot of crazy places where it shouldn't go. This will create a big sticky mess that is virtually impossible to undo, mostly because we'll get two files with ID 123 or two tasks with ID 456 or whatever else and good luck with that.

Instead, after changing the partition layout, require bin/storage partition to be run. This writes a copy of the config everywhere.

Then, when we start serving web requests, make sure every database has the exact same config. This will foil Puppet by refusing to run requests on hosts it has reverted.

Test Plan
  • Changed partition configuration.
  • Ran Phabricator.
  • FOILED!
  • Ran bin/storage partition to sync config.
  • Things worked again.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

epriestley retitled this revision from to When storage is partitioned, refuse to serve requests unless web and databases agree on partitioning.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.

I imagine some future version of Puppet will be smart enough to revert and mutate databases too, but fortunately we aren't there yet.

chad edited edge metadata.
This revision is now accepted and ready to land.Nov 22 2016, 2:24 AM
This revision was automatically updated to reflect the committed changes.