Page MenuHomePhabricator

Automatically sever databases after prolonged unreachability

Authored by epriestley on Apr 10 2016, 9:51 PM.
Referenced Files
Unknown Object (File)
Thu, Aug 11, 9:04 PM
Unknown Object (File)
Sun, Aug 7, 8:07 AM
Unknown Object (File)
Fri, Aug 5, 10:12 PM
Unknown Object (File)
Fri, Aug 5, 10:02 AM
Unknown Object (File)
Wed, Aug 3, 2:41 PM
Unknown Object (File)
Mon, Jul 25, 3:41 AM
Unknown Object (File)
Sun, Jul 24, 12:15 PM
Unknown Object (File)
Fri, Jul 22, 6:56 PM



Ref T4571. When a database goes down briefly, we fall back to replicas.

However, this fallback is slow (not good for users) and keeps sending a lot of traffic to the master (might be bad if the root cause is load-related).

Keep track of recent connections and fully degrade into "severed" mode if we see a sequence of failures over a reasonable period of time. In this mode, we send much less traffic to the master (faster for users; less load for the database).

We do send a little bit of traffic still, and if the master recovers we'll recover back into normal mode seeing several connections in a row succeed.

This is similar to what most load balancers do when pulling web servers in and out of pools.

For now, the specific numbers are:

  • We do at most one health check every 3 seconds.
  • If 5 checks in a row fail or succeed, we sever or un-sever the database (so it takes about 15 seconds to switch modes).
  • If the database is currently marked unhealthy, we reduce timeouts and retries when connecting to it.
Test Plan
  • Configured a bad master.
  • Browsed around for a bit, initially saw "unrechable master" errors.
  • After about 15 seconds, saw "major interruption" errors instead.
  • Fixed the config for master.
  • Browsed around for a while longer.
  • After about 15 seconds, things recovered.
  • Used "Cluster Databases" console to keep an eye on health checks: it now shows how many recent health checks were good:

Screen Shot 2016-04-10 at 2.50.41 PM.png (136×658 px, 25 KB)

Diff Detail

rP Phabricator
Lint Not Applicable
Tests Not Applicable

Event Timeline

epriestley retitled this revision from to Automatically sever databases after prolonged unreachability.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.
chad edited edge metadata.
This revision is now accepted and ready to land.Apr 10 2016, 10:36 PM
This revision was automatically updated to reflect the committed changes.