Page MenuHomePhabricator
Create Task
Maniphest T11360

Detect MySQL `read_only` and degrade masters to replicas
Open, NormalPublic
Actions

Description

mysqld can be set to read-only with --read-only at startup, which is reflected in the read_only global.

When we attempt to perform writes and they fail because this mode is set, we should degrade to read-only mode as though cluster.read-only was set.

In a true master + replica cluster configuration this is probably moot (you'd just take down whichever database you needed to, probably?) but if operations and Phabricator administration aren't the same teams (or the ops team is just more familiar with MySQL than advanced Phabricator options) the intent of putting MySQL into --read-only is unambiguous.

Event Timeline

epriestley created this task.Jul 22 2016, 3:21 AM
Herald added a subscriber: eadler. · View Herald TranscriptJul 22 2016, 3:21 AM
eadler added a project: Restricted Project.Jul 25 2016, 4:23 PM
20after4 added a subscriber: 20after4.Jul 27 2016, 10:39 AM
epriestley added a project: Contributor Onboarding.Aug 22 2016, 5:34 PM

Although this is cluster-related, it can be tested with a single database host and is probably not exceptionally difficult to handle better. Briefly:

  • Connect to your MySQL server and put it in read_only mode. Maybe you can do this with set global read_only = 1? That doesn't actually seem to work for me. Otherwise you might have to change config and restart it.
  • Try to perform a write (e.g., edit a task).
  • Observe that Phabricator breaks in a sort-of confusing way.

We could improve this behavior and treat read_only mode more like the setting cluster.read-only.

However, detecting this mode is a little tricky -- I don't want to add a show variables like @@read_only call to every page, since 99.999% of the time this would just slow things down. We probably need to catch whatever exception read_only raises somewhere and flag the database as iffy in cache until we get a good health check.

epriestley moved this task from Backlog to Ready on the Contributor Onboarding board.Apr 7 2017, 1:07 PM
Herald added a subscriber: tycho.tatitscheff. · View Herald TranscriptApr 7 2017, 1:07 PM
epriestley moved this task from Ready to Intermediate on the Contributor Onboarding board.Apr 9 2017, 10:53 AM
Phabricator · Built by Phacility