Page MenuHomePhabricator
Create Task
Maniphest T11042

Cluster documentation leads you slightly awry about "read only mode"
Closed, ResolvedPublic
Actions

Description

The database clustering doc prescribes a replica data export method by which you:

  • Turn off the daemons.
  • Turn on read only mode
  • ./bin/storage dump --for-replica
  • Bring stuff back online.

When I ran ./bin/storage dump i got a message that told me I couldn't do that because my database was in read only mode. I might have had a different setup than what you were expecting, and did something not explicitly documented, although I assumed "correct".

  • Turn off daemons
  • Run ./bin/config set cluster.read-only true on both of the machines that were in my setup, I assumed that I should be setting the "read only" flag on all of my phabricator clones (one db node, one web node) since the idea of having two different configs made my spidey sense tingle
  • ./bin/storage dump --for-replica throws an exception

I just turned off the flag on the db node and everything went swimmingly, I'm not sure if this would also manifest itself in a single-node setup.

Revisions and Commits

rP Phabricator
D15987rPb352cae97de4 Don't stop on read-only mode for read-only storage workflows

Related Objects

Event Timeline

yelirekim created this task.May 26 2016, 6:02 PM
epriestley claimed this task.May 26 2016, 7:20 PM
epriestley triaged this task as Normal priority.
epriestley added a project: Clusters.
Herald added a subscriber: eadler. · View Herald TranscriptMay 26 2016, 7:20 PM
epriestley added a comment.May 26 2016, 7:45 PM

I think I didn't actually do these things, I did a different and more horrible set of things and then wrote these great instructions to avoid all the msitakes I made. I'll attempt to let dump pierce through read-only, or find some other solution if that isn't straightforward.

hach-que added a subscriber: hach-que.May 27 2016, 12:29 AM

I was kind of expecting that 'cluster.read-only' would be a database configuration value, not a local configuration value. Otherwise you need to update all nodes in the cluster instead of just setting the flag in the DB. My expectation when the cluster DB is read-only is that only the following operations work:

  • bin/storage related tasks, and
  • bin/config set cluster.read-only false
Herald added a subscriber: faulconbridge. · View Herald TranscriptMay 27 2016, 12:29 AM
epriestley added a revision: D15987: Don't stop on read-only mode for read-only storage workflows.May 30 2016, 3:34 PM
epriestley added a comment.May 30 2016, 3:37 PM

Prior to D15987, the easiest fix for this was to add --force:

$ ./bin/storage dump ...
Usage Exception: Phabricator is currently in read-only mode. Use --force to override this mode.

However, that's scary and there's no need for us to be scary here. After D15987, we won't prompt/warn when running read-only storage operations (databases, dump, probe, renamespace, shell, status).

You can still run storage write operations in read-only mode with --force.

epriestley closed this task as Resolved by committing rPb352cae97de4: Don't stop on read-only mode for read-only storage workflows.May 30 2016, 4:30 PM
epriestley added a commit: rPb352cae97de4: Don't stop on read-only mode for read-only storage workflows.
Phabricator · Built by Phacility