Page MenuHomePhabricator
Create Task
Maniphest T11577

Initializing databases for the first time while already in cluster mode fails
Closed, ResolvedPublic
Actions

Description

Hello,

I'm reinstalling a test infra using the MySQL cluster mode, and apparently I can't create the database structure. I'm using the latest stable branch to date of Phabricator (be84c051e5ba05296be4070bdfbf6f1fdd44355b). I have a clean mysql setup without phabricator database and a working replication setup.

I had to temporarily add mysq.host so I can run ./bin/storage upgrade otherwise I'd receive this error message:

# ./bin/storage upgrade
[2016-09-02 14:49:27] EXCEPTION: (PhabricatorClusterStrandedException) Unable to establish a connection to any database host (while trying "phabricator_config"). All masters and replicas are completely unreachable. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:166]
arcanist(head=stable, ref.stable=10e519475290), phabricator(head=stable, ref.stable=be84c051e5ba, custom=13), phutil(head=stable, ref.stable=d6818e59c176, custom=1)
  #0 PhabricatorLiskDAO::raiseUnreachable(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:137]
  #1 PhabricatorLiskDAO::newClusterConnection(string, string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:65]
  #2 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1008]
  #3 LiskDAO::establishConnection(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:516]
  #4 LiskDAO::loadRawDataWhere(string, string)
  #5 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:476]
  #6 LiskDAO::loadAllWhere(string, string) called at [<phabricator>/src/infrastructure/env/PhabricatorConfigDatabaseSource.php:19]
  #7 PhabricatorConfigDatabaseSource::loadConfig(string) called at [<phabricator>/src/infrastructure/env/PhabricatorConfigDatabaseSource.php:7]
  #8 PhabricatorConfigDatabaseSource::__construct(string) called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:232]
  #9 PhabricatorEnv::buildConfigurationSourceStack() called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:95]
  #10 PhabricatorEnv::initializeCommonEnvironment() called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:75]
  #11 PhabricatorEnv::initializeScriptEnvironment() called at [<phabricator>/scripts/__init_script__.php:21]
  #12 init_phabricator_script() called at [<phabricator>/scripts/__init_script__.php:24]
  #13 require_once(string) called at [<phabricator>/scripts/sql/manage_storage.php:5]

The servers are reachable because it would raiseUnconfigured instead of raiseUnreachable no ?

I see in some of your commit (6a4a9bb2d284) that it's supposed to work in cluster mode though. Adding mysql.host along with the cluster.databases config doesn't help. Is there a flag or something for the first install ? I only reinstalled the databases, the rest is from a previous install.

Best regard,
Jean-Baptiste

Revisions and Commits

rP Phabricator
D16489rPd0013d08983a Distinguish between unreachable cluster database hosts and missing MySQL…

Related Objects

Event Timeline

aubort created this task.Sep 2 2016, 2:34 PM
epriestley added a subscriber: epriestley.Sep 2 2016, 2:56 PM

I can reproduce this. Fixing it will take a bit of work.

The workaround (using single-node mysql.host configuration for the first time run) should work correctly.

epriestley renamed this task from Create db schema in cluster mode to Initializing databases for the first time while already in cluster mode fails.Sep 2 2016, 2:56 PM
epriestley added a project: Clusters.
Herald added a subscriber: eadler. · View Herald TranscriptSep 2 2016, 2:56 PM
epriestley added a comment.Sep 2 2016, 3:01 PM

A bad half-fix for this is here:

diff --git a/src/infrastructure/env/PhabricatorEnv.php b/src/infrastructure/env/PhabricatorEnv.php
index dd8c11e..b482bd4 100644
--- a/src/infrastructure/env/PhabricatorEnv.php
+++ b/src/infrastructure/env/PhabricatorEnv.php
@@ -235,6 +235,8 @@ final class PhabricatorEnv extends Phobject {
       // If the database is not available, just skip this configuration
       // source. This happens during `bin/storage upgrade`, `bin/conf` before
       // schema setup, etc.
+    } catch (PhabricatorClusterStrandedException $exception) {
+      // This is the cluster flavor of the exception above.
     }
   }

However, this is a pretty garbage fix. Better is probably having PhabricatorConfigDatabaseSource actually check if the table exists and just act like an empty config source if it does not yet. I think this gets us into much less trouble.

We also shouldn't be throwing PhabricatorClusterStrandedException when the connection fails only because the target database does not exist.

epriestley added a revision: D16489: Distinguish between unreachable cluster database hosts and missing MySQL databases.Sep 2 2016, 3:08 PM
epriestley closed this task as Resolved by committing rPd0013d08983a: Distinguish between unreachable cluster database hosts and missing MySQL….Sep 2 2016, 3:23 PM
epriestley added a commit: rPd0013d08983a: Distinguish between unreachable cluster database hosts and missing MySQL….
epriestley added a comment.Sep 2 2016, 3:23 PM

I believe this should be fixed in HEAD of master. It should promote to stable in about 24 hours.

Let us know if you're still seeing issues after updating. Thanks for the report!

Phabricator · Built by Phacility