Page MenuHomePhabricator
Create Task
Maniphest T12081

"Unable to establish a connection to any database host" via ssh git push
Closed, InvalidPublic
Actions

Description

I'm currently unable to push to any repo. following an update to the latest stable (ea9c0607e1fb252ad5d439311527b05c3ce58c38)
A git push yields:

Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 612 bytes | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
remote: [2017-01-08 14:16:52] EXCEPTION: (PhabricatorClusterStrandedException) Unable to establish a connection to any database host (while trying "phabricator_config"). All masters and replicas are completely unreachable. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:140]
remote: arcanist(head=stable, ref.master=483e985d08d2, ref.stable=9503b941cc02), phabricator(head=stable, ref.master=8a85ee7c1575, ref.stable=ea9c0607e1fb), phutil(head=stable, ref.master=1a0371a2247a, ref.stable=35b57f7ab524)
remote:   #0 PhabricatorLiskDAO::raiseUnreachable(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:111]
remote:   #1 PhabricatorLiskDAO::newClusterConnection(string, string, string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:64]
remote:   #2 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1008]
remote:   #3 LiskDAO::establishConnection(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:516]
remote:   #4 LiskDAO::loadRawDataWhere(string, string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:476]
remote:   #5 LiskDAO::loadAllWhere(string, string) called at [<phabricator>/src/infrastructure/env/PhabricatorConfigDatabaseSource.php:18]
remote:   #6 PhabricatorConfigDatabaseSource::loadConfig(string) called at [<phabricator>/src/infrastructure/env/PhabricatorConfigDatabaseSource.php:7]
remote:   #7 PhabricatorConfigDatabaseSource::__construct(string) called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:249]
remote:   #8 PhabricatorEnv::buildConfigurationSourceStack(boolean) called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:95]
remote:   #9 PhabricatorEnv::initializeCommonEnvironment(boolean) called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:75]
remote:   #10 PhabricatorEnv::initializeScriptEnvironment(boolean) called at [<phabricator>/scripts/init/lib.php:22]
remote:   #11 init_phabricator_script(array) called at [<phabricator>/scripts/init/init-script.php:9]
remote:   #12 require_once(string) called at [<phabricator>/scripts/__init_script__.php:3]
remote:   #13 require_once(string) called at [<phabricator>/scripts/repository/commit_hook.php:32]
To ssh://dev.host.com:2222/diffusion/TCT/repo.git
 ! [remote rejected] develop -> develop (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@dev.host.com:2222/diffusion/TCT/repo.git'

Everything else seems to be working fine, I can access the UI, browse etc without issue. git pull, git clone, git remote show origin work as expected.

I can run ssh -T successfully.

➜  build git:(develop) ssh -T git@dev.host.com -p 2222
phabricator-ssh-exec: Welcome to Phabricator.

You are logged in as zcourts.

You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.

Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.

Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.

As well as echo {} | ssh git@dev.host.com -p 2222 conduit conduit.ping which gives
{"result":"vps296374","error_code":null,"error_info":null}

Looking around I found this Stackoverflow post
http://stackoverflow.com/a/40764145/400048

My version was from around Sept/Oct 16 (unfortunately don't know the exact commit hash), so I rolled back to 27006fedccc2 - this lead to a different error

>>> UNRECOVERABLE FATAL ERROR <<<

Class PhabricatorUser contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (PhutilPerson::getGender)

/var/www/phabricator/src/applications/people/storage/PhabricatorUser.php:1508


┻━┻ ︵ ¯\_(ツ)_/¯ ︵ ┻━┻

I then checked out master to try but got the database error again.
FYI to anyone upgrading master has a new DB field search_profilepanelconfiguration.customPHID so I had to drop that when switch from master back to stable.
Currently on stable with the original DB error.
https://dev.host.com/config/dbissue/ shows No databases have any issues.
https://dev.host.com/config/database/ shows No Schema Issues and everything is green.

I tried creating an instance on https://admin.phacility.com to see if I could reproduce it but I haven't been able to.
arcanist, libphutil and phabricator are all on stable. sshd, phabricator daemons and php-fpm all restarted, I even restarted nginx for good measure.

uname -a
Linux vps296374 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04 LTS
Release:	16.04
Codename:	xenial

Related Objects

Event Timeline

zcourts created this task.Jan 8 2017, 2:47 PM
zcourts updated the task description. (Show Details)Jan 8 2017, 2:53 PM
epriestley added a subscriber: epriestley.Jan 8 2017, 3:37 PM

This can either move forward as a bug report or a consulting request.

As a bug report: we require reproduction instructions, per Contributing Bug Reports and Providing Reproduction Steps. I can't reproduce this and this report doesn't have enough information for me to reproduce it. You can either:

  • provide complete, detailed instructions which allow the issue to be reproduced starting from a clean environment; or
  • provide root SSH access to a host where this issue reproduces -- that I can freely modify and break -- to debug the issue. If the host is a virtual host, you may be able to copy it using something like EBS snapshots and then give me access to the copy. Here's my public key:
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqtgXJJeboECX1s5/Esdj+8Tdfeu73CezkAodgbld62QfAQ8UG8x4ZPZgni8j7lwnIpLyNFsQUHGf0mPpQE+KIyT7sA/ICIbXNI74qIWsO2oe+d5OZhDA9c7LUx4TdhzSzdI9LHx99zH2Q74l/L4bIjMKFtKBxuLBsTWZOTB5d1J63YFOKSDn0RtXmYND1i5iaEjQTTxGZHCJiECFyBCl1Gtkjlh04caJQebXH7Wzipwa2yTwddDOuARTT/48uI5WvbwYmVi8JiqSdDr8DoYUaBkI8vpvSj3iHa8Br+runZLxNf1JEY/etM9V0dAICYaAmdVA5s5cFGEfPYE5mGHaIQ== epriestley@orbital

As a consulting request: see Consulting to move forward.

Phabricator can not be safely downgraded because migrations can not run in reverse.

Because neither we nor other installs have experienced this, I think it is very likely to be a configuration issue, but there is some possibility that it is somehow entangled with T12071 or other changes.

The most recent stable has not deployed to the Phacility cluster yet (it is deploying it shortly) so it's possible that the cluster will exhibit this issue after deployment and you'll be off the hook for a repro case. However, it is already deployed on the secure cluster without issues, so I think it's probably going to deploy cleanly.

epriestley edited projects, added Bug Report (Needs Information); removed Bug Report.Jan 8 2017, 4:28 PM

The Phacility cluster is on 2017 Week 1 now and I can't reproduce this there, either:

Screen Shot 2017-01-08 at 8.27.47 AM.png (1×1 px, 208 KB)

zcourts added a comment.Jan 9 2017, 6:27 PM

I'm using a VPS but don't have the facility to easily clone it as is and can't risk it being broken anymore than is now. I will get a small instance setup which reproduces the problem (using ansible to roll out so should be straightforward), that probably won't be until in the week, right now I'm working around it by mirroring and pushing to another remote temporarily.

epriestley added a comment.Jan 9 2017, 6:38 PM

Alright. If we don't hear back from you in a few days we'll close this since we can't move forward on it, but you're free to file a new task when you're ready to move forward.

aurelijus added a subscriber: aurelijus.Jan 9 2017, 10:51 PM

First thing to check is if your DB configs are in /conf/local/local.json, I went through the same issue, because my mysql configs where in /conf/custom/myconfig.php and for some reason newer versions of phabricator do not evaluate PHABRICATOR_ENV properly. So basically run

./bin/config set mysql.user blabla
./bin/config set mysql.host 127.0.0.1
./bin/config set mysql.pass blabla
zcourts added a comment.Jan 13 2017, 1:39 AM

@aurelijus unfortunately that wasn't it. I had local.json configured.

@epriestley I finally got around to doing this. I ran the ansible scripts on a new server
IP: 79.137.73.41
username: epriestley
your public key's been added.

Having run the scripts on a new server the only thing I can think of is the fact that the scripts change
#!/usr/bin/env php to #!/usr/bin/env php5.6

See /var/www/phabricator and ..

Destroy at will.

epriestley closed this task as Invalid.Jan 13 2017, 1:46 AM

Having run the scripts on a new server the only thing I can think of is the fact that the scripts change #!/usr/bin/env php to #!/usr/bin/env php5.6

Yes, this is the issue. We write a commit hook:

$ cat /var/repo/1/hooks/pre-receive
#!/bin/sh
export TERM=dumb
exec '/usr/bin/php' -f '/var/www/phabricator/bin/commit-hook' -- 'PHID-REPO-gw3jvlnsjcwvb54dg7nz'  "$@"

Note that the hook generates a /usr/bin/php path. This hook executes with PHP7:

$ /var/repo/1/hooks/pre-receive
[2017-01-13 01:43:04] EXCEPTION: (PhabricatorClusterStrandedException) Unable to establish a connection to any database host (while trying "phabricator_config"). All masters and replicas are completely unreachable. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:140]

I think this is because PHP7 does not have mysql or mysqli.

If you edit the hook to execute php5.6 explicitly instead, it runs normally.

$ /var/repo/1/hooks/pre-receive
[2017-01-13 01:43:42] EXCEPTION: (Exception) No Direct Pushes: You are pushing directly to a repository hosted by Phabricator. This will not work. See "No Direct Pushes" in the documentation for more information. at [<phabricator>/scripts/repository/commit_hook.php:115]

This is not a bug in Phabricator.

Note that PHP7 is now supported (T12101) at HEAD of master, although support is about two hours hold.

epriestley added a comment.Jan 13 2017, 1:48 AM

(I no longer need access to the server.)

zcourts added a comment.Jan 13 2017, 6:38 AM

Thanks. I was modifying php version because when it was last deployed PHP 7 wasn't supported so 5.6 was installed as well so there were two PHP versions on the server.

epriestley mentioned this in T12134: Develop a Nuance-based Phabricator reporting/support flow.Jan 20 2017, 5:32 PM
Phabricator · Built by Phacility