config set mysql.*` are perplexing
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	mim
	Aug 1 2014, 10:12 PM

Description

I proceed with upgrade of storage after git pull all from master. But unfortunately it failed with such error:

P1220 Problems during upgrade

1	# ./phabricator/bin/storage upgrade --force --user root --password password
2
3	Applying patch 'phabricator:20140722.appname.php'...
4	Migrating user preferences...
5	Migrating dashboard installs...
6	Migrating `phabricator.uninstalled-applications` config...
7	[2014-08-02 01:02:07] EXCEPTION: (AphrontQueryConnectionException) Attempt to connect to root@localhost failed with error #1045: Access denied for user 'root'@'localhost' (using password: NO). at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:65]
8	#0 AphrontMySQLiDatabaseConnection::connect() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:97]
9	#1 AphrontMySQLDatabaseConnectionBase::establishConnection() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:120]
10	#2 AphrontMySQLDatabaseConnectionBase::requireConnection() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:15]
11	#3 AphrontMySQLiDatabaseConnection::escapeBinaryString(string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:11]
12	#4 AphrontMySQLiDatabaseConnection::escapeUTF8String(string) called at [<phutil>/src/xsprintf/qsprintf.php:170]
13	#5 xsprintf_query(AphrontMySQLiDatabaseConnection, string, integer, string, integer) called at [<phutil>/src/xsprintf/xsprintf.php:63]
14	#6 xsprintf(string, AphrontMySQLiDatabaseConnection, array) called at [<phutil>/src/xsprintf/qsprintf.php:64]
15	#7 qsprintf(AphrontMySQLiDatabaseConnection, string, string, string, string, string)
16	#8 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:5]
17	#9 queryfx(AphrontMySQLiDatabaseConnection, string, string, string, string, string)
18	#10 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:16]
19	#11 queryfx_all(AphrontMySQLiDatabaseConnection, string, string, string, string, string)
20	#12 call_user_func_array(string, array) called at [<phutil>/src/aphront/storage/connection/AphrontDatabaseConnection.php:20]
21	#13 AphrontDatabaseConnection::queryData(string, string, string, string, string)
22	#14 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:496]
23	#15 LiskDAO::loadRawDataWhere(string, string, string)
24	#16 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:460]
25	#17 LiskDAO::loadOneWhere(string, string, string) called at [<phabricator>/src/applications/config/storage/PhabricatorConfigEntry.php:30]
26	#18 PhabricatorConfigEntry::loadConfigEntry(string) called at [<phabricator>/resources/sql/autopatches/20140722.appname.php:131]
27	#19 require_once(string) called at [<phabricator>/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:196]
28	#20 PhabricatorStorageManagementAPI::applyPatchPHP(string) called at [<phabricator>/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:171]
29	#21 PhabricatorStorageManagementAPI::applyPatch(PhabricatorStoragePatch) called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementUpgradeWorkflow.php:168]
30	#22 PhabricatorStorageManagementUpgradeWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:394]
31	#23 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:290]
32	#24 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/sql/manage_storage.php:115]

It looks like at some point upgrade utility loses password to mysql db

Revisions and Commits

rP Phabricator
	D10199	rP237b1d769b00 Improve `bin/storage upgrade` behavior when run out-of-order

Event Timeline

mim created this task.Aug 1 2014, 10:12 PM

mim raised the priority of this task from to Needs Triage.

mim updated the task description. (Show Details)

mim added a subscriber: mim.

I would try running the upgrade script with the correct flags for user and password:

https://secure.phabricator.com/book/phabricator/article/configuration_guide/#storage-configuring-mysq

If you still have problems, let us know!

Originally it was with correct flags. That one was after experiments of trying to do something different.

I updated paste and also rechecked with git pull of all repositories. Still the same problem.

If i try to change something to wrong login i do find that script end simultaneously saying that that login was wrong. But in this case it starts some upgrade process and end up at some stage with ": Access denied for user 'root'@'localhost' (using password: NO)" when i definitely uses password to access mysql root shell.

We haven't had any other reports of this, so it'll be some digging to find the issue.

What is your upgrade process?
Is this a new install/upgrade or do you regularly update and this was the first failure?
Did you verify the mysql connection manually on the machine?

chad reopened this task as Open.Aug 2 2014, 4:20 PM

chad added subscribers: joshuaspence, epriestley.Aug 2 2014, 4:45 PM

P1220 - the original version ;) - has an error about opcache.so

Just pointing that out incase that is involved with keeping this data in memory or something fancy.

Can you run the upgrade script with --trace? That should give some more information.

I'm using Phabricator for 1.5 years and update it approximately once in a month. I have small script for upgrade and and start/stop of phd daemons:

P1222 Upgrade phabricator, start, stop phd daemons

1	#!/bin/sh
2
3	### BEGIN INIT INFO
4	# Provides: phd
5	# Required-Start: $network
6	# Required-Stop: $network
7	# Default-Start: 2 3 4 5
8	# Default-Stop:
9	# Short-Description: starts\|stops phd daemon for phabricator
10	# Description: Starts\|stops phd daemon for phabricator also provides way to upgrade phabricator totally
11	### END INIT INFO
12
13
14	ROOT=`pwd`
15	PHABRICATOR_ENV=custom/myconfig
16
17	stop() {
18	sudo -u www-data PHABRICATOR_ENV=$PHABRICATOR_ENV $ROOT/phabricator/bin/phd stop
19	}
20
21	upgrade(){
22	/etc/init.d/php5-fpm stop
23
24	git -C $ROOT/phabricator pull
25	git -C $ROOT/arcanist pull
26	git -C $ROOT/libphutil pull
27	$ROOT/phabricator/bin/storage upgrade -f --user root --password password
28
29	/etc/init.d/php5-fpm start
30	}
31
32	start() {
33	sudo -u www-data PHABRICATOR_ENV=$PHABRICATOR_ENV $ROOT/phabricator/bin/phd start
34	}
35
36	case "$1" in
37
38	start)
39	log_begin_msg "Starting phd daemon..."
40	start
41	log_end_msg $?
42	;;
43
44	stop)
45	log_begin_msg "Stopping phd daemon..."
46	stop
47	log_end_msg 0
48	;;
49
50	log_begin_msg "Restarting phd daemon..."
51	stop
52	start
53	log_end_msg $?
54	;;
55
56	upgrade)
57	log_begin_msg "Restarting phd daemon..."
58	stop
59	upgrade
60	start
61	log_end_msg $?
62	;;
63
64
65	*)
66	echo "Usage: /etc/init.d/iptables {start\|stop\|restart\|upgrade}" >&2
67	exit 1
68	;;
69	esac
70
71	exit 0

It worked perfectly before but failed at this time.

P1223 Upgrade with trace

1	# ./phabricator/bin/storage upgrade --force --trace --user root --password password
2
3	>>> [1] <connect>
4	<<< [1] <connect> 843 us
5	>>> [2] <query> SELECT 1
6	<<< [2] <query> 294 us
7	>>> [3] <connect> phabricator_meta_data
8	<<< [3] <connect> 523 us
9	>>> [4] <query> SELECT patch FROM patch_status
10	<<< [4] <query> 239 us
11	>>> [5] <query> SELECT patch FROM patch_status
12	<<< [5] <query> 264 us
13	Applying patch 'phabricator:20140722.appname.php'...
14	Migrating user preferences...
15	>>> [6] <connect> phabricator_user
16	<<< [6] <connect> 494 us
17	>>> [7] <query> SELECT * FROM `user` WHERE id > 0 ORDER BY id ASC LIMIT 100
18	<<< [7] <query> 268 us
19	>>> [8] <query> SELECT * FROM `user_preferences` WHERE userPHID = 'PHID-USER-w3k3d53akbugmhdh5qp2'
20	<<< [8] <query> 146 us
21	>>> [9] <query> UPDATE `user_preferences` SET preferences = '{\"titles\":\"glyph\",\"editor\":\"\",\"monospaced\":\"\",\"dark_console\":true,\"nav-collapsed\":false,\"diff-filetree\":1,\"time-format\":\"H:i\",\"app-tiles\":{\"PhabricatorApplicationDifferential\":\"hide\"},\"re-prefix\":true,\"self-mail\":\"1\",\"mailtags\":{\"differential-review-request\":true,\"differential-updated\":true,\"differential-comment\":true,\"differential-reviewers\":true,\"differential-committed\":true,\"differential-cc\":true,\"differential-o
22	<<< [9] <query> 34,334 us
23	>>> [10] <query> SELECT * FROM `user_preferences` WHERE userPHID = 'PHID-USER-oqh4ssxbo2xjw7vsqtfr'
24	<<< [10] <query> 241 us
25	>>> [11] <query> SELECT * FROM `user_preferences` WHERE userPHID = 'PHID-USER-ghvj4wfkhzkyitjs7pku'
26	<<< [11] <query> 184 us
27	>>> [12] <query> SELECT * FROM `user_preferences` WHERE userPHID = 'PHID-USER-a6nyjdduxb55xn6tvji2'
28	<<< [12] <query> 195 us
29	>>> [13] <query> SELECT * FROM `user` WHERE id > 4 ORDER BY id ASC LIMIT 100
30	<<< [13] <query> 245 us
31	Migrating dashboard installs...
32	>>> [14] <connect> phabricator_dashboard
33	<<< [14] <connect> 580 us
34	>>> [15] <query> SELECT * FROM `dashboard_install` WHERE id > 0 ORDER BY id ASC LIMIT 100
35	<<< [15] <query> 232 us
36	Migrating `phabricator.uninstalled-applications` config...
37	>>> [16] <connect> phabricator_config
38	<<< [16] <connect> 504 us
39	[2014-08-03 10:18:28] EXCEPTION: (AphrontQueryConnectionException) Attempt to connect to root@localhost failed with error #1045: Access denied for user 'root'@'localhost' (using password: NO). at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:65]
40	#0 AphrontMySQLiDatabaseConnection::connect() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:97]
41	#1 AphrontMySQLDatabaseConnectionBase::establishConnection() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:120]
42	#2 AphrontMySQLDatabaseConnectionBase::requireConnection() called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:15]
43	#3 AphrontMySQLiDatabaseConnection::escapeBinaryString(string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:11]
44	#4 AphrontMySQLiDatabaseConnection::escapeUTF8String(string) called at [<phutil>/src/xsprintf/qsprintf.php:170]
45	#5 xsprintf_query(AphrontMySQLiDatabaseConnection, string, integer, string, integer) called at [<phutil>/src/xsprintf/xsprintf.php:63]
46	#6 xsprintf(string, AphrontMySQLiDatabaseConnection, array) called at [<phutil>/src/xsprintf/qsprintf.php:64]
47	#7 qsprintf(AphrontMySQLiDatabaseConnection, string, string, string, string, string)
48	#8 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:5]
49	#9 queryfx(AphrontMySQLiDatabaseConnection, string, string, string, string, string)
50	#10 call_user_func_array(string, array) called at [<phutil>/src/xsprintf/queryfx.php:16]
51	#11 queryfx_all(AphrontMySQLiDatabaseConnection, string, string, string, string, string)
52	#12 call_user_func_array(string, array) called at [<phutil>/src/aphront/storage/connection/AphrontDatabaseConnection.php:20]
53	#13 AphrontDatabaseConnection::queryData(string, string, string, string, string)
54	#14 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:496]
55	#15 LiskDAO::loadRawDataWhere(string, string, string)
56	#16 call_user_func_array(array, array) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:460]
57	#17 LiskDAO::loadOneWhere(string, string, string) called at [<phabricator>/src/applications/config/storage/PhabricatorConfigEntry.php:30]
58	#18 PhabricatorConfigEntry::loadConfigEntry(string) called at [<phabricator>/resources/sql/autopatches/20140722.appname.php:131]
59	#19 require_once(string) called at [<phabricator>/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:196]
60	#20 PhabricatorStorageManagementAPI::applyPatchPHP(string) called at [<phabricator>/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:171]
61	#21 PhabricatorStorageManagementAPI::applyPatch(PhabricatorStoragePatch) called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementUpgradeWorkflow.php:168]
62	#22 PhabricatorStorageManagementUpgradeWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:394]
63	#23 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:290]
64	#24 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/sql/manage_storage.php:115]

Also i rechecked access to mysql via my root login an it is ok. I had some migration process from old system, but that one is definitely over and i performed upgrade after migration.

Ok i finally done with this bug.
It appeared this bug came with me using PHABRICATOR_ENV environment variable in my setup. When i configured my phabricator long time ago there wasn't any of conf/local/ENVIRONMENT option. After setting my environment configuration in that file i finally get all working.

But unfortunately i can't close this bug, as you need either to fix it for such old user as me, or drop down use of PHABRICATOR_ENV.

Thanks for digging in. Will see if there's anything we can detect here in the future.

T2221 was the task that tracked the work here. It was finished way back Jan 23 2013. I think the intersection of users who

were users before Jan 2013
wrote an upgrade script that depends on PHABRICATOR_ENV
haven't fixed this yet

...is very, very small. Rather than a programmatic fix - which I'm not even sure how to make it work well fwiw - @chad, myself, @joshuaspence etc. will remember this if someone else does in fact have this problem.

btrahan closed this task as Wontfix.Aug 4 2014, 7:09 PM

btrahan claimed this task.

I also spent some time trying to reproduce this (--password being ignored) and was not able to.

I am experiencing the same error when doing a fresh install. It seems to be losing the mysql credentials provided (both user and password) at some point. I have PHP 5.5.9 in case that's relevant.

Can you paste the specific error.

Here's @ADmad's trace:

https://gist.github.com/ADmad/2fe615e6cf70802eeeb6

Well, the two traces have the same migration script in common. I'll take a look at it and see if its being funky.

◀ Merged tasks: T2517.

Sorry, there was a bunch of chatter on this on IRC. It looks like a modern flavor of T2517. Patch in a sec..

epriestley claimed this task.Aug 8 2014, 10:31 PM

Ah okay.

(I was looking at D9982 which added this patch. The part that breaks is fairly substandard - uses a static method to load an object, updates the object, then saves the object - versus the more standard version of writing a select and an update. That said, I don't think it is supposed to break this way; just maybe break with schema changes and stuff. So yeah, you have a much better / actual lead. =D )

epriestley added a revision: D10199: Improve `bin/storage upgrade` behavior when run out-of-order.Aug 8 2014, 10:36 PM

Closed by commit rP237b1d769b00.

epriestley renamed this task from Problems during upgrade to Errors for `bin/storage upgrade` before `bin/config set mysql.*` are perplexing.Aug 8 2014, 10:45 PM

I think this is fixed now. Specifically, the root cause is that the standard credentials (mysql.user / mysql.pass) are not configured. For various technical reasons, we need both these credentials and any provided administrative credentials (with --user and --password) to be valid before we can run bin/storage upgrade.

The web UI sends you through these steps in the correct order and the install guide points you there, but it's easy to end up running them out of order, and the failure mode was extremely confusing. After D10199, we'll provide instructions for configuring the standard credentials instead of failing mysteriously.

To fix this issue, use bin/config set ... to set standard credentials for mysql.user, mysql.host and mysql.pass before running bin/storage upgrade. Detailed instructions should be available in the web UI if your install is in this state (you should see a setup screen telling you to configure these settings) or, after D10199, when running bin/storage upgrade.

Let us know if this still happens or this doesn't seem like the right fix.

This issue is fixed, thanks.Perhaps the docs should be updated to configure the "standard" credentials first before trying to run storage upgrade with "administrative" credentials.

Can you point me at what you think is confusing in the docs? I looked through them but wasn't immediately sure.

In particular, the Install Guide points you at the Configuration Guide, which guides you through webserver setup and then says:

Now, navigate to whichever subdomain you set up. You should see instructions to continue setup. The rest of this document contains additional instructions for specific setup steps.

The intent is that users will continue the install flow from the web UI at this point, since we can run code and give you much better step-by-step instructions based on which commands you actually need to run. At this point, the instruction the web UI gives you should be "configure MySQL".

(The docs haven't been reorganized in a while and could probably be clearer in general, but I couldn't find anything that seemed specifically misleading.)

Errors for `bin/storage upgrade` before `bin/config set mysql.*` are perplexingClosed, ResolvedPublicActions

Description

Revisions and Commits

Event Timeline

Errors for `bin/storage upgrade` before `bin/config set mysql.*` are perplexing
Closed, ResolvedPublic
Actions