Page MenuHomePhabricator
Create Task
Maniphest T11893

storage upgrade claims it lacks permissions, but is running as root mysql user
Closed, ResolvedPublic
Actions

Description

Hi,

I just tried to upgrade to latest stable from last weeks stable and am running into a weird issue. Here is my shell history...

Notes: The regular phabricator user doesn't have ddl rights so I run the upgrade as root. Yes I realise that password in history is lame, I'll change my habits at some point.

843  sudo systemctl stop nginx phd aphlict php-fpm
844  cd /opt/phabricator/arcanist/
845  git pull 
846  cd ..
847  cd libphutil/
848  git pull
849  cd ..
850  cd phabricator/
851  git pull
852  ./bin/storage -p ***** -u root upgrade

Which is a repeat of what I do most Saturday mornings whilst browsing reddit. This time however I got an error at the end of the upgrade. I re-ran it again with trace attached hoping to see some kind of extra errors, but I get nothing. Just a complaint about access denied for two tables. That doesn't really make much sense to me. Ideas?

Trace:

trace.txt45 KBDownload

Revisions and Commits

rP Phabricator
D16901rP74a0e5cd7930 (stable) Respect user/pass flags to bin/storage for direct DatabaseRef-based…
D16901rP8fbb411a9a72 Respect user/pass flags to bin/storage for direct DatabaseRef-based queries

Related Objects

Event Timeline

benjumanji created this task.Nov 19 2016, 12:29 PM
benjumanji updated the task description. (Show Details)
benjumanji updated the task description. (Show Details)
epriestley added a subscriber: epriestley.Nov 19 2016, 3:42 PM

If you run this from mysql directly, as root, does it work?

SHOW TABLES IN phabricator_xhpast

Does it work if you run as the normal user (not root)?

We've made some changes here in connection with T11044, but none of the interesting stuff is in master yet, so the most likely failure mode I can imagine is that we might have stopped respecting -u / -p somehow.

If that's the case, we'd fall back to normal credentials.

This could explain what you're seeing if the normal credentials can't access those databases. The applications are rarely-used / new enough that you might not notice if that was the case.

epriestley added a comment.Nov 19 2016, 3:44 PM

Maybe also: do you have cluster.databases configured?

epriestley added a comment.Nov 19 2016, 3:49 PM

Yeah, looks like some credential shenanigans are afoot. I instrumented the user we're actually connecting as and am getting a mixture of what I passed with -u ("trunk") and the default user ("root").

$ ./bin/storage upgrade -u trunk -f
>>> CONNECTING AS USER root!
>>> CONNECTING AS USER root!
>>> CONNECTING AS USER trunk!
>>> CONNECTING AS USER trunk!
Storage is up to date on "localhost". Use "storage status" for details.
>>> CONNECTING AS USER trunk!
Verifying database schemata on "localhost"...
>>> CONNECTING AS USER root!

At a minimum, this is a bug. I think there's an extremely good likelihood that it explains the behavior you're seeing, too.

benjumanji added a comment.Nov 19 2016, 3:55 PM

I can confirm that SHOW TABLES IN phabricator_xhpast works just fine as root mysql user, and dropping the credentials from the upgrade command yields exactly the same results as running under root. So that seems like it all ties up. I can work around this any number of ways and I will do just that.

benjumanji added a comment.Nov 19 2016, 4:19 PM

Fixed and I'm back up and running, thanks!

For the record, my normal credentials didn't have rights to those application tables and we don't use either of them, hence not noticing earlier.

epriestley added a revision: D16901: Respect user/pass flags to bin/storage for direct DatabaseRef-based queries.Nov 19 2016, 4:38 PM
epriestley added a comment.Nov 19 2016, 4:38 PM

Cool. D16901 should fix this properly. Thanks for the report!

epriestley added a commit: rP8fbb411a9a72: Respect user/pass flags to bin/storage for direct DatabaseRef-based queries.Nov 19 2016, 4:39 PM
epriestley added a commit: rP74a0e5cd7930: (stable) Respect user/pass flags to bin/storage for direct DatabaseRef-based….
epriestley closed this task as Resolved.Nov 19 2016, 4:43 PM
epriestley claimed this task.

This should now be fixed in HEAD of master and stable.

I think all of the edit/mutation commands were still respecting -u, so this should (probably?) only have caused problems if the normal user was unable to access some databases, as here.

benjumanji added a comment.Nov 19 2016, 6:13 PM

Great thanks!

epriestley mentioned this in T11044: Support partitioning application databases across multiple database hosts.Nov 19 2016, 10:11 PM
Phabricator · Built by Phacility