Page MenuHomePhabricator

Make cause/resolution for PhutilLockException during repository import more clear
Closed, ResolvedPublic

Description

If you run multiple PullLocal daemons on the same repositories or run bin/repository update on the repository a daemon is running on, you can get lock exceptions. These are expected during normal operation (if multiple processes are really updating, at least) but the stack trace you hit when they occur doesn't explain why they happened or how to resolve them.

We should catch this and rethrow a proxy exception with a more admin-friendly description:

  • Add a troubleshooting section to the documentation somewhere, see below in this task for some steps.
  • Catch this exception, rethrow it with a proxy wrapper that says "everything is OK, another process is updating that repository right now, try again in a little while or see <the documentation> for more information".

After adding a SVN repository for testing, I got to the import stage but the daemon was failing. Logs showed the following:

1$ ./bin/repository update SVN
2[Tue Jan 13 14:39:00 2015] [hphp] [5139:7f8ded14bc00:0:000001] [8329bf:832aec:9c54a4:9c5773:9c9e8a:9cb46f:a69181:8544fd:7f8de4dc1ec5:907794]
3Fatal error: Uncaught exception 'PhutilLockException' with message 'global:phabricator:PhabricatorRepositoryManagementUpdateWorkflow:SVN' in /vagrant/phab/phabricator/src/infrastructure/util/PhabricatorGlobalLock.php:94
4Stack trace:
5#0 /vagrant/phab/libphutil/src/filesystem/PhutilLock.php(167): PhabricatorGlobalLock->doLock()
6#1 /vagrant/phab/phabricator/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php(74): PhutilLock->lock()
7#2 /vagrant/phab/libphutil/src/parser/argument/PhutilArgumentParser.php(394): PhabricatorRepositoryManagementUpdateWorkflow->execute()
8#3 /vagrant/phab/libphutil/src/parser/argument/PhutilArgumentParser.php(290): PhutilArgumentParser->parseWorkflowsFull()
9#4 /vagrant/phab/phabricator/scripts/repository/manage_repositories.php(22): PhutilArgumentParser->parseWorkflows()
10#5 {main}
11[Tue Jan 13 14:39:00 2015] [hphp] [5139:7f8ded14bc00:0:000002] [c6c4bf:c4bd2e:c4bf9c:96540f:9a3421:8329bf:832aec:9c54a4:9c5773:9c9e8a:9cb46f:a69181:8544fd:7f8de4dc1ec5:907794] [2015-01-13 14:39:00] EXCEPTION: (PhutilLockException) global:phabricator:PhabricatorRepositoryManagementUpdateWorkflow:SVN at [<phabricator>/src/infrastructure/util/PhabricatorGlobalLock.php:94]
12[Tue Jan 13 14:39:00 2015] [hphp] [5139:7f8ded14bc00:0:000003] [c6c4bf:c4bd2e:c4bf9c:96540f:9a3421:8329bf:832aec:9c54a4:9c5773:9c9e8a:9cb46f:a69181:8544fd:7f8de4dc1ec5:907794] #0 PhabricatorGlobalLock::doLock(double) called at [<phutil>/src/filesystem/PhutilLock.php:167]
13[Tue Jan 13 14:39:00 2015] [hphp] [5139:7f8ded14bc00:0:000004] [c6c4bf:c4bd2e:c4bf9c:96540f:9a3421:8329bf:832aec:9c54a4:9c5773:9c9e8a:9cb46f:a69181:8544fd:7f8de4dc1ec5:907794] #1 PhutilLock::lock() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:74]
14[Tue Jan 13 14:39:00 2015] [hphp] [5139:7f8ded14bc00:0:000005] [c6c4bf:c4bd2e:c4bf9c:96540f:9a3421:8329bf:832aec:9c54a4:9c5773:9c9e8a:9cb46f:a69181:8544fd:7f8de4dc1ec5:907794] #2 PhabricatorRepositoryManagementUpdateWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:394]
15[Tue Jan 13 14:39:00 2015] [hphp] [5139:7f8ded14bc00:0:000006] [c6c4bf:c4bd2e:c4bf9c:96540f:9a3421:8329bf:832aec:9c54a4:9c5773:9c9e8a:9cb46f:a69181:8544fd:7f8de4dc1ec5:907794] #3 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:290]
16[Tue Jan 13 14:39:00 2015] [hphp] [5139:7f8ded14bc00:0:000007] [c6c4bf:c4bd2e:c4bf9c:96540f:9a3421:8329bf:832aec:9c54a4:9c5773:9c9e8a:9cb46f:a69181:8544fd:7f8de4dc1ec5:907794] #4 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22]

Event Timeline

WikiChad raised the priority of this task from to Needs Triage.
WikiChad updated the task description. (Show Details)
WikiChad added a subscriber: WikiChad.

This is expected if you try to update the repository at the same time the daemons are updating it. Is it possible that's the issue?

Well the daemons were failing with the same message. I tried running it manually to see if phd was at fault at all. Same stacktrace either way.

You can figure out what's holding the lock by running this command:

mysql> SELECT IS_USED_LOCK('global:phabricator:PhabricatorRepositoryManagementUpdateWorkflow:SVN');

If the lock is currently held, that will return a connection ID. You can find more information about the connection in:

mysql> SHOW FULL PROCESSLIST;

The connection IDs are listed in the first column. The "Host" might be enough to tell you what's holding the lock, or you could use something like netstat -t to figure out which process is connected on the given port.

In general, this behavior is expected if multiple processes are trying to update a repository. It just means "some other process is updating this repository right now, and holding a global lock in MySQL while it does". This can happen if you accidentally launch several PullLocal daemons, or someone runs a rogue PullLocal daemon somewhere, or multiple users are running repository update, or some combination of those things.

WikiChad claimed this task.

Ah, I think I had 2 PullLocal daemons running and one of them was fighting while the other was doing its job. All seems fine now.

Cool. I'll repurpose this to make the exception more clear.

epriestley renamed this task from Uncaught exception (PhutilLockException) when importing SVN repo to Make cause/resolution for PhutilLockException during repository import more clear.Jan 13 2015, 3:50 PM
epriestley reopened this task as Open.
epriestley triaged this task as Low priority.
epriestley updated the task description. (Show Details)
epriestley added projects: Repositories, Daemons.