Page MenuHomePhabricator
Create Task
Maniphest T5897

Consistent `PhutilLockException` in `PhabricatorRepositoryPullLocalDaemon` log
Closed, ResolvedPublic
Actions

Description

I see multiple PhutilLockException logs in the daemon log for PhabricatorRepositoryPullLocalDaemon. They look like this:

[2014-08-18 10:23:06] EXCEPTION: (PhutilProxyException) Error while updating the "rOH" repository. {>} (CommandException) Command failed with error #255!
COMMAND
'/opt/phabricator/phabricator/bin/repository' update  -- 'OH'

STDOUT
(empty)

STDERR
[2014-08-18 10:23:06] EXCEPTION: (PhutilLockException) global:phabricator:PhabricatorRepositoryManagementUpdateWorkflow:OH at [<phabricator>/src/infrastructure/util/PhabricatorGlobalLock.php:94]
  #0 PhabricatorGlobalLock::doLock(double) called at [<phutil>/src/filesystem/PhutilLock.php:167]
  #1 PhutilLock::lock() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:74]
  #2 PhabricatorRepositoryManagementUpdateWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:394]
  #3 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:290]
  #4 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22]
 at [<phutil>/src/future/exec/ExecFuture.php:397]
  #0 phlog(PhutilProxyException) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:335]
  #1 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:198]
  #2 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:91]
  #3 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:111]

Following @hach-que's suggestion on IRC, I'm opening a task for this.

Event Timeline

Avish created this task.Aug 18 2014, 11:06 AM
Avish raised the priority of this task from to Needs Triage.
Avish updated the task description. (Show Details)
Avish added projects: Daemons, Repositories.
Avish added subscribers: Avish, hach-que, epriestley.
hach-que added a comment.Aug 18 2014, 11:44 AM

I hit this same issue when trying to use PhabricatorGlobalLock inside Drydock, which appeared to only occur with lots of frequent requests to Drydock. I ended up using beginReadLocking instead, but obviously this is a problem that impacts existing code as well.

epriestley added a comment.Aug 18 2014, 4:51 PM

Are you running multiple copies of the PullLocal daemon?

Avish added a comment.Sep 2 2014, 10:04 AM

I might have been, yes. Is this not supported?

epriestley added a comment.Sep 2 2014, 5:53 PM

If you run multiple copies of the daemon, it's expected that they'll occasionally fail to acquire locks, because another copy of the daemon is busy updating the repository.

Are you running multiple copies of the daemon for any specific reason? If it's to improve responsiveness, they shouldn't have any effect in the modern codebase -- see:

https://secure.phabricator.com/book/phabricator/article/diffusion_updates/

If it's because you're running multiple machines, things are a bit more complicated (see T4209).

In any case, it's safe, but if you're running multiple copies these lock failures are expected. They just mean that the daemon declined to update a repository which was being updated by another daemon, to avoid errors in how the data is imported.

epriestley closed this task as Resolved.Sep 5 2014, 2:19 PM
epriestley claimed this task.

Assuming that explanation covers things, but yell if stuff is still unclear.

Phabricator · Built by Phacility