Page MenuHomePhabricator
Create Task
Maniphest T10029

PhabricatorRepositoryGitCommitChangeParserWorker is unable to find a commit
Closed, ResolvedPublic
Actions

Description

In our Phabricator setup, we have two distinct hosts:

  • A "worker" host which runs the PhabricatorTaskmasterDaemon daemons, as well as an instance of PhabricatorRepositoryPullLocalDaemon with the --no-discovery flag.
  • A "repository" host which runs the PhabricatorRepositoryPullLocalDaemon daemon (without the --no-discovery flag).

We have a repository which doesn't receive many updates. I just pushed a commit to this repository a few hours ago and the update frequency for this repository is now at 20s.

In the Daemons console, I can see that there is a failing PhabricatorRepositoryGitCommitChangeParserWorker task:

-14 ☯ Task 21200466.png (430×1 px, 66 KB)

Trying to execute the command manually from the worker host:

josh@ip-10-63-53-226:/usr/src/phabricator$ ./bin/worker execute --id 21200466
Executing task 21200466 (PhabricatorRepositoryGitCommitChangeParserWorker)...
[2015-12-21 22:41:43] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git log -n1 --format='%P' '3515f0da8614e8eda5d5c2183d2885510dffd945'

STDOUT
(empty)

STDERR
fatal: bad object 3515f0da8614e8eda5d5c2183d2885510dffd945
 at [<phutil>/src/future/exec/ExecFuture.php:416]
arcanist(head=master, ref.master=8762e3f36715), phabricator(head=master, ref.master=006321cce76b), phlab(head=master, ref.master=f692a32769cb), phutil(head=master, ref.master=b0843cba1270)
  #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:346]
  #1 PhabricatorRepository::execxLocalCommand(string, string, string) called at [<phabricator>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryGitCommitChangeParserWorker.php:19]
  #2 PhabricatorRepositoryGitCommitChangeParserWorker::parseCommitChanges(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryCommitChangeParserWorker.php:30]
  #3 PhabricatorRepositoryCommitChangeParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:44]
  #4 PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
  #5 PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
  #6 PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/management/PhabricatorWorkerManagementExecuteWorkflow.php:52]
  #7 PhabricatorWorkerManagementExecuteWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:406]
  #8 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:301]
  #9 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/setup/manage_worker.php:21]

Related Objects

Event Timeline

joshuaspence created this task.Dec 21 2015, 11:43 AM

Seeing this on the repository host:

[21-Dec-2015 22:39:29 Australia/Sydney] [2015-12-21 22:39:29] EXCEPTION: (PhutilProxyException) Error while updating the "rDOCS" repository. {>} (CommandException) Command failed with error #
255!
COMMAND
'/usr/src/phabricator/bin/repository' update  -- 'DOCS'

STDOUT
(empty)

STDERR
[2015-12-21 22:39:29] EXCEPTION: (PhutilProxyException) Another process is currently holding the update lock for repository "rDOCS". Repositories may only be updated by one process at a time.
 This can happen if you are running multiple copies of the daemons. This can also happen if you manually update a repository while the daemons are also updating it (in this case, just try aga
in in a few moments). {>} (PhutilLockException) ph:phabric-6rILUDHKjHKu:repository.update:432 at [<phabricator>/src/infrastructure/util/PhabricatorGlobalLock.php:120]
arcanist(head=master, ref.master=8762e3f36715), phabricator(head=master, ref.master=006321cce76b), phlab(head=master, ref.master=f692a32769cb), phutil(head=master, ref.master=b0843cba1270)
  #0 <#2> PhabricatorGlobalLock::doLock(double) called at [<phutil>/src/filesystem/PhutilLock.php:168]
  #1 <#2> PhutilLock::lock() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:60]
  #2 Phabric... (384 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:416]
[21-Dec-2015 22:39:29 Australia/Sydney] arcanist(head=master, ref.master=8762e3f36715), phabricator(head=master, ref.master=006321cce76b), phlab(head=master, ref.master=f692a32769cb), phutil(
head=master, ref.master=b0843cba1270)
[21-Dec-2015 22:39:29 Australia/Sydney]   #0 <#3> ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:361]
[21-Dec-2015 22:39:29 Australia/Sydney]   #1 phlog(PhutilProxyException) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:368]
[21-Dec-2015 22:39:29 Australia/Sydney]   #2 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/re
pository/daemon/PhabricatorRepositoryPullLocalDaemon.php:201]
[21-Dec-2015 22:39:29 Australia/Sydney]   #3 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:183]
[21-Dec-2015 22:39:29 Australia/Sydney]   #4 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:125]
hach-que added a subscriber: hach-que.Dec 21 2015, 2:07 PM

You probably need to apply D11874, which fixes this issue.

joshuaspence added a comment.Dec 21 2015, 9:46 PM

I don't think that would fix the deadlock issue, which seems to be the root cause?

hach-que added a comment.Dec 21 2015, 10:24 PM

It means you don't need to run the taskmaster daemons on the same machines as the repositories because it can just make a Conduit call instead. You no longer need to run separate daemon tiers or use the --no-discovery flag.

joshuaspence added a comment.Dec 21 2015, 10:27 PM

Sure, I get that... But the import is failing even on the instance running without the --no-discovery flag.

hach-que added a comment.Dec 21 2015, 10:31 PM

Right, try stopping the pull daemons on the other tiers and leave only one running on the repository machine and see if that resolves your issue.

Most of the task description is dedicated to talking about PhabricatorRepositoryGitCommitChangeParserWorker, which is not expected to work from non-repository hosts until D11874 lands, and currently there's no way of filtering tasks so that PhabricatorRepositoryGitCommitChangeParserWorker only happens on repository hosts.

joshuaspence added a comment.Dec 21 2015, 11:35 PM

I thought that it should still work because I'm running the repository pull daemon on the worker host?

hach-que added a comment.EditedDec 22 2015, 12:59 AM

I don't believe so, https://secure.phabricator.com/T4209#146794 clearly states that until that patch lands, daemons and repos must run on the same host.

eadler added a subscriber: eadler.Apr 10 2016, 6:15 AM
epriestley closed this task as Resolved.Aug 22 2016, 6:18 PM
epriestley claimed this task.
epriestley added a subscriber: epriestley.

I believe this is resolved with modern clustered repositories. secure has been spread across two hosts for some time now without encountering any of this.

Phabricator · Built by Phacility