Page MenuHomePhabricator

Commit is no longer reachable
Closed, InvalidPublic

Description

Our commit import queue is way backed up. I ran through them manually and found this:

Any help would be greatly appreciated.

sh-4.1# ./repository reparse --importing --trace rT7189107aadafc000d6693dd7c578b4579f78ecf7
>>> [2] <connect> phabricator_repository
<<< [2] <connect> 1,433 us
>>> [3] <query> SELECT `r`.*, r.* FROM `repository` r  WHERE ((r.callsign IN ('T')))   ORDER BY `r`.`id` DESC
<<< [3] <query> 578 us
>>> [4] <query> SELECT `commit`.* FROM `repository_commit` commit  WHERE (((commit.repositoryID = 2
                  AND commit.commitIdentifier = '7189107aadafc000d6693dd7c578b4579f78ecf7')))   ORDER BY `commit`.`id` DESC
<<< [4] <query> 334 us
>>> [5] <query> SELECT `r`.*, r.* FROM `repository` r  WHERE (r.id IN (2))   ORDER BY `r`.`id` DESC
<<< [5] <query> 329 us
>>> [6] <query> SELECT `commit`.* FROM `repository_commit` commit  WHERE (commit.id IN (592381))   ORDER BY `commit`.`id` DESC
<<< [6] <query> 452 us
>>> [7] <query> SELECT `r`.*, r.* FROM `repository` r  WHERE (r.id IN (2))   ORDER BY `r`.`id` DESC
<<< [7] <query> 320 us
[2016-11-10 14:06:58] EXCEPTION: (PhabricatorWorkerPermanentFailureException) Commit "rT7189107aadafc000d6693dd7c578b4579f78ecf7" (with internal ID "592381") is no longer reachable from any branch, tag, or ref in this repository, so it will not be imported. This usually means that the branch the commit was on was deleted or overwritten. at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:30]
arcanist(head=stable, ref.master=9c056c5cc887, ref.stable=e17fe43ca3fe), phabricator(head=stable, ref.master=b441e8b81e31, ref.stable=111c63955155), phutil(head=stable, ref.master=c72eb747e706, ref.stable=40e473e05788)
  #0 PhabricatorRepositoryCommitParserWorker::loadCommit() called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:46]
  #1 PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:123]
  #2 PhabricatorWorker::executeTask() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementReparseWorkflow.php:325]
  #3 PhabricatorRepositoryManagementReparseWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:438]
  #4 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:330]
  #5 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22]

Event Timeline

What can I do next to get the commit parser to stop trying to parse this.

When I run:

./repository reparse --force-local --trace --importing --all rT

it bails out at that commit

The task for parsing this commit should auto-cancel because it encountered a PhabricatorWorkerPermanentFailureException. In this case, you're manually re-parsing it, but it shouldn't be happening automatically.

I believe that if you drop --force-local, the reparse code will schedule the reparse to run using the daemons, which should fail to parse this commit and continue to the next one.

OK, Here's the issue (from what I can tell)
PhabricatorRepositoryGitCommitMessageParserWorker has a count of 105,000 in the queue, but for some reason those commits are not getting parsed. I don't want to 're-queue' them by removing the --force-local, because (according to the help) that will duplicate the work that is already queued. How can I get phd to start working on those commits?

One other thing that I should note, these commits are all in a repository that is no longer active. (if that matters)

are the daemons running? You might need to activate the repository for the commits to be parsed.

Yes.. interesting, I had not considered that. I'll re-activate the repo

Nope, phd is definitely *not* making any progress on the PhabricatorRepositoryGitCommitMessageParserWorker queued tasks I'm stumped. When I run this:

./phd debug --as-current-user PhabricatorTaskmasterDaemon

from the command line, it grabs a couple jobs (herald worker jobs etc..), then just sits there. Is that the right daemon to run to process git commit messages?

./bin/phd start will start all the right daemons. Look in the Daemons app in the web interface to see what's going on.

Sorry, I should have been more clear..

  • I re-activated the repo
  • I restarted the phd daemons
  • I checked the queue in the web interface
  • I observed that none of the commit queue jobs were being picked up (even though they are in the queue)
  • I stopped the daemons
  • I ran the phd debug command (above) to see if I could observe what phd is doing
  • I restarted phd (still no progress on the jobs in the commit queue)

Are the daemons doing anything? Are any tasks, at all, getting done? Are some tasks failing?

First, thank you for taking the time to answer...

I'm sitting on the 'daemons/console' page and I have a couple observations:

  1. jobs such as: PhabricatorApplicationTransactionPublishWorker and PhabricatorRepositoryCommitHeraldWorker are getting processed
  2. the commit queue actually went down... 2 from 105, 203 to 105,201 At this rate, it will be finished sometime in the year 2018. even though CPU utilization is only 47%.

Most of the work is disk/DB bound, so not having a high cpu load makes sense. That said, processing a lot of tasks will take a while.
When a task fails in an unexpected way, the daemon thread backs-off for a while, so it takes much longer to fail a task than to complete it; Unless most of the commits are failing, it should speed up at some point.
These tasks are also lower-priority than Herald and Publish tasks, which is why those get executed before these; Once these tasks are out of the queue, expect work to happen faster on the commit parsing tasks.

This is way outside the realm of an upstreamable bug report with reproduction steps, but run this again:

$ ./phd debug --as-current-user PhabricatorTaskmasterDaemon --trace

If it hangs, paste the output from before the hang here. Then use ps auxwww in another window to find the PID of the Taskmaster process and send it SIGHUP:

$ kill -HUP <pid>

Find the backtrace in /tmp/phabricator_backtrace_<pid> and paste it here. For details, see "Command-Line Hangs" here:

https://secure.phabricator.com/book/phabricator/article/performance/

It's not 'Hanging', it just doesn't appear to be doing with queued tasks and this is the output (repeated ad nauseam)

>>> [1425] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1425] <query> 580 us
>>> [1426] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1426] <query> 601 us
>>> [1427] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1427] <query> 585 us
>>> [1428] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1428] <query> 602 us
>>> [1429] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1429] <query> 591 us
>>> [1430] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1430] <query> 582 us
>>> [1431] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1431] <query> 614 us
>>> [1432] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1432] <query> 605 us
>>> [1433] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1433] <query> 568 us
>>> [1434] <query> SELECT MAX(id) FROM `config_transaction`
<<< [1434] <query> 611 us

What happened before that?

What is the backtrace for the actual Taskmaster process?

More detail:

The full output before it hit that 'config_transaction' line has confidential information, I can direct message you a 'best effort' redacted version of it if you'd like:

The backtrace is:

#0 /opt/libphutil/src/future/exec/PhutilSignalRouter.php(61): PhutilBacktraceSignalHandler->handleSignal(Object(PhutilSignalRouter), 1)
#1 /opt/libphutil/src/error/PhutilErrorHandler.php(187): PhutilSignalRouter->routeSignal(1)
#2 [internal function]: PhutilErrorHandler::handleError(2, 'stream_select()...', '/opt/libphutil/...', 144, Array)
#3 /opt/libphutil/src/future/Future.php(144): stream_select(Array, Array, Array, 1, 0)
#4 /opt/libphutil/src/future/Future.php(56): Future::waitForSockets(Array, Array, 1)
#5 /opt/libphutil/src/future/exec/ExecFuture.php(366): Future->resolve(NULL)
#6 /opt/phabricator/src/applications/repository/storage/PhabricatorRepository.php(523): ExecFuture->resolvex()
#7 /opt/phabricator/src/applications/diffusion/conduit/DiffusionBranchQueryConduitAPIMethod.php(38): PhabricatorRepository->execxLocalCommand('branch --verbos...', '823e1e00031fd8e...')
#8 /opt/phabricator/src/applications/diffusion/conduit/DiffusionQueryConduitAPIMethod.php(167): DiffusionBranchQueryConduitAPIMethod->getGitResult(Object(ConduitAPIRequest))
#9 /opt/phabricator/src/applications/diffusion/conduit/DiffusionQueryConduitAPIMethod.php(158): DiffusionQueryConduitAPIMethod->getResult(Object(ConduitAPIRequest))
#10 /opt/phabricator/src/applications/conduit/method/ConduitAPIMethod.php(122): DiffusionQueryConduitAPIMethod->execute(Object(ConduitAPIRequest))
#11 /opt/phabricator/src/applications/conduit/call/ConduitCall.php(131): ConduitAPIMethod->executeMethod(Object(ConduitAPIRequest))
#12 /opt/phabricator/src/applications/conduit/call/ConduitCall.php(81): ConduitCall->executeMethod()
#13 /opt/phabricator/src/applications/diffusion/herald/DiffusionCommitBranchesHeraldField.php(23): ConduitCall->execute()
#14 /opt/phabricator/src/applications/herald/adapter/HeraldAdapter.php(102): DiffusionCommitBranchesHeraldField->getHeraldFieldValue(Object(PhabricatorRepositoryCommit))
#15 /opt/phabricator/src/applications/herald/engine/HeraldEngine.php(370): HeraldAdapter->getHeraldField('diffusion.commi...')
#16 /opt/phabricator/src/applications/herald/engine/HeraldEngine.php(365): HeraldEngine->getObjectFieldValue('diffusion.commi...')
#17 /opt/phabricator/src/applications/herald/engine/HeraldEngine.php(277): HeraldEngine->getConditionObjectValue(Object(HeraldCondition), Object(HeraldCommitAdapter))
#18 /opt/phabricator/src/applications/herald/engine/HeraldEngine.php(91): HeraldEngine->doesRuleMatch(Object(HeraldRule), Object(HeraldCommitAdapter))
#19 /opt/phabricator/src/applications/herald/engine/HeraldEngine.php(43): HeraldEngine->applyRules(Array, Object(HeraldCommitAdapter))
#20 /opt/phabricator/src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php(3106): HeraldEngine::loadAndApplyRules(Object(HeraldCommitAdapter))
#21 /opt/phabricator/src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php(1008): PhabricatorApplicationTransactionEditor->applyHeraldRules(Object(PhabricatorRepositoryCommit), Array)
#22 /opt/phabricator/src/applications/repository/worker/PhabricatorRepositoryCommitHeraldWorker.php(102): PhabricatorApplicationTransactionEditor->applyTransactions(Object(PhabricatorRepositoryCommit), Array)
#23 /opt/phabricator/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php(51): PhabricatorRepositoryCommitHeraldWorker->parseCommit(Object(PhabricatorRepository), Object(PhabricatorRepositoryCommit))
#24 /opt/phabricator/src/infrastructure/daemon/workers/PhabricatorWorker.php(123): PhabricatorRepositoryCommitParserWorker->doWork()
#25 /opt/phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php(171): PhabricatorWorker->executeTask()
#26 /opt/phabricator/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php(22): PhabricatorWorkerActiveTask->executeTask()
#27 /opt/libphutil/src/daemon/PhutilDaemon.php(181): PhabricatorTaskmasterDaemon->run()
#28 /opt/libphutil/scripts/daemon/exec/exec_daemon.php(127): PhutilDaemon->execute()

From that trace, it's possible that there's an issue in the vein of T9898, if the repository has an enormous number of branches (tens of thousands?).

If not, this is unfortunately too far afield from a reproducible bug report (e.g., as documented in Contributing Bug Reports) for us to accept upstream.

If you'd like dedicated support for problems unique to your environment, it's available for $1,500/hr. See Consulting for details.

Thank you for the response, it is very likely a dup of T9898