Daemon never recovers with Git commit parsing task, but manually reparsing succeeds
Closed, InvalidPublic
Actions

Assigned To

None

Authored By

	mbishopim3
	Mar 7 2017, 4:10 PM

Description

Running HEAD on the stable branch.

Notice that the commit parser daemon is stuck with a certain task.
Log onto server, execute bin/repository reparse --importing --trace rREPO; note no errors reported.
Watch commit successfully import.

Related Objects

Mentioned Here: T4752: Daemons may not have permission to remove files from local disk storage engine

Event Timeline

mbishopim3 created this task.Mar 7 2017, 4:10 PM

Herald added a subscriber: eadler. · View Herald TranscriptMar 7 2017, 4:10 PM

There isn't enough information here for us to reproduce this, or even begin.

You can either provide complete reproduction steps which work in a clean environment, or we'll close this after a few days if we aren't able to move forward.

Is there a way to recall the logs from a failing task? Is there any value in providing the logs for the successful task? Not really sure what to say here besides that it's a consistent and random issue on our repositories. When we run the trace, it then "bumps" the task somehow and things succeed, so it would be difficult to provide detail on an error.

We now have one that's failing even when forcing a reparse:

    - PhabricatorLocalDiskFileStorageEngine: CommandException: Command failed with error #1!
      COMMAND
      mkdir -p '/var/storage/3f/49'
      
      STDOUT
      (empty)
      
      STDERR
      mkdir: cannot create directory ‘/var/storage/3f/49’: Permission denied
       at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:384]
arcanist(head=stable, ref.master=b87138356a9c, ref.stable=822bc53ca306), phabricator(head=stable, ref.stable=9124bb416241), phutil(head=stable, ref.master=c3fd3a8bb2c3, ref.stable=796cb1c2ee27)
  #0 <#2> ExecFuture::resolvex() called at [<phutil>/src/future/exec/execx.php:17]
  #1 <#2> execx(string, string) called at [<phabricator>/src/applications/files/engine/PhabricatorLocalDiskFileStorageEngine.php:58]
  #2 <#2> PhabricatorLocalDiskFileStorageEngine::writeFile(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:524]
  #3 <#2> PhabricatorFile::writeToEngine(PhabricatorLocalDiskFileStorageEngine, string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:365]
  #4 PhabricatorFile::buildFromFileData(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:422]
  #5 PhabricatorFile::newFromFileData(string, array) called at [<phabricator>/src/applications/files/uploadsource/PhabricatorFileUploadSource.php:198]
  #6 PhabricatorFileUploadSource::writeChunk(PhabricatorFile, PhabricatorChunkedFileStorageEngine) called at [<phabricator>/src/applications/files/uploadsource/PhabricatorFileUploadSource.php:163]
  #7 PhabricatorFileUploadSource::writeChunkedFile() called at [<phabricator>/src/applications/files/uploadsource/PhabricatorFileUploadSource.php:47]
  #8 PhabricatorFileUploadSource::uploadFile() called at [<phabricator>/src/applications/diffusion/query/DiffusionFileFutureQuery.php:109]
  #9 DiffusionFileFutureQuery::executeQuery() called at [<phabricator>/src/applications/diffusion/query/DiffusionQuery.php:96]
  #10 DiffusionQuery::execute() called at [<phabricator>/src/applications/diffusion/query/DiffusionFileFutureQuery.php:60]
  #11 DiffusionFileFutureQuery::respondToConduitRequest(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionFileContentQueryConduitAPIMethod.php:29]
  #12 DiffusionFileContentQueryConduitAPIMethod::getResult(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionQueryConduitAPIMethod.php:158]
  #13 DiffusionQueryConduitAPIMethod::execute(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/method/ConduitAPIMethod.php:123]
  #14 ConduitAPIMethod::executeMethod(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:131]
  #15 ConduitCall::executeMethod() called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:81]
  #16 ConduitCall::execute() called at [<phabricator>/src/applications/diffusion/query/DiffusionQuery.php:82]
  #17 DiffusionQuery::callConduitWithDiffusionRequest(PhabricatorUser, DiffusionGitRequest, string, array) called at [<phabricator>/src/applications/differential/engine/DifferentialDiffExtractionEngine.php:187]
  #18 DifferentialDiffExtractionEngine::isDiffChangedBeforeCommit(PhabricatorRepositoryCommit, DifferentialDiff, DifferentialDiff) called at [<phabricator>/src/applications/differential/engine/DifferentialDiffExtractionEngine.php:255]
  #19 DifferentialDiffExtractionEngine::updateRevisionWithCommit(DifferentialRevision, PhabricatorRepositoryCommit, array, PhabricatorDaemonContentSource) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:251]
  #20 PhabricatorRepositoryCommitMessageParserWorker::updateCommitData(DiffusionCommitRef) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:42]
  #21 PhabricatorRepositoryCommitMessageParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:51]
  #22 PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:123]
  #23 PhabricatorWorker::executeTask() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementReparseWorkflow.php:325]
  #24 PhabricatorRepositoryManagementReparseWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441]
  #25 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333]
  #26 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22]

The directory attempting to be created changes on each run. There is no shortage of disk space and the command is being executed as the daemon user.

More info: the commit contained binary files (executables actually).

Occasionally, even after running the re-parse command, the basics of the commit will import but the content itself will not. A phd restart will almost immediately allow the commit to then import.

mkdir: cannot create directory ‘/var/storage/3f/49’: Permission denied

/var/storage is currently supposed to be owned by www-user, so the daemons are not supposed to executes any file-writes - this might be an instance of T4752.
Your deamon-user might have partial write access, so this sometimes works, or if it's trying to upload a file that was already uploaded (e.g. via arc diff), it will also not fail.

In our case these are commits from landed Differential diffs. Permissions
for the web user are there and correct, but I'm not sure I understand all
the details around the daemon vs. web user, and the daemon is doing the
work for this git commit parser.

We don't know how to reproduce this, so we can't move forward.

Daemon never recovers with Git commit parsing task, but manually reparsing succeedsClosed, InvalidPublicActions

Description

Related Objects

Event Timeline

Daemon never recovers with Git commit parsing task, but manually reparsing succeeds
Closed, InvalidPublic
Actions