File write failure during daemon operations caused daemons to populate a few too many diffs
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	epriestley
	May 13 2016, 10:29 PM

Description

@csilvers ran into an issue similar to T10210 in a more recent version of Phabricator.

Daemons could not write to the local file storage engine.
This caused the --message phase of parsing a large commit to retry repeatedly, creating new diffs every time.

A general fix for this is probably to look for an existing persisted diff and reuse it if it already exists, building a new one only if it does not. Then we don't have to worry too much about getting the exception behavior exactly right.

Relevant trace:

Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700] [2016-05-13 15:05:34] EXCEPTION: (FilesystemException) Requested path '/opt/phabricator_files/c0/ce' is not writable. at [<phutil>/src/filesystem/Filesystem.php:1081]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700] arcanist(), phabricator(), phutil()
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #0 <#2> Filesystem::assertWritable(string) called at [<phutil>/src/filesystem/Filesystem.php:73]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #1 <#2> Filesystem::assertWritableFile(string) called at [<phutil>/src/filesystem/Filesystem.php:89]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #2 <#2> Filesystem::writeFile(string, string) called at [<phabricator>/src/applications/files/engine/PhabricatorLocalDiskFileStorageEngine.php:62]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #3 <#2> PhabricatorLocalDiskFileStorageEngine::writeFile(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:437]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #4 <#2> PhabricatorFile::writeToEngine(PhabricatorLocalDiskFileStorageEngine, string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:334]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #5 phlog(FilesystemException) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:344]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #6 PhabricatorFile::buildFromFileData(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:395]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #7 PhabricatorFile::newFromFileData(string, array) called at [<phabricator>/src/applications/files/uploadsource/PhabricatorFileUploadSource.php:198]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #8 PhabricatorFileUploadSource::writeChunk(PhabricatorFile, PhabricatorChunkedFileStorageEngine) called at [<phabricator>/src/applications/files/uploadsource/PhabricatorFileUploadSource.php:163]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #9 PhabricatorFileUploadSource::writeChunkedFile() called at [<phabricator>/src/applications/files/uploadsource/PhabricatorFileUploadSource.php:47]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #10 PhabricatorFileUploadSource::uploadFile() called at [<phabricator>/src/applications/diffusion/query/filecontent/DiffusionFileContentQuery.php:73]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #11 DiffusionFileContentQuery::executeQuery() called at [<phabricator>/src/applications/diffusion/query/DiffusionQuery.php:88]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #12 DiffusionQuery::execute() called at [<phabricator>/src/applications/diffusion/conduit/DiffusionFileContentQueryConduitAPIMethod.php:42]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #13 DiffusionFileContentQueryConduitAPIMethod::getResult(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionQueryConduitAPIMethod.php:158]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #14 DiffusionQueryConduitAPIMethod::execute(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/method/ConduitAPIMethod.php:122]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #15 ConduitAPIMethod::executeMethod(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:131]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #16 ConduitCall::executeMethod() called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:81]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #17 ConduitCall::execute() called at [<phabricator>/src/applications/diffusion/query/DiffusionQuery.php:81]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #18 DiffusionQuery::callConduitWithDiffusionRequest(PhabricatorUser, DiffusionGitRequest, string, array) called at [<phabricator>/src/applications/differential/engine/DifferentialDiffExtractionEngine.php:158]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #19 DifferentialDiffExtractionEngine::isDiffChangedBeforeCommit(PhabricatorRepositoryCommit, DifferentialDiff, DifferentialDiff) called at [<phabricator>/src/applications/differential/engine/DifferentialDiffExtractionEngine.php:226]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #20 DifferentialDiffExtractionEngine::updateRevisionWithCommit(DifferentialRevision, PhabricatorRepositoryCommit, array, PhabricatorDaemonContentSource) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:233]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #21 PhabricatorRepositoryCommitMessageParserWorker::updateCommitData(DiffusionCommitRef) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryGitCommitMessageParserWorker.php:11]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #22 PhabricatorRepositoryGitCommitMessageParserWorker::parseCommitWithRef(PhabricatorRepository, PhabricatorRepositoryCommit, DiffusionCommitRef) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:41]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #23 PhabricatorRepositoryCommitMessageParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:40]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #24 PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #25 PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #26 PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #27 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:183]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]   #28 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:125]
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700] [2016-05-13 15:05:34] EXCEPTION: (PhutilProxyException) Error while executing Task ID 11477894. {>} (PhutilAggregateException) All storage engines failed to write file:
Daemon 108549 STDE [Fri, 13 May 2016 15:05:34 -0700]     - PhabricatorLocalDiskFileStorageEngine: FilesystemException: Requested path '/opt/phabricator_files/c0/ce' is not writable. at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:353]