We had a developer mistakenly push a large branch to a repository tracked (but not hosted) by Diffusion, and delete it after realising. Unfortunately this seemed to fall into a gap between the repository being fetched by Diffusion, and the commit message parser workers being scheduled.
This resulted in a load of:
fatal: bad object f6e289c666b03d28c3f66ce028ab9c7963a40fa3 at [<phutil>/src/future/exec/ExecFuture.php:416] arcanist(head=master, ref.master=d0fa7cd4abf6), phabricator(head=master, ref.master=578b45d4a147), phutil(head=master, ref.master=3b27c5a1bc40), sprint(head=master, ref.master=11655849c3a2) #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:346] #1 PhabricatorRepository::execxLocalCommand(string, string, string, string) called at [<phabricator>/src/applications/diffusion/query/lowlevel/DiffusionLowLevelCommitQuery.php:58] #2 DiffusionLowLevelCommitQuery::loadGitCommitRef() called at [<phabricator>/src/applications/diffusion/query/lowlevel/DiffusionLowLevelCommitQuery.php:26] #3 DiffusionLowLevelCommitQuery::executeQuery() called at [<phabricator>/src/applications/diffusion/query/lowlevel/DiffusionLowLevelQuery.php:23] #4 DiffusionLowLevelQuery::execute() called at [<phabricator>/src/applications/diffusion/conduit/DiffusionQueryCommitsConduitAPIMethod.php:102] #5 DiffusionQueryCommitsConduitAPIMethod::execute(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/method/ConduitAPIMethod.php:113] #6 ConduitAPIMethod::executeMethod(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:135] #7 ConduitCall::executeMethod() called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:85] #8 ConduitCall::execute() called at [<phabricator>/src/applications/diffusion/query/DiffusionQuery.php:81] #9 DiffusionQuery::callConduitWithDiffusionRequest(PhabricatorUser, DiffusionGitRequest, string, array) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:29] #10 PhabricatorRepositoryCommitMessageParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:44] #11 PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
tasks failing in a tight loop, as the commits in question had since vanished from the repository.
Seems like the fix would be to detect the query failing in loadGitCommitRef, and throwing a permanent error from it.