Page MenuHomePhabricator
Create Task
Maniphest T4159

Exceptions occurring in Diffusion which result in PHP-FPM worker processes crashing
Closed, InvalidPublic
Actions

Description

I'm seeing a lot of errors in my Nginx log. Eventually the errors cause the PHP-FPM worker process to crash, which means occasionally page requests result in Nginx returning 503. Here's a portion of the error log:

2013/11/25 00:24:23 [error] 11817#0: *12471 FastCGI sent in stderr: "PHP message: [2013-11-25 16:24:23] EXCEPTION: (Exception) Ref "origin/origin/HEAD" is ambiguous or does not exist. at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/request/DiffusionRequest.php:655]                                                                                                                                                                                                                                                    
PHP message:   #0 DiffusionRequest::queryStableCommitName() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/request/DiffusionRequest.php:320]                                                                                          
PHP message:   #1 DiffusionRequest::getStableCommitName() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_readmequery_Method.php:58]                                                                      
PHP message:   #2 ConduitAPI_diffusion_readmequery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:157]                              
PHP message:   #3 ConduitAPI_diffusion_abstractquery_Method::execute(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/method/ConduitAPIMethod.php:68]                                                           
PHP message:   #4 ConduitAPIMethod::executeMethod(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:163]                                                                                    
PHP message:   #5 ConduitCall::executeMethod() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:90]                                                                                                                  
PHP message:   #6 ConduitCall::execute() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/query/DiffusionQuery.php:70]                                                                                                                  
PHP message:   #7 DiffusionQuery::callConduitWithDiffusionRequest(Object PhabricatorUser, Object DiffusionGitRequest, diffusion.readmequery, Array { paths => Array of size 19 starting with: { 0 => Array of size 8 starting with: { fullPath => .hgignore } } }) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/controller/DiffusionController.php:171]                                                                                                                                                            
PHP message:   #8                                                                                                                                                                                                                                                              
2013/11/25 00:46:59 [error] 11817#0: *12553 FastCGI sent in stderr: "PHP message: [2013-11-25 16:46:59] ERROR 8: Undefined offset: 4 at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:106] 
PHP message:   #0 ConduitAPI_diffusion_browsequery_Method::getGitResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:165]                           
PHP message:   #1 ConduitAPI_diffusion_abstractquery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:28]                               
PHP message:   #2 ConduitAPI_diffusion_browsequery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:157]                              
PHP message:   #3 ConduitAPI_diffusion_abstractquery_Method::execute(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/method/ConduitAPIMethod.php:68]
PHP message:   #4 ConduitAPIMethod::executeMethod(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:163]
PHP message:   #5 ConduitCall::executeMethod() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:90]
PHP message:   #6 ConduitCall::execute() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/query/DiffusionQuery.php:70]
PHP message:   #7 DiffusionQuery::callConduitWithDiffusionRequest(Object PhabricatorUser, Object DiffusionGitRequest, diffusion.browsequery, Array of size 2 starting with: { path => null }) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/controller/DiffusionController.php:171]
PHP message: 
2013/11/25 00:46:59 [error] 11817#0: *12553 FastCGI sent in stderr: "thDiffusionRequest(Object PhabricatorUser, Object DiffusionGitRequest, diffusion.browsequery, Array of size 2 starting with: { path => null }) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/controller/DiffusionController.php:171]
PHP message:   #8 DiffusionController::callConduitWithDiffusionRequest(diffusion.browsequery, Array of size 2 starting with: { path => null }) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/controller/DiffusionBrowseMainController.php:22]
PHP message:   #9 DiffusionBrowseMainController::processRequest() called at [/srv/www/redpointsoftware.com.au/code/phabricator/webroot/index.php:84]
PHP message: [2013-11-25 16:46:59] ERROR 8: Undefined offset: 2 at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:106]
PHP message:   #0 ConduitAPI_diffusion_browsequery_Method::getGitResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:165]
PHP message:   #1 ConduitAPI_diffusion_abstractquery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:28]
PHP message:   #2 ConduitAPI_diffusion_browsequery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:157]
PHP message:   #3 ConduitAPI_diffusion_abstractquery_Method::execute(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/method/ConduitAPIMethod.php:68]
PHP message:   #4 ConduitAPIMethod::executeMethod(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricato
2013/11/25 00:46:59 [error] 11817#0: *12553 FastCGI sent in stderr: "ect ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/method/ConduitAPIMethod.php:68]
PHP message:   #4 ConduitAPIMethod::executeMethod(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:163]
PHP message:   #5 ConduitCall::executeMethod() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:90]
PHP message:   #6 ConduitCall::execute() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/query/DiffusionQuery.php:70]
PHP message:   #7 DiffusionQuery::callConduitWithDiffusionRequest(Object PhabricatorUser, Object DiffusionGitRequest, diffusion.browsequery, Array of size 2 starting with: { path => null }) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/controller/DiffusionController.php:171]
PHP message:   #8 DiffusionController::callConduitWithDiffusionRequest(diffusion.browsequery, Array of size 2 starting with: { path => null }) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/controller/DiffusionBrowseMainController.php:22]
PHP message:   #9 DiffusionBrowseMainController::processRequest() called at [/srv/www/redpointsoftware.com.au/code/phabricator/webroot/index.php:84]" while reading response header from upstream, client: 127.0.0.1, server: code.redpointsoftware.com.au, request: "GET /diffusion/SUB/browse/master/;3ee21521b3a00d3a87388be427da86566c799f3c HTTP/1.0", upstream: "fastcgi://unix:/tmp/php-fpm-code.redpointsoftware.com.au.sock:", host: "code.redpointsoftware.com.au"
2013/11/25 01:36:06 [error] 11817#0: *12834 FastCGI sent in stderr: "PHP message: [2013-11-25 17:36:05] EXCEPTION: (Exception) Ref "origin/origin/master" is ambiguous or does not exist. at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/request/DiffusionRequest.php:655]
PHP message:   #0 DiffusionRequest::queryStableCommitName() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/request/DiffusionRequest.php:320]
PHP message:   #1 DiffusionRequest::getStableCommitName() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_readmequery_Method.php:58]
PHP message:   #2 ConduitAPI_diffusion_readmequery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:157]
PHP message:   #3 ConduitAPI_diffusion_abstractquery_Method::execute(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/method/ConduitAPIMethod.php:68]
PHP message:   #4 ConduitAPIMethod::executeMethod(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:163]
PHP message:   #5 ConduitCall::executeMethod() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:90]
PHP message:   #6 ConduitCall::execute() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/query/DiffusionQuery.php:70]
PHP message:   #7 DiffusionQuery::callConduitWithDiffusionRequest(Object PhabricatorUser, Object DiffusionGitRequest, diffusion.readmequery, Array { paths => Array of size 11 starting with: { 0 => Array of size 8 starting with: { fullPath => .arcconfig } } }) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/controller/DiffusionController.php:171]
PHP message:   
2013/11/25 02:11:06 [error] 11817#0: *12982 FastCGI sent in stderr: "PHP message: [2013-11-25 18:11:06] EXCEPTION: (Exception) Ref "origin/origin/HEAD" is ambiguous or does not exist. at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/request/DiffusionRequest.php:655]
PHP message:   #0 DiffusionRequest::queryStableCommitName() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/request/DiffusionRequest.php:320]
PHP message:   #1 DiffusionRequest::getStableCommitName() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_readmequery_Method.php:58]
PHP message:   #2 ConduitAPI_diffusion_readmequery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:157]
PHP message:   #3 ConduitAPI_diffusion_abstractquery_Method::execute(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/method/ConduitAPIMethod.php:68]
PHP message:   #4 ConduitAPIMethod::executeMethod(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:163]
PHP message:   #5 ConduitCall::executeMethod() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:90]
PHP message:   #6 ConduitCall::execute() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/query/DiffusionQuery.php:70]
PHP message:   #7 DiffusionQuery::callConduitWithDiffusionRequest(Object PhabricatorUser, Object DiffusionGitRequest, diffusion.readmequery, Array { paths => Array of size 10 starting with: { 0 => Array of size 8 starting with: { fullPath => .gitignore } } }) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/controller/DiffusionController.php:171]
PHP message:   #8
2013/11/25 02:49:24 [error] 11818#0: *13114 FastCGI sent in stderr: "PHP message: [2013-11-25 18:49:24] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git cat-file -t '10b132b28ae8687ea01d4bdab815b2dbcac0493c':'Protogame'

STDOUT
(empty)

STDERR
fatal: git cat-file 10b132b28ae8687ea01d4bdab815b2dbcac0493c:Protogame: bad file
 at [/srv/www/redpointsoftware.com.au/code/libphutil/src/future/exec/ExecFuture.php:349]
PHP message:   #0 ExecFuture::resolvex() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/repository/storage/PhabricatorRepository.php:245]
PHP message:   #1 PhabricatorRepository::execxLocalCommand(cat-file -t %s:%s, 10b132b28ae8687ea01d4bdab815b2dbcac0493c, Protogame) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:48]
PHP message:   #2 ConduitAPI_diffusion_browsequery_Method::getGitResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:165]
PHP message:   #3 ConduitAPI_diffusion_abstractquery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:28]
PHP message:   #4 ConduitAPI_diffusion_browsequery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:157]
PHP message:   #5 ConduitAPI_diffusion_abstractquery_Method::execute(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/method/ConduitAPIMethod.php:68]
PHP message:   #6 ConduitAPIMethod::executeMethod(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:163]
PHP message:   #7 ConduitCall::exec
2013/11/25 03:26:23 [error] 11818#0: *13289 FastCGI sent in stderr: "PHP message: [2013-11-25 19:26:23] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git cat-file -t '0ccec247333ccaacb64fc66a6c159003dd3049a7':'Protogame'

STDOUT
(empty)

STDERR
fatal: git cat-file 0ccec247333ccaacb64fc66a6c159003dd3049a7:Protogame: bad file
 at [/srv/www/redpointsoftware.com.au/code/libphutil/src/future/exec/ExecFuture.php:349]
PHP message:   #0 ExecFuture::resolvex() called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/repository/storage/PhabricatorRepository.php:245]
PHP message:   #1 PhabricatorRepository::execxLocalCommand(cat-file -t %s:%s, 0ccec247333ccaacb64fc66a6c159003dd3049a7, Protogame) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:48]
PHP message:   #2 ConduitAPI_diffusion_browsequery_Method::getGitResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:165]
PHP message:   #3 ConduitAPI_diffusion_abstractquery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:28]
PHP message:   #4 ConduitAPI_diffusion_browsequery_Method::getResult(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_abstractquery_Method.php:157]
PHP message:   #5 ConduitAPI_diffusion_abstractquery_Method::execute(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/method/ConduitAPIMethod.php:68]
PHP message:   #6 ConduitAPIMethod::executeMethod(Object ConduitAPIRequest) called at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/conduit/call/ConduitCall.php:163]

Revisions and Commits

rP Phabricator
D7744rP52462a46c00c Raise a better error for malformed `git ls-tree`

Event Timeline

hach-que created this task.Nov 25 2013, 8:54 AM
hach-que assigned this task to epriestley.
hach-que raised the priority of this task from to Needs Triage.
hach-que updated the task description. (Show Details)
hach-que added projects: Phabricator, Diffusion.
hach-que added a subscriber: hach-que.
epriestley added a comment.Dec 8 2013, 5:24 PM

Picking away at these, this one is interesting:

2013/11/25 00:46:59 [error] 11817#0: *12553 FastCGI sent in stderr: "PHP message: [2013-11-25 16:46:59] ERROR 8: Undefined offset: 4 at [/srv/www/redpointsoftware.com.au/code/phabricator/src/applications/diffusion/conduit/ConduitAPI_diffusion_browsequery_Method.php:106]

The code is:

// NOTE: Limit to 5 components so we parse filenames with spaces in them
// correctly.
list($mode, $type, $hash, $size, $name) = preg_split('/\s+/', $line, 5);

That comes out of:

list($stdout) = $repository->execxLocalCommand(
  'ls-tree -z -l %s:%s',
  $commit,
  $path);

The other "undefined offset" below is similar. I'm not sure what the circumstances are which make git ls-tree -z -l x:y emit output which isn't in the expected format, but that's easy enough to check for.

epriestley edited this Maniphest Task.Dec 8 2013, 5:28 PM
epriestley added a comment.Dec 8 2013, 5:43 PM

The "ambiguous or does not exist" is a rough edge from T1493 which hasn't been cleaned up yet. You can hit that one normally by trying to access a branch which doesn't exist, although I'm not sure how it's being hit on your install. It looks like diffusion.readmequery might have some issues. That should be going away anyway.

epriestley added a comment.Dec 8 2013, 5:46 PM

The cat-file thing can happen if the target is a submodule, it looks like? Is "Protogame" a directory?

Our production log is otherwise pretty similar, with two other errors:

  • gd not being happy about a .ico file.
  • A bunch of people/bots trying to access the install from the wrong domains.
hach-que added a comment.Dec 8 2013, 11:20 PM

We use submodules a lot, so that's probably like 90% of the issues.

epriestley edited this Maniphest Task.Dec 9 2013, 9:22 PM
chad removed a project: Phabricator.Nov 3 2014, 1:00 AM
epriestley closed this task as Invalid.Jan 5 2016, 9:50 PM

This is very old, possibly fixed by now, doesn't include reproduction steps, and other users haven't reported issues. If you have specific issues with externals and reproduction steps, feel free to file new tasks.

Phabricator · Built by Phacility