Page MenuHomePhabricator

Out of memory issue with Diffusion when Phabricator daemons are running for at least a day
Closed, InvalidPublic

Description

I believe there is a memory leak problem with one or several of the Phabricator daemons. I usually can use Phabricator and Diffusion without a problem. After about a day I am greeted with the following exception for each file Diffusion tries to create diffs for:

[12-Jun-2015 09:59:44 America/Los_Angeles] [2015-06-12 09:59:44] EXCEPTION: (Exception) Failed to proc_open(): proc_open(): fork failed - Cannot allocate memory at [<phutil>/src/future/exec/ExecFuture.php:738]
[12-Jun-2015 09:59:44 America/Los_Angeles] arcanist(head=master, ref.master=7d15b85a1bc0), phabricator(head=master, ref.master=52f8756c3c7c), phutil(head=master, ref.master=92882eb9404d)
[12-Jun-2015 09:59:44 America/Los_Angeles]   #0 <#2> ExecFuture::isReady() called at [<phutil>/src/future/Future.php:39]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #1 <#2> Future::resolve(NULL) called at [<phutil>/src/future/exec/ExecFuture.php:413]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #2 <#2> ExecFuture::resolvex() called at [<phabricator>/src/applications/diffusion/query/rawdiff/DiffusionGitRawDiffQuery.php:40]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #3 <#2> DiffusionGitRawDiffQuery::executeQuery() called at [<phabricator>/src/applications/diffusion/query/rawdiff/DiffusionRawDiffQuery.php:17]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #4 <#2> DiffusionRawDiffQuery::loadRawDiff() called at [<phabricator>/src/applications/diffusion/conduit/DiffusionDiffQueryConduitAPIMethod.php:210]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #5 <#2> DiffusionDiffQueryConduitAPIMethod::getGitOrMercurialResult(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionDiffQueryConduitAPIMethod.php:39]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #6 <#2> DiffusionDiffQueryConduitAPIMethod::getGitResult(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionQueryConduitAPIMethod.php:144]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #7 <#2> DiffusionQueryConduitAPIMethod::getResult(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionDiffQueryConduitAPIMethod.php:30]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #8 <#2> DiffusionDiffQueryConduitAPIMethod::getResult(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionQueryConduitAPIMethod.php:135]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #9 <#2> DiffusionQueryConduitAPIMethod::execute(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/method/ConduitAPIMethod.php:90]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #10 <#2> ConduitAPIMethod::executeMethod(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:134]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #11 <#2> ConduitCall::executeMethod() called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:84]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #12 <#2> ConduitCall::execute() called at [<phabricator>/src/applications/diffusion/query/DiffusionQuery.php:81]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #13 <#2> DiffusionQuery::callConduitWithDiffusionRequest(PhabricatorUser, DiffusionGitRequest, string, array) called at [<phabricator>/src/applications/diffusion/controller/DiffusionController.php:184]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #14 <#2> DiffusionController::callConduitWithDiffusionRequest(string, array) called at [<phabricator>/src/applications/diffusion/controller/DiffusionDiffController.php:60]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #15 <#2> DiffusionDiffController::processDiffusionRequest(AphrontRequest) called at [<phabricator>/src/applications/diffusion/controller/DiffusionController.php:51]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #16 <#2> DiffusionController::handleRequest(AphrontRequest) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:226]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #17 phlog(Exception) called at [<phabricator>/src/aphront/configuration/AphrontDefaultApplicationConfiguration.php:229]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #18 AphrontDefaultApplicationConfiguration::handleException(Exception) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:230]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #19 AphrontApplicationConfiguration::processRequest(AphrontRequest, PhutilDeferredLog, AphrontPHPHTTPSink, MultimeterControl) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:140]
[12-Jun-2015 09:59:44 America/Los_Angeles]   #20 AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phabricator>/webroot/index.php:21]

Once the daemons have been running for a day, Diffusion has also a problem with pulling the latest state from the repositories. When I restart phd the "out of memory" problem is gone for another day.

My Phabricator installation is actually pretty minimal at the moment. There are just 100 Maniphest tasks and 4 repositories. 3 of them are mostly empty. One of them has just 5 commits so far, so I doubt it's a repository size issue.

Event Timeline

tiguchi raised the priority of this task from to Needs Triage.
tiguchi updated the task description. (Show Details)
tiguchi added projects: Diffusion, Daemons.
tiguchi added a subscriber: tiguchi.
epriestley claimed this task.
epriestley added a subscriber: epriestley.

I'm unable to reproduce this problem. We routinely run daemons for long periods of time without issues. See:

https://secure.phabricator.com/book/phabcontrib/article/bug_reports/