Page MenuHomePhabricator

launching daemons (phd) locks server (high load) or start is failing
Closed, ResolvedPublic

Description

Launching daemons (phd) locks server (high load) or start is failing.

Output:

xyz@server:/var/www/phabricator/phabricator/bin# ./phd status
There are no running Phabricator daemons.

xyz@server:/var/www/phabricator/phabricator/bin# ./phd start
Freeing active task leases...
Freed 0 task lease(s).
Preparing to launch daemons.
NOTE: Logs will appear in '/var/tmp/phd/log/daemons.log'.

Launching daemon "PhabricatorRepositoryPullLocalDaemon".
Launching daemon "PhabricatorGarbageCollectorDaemon".
Launching daemon "PhabricatorTaskmasterDaemon".
[2014-12-11 18:34:17] EXCEPTION: (CommandException) Command failed with error #255!
COMMAND
exec ./phd-daemon 'PhabricatorTaskmasterDaemon' --daemonize --log='/var/tmp/phd/log/daemons.log' --phd='/var/tmp/phd/pid' 

STDOUT
(empty)

STDERR
(empty) at [<phutil>/src/future/exec/ExecFuture.php:397]
  #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php:193]
  #1 PhabricatorDaemonManagementWorkflow::launchDaemon(string, array, boolean) called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementWorkflow.php:282]
  #2 PhabricatorDaemonManagementWorkflow::executeStartCommand(boolean) called at [<phabricator>/src/applications/daemon/management/PhabricatorDaemonManagementStartWorkflow.php:26]
  #3 PhabricatorDaemonManagementStartWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:394]
  #4 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:290]
  #5 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/daemon/manage_daemons.php:23]
....

LOG:

[11-Dec-2014 18:32:22 UTC] Bringing daemon 'PhabricatorRepositoryPullLocalDaemon' online...
[11-Dec-2014 18:32:25 UTC] Bringing daemon 'PhabricatorGarbageCollectorDaemon' online...
[11-Dec-2014 18:33:18 UTC] Bringing daemon 'PhabricatorTaskmasterDaemon' online...
[11-Dec-2014 18:33:22 UTC] [2014-12-11 18:33:21] ERROR 2: pcntl_fork(): Error 12 at [/var/www/phabricator/libphutil/src/daemon/PhutilDaemonOverseer.php:142]
[11-Dec-2014 18:33:31 UTC]   #0 pcntl_fork() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:142]
[11-Dec-2014 18:33:33 UTC]   #1 PhutilDaemonOverseer::__construct(array) called at [<phabricator>/scripts/daemon/launch_daemon.php:26]
[11-Dec-2014 18:33:34 UTC] [2014-12-11 18:33:34] EXCEPTION: (Exception) Unable to fork! at [<phutil>/src/daemon/PhutilDaemonOverseer.php:144]
[11-Dec-2014 18:33:34 UTC]   #0 PhutilDaemonOverseer::__construct(array) called at [<phabricator>/scripts/daemon/launch_daemon.php:26]

DEBUG:

launching daemon via phd debug do not cause a massive load

./phd debug PhabricatorTaskmasterDaemon

Event Timeline

rphl assigned this task to epriestley.
rphl raised the priority of this task from to Needs Triage.
rphl updated the task description. (Show Details)
rphl added a project: Phabricator.
rphl changed the edit policy from "All Users" to "Custom Policy".
rphl added a subscriber: rphl.
chad edited projects, added Daemons; removed Phabricator.

Can you tell us more about your environment? How much memory does this machine have? What is the OS?

pcntl_fork(): Error 12

This bit means out of memory.

it is a small vm for testing (only phabricator is running on it)

  • VM: google compute engine (0.6GB RAM)
  • OS: Debian/Wheezy 7.7
  • Webserver: default apache2 (php5+apc) / mysql

I noticed this as well, whilst testing in a small VM. I'd recommend at least 2GB.

epriestley claimed this task.
epriestley added a subscriber: epriestley.

With autoscaling pools and reduced overseer memory requirements this is probably indirectly fixed -- phd start is ~3x less memory hungry or something in that vein. Haven't seen any more reports of it recently.