Why do my daemons keep <dying> ?
OpenPublic
Actions

Asked by alawson on May 25 2016, 10:16 PM.

Details

I have installed an init script (below) and my daemons start with this script just fine. I can also start them manually. However, the consistently die after around 5 minutes. After this happens, I naturally get the "You have an unresolved setup issue..." and Phabricator reports that my daemons are dead. If I do a phd status, I get something like this:

Log Daemon       Host      Overseer Started                 Class                                       Arguments
363 282720:fjp6a localhost 282720   May 25 2016, 5:50:28 PM <DEAD> PhabricatorRepositoryPullLocalDaemon
364 282720:gpdw4 localhost 282720   May 25 2016, 5:50:28 PM <DEAD> PhabricatorTriggerDaemon
365 282720:hu7io localhost 282720   May 25 2016, 5:50:28 PM <DEAD> PhabricatorTaskmasterDaemon

Everything else seems to be working fine. The system is running CloudLinux (CentOS 6.8) and it's in its own account with a ton of resources given to it. I do have root access, but I like to keep everything in the server isolated as much as possible.

I check the logs and they say:

[25-May-2016 18:05:01 America/New_York] [2016-05-25 18:05:01] ERROR 8: ob_end_flush(): failed to delete and flush buffer. No buffer to delete or flush at [/home/kmdgdev/public_html/libphutil/src/daemon/PhutilDaemon.php:246]
[25-May-2016 18:05:01 America/New_York] arcanist(head=master, ref.master=c58f1b9a2507), phabricator(head=master, ref.master=0b5ab2330d99), phutil(head=master, ref.master=180f0999d094)
[25-May-2016 18:05:01 America/New_York]   #0 ob_end_flush() called at [<phutil>/src/daemon/PhutilDaemon.php:246]
[25-May-2016 18:05:01 America/New_York]   #1 PhutilDaemon::endStdoutCapture() called at [<phutil>/src/daemon/PhutilDaemon.php:95]
[25-May-2016 18:05:01 America/New_York]   #2 PhutilDaemon::__destruct()
[25-May-2016 18:05:02 America/New_York] [2016-05-25 18:05:02] ERROR 8: ob_end_flush(): failed to delete and flush buffer. No buffer to delete or flush at [/home/kmdgdev/public_html/libphutil/src/daemon/PhutilDaemon.php:246]
[25-May-2016 18:05:02 America/New_York] arcanist(head=master, ref.master=c58f1b9a2507), phabricator(head=master, ref.master=0b5ab2330d99), phutil(head=master, ref.master=180f0999d094)
[25-May-2016 18:05:02 America/New_York]   #0 ob_end_flush() called at [<phutil>/src/daemon/PhutilDaemon.php:246]
[25-May-2016 18:05:02 America/New_York]   #1 PhutilDaemon::endStdoutCapture() called at [<phutil>/src/daemon/PhutilDaemon.php:95]
[25-May-2016 18:05:02 America/New_York]   #2 PhutilDaemon::__destruct()
[25-May-2016 18:05:03 America/New_York] [2016-05-25 18:05:03] ERROR 8: ob_end_flush(): failed to delete and flush buffer. No buffer to delete or flush at [/home/kmdgdev/public_html/libphutil/src/daemon/PhutilDaemon.php:246]
[25-May-2016 18:05:03 America/New_York] arcanist(head=master, ref.master=c58f1b9a2507), phabricator(head=master, ref.master=0b5ab2330d99), phutil(head=master, ref.master=180f0999d094)
[25-May-2016 18:05:03 America/New_York]   #0 ob_end_flush() called at [<phutil>/src/daemon/PhutilDaemon.php:246]
[25-May-2016 18:05:03 America/New_York]   #1 PhutilDaemon::endStdoutCapture() called at [<phutil>/src/daemon/PhutilDaemon.php:95]
[25-May-2016 18:05:03 America/New_York]   #2 PhutilDaemon::__destruct()

Inside my daemon (while it's running):

{"pid":282720,"start":1464213028,"config":{"daemonize":true,"log":"\/home\/kmdgdev\/phab.working\/var\/tmp\/phd\/log\/daemons.log","piddir":"\/home\/kmdgdev\/phab.working\/var\/tmp\/phd\/pid","daemons":[{"class":"PhabricatorRepositoryPullLocalDaemon"},{"class":"PhabricatorTriggerDaemon"},{"class":"PhabricatorTaskmasterDaemon","autoscale":{"group":"task","pool":7,"reserve":0}}]},"daemons":[{"pid":282722,"id":"282720:fjp6a","config":{"class":"PhabricatorRepositoryPullLocalDaemon","argv":[],"autoscale":[]}},{"pid":282723,"id":"282720:gpdw4","config":{"class":"PhabricatorTriggerDaemon","argv":[],"autoscale":[]}},{"pid":282724,"id":"282720:hu7io","config":{"class":"PhabricatorTaskmasterDaemon","autoscale":{"group":"task","pool":7,"reserve":0},"argv":[]}}]}

/etc/init.d/phd :

#!/bin/sh
### BEGIN INIT INFO
# Provides: phd
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
### END INIT INFO
# phabricator-phd      Wrapper above the phabricator-phd daemon (for CentOS 6.x)
#
# chkconfig: 2345 90 90

# Source function library.
. /etc/init.d/functions

start() {
        echo -n "Starting phabricator PHD daemon: "
        su - root -s /bin/sh -c '/home/kmdgdev/public_html/phabricator/bin/phd start'
	stat=$?
        touch /var/lock/subsys/phabricator-phd
        return $stat
}

stop() {
        echo -n "Shutting down Phabricator PHD daemon: "
        su - root -s /bin/sh -c '/home/kmdgdev/public_html/phabricator/bin/phd stop'
	stat=$?
        rm -f /var/lock/subsys/phabricator-phd
        return $stat
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    status)
       su - root -s /bin/sh -c '/home/kmdgdev/public_html/phabricator/bin/phd status'
        ;;
    restart)
        stop
        start
        ;;
    reload)
        restart
        ;;
    *)
        echo "Usage: phabricator-phd {start|stop|status|reload|restart"
        exit 1
        ;;
esac
exit $?

Event Timeline

Do they die after 5 minutes if you start them manually?
Can you give us access to this system, or provide instructions for us to construct a similar system? We haven't seen this on any production system we manage, but do have other installs reporting some possibly-related issues.

(By "provide instructions to construct a similar system", I mean something we can do in a few minutes, like launch a VM in AWS. I don't see any CloudLinux 6.8 AMIs offhand there, though.)

Isn't CloudLinux a shared-host?

I actually have no clue. There are AMIs for it in AWS but maybe that's unrelated or related in some complicated way.

Yes, they die when called manually.

Yes, I can grant access to the server.

I had this same problem with a similar server where it was installed under
root. I was using it as a test platform to test Phabricator and it also had
this problem.

CloudLinux is very similar to CentOS. Resources aren't an issue. I can
allocate whatever is necessary.
chad added a comment.

Isn't CloudLinux a shared-host?

*QUESTION DETAIL*
https://secure.phabricator.com/Q402

*EMAIL PREFERENCES*
https://secure.phabricator.com/settings/panel/emailpreferences/

*To: *alawson, chad
*Cc: *chad, epriestley

This is almost certainly not a resource problem, but I have no idea what it is.

Here's my SSH public key:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqtgXJJeboECX1s5/Esdj+8Tdfeu73CezkAodgbld62QfAQ8UG8x4ZPZgni8j7lwnIpLyNFsQUHGf0mPpQE+KIyT7sA/ICIbXNI74qIWsO2oe+d5OZhDA9c7LUx4TdhzSzdI9LHx99zH2Q74l/L4bIjMKFtKBxuLBsTWZOTB5d1J63YFOKSDn0RtXmYND1i5iaEjQTTxGZHCJiECFyBCl1Gtkjlh04caJQebXH7Wzipwa2yTwddDOuARTT/48uI5WvbwYmVi8JiqSdDr8DoYUaBkI8vpvSj3iHa8Br+runZLxNf1JEY/etM9V0dAICYaAmdVA5s5cFGEfPYE5mGHaIQ== epriestley@orbital

If you give me access to an account on the server so I can reproduce the problem, I can likely resolve it.

Sure thing. I left the office shortly after posting on Ponder (hence the
answer delay).

I have an idea I'm going to test tomorrow and will post results if it works.

Otherwise, expect me to reply around the same time I posted today,
tomorrow. Moving to Phabricator entirely is a near term goal of ours, but I
only have certain times during the day when I can work on it.

We're extremely excited about the platform and it's potential. Thank you
for your expedient replies as always. I never would have expected such a
level of devotion.

I worked on this problem for a couple hours last night, tailing log files
and checking various security features. This is our development server, the
lead Dev and I have complete control over it. I also had this problem on
another machine which is setup very similarly (CloudLinux, but locked done
a ton as its our hosting server), but on that machine I did not put the
Phabricator installation in a cPanel account. Had the same problem there,
so I don't think it's the security / CageFS getting in the way. Something
is timing the script out or killing it, and it's not the Kernel.

I didn't get the problem fixed, but I did notice a pattern. The init script
(I have two versions which work completely differently) kicks off the php
link/program phd which then launches a PHP instance.

I noticed that the launched PHP process dies after exactly 15 minutes. I
thought this may have something to do with max_execution_time or something,
but modifying those settings did nothing.

I will add your certificate to my server this evening... is there a more
secure method I can use to send you other information you will need to look
at it?

You can email me directly at epriestley@phacility.com, or create a private thread in Conpherence on this install via the icon in the header.

I have the similar problem.. Sometimes web UI shows setup issue warning "Phabricator Daemons Are Not Running". Then I restart daemons. They work fine for some period (may be a week or two). And then again "Phabricator Daemons Are Not Running" is showed...

Log   Daemon       Host      Overseer Started                 Class                                      Arguments
37953 13371:rbg2dl localhost 13371    Jul 27 2016, 8:33:37 AM <DEAD> PhabricatorRepositoryPullLocalDaeon
37954 13371:heizcr localhost 13371    Jul 27 2016, 8:33:37 AM <DEAD> PhabricatorTriggerDaemon         
37955 13371:3hev6f localhost 13371    Jul 27 2016, 8:33:37 AM <DEAD> PhabricatorTaskmasterDaemon

In daemons.log I see a lot of messages (every 5-10 seconds):

[03-Aug-2016 15:44:45 UTC] [2016-08-03 15:44:45] EXCEPTION: (PhutilProxyException) Error while executing Task ID 4899. {>} (AphrontCharacterSetQueryException) Attempting to construct a query using a non-utf8 string when utf8 is expected. Use the `%B` conversion to escape binary strings data. at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:377]
[03-Aug-2016 15:44:45 UTC] arcanist(head=master, ref.master=2234c8cacc21), phabricator(head=master, ref.master=03e54afc146c), phutil(head=master, ref.master=4a2238fc8605)
[03-Aug-2016 15:44:45 UTC]   #0 <#2> AphrontBaseMySQLDatabaseConnection::validateUTF8String(string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:10]
[03-Aug-2016 15:44:45 UTC]   #1 <#2> AphrontMySQLiDatabaseConnection::escapeUTF8String(string) called at [<phutil>/src/xsprintf/qsprintf.php:178]
[03-Aug-2016 15:44:45 UTC]   #2 <#2> xsprintf_query(AphrontMySQLiDatabaseConnection, string, integer, string, integer) called at [<phutil>/src/xsprintf/xsprintf.php:70]
[03-Aug-2016 15:44:45 UTC]   #3 <#2> xsprintf(string, AphrontMySQLiDatabaseConnection, array) called at [<phutil>/src/xsprintf/qsprintf.php:64]
[03-Aug-2016 15:44:45 UTC]   #4 <#2> qsprintf(AphrontMySQLiDatabaseConnection, string, string, string) called at [<phabricator>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryCommitChangeParserWorker.php:54]
[03-Aug-2016 15:44:45 UTC]   #5 <#2> PhabricatorRepositoryCommitChangeParserWorker::lookupOrCreatePaths(array) called at [<phabricator>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryMercurialCommitChangeParserWorker.php:255]
[03-Aug-2016 15:44:45 UTC]   #6 <#2> PhabricatorRepositoryMercurialCommitChangeParserWorker::parseCommitChanges(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryCommitChangeParserWorker.php:26]
[03-Aug-2016 15:44:45 UTC]   #7 <#2> PhabricatorRepositoryCommitChangeParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:40]
[03-Aug-2016 15:44:45 UTC]   #8 <#2> PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
[03-Aug-2016 15:44:45 UTC]   #9 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
[03-Aug-2016 15:44:45 UTC]   #10 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[03-Aug-2016 15:44:45 UTC]   #11 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:184]
[03-Aug-2016 15:44:45 UTC]   #12 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:127]

Why do my daemons keep <dying> ?
OpenPublic
Actions

Details

Event Timeline

New Answer

Answer

Why do my daemons keep <dying> ?OpenPublicActions

Details

Event Timeline

New Answer

Answer

Why do my daemons keep <dying> ?
OpenPublic
Actions