Why do my daemons keep <dying> ?
Event Timeline
- Do they die after 5 minutes if you start them manually?
- Can you give us access to this system, or provide instructions for us to construct a similar system? We haven't seen this on any production system we manage, but do have other installs reporting some possibly-related issues.
(By "provide instructions to construct a similar system", I mean something we can do in a few minutes, like launch a VM in AWS. I don't see any CloudLinux 6.8 AMIs offhand there, though.)
I actually have no clue. There are AMIs for it in AWS but maybe that's unrelated or related in some complicated way.
Yes, they die when called manually.
Yes, I can grant access to the server.
I had this same problem with a similar server where it was installed under
root. I was using it as a test platform to test Phabricator and it also had
this problem.
CloudLinux is very similar to CentOS. Resources aren't an issue. I can
allocate whatever is necessary.
chad added a comment.
Isn't CloudLinux a shared-host?
*QUESTION DETAIL*
https://secure.phabricator.com/Q402
*EMAIL PREFERENCES*
https://secure.phabricator.com/settings/panel/emailpreferences/
*To: *alawson, chad
*Cc: *chad, epriestley
This is almost certainly not a resource problem, but I have no idea what it is.
Here's my SSH public key:
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqtgXJJeboECX1s5/Esdj+8Tdfeu73CezkAodgbld62QfAQ8UG8x4ZPZgni8j7lwnIpLyNFsQUHGf0mPpQE+KIyT7sA/ICIbXNI74qIWsO2oe+d5OZhDA9c7LUx4TdhzSzdI9LHx99zH2Q74l/L4bIjMKFtKBxuLBsTWZOTB5d1J63YFOKSDn0RtXmYND1i5iaEjQTTxGZHCJiECFyBCl1Gtkjlh04caJQebXH7Wzipwa2yTwddDOuARTT/48uI5WvbwYmVi8JiqSdDr8DoYUaBkI8vpvSj3iHa8Br+runZLxNf1JEY/etM9V0dAICYaAmdVA5s5cFGEfPYE5mGHaIQ== epriestley@orbital
If you give me access to an account on the server so I can reproduce the problem, I can likely resolve it.
Sure thing. I left the office shortly after posting on Ponder (hence the
answer delay).
I have an idea I'm going to test tomorrow and will post results if it works.
Otherwise, expect me to reply around the same time I posted today,
tomorrow. Moving to Phabricator entirely is a near term goal of ours, but I
only have certain times during the day when I can work on it.
We're extremely excited about the platform and it's potential. Thank you
for your expedient replies as always. I never would have expected such a
level of devotion.
I worked on this problem for a couple hours last night, tailing log files
and checking various security features. This is our development server, the
lead Dev and I have complete control over it. I also had this problem on
another machine which is setup very similarly (CloudLinux, but locked done
a ton as its our hosting server), but on that machine I did not put the
Phabricator installation in a cPanel account. Had the same problem there,
so I don't think it's the security / CageFS getting in the way. Something
is timing the script out or killing it, and it's not the Kernel.
I didn't get the problem fixed, but I did notice a pattern. The init script
(I have two versions which work completely differently) kicks off the php
link/program phd which then launches a PHP instance.
I noticed that the launched PHP process dies after exactly 15 minutes. I
thought this may have something to do with max_execution_time or something,
but modifying those settings did nothing.
I will add your certificate to my server this evening... is there a more
secure method I can use to send you other information you will need to look
at it?
You can email me directly at epriestley@phacility.com, or create a private thread in Conpherence on this install via the icon in the header.
I have the similar problem.. Sometimes web UI shows setup issue warning "Phabricator Daemons Are Not Running". Then I restart daemons. They work fine for some period (may be a week or two). And then again "Phabricator Daemons Are Not Running" is showed...
Log Daemon Host Overseer Started Class Arguments 37953 13371:rbg2dl localhost 13371 Jul 27 2016, 8:33:37 AM <DEAD> PhabricatorRepositoryPullLocalDaeon 37954 13371:heizcr localhost 13371 Jul 27 2016, 8:33:37 AM <DEAD> PhabricatorTriggerDaemon 37955 13371:3hev6f localhost 13371 Jul 27 2016, 8:33:37 AM <DEAD> PhabricatorTaskmasterDaemon
In daemons.log I see a lot of messages (every 5-10 seconds):
[03-Aug-2016 15:44:45 UTC] [2016-08-03 15:44:45] EXCEPTION: (PhutilProxyException) Error while executing Task ID 4899. {>} (AphrontCharacterSetQueryException) Attempting to construct a query using a non-utf8 string when utf8 is expected. Use the `%B` conversion to escape binary strings data. at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:377]
[03-Aug-2016 15:44:45 UTC] arcanist(head=master, ref.master=2234c8cacc21), phabricator(head=master, ref.master=03e54afc146c), phutil(head=master, ref.master=4a2238fc8605)
[03-Aug-2016 15:44:45 UTC] #0 <#2> AphrontBaseMySQLDatabaseConnection::validateUTF8String(string) called at [<phutil>/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:10]
[03-Aug-2016 15:44:45 UTC] #1 <#2> AphrontMySQLiDatabaseConnection::escapeUTF8String(string) called at [<phutil>/src/xsprintf/qsprintf.php:178]
[03-Aug-2016 15:44:45 UTC] #2 <#2> xsprintf_query(AphrontMySQLiDatabaseConnection, string, integer, string, integer) called at [<phutil>/src/xsprintf/xsprintf.php:70]
[03-Aug-2016 15:44:45 UTC] #3 <#2> xsprintf(string, AphrontMySQLiDatabaseConnection, array) called at [<phutil>/src/xsprintf/qsprintf.php:64]
[03-Aug-2016 15:44:45 UTC] #4 <#2> qsprintf(AphrontMySQLiDatabaseConnection, string, string, string) called at [<phabricator>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryCommitChangeParserWorker.php:54]
[03-Aug-2016 15:44:45 UTC] #5 <#2> PhabricatorRepositoryCommitChangeParserWorker::lookupOrCreatePaths(array) called at [<phabricator>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryMercurialCommitChangeParserWorker.php:255]
[03-Aug-2016 15:44:45 UTC] #6 <#2> PhabricatorRepositoryMercurialCommitChangeParserWorker::parseCommitChanges(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryCommitChangeParserWorker.php:26]
[03-Aug-2016 15:44:45 UTC] #7 <#2> PhabricatorRepositoryCommitChangeParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:40]
[03-Aug-2016 15:44:45 UTC] #8 <#2> PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
[03-Aug-2016 15:44:45 UTC] #9 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
[03-Aug-2016 15:44:45 UTC] #10 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[03-Aug-2016 15:44:45 UTC] #11 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:184]
[03-Aug-2016 15:44:45 UTC] #12 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:127]