I am having isues with the background daemons, causing over 100% in CPU load.
When I start the daemons (./bin/phd start) at first nothing seems wrong. But on occasion the load spikes to 100% for longer periods of time. Usually for more than 15 minutes.
Then after a while it disappears to reappear some seemingly random time later.
The command that is used to spawn the process that has this load is;
php /var/vhosts/phabricator/libphutil/scripts/daemon/exec/exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/var/vhosts/phabricator/arcanist/src --load-phutil-library=/var/vhosts/phabricator/phabricator/src --log=/var/vhosts/phabricator/logs/daemons.log --
When I strace the daemon while the load is high, I see an endless loop of these;
19444 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
19444 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
19444 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
19444 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
19444 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
19444 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
19444 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
19444 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
19444 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
19444 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
19444 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
19444 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
19444 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
19444 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
19444 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
19444 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
I have 149.508 of these lines captured in less than 10 seconds, which explain the high CPU.
Now as far as I know the number 8 refers a file descriptor for this proces. But when I view /proc/19444/fd/ there is no reference 8;
0 lr-x------ 1 root root 64 Mar 17 14:08 0 -> pipe:[857844016]
0 l-wx------ 1 root root 64 Mar 17 14:08 1 -> pipe:[857844017]
0 l-wx------ 1 root root 64 Mar 17 14:08 2 -> pipe:[857844018]
0 lrwx------ 1 root root 64 Mar 17 14:08 3 -> socket:[776376581]
0 lrwx------ 1 root root 64 Mar 17 14:08 4 -> socket:[776376691]
0 lrwx------ 1 root root 64 Mar 17 14:08 5 -> socket:[857844034]
0 lrwx------ 1 root root 64 Mar 17 14:08 6 -> socket:[857844036]
I have tried to increase some sleep() calls in the source code to slow down the number of calls, but had no success. And have in no way been able to locate where exactly this is happening. It does not seem to occur until I add a SVN repository to a fresh install. Even when Phabricator reports to have fully imported, and no commits are made, the problem appears.
I am running the lastest versions of arcanist, libphutil and phabricator from the Git repository on a DELL PowerEdge R310 running CentOS 5.10 x64.
All storage is on local disks.
The repositories are "Hosted elsewhere" and loaded from HTTP, though actually hosted on the same machine.
Any hints or tips on how to resolve or debug this issue?