I've noticed in the past few days that the daemons are regularly stopping (and also starting back up) -- I'll get the red banner about an unresolved issue, but usually by the time I ssh into the server and do a `./bin/phd status` they will be back up and running.
|libphutil|c5a7d67db29431dc67f930473f967c46477a7ad7|
|arcanist|1773aad8559965f8896cd66087710ddbeee153ae|
|phabricator|be4752f05a708d46000dd0be6964d1b1d51fae56|
From the daemons.log file, this exception has been appearing since the 14th and has occurred >1500 times. I've only seen evidence of this one exception repeating numerous times, I'm not sure if there are other exceptions happening -- I'm still digging through which is a little difficult as the daemons.log file is ~2GB.
```lang=console,name=/var/tmp/phd/log/daemons.log
[20-Oct-2015 10:46:53 America/New_York] [2015-10-20 10:46:53] EXCEPTION: (PhutilProxyException) Error while executing Task ID 3671295. {>} (Exception) Diff "PHID-DIFF-fyfkn25bo6gtkutuepax" does not exist! at [<phabricator>/src/applications/differential/editor/DifferentialTransactionEditor.php:1535]
[20-Oct-2015 10:46:53 America/New_York] arcanist(head=stable, ref.master=d54cb072facd, ref.stable=1773aad85599), phabricator(head=stable, ref.master=7cc8a73d1efd, ref.stable=be4752f05a70, custom=1), phutil(head=stable, ref.master=83f09f6c5a03, ref.stable=c5a7d67db294)
[20-Oct-2015 10:46:53 America/New_York] #0 <#2> DifferentialTransactionEditor::requireDiff(string, boolean) called at [<phabricator>/src/applications/differential/editor/DifferentialTransactionEditor.php:1243]
[20-Oct-2015 10:46:53 America/New_York] #1 <#2> DifferentialTransactionEditor::buildMailBody(DifferentialRevision, array) called at [<phabricator>/src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php:2331]
[20-Oct-2015 10:46:53 America/New_York] #2 <#2> PhabricatorApplicationTransactionEditor::buildMailForTarget(DifferentialRevision, array, PhabricatorMailTarget) called at [<phabricator>/src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php:2288]
[20-Oct-2015 10:46:53 America/New_York] #3 <#2> PhabricatorApplicationTransactionEditor::buildMail(DifferentialRevision, array) called at [<phabricator>/src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php:1048]
[20-Oct-2015 10:46:53 America/New_York] #4 <#2> PhabricatorApplicationTransactionEditor::publishTransactions(DifferentialRevision, array) called at [<phabricator>/src/applications/transactions/worker/PhabricatorApplicationTransactionPublishWorker.php:21]
[20-Oct-2015 10:46:53 America/New_York] #5 <#2> PhabricatorApplicationTransactionPublishWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
[20-Oct-2015 10:46:53 America/New_York] #6 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
[20-Oct-2015 10:46:53 America/New_York] #7 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[20-Oct-2015 10:46:53 America/New_York] #8 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:183]
[20-Oct-2015 10:46:53 America/New_York] #9 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:125]
```
There don't seem to be any related logs in nginx/php-fpm. I'm continuing investigation on my end.