I've recently come across a glitch where sometimes a daemon (generally the Taskmaster) stops running (I suspect it gets killed due to memory constraints in the fairly resource limited environment where I currently run our small instance), but others continue.
The web UI shows a warning that daemons are not (all) running, however bin/phd start does not work to resolve the issue because some other daemons are still running. The Overseer doesn't seem to detect or fix this itself either, though it continues to run.
I have already rolled some monitoring based on the output of bin/phd status which can detect if no daemons are running and automatically start them/notify me. However, detecting the 'not all daemons that should be running are running' case is more difficult.
It would be good either if Phabricator could look after itself in this circumstance (i.e. the Overseer detect and reinstate killed/unexpectedly dead instances). Failing that, at least if this situation could be detected from the commandline (not just the web UI) external means could be used to fix it up; preferably running bin/phd start would resolve the issue, not requiring bin/phd restart.
phabricator
8f7983a5be3a56db5b79dc7c3a0eb470f1d7ca02 (Sat, Mar 25) (branched from b4effdf26c3e7d5de0d010cf14626c5d8d404e04 on origin)
arcanist
60aaee0ed3f5a1e4384ac7d7f2efd2c64cecbe44 (Sat, Mar 25) (branched from d1db9a72b552151613a918e3d49fa72433387a68 on origin)
phutil
b133c277014868d476f08b4ebecde2ea795509e4 (Sat, Mar 25) (branched from c0bc116bedc895fd617799a13549f8707edfd3fb on origin)