The phd user has /bin/false as it's shell on our production instance, and it seems to work fine.

Hmm, this is pretty interesting. There was a configuration change right at the time of impact:

Yeah, I'm familiar with monit, and I think it will work decently well for this situation. Unfortunately, the current environment standardized on daemontools instead of monit.

@epriestley thanks for the feedback. I've read through the new docs, and I think have a section explaining phd-daemon, and incorporating some of the notes from T4181 (about setting a startup script for your OS) could be really useful.

Hey @eadler, thanks for looking into this. I feel this ticket is different from T4181. The former covers starting phd at boot, this ticket covers using process supervisors (like daemontools) which monitor processes on an ongoing basis.