Yes, they die when called manually.

I actually have no clue. There are AMIs for it in AWS but maybe that's unrelated or related in some complicated way.

Isn't CloudLinux a shared-host?

(By "provide instructions to construct a similar system", I mean something we can do in a few minutes, like launch a VM in AWS. I don't see any CloudLinux 6.8 AMIs offhand there, though.)

• Do they die after 5 minutes if you start them manually?
• Can you give us access to this system, or provide instructions for us to construct a similar system? We haven't seen this on any production system we manage, but do have other installs reporting some possibly-related issues.

Maybe also reasonable would be something like this:

T4752 is the detail on that.

When using local disk storage, the file is owned by the web-server user, not the daemon user.

Kicking this to the daemons should be like 20 lines of code, but we probably also need to flag the file as "queued for deletion" in the UI in the meantime and prevent you from interacting with it, which touches a few other interfaces, so this isn't completely trivial.

The phd user has /bin/false as it's shell on our production instance, and it seems to work fine.

T6806 might be ripe to resolve one way or another, too.

This "shouldn't" be possible: the overseer should restart them if they fail. And if the overseer dies, all the other daemons should die.

You can now start daemons at a low level by piping JSON into scripts/daemon/phd-daemon:

This is almost a year old and we have no idea how to reproduce it. If anyone hits this, feel free to file a new issue with reproduction steps (see Providing Reproduction Steps).

I fixed the totally unrelated half of this (bad default verbosity of phd debug) at some point in the recent-ish few months.

We got rid of this warning in favor of automatically reloading after config changes, which should survive HA setups without additional work.

I don't think this made it to 100% of callsites but we have reasonable saturation now, are populating it on newer tasks, and haven't hit any issues where this information would be useful in a long time. I'm just going to call this resolved-enough, we can add more of them if we're still missing some as needs arise.

This was a very long time ago, but I think the root cause of whatever was happening was really conflated with some other issue.

This should now be fully resolved in HEAD of master. Thanks for the report! Your details and stack traces were very helpful in understanding and resolving the issue.

After D15632:

Cool, let me see if I can repro that locally.

You're right; our amazon-ses.endpoint wasn't set. We just upgraded to HEAD and (incorrectly) configured amazon-ses.endpoint to email-smtp.us-east-1.amazonaws.com and got some exciting new errors that also might be worth fixing:

You should get a little further now in HEAD of master.

It looks like this is occurring during error handling so there's likely some other root error that this is masking, but we can fix this and see what it unveils.

Hi guys,
Same issue here!

Thanks !

At least some of these were likely T10214. I'm just going to merge this there until more evidence crops up.

Great, it works like a charm !

Specifically, the issue was that these commits contained strings like R2/R13, which the parser was getting confused about because of an improperly grouped regular expression.

This should be fixed by D15049. Thanks for the detailed reproduction instructions, including a repository I could clone locally.

You likely have a diff which isn't JSON-encodable (for example, it affects non-utf8 files). T5955 is probably the pathway forward here.

We haven't seen requests for this from other installs and I don't particularly favor this mechanism. Prior alerts in this vein like "daemon and web config differ" have created a lot of noise and little value. T7338 should eventually provide more monitoring tools.

I think it's good that destruction happens in the foreground, since I think this should be a very rare operation and errors should always be reported to the user. Users who want to background long destruction operations for some reason can use tools like screen.

I don't believe so, https://secure.phabricator.com/T4209#146794 clearly states that until that patch lands, daemons and repos must run on the same host.

I thought that it should still work because I'm running the repository pull daemon on the worker host?