Page MenuHomePhabricator

Herald stopped sending commit emails
Closed, ResolvedPublic

Description

We've been using a herald rule that emails a mailing list each time a commit occurs.

Since yesterday, the emails have stopped. There have been no changes to Phabricator or the machine hosting it.

Other Phabricator emails, such as Differential emails, as well as a personal Herald rule for Differential, are working fine.

I've already tried restarting the daemons via phd and that didn't fix it.

Event Timeline

adityar7 raised the priority of this task from to Needs Triage.
adityar7 updated the task description. (Show Details)
adityar7 added a project: Herald.
adityar7 added a subscriber: adityar7.

Is the repository you're expecting email from in the "Importing..." state? You can check on /diffusion/X/, where "X" is the repository callsign. If the repository is importing, the header will say "Importing..." in red. It's also possible T4086 is related to this.

Nope, it doesn't appear to be in "Importing" state.

And Diffusion does have the latest commits.

Also, T4086 has been happening for several weeks and was caused by a different repository, while this one started yesterday on our main repository that gets several commits every day.

Okay, here are some steps to diagnose this.

First, identify a commit which you expected to receive email for but did not. Let's call this rXabcd, where X is the repository callsign.

(1) Find the commit in the Diffusion history view (normally, on the /diffusion/X/ page, in the "Recent Commits" table) and verify the "Change" column says "Contents Modified" (or similar), not "Importing...". Is the commit "Importing.."?

(2) In either case, run:

phabricator/ $ ./bin/repository importing X

...and look for the commit. If it's there, the output should look like:

rXabcdefghijklmnopqrstuvwxyz Apple Banana

The right columns show remaining import steps. Is the commit present in the output? If so, what's the text of the line it appears on?

(3) Go to Herald and click Transcripts. Find the commit in the transcript list. Is the commit present? If you can't find a transcript for it, you can stop here.

(4) If you found a transcript, click "View Transcript", then "All Rules". Did the rules you expected to match match? Does the "Actions" table say that it sent email?

(5) Run this:

phabricator/ $ ./bin/mail list-outbound

...and look for the mail. What does the status column say (sent, queued, void)?

(6) Examine the mail in detail:

phabricator/ $ ./bin/mail show-outbound --id <mail-id>

Does the "Recipients" section indicate the mail was undeliverable?

(1) It says "Contents Modified".

(2) No output.

(3) Yes, it's present.

(4) Yes, rule matched and "Outcome: SUCCESS Added address to email targets."

(5) Sent.

(6) It does not say that the mail was undeliverable.

All of the above apply to all the commits since the email stopped coming :/

The mailing list is functioning, and it's not spam-related. In fact, in order to confirm that Phabricator can send emails to it I just created a Maniphest rule in Herald that CCs that mailing list. It works i.e. Maniphest emails are being sent to the mailing list.

I had to upgrade Phabricator in order to run these steps, so I decided to test the commit email again by landing a test commit. No email. The ./bin/mail show-outbound output shows no errors and is similar to that for the successful Maniphest email that was sent to the same mailing list.

If the mail shows "sent", that means we handed it to the MTA and the MTA confirmed receipt or otherwise responded positively. I don't have any theories on why Maniphest mail might work but commit mail doesn't -- if you add a Herald rule which sends to a user instead of a list, does that work?

adityar7 claimed this task.

I recreated the herald rules and also re-setup the Amazon SES config. Something fixed it, it's back in business :/