Write Herald rules for outbound mail
Open, NormalPublic

Description

When I set up a herald rule, it doesn't include the repository as a separate token, so it's not possible to do things like set up a gmail filter based on the particular repository. It would be great to have the repository separated out so that I could filter different repositories in different ways.

Alternatively, a way to customizing herald emails (either globally or on a per-rule basis) would be a more general solution to the problem.

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes

I think T6367 effectively made the "logical" vs "physical" separation irrelevant: we can just operate on "physical" messages, since we no longer do any of the weird/bad stuff we used to do when splitting messages. We'll run rules on each "physical" message, but that's desirable and expected. Possibly we'll put some kind of new field on them to tie them all together, but I'm not sure we even need that.

We do need to make some changes: to formally record actors, and to denormalize recipients so we can query for them efficiently. Mail objects also don't currently have a PHID, but need one. These changes are straightforward.

After that, we can build a UI. This is straightforward except that we need to flag certain messages (like password reset email and welcome mail) as never viewable.

So my rough guess is that this is something like:

EstimateTask
2 HoursAdd PHIDs, actors, edges to mail.
3 HoursBuild new policy-respecting "mail" application.
1 HourBuild a Herald adapter for outbound mail.
1-3 Hours?Build rules -- depends how much stuff we want.

So I'd ballpark this roughly in the realm of 7-9 hours, mostly depending on how many rules we implement now vs eventually.

angie moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Jun 16 2015, 9:39 PM
epriestley moved this task from Backlog to The Queue on the Prioritized board.

Concrete stuff here is probably something like this (this might not be the optimal order):

  • Add PHIDs to MetaMTAMail, with a PHIDType, and backpopulate them.
  • Make PhabricatorMetaMTAMail implement PolicyInterface.
    • View policy is NOONE and the author and recipients have automatic view capabilities: you can see stuff you sent or received.
    • No edit capability.
  • Add a Query and SearchEngine for mail.
  • Build new ListViewController + DetailViewController UIs at /mail/. These are your "outbox" and "message detail" views. If this happens bit by bit, don't show any mail details yet (subjects/bodies) since we can't let web users see password reset email (even their own).
  • Add an actorPHID to PhabricatorMetaMTAMail. This probably has to be nullable since some mail won't have a meaningful actor. This will let us query for stuff you can view more efficiently. This data is usually present in the 'from' key on the mail properties right now, but that isn't queryable. Populate this in either setFrom() or transaction publisher.
  • Add an edge type and start writing edges for recipients when we initially save mail. This will be the other half of efficiently querying for stuff you can view.
  • Change the Query to be smart about querying for stuff you can view so we aren't doing a ton of application-level work filtering it (join edges, do WHERE dst = you OR authorPHID = you or something)
  • Add a parameter like "nonsensitive", and require it to be set before any mail details are shown. Start populating it from the TransactionPublisher. The idea is that stuff is "sensitive" by default unless we mark it as OK to show, so we don't leak welcome links, password reset links, etc.
  • Show mail details on "nonsensitive" mail.

Theoretically that gives us a reasonable "Outbox" kind of view that we can build on to show what happened with Herald and why mail got dropped.

A few messages won't show up in this view (e.g., confirmation mail we send when you add a new email address is probably sent directly to the address, so the user won't be a recipient). I think this stuff doesn't matter and will be immune to outbound rules anyway.

A partial solution is to (ab)use the List-ID header, which Gmail does understand. GitHub took this approach.

avivey added a subscriber: avivey.Jun 29 2015, 5:28 PM

A use-case I have with JIRA:
We have a bunch of custom fields, most of them are of no interest to me, so I'd like to ignore them when they change; But I'd rather black-list uninteresting fields than white-list interesting ones (Which risks missing interesting changes).
So the condition I'd like is Transaction is limited to these fields: [field], [field], [field].

Do users manually update the JIRA fields, or does something synchronize them?

That is, would [Content source][is][synchronizing thing] (or [actor][is][jira-bot]) be equivalent? Or is that not the same in this use case?

In this case, they manually update them (And they find them interesting - these are things like "when we expect a fix to be released").

nickz added a subscriber: nickz.Jul 16 2015, 9:39 PM
epriestley raised the priority of this task from Wishlist to Normal.Aug 11 2015, 9:25 PM

This sort of exists in HEAD now, but with major limitations:

  • There are no meaningful fields available yet (only a "subject" field and some other builtin fields).
  • You can't write global rules.

These are both somewhat blocked on philosophical/product discussion in T9161. In particular, the outcome there affects how "mail tags" are represented in the future, which needs to be resolved before we can write a "mail tags" field. This is probably one of the most useful/important fields.

I'd also like to have a clearer picture of our pathway forward there before enabling global rules. I think personal outbound rules are the correct solution for the narrow motivating problem here and elsehwere (roughly, "Gmail isn't powerful enough to write the rules I want") but global outbound rules probably aren't the correct solution for the larger problems in T9161 in the general case (roughly, "users complain to administrators who complain to us that Phabricator sends too much mail"). It will be harder for us to move forward in the future if installs work around this problem by adding global outbound rules today.

epriestley moved this task from The Queue to Paused on the Prioritized board.Sep 6 2015, 5:49 PM
eadler added a project: Restricted Project.Feb 9 2016, 12:15 AM
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Feb 23 2016, 6:13 PM
jparise removed a subscriber: jparise.Feb 23 2016, 6:18 PM
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Apr 7 2016, 6:08 PM
kfa added a subscriber: kfa.Jun 7 2016, 8:31 PM

Bookkeeping; there's no longer an external driver for this.

eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Jul 4 2016, 9:19 PM
epriestley moved this task from Backlog to v3 on the Mail board.Jul 16 2016, 2:26 PM
eadler moved this task from Important to Nits on the FreeBSD board.Oct 24 2016, 6:42 AM
urzds added a subscriber: urzds.Jul 12 2017, 11:12 AM
sfrench removed a subscriber: sfrench.Nov 28 2017, 8:37 PM