Best of Both Worlds: one email to everyone with privacy controls and good threading
Open, Needs TriagePublic

Description

I believe its possible to solve the vast majority of the problems that metamta.one-mail-per-recipient=false has and possibly remove some options including possibly metamta.recipients.show-hints metamta.public-replies and metamta.public-replies.

Best Of Both Worlds Config:

Create one email per user:

  • Set the reply-to header to a secure per-user value. This overrides both "reply" and "reply all"
  • Set the TO and CC header to the complete set of users
  • (optional) Don't show TO and CC hints as the they are now in the header field
  • mail to Mailing List users should generate reply-to with a hint to the user that no action was taken.

If you construct a different email per recipient, but include the complete header information you will get the following pros and cons:

Pros:

Policy controls work correctly and are enforced per-user.
Recipients can see To/Cc at a glance.
If you use mailing lists, you won't get duplicate mail if you're a normal recipient and also Cc'd on a mailing list.
"Reply All" no longer spams all other users.
both public and private reply-to addresses work.
Mails are sent in the language of user preference.

Cons:

Getting threading to work properly is harder, and probably requires making mail less useful by turning off options.

This is actually questionable. This makes threading work better for me on some clients.

eadler created this task.Thu, Feb 9, 6:27 PM
eadler edited the task description. (Show Details)
eadler edited the task description. (Show Details)Thu, Feb 9, 6:31 PM
  • I think this sends the recipients many copies of the email, particularly if they are on mailing lists? Why do you believe that users will receive only one copy of the mail if we send ten messages with their address in "To"?
  • Are you sure "Reply-To" overrides "Reply All" in all major clients? I haven't tested this personally, but I can't immediately find any references to this behavior on Google. I believe we already use "Reply-To", so I would expect "Reply All" to already be a non-problem if "Reply-To" overrode it. My pre-existing belief was that "Reply-To" replaced only "From", not the entire recipient list, under "Reply All".

Or, more generally, if we send two mails:

To: X, Y
Reply-To: Secret-Address-For-X
To: X, Y
Reply-To: Secret-Address-For-Y

...what guarantees that user X receives only one of them, and that the one they receive is the one with the right secret?

Unless I'm missing something, I think the instructions here cause X to receive two mails (they are on "To" for both).

If some header like Message-ID could de-duplicate them on the client (or in the mailbox adjacent to the client), they might only see one in their mailbox, but nothing would guarantee that X and Y saw the versions with the right secrets -- they'd probably see whichever one arrived first? (Presumably, usually the one we sent first.)

Also note that we can not control Message-ID (or some other headers) with some mail providers (notably, SES).

I don't think there's a header like Ignore-The-"To"-Header-And-Only-Send-To-These-Recipients: X that lets us lie in To but really send to a different recipient list, but that seems like it's required for this approach to be viable.

eadler added a comment.Thu, Feb 9, 8:51 PM

Unless I'm missing something, I think the instructions here cause X to receive two mails (they are on "To" for both).

This is the difference between the envelope TO and the header TO. Here is an example

Here is how I sent it

∴sendmail -i -- mymail@my-real-domain.example.com < to-eax
[44007 12:51:03.343 eax@tw-mpb-eax ~/exps/mail]∴cat to-eax
From: yelo@website.com
To: someone@important.com
Subject: Yellow Blosums 2

EOF
eadler added a comment.Thu, Feb 9, 9:02 PM

Some more thoughts: even if we can't control the Reply-to in all cases, phabricator can look at the TO header and perhaps choose not to send mail if it looks like it was already sent to a given user. Of course, given the above, this is spoofable.

eadler added a comment.Thu, Feb 9, 9:04 PM
From: yelo@website.com
To: someone@important.com, bob@bob.com
Reply-to: only-me@somewhereelse.com
Subject: Yellow Blosums 2

EOF

works for me on some mail clients: both Reply and Reply All send only to only-me@somewhereelse.com. On other clients Reply works but Reply-All does not (it sends it to everyone, including the reply-all address). We may not be able to eliminate all the options then, but I do think we can get closer to being better and at least not violating policy when using this option.

Oh, I didn't even realize that was a thing. I assumed SMTP went like "okay here is a message: <message body>".

I don't think we have control over the SMTP envelope for at least some of of the APIs we implement, although perhaps I'm mistaken. In some cases I chose the non-SMTP version of things (e.g., POST some JSON) when an SMTP version was available since I mostly know how JSON works but do not mostly know how SMTP works. It's possible we could revisit those decisions and do more SMTP to get greater access to envelopes, although I'd guess not all adapters provide us envelope access.

Some more thoughts: even if we can't control the Reply-to in all cases, phabricator can look at the TO header and perhaps choose not to send mail if it looks like it was already sent to a given user.

We already attempt to do this -- are you seeing it not work in practice? Rough pathway is:

  • PhabricatorMetaMTAReceivedMail loads all objects associated with addresses which were present in the "To" or "CC" fields of the mail.
  • These are passed to the response via setExcludeMailRecipientPHIDs().
  • This mail is dropped with PhabricatorMetaMTAActor::REASON_RESPONSE:

'This message is a response to another email message, and this '.
'recipient received the original email message, so we are not '.
'sending them this substantially similar message (for example, '.
'the sender used "Reply All" instead of "Reply" in response to '.
'mail from Phabricator).'


To step back here, what are the concrete issues you're running into with one-mail-per-recipient?

eadler added a comment.Thu, Feb 9, 9:43 PM

Oh, I didn't even realize that was a thing. I assumed SMTP went like "okay here is a message: <message body>".

I don't think we have control over the SMTP envelope for at least some of of the APIs we implement, although perhaps I'm mistaken. In some cases I chose the non-SMTP version of things (e.g., POST some JSON) when an SMTP version was available since I mostly know how JSON works but do not mostly know how SMTP works. It's possible we could revisit those decisions and do more SMTP to get greater access to envelopes, although I'd guess not all adapters provide us envelope access.

Some more thoughts: even if we can't control the Reply-to in all cases, phabricator can look at the TO header and perhaps choose not to send mail if it looks like it was already sent to a given user.

We already attempt to do this -- are you seeing it not work in practice? Rough pathway is:

'This message is a response to another email message, and this '.
'recipient received the original email message, so we are not '.
'sending them this substantially similar message (for example, '.
'the sender used "Reply All" instead of "Reply" in response to '.
'mail from Phabricator).'

We don't have public-reply-to turned on because we are worried about reply-to spam. In particular the belief is that turning on public-reply-to will cause duplicate emails. If this is not the case we can likely just change this setting.
The text that is causing us to believe this is:

Sometimes people will "Reply All" and everyone will get two mails, one from the user and one from Phabricator turning their mail into a comment.

To step back here, what are the concrete issues you're running into with one-mail-per-recipient?

some of the problems are concrete and known and thus solvable.

This violates policy controls. The body of the mail is generated without respect for object policies.

As a result of having public reply-to turned off, and metamta.one-mail-per-recipient turned off is that there is no way to reply to the email and phabricator process it at all.

To jump back further, the root-ier cause is that you're using mailing lists?

If we implemented T3980, could you create projects for mailing lists instead of "Mailing List" users, then do one-mail-per-recipient + private-replies and be generally OK?

Sometimes people will "Reply All" and everyone will get two mails, one from the user and one from Phabricator turning their mail into a comment.

I think this is mostly just out of date (I wrote it before we wrote the "exclude recipients" stuff) but you'll still probably at least get some cases like this:

User [Reply-All]: !acpetpt looks great
Phabricator [To User]: Sorry, that's not a valid command.
User [Reply-All]: !accept looks great

(D17331 should fix the outdated discussion of the option in Config.)

To jump back further, the root-ier cause is that you're using mailing lists?

If we implemented T3980, could you create projects for mailing lists instead of "Mailing List" users, then do one-mail-per-recipient + private-replies and be generally OK?

We actually implemented this ourselves and moved back to mailing list users. One issue with one-mail-per-recipient is that gmail does not permit muting an email conversation if you are the only TO recipient. Another issue is that it makes filtering mail to specific mailing lists impossible if you don't have the ability to filter on arbitrary headers. Sadly a lot of this problem is due to the inferiority of the gmail email client.

FWIW I think implementing part of this proposal would at least solve this issue:

Policy controls work correctly and are enforced per-user.

so even if my root cause can't be fixed, this should remain open.

One issue with one-mail-per-recipient is that gmail does not permit muting an email conversation if you are the only TO recipient.

I don't necessarily want to do actually this, but could we hypothetically fix this in a technical sense by adding a dummy recipient to every message ("Dummy-Recipient-So-Gmail-Will-Let-You-Mute-This@gmail-mandated-by-intergalactic-law.com")?

Sadly a lot of this problem is due to the inferiority of the gmail email client.

Perhaps now that we have a new administration, Trump will overturn the galactic law mandating all users use Gmail and the Gmail web client for all email.

If we use SMTP envelope To, doesn't Alice potentially get multiple messages still (envelope "To: Alice", envelope "To: All-About-Alice@lists.company.com")? The list rewrites the envelope to go to all the list members, right? Or is there more magic there I don't know about?

We could also implement "Mute Thread" in Phabricator, but I assume no one would be willing to click twice to mute a thread.

(dropped this since I don't have time to fully test, but I might come back to this in the future)

One issue with one-mail-per-recipient is that gmail does not permit muting an email conversation if you are the only TO recipient.

I don't necessarily want to do actually this, but could we hypothetically fix this in a technical sense by adding a dummy recipient to every message ("Dummy-Recipient-So-Gmail-Will-Let-You-Mute-This@gmail-mandated-by-intergalactic-law.com")?

I think so. I've considered implementing a local patch to try this too. :)

If we use SMTP envelope To, doesn't Alice potentially get multiple messages still (envelope "To: Alice", envelope "To: All-About-Alice@lists.company.com")? The list rewrites the envelope to go to all the list members, right? Or is there more magic there I don't know about?

With appropriate message-ids this shouldn't be a concern but if don't have control over that then it still becomes an issue. I'd need to spend some time with different clients and different messages to fully test :(

I'm pretty sure we can't generate our own Message-ID with Amazon SES, at least -- although we use the HTTP API, not the SMTP API, and maybe the rules differ. Not sure about other providers.

We could also implement "Mute Thread" in Phabricator, but I assume no one would be willing to click twice to mute a thread.

In a monorepo the size of ours, you'd be surprised. Large changes still have to go through review for compliance reasons, even though you guys say things like renames should not be reviewed in the context of a Differential.

This also comes into play with a feature request that we have for "Resign from this review, even though I'm in an owners package, and stop spamming me" but there's more to come on that.

We'll probably implement a Phabricator-side mute-this-thread in connection with T10448, I'm just guessing that some of the users complaining that GMail mute doesn't work won't consider clicking a link in the email body, then clicking "Mute Thread" to be an acceptable solution.