Page MenuHomePhabricator

Long list of reviewers breaks to: field when metamta.one-mail-per-recipient set to false
Closed, ResolvedPublic

Description

Versions:

phabricator 92d520cdc2160b95f03b89c10ebf0c9edaf7c547 (Mon, Feb 27)
arcanist 467dc3ae4286df6d42730d472b7b52d3afa33ec5 (Mon, Feb 27)
phutil 6c902414c30dfaad64f24b3bc950857cebbcf8fe (Mon, Feb 27)
libtwitter 65e64977fd93383256ed3a0292aaf9ef1502bb9c (Wed, Feb 15)

Reproduction steps:

  1. Set metamta.one-mail-per-recipient to false
  2. Create a revision and add hundreds of reviewers (enough such that their emails exceeds 989 characters)

Expected:

To: field in revision emails contains a comma separated list of email addresses for all reviewers, with every email correctly formatted

Actual:

To: field in revision emails contains comma separated list of email addresses, but adds a newline every 989 characters. So at around the 989 character mark, the to: field looks something like

..., donald@whitehouse.gov, sba
 nnon@whitehouse.gov, notputin@whitehouse.gov, ...

We can privately send you a raw email example if that helps. This is consistently happening to revisions with a large number of reviewers.

Event Timeline

See also T12046 for why all open source is sketchy voodoo and the eventual pathway forward here is likely replacing it with first-party code.

I guess this technically qualifies as a scalability bug so we'll prioritize resolving it.

What is metamta.mail-adapter set to?

The default -- PhabricatorMailImplementationPHPMailerLiteAdapter

I think the actual issue is that we're not wrapping the header. When I generate a synthetic message locally with a "To:" header with a length of ~50,000 bytes, the raw output is an unbroken line ~50,00 bytes long. I suspect this is RFC-violating and that whatever the mail gets handed to next gets upset by it.

Header construction in PHPMailer is fairly ad-hoc and I think some headers just don't get wrapped.

I'm going to try to fix this very narrowly. I think we're headed toward severing this dependency, but that's a substantial amount of work.

There are also various issues here if a single address is more than 1,000 bytes in length but presumably that's not much of an issue in practice.

I think this is the controlling specification -- 2008 was a simpler time when no sequence of non-whitespace characters could ever be longer than 78 bytes, and a thousand bytes of memory was an untold wealth of riches:

https://tools.ietf.org/html/rfc5322#section-2.2.3

Can you test if this fixes the issue in your environment?

diff --git a/externals/phpmailer/class.phpmailer-lite.php b/externals/phpmailer/class.phpmailer-lite.php
index 92601178a5..adde2ea3cf 100644
--- a/externals/phpmailer/class.phpmailer-lite.php
+++ b/externals/phpmailer/class.phpmailer-lite.php
@@ -662,6 +662,10 @@ class PHPMailerLite {
     $addr_str .= implode(', ', $addresses);
     $addr_str .= $this->LE;
 
+    // NOTE: This is a narrow hack to fix an issue with 1000+ characters of
+    // recipients, described in T12372.
+    $addr_str = wordwrap($addr_str, 75, "\n ");
+
     return $addr_str;
   }

Internally, PHPMailer sends headers through two adjustment phases:

  • SecureHeader(), which makes headers "secure" by silently discarding newlines. I'll set this aside for now.
  • EncodeHeader(), which encodes and, if necessary, wraps headers, some of the time (but not always: see https://github.com/PHPMailer/PHPMailer/issues/36).

Among many others, the narrowest issue here is that PHPMailer calls EncodeHeader() on each address individually for headers containing lists of email addresses, so the wrapping logic never applies to the header (with the full list of addresses) as a whole.

The patch above wraps those headers after they are built, approximately using the folding algorithm described in RFC5322 Section 2.2.3. wordwrap() is not correct exhaustively, but should produce correct-enough output for reasonable inputs here, I believe.

Thanks for the quick fix and the huge amount of detail under the hood, we'll give that patch a shot!

And agreed, an address exceeding 1000 bytes in length shouldn't be much of an issue in practice. I'm guessing there would be other worse symptoms if we had any of those sorts of big addresses.

Any suggestions to test this on staging without mass emailing everyone / patching prod? I am trying to test this on staging with bin/mail show-outbound --id xxxx (staging has mail disabled), but the recipients / To: list look fine with or without the patch.

Hrrm -- I can adjust bin/mail send-test to accept raw email addresses, and then you can use it to send a message to test+00001@gmail.com, test+00002@gmail.com and similar. Shouldn't be too involved.

After D17489 you should be able to do this:

echo "mail body" | ./bin/mail send-test --subject hi --to test+0001@gmail.com --to test+0002@gmail.com ...

...until you've added more than 1,000 characters of fake recipients, then maybe put a real address at the end and see if it arrives there correctly if you don't have a better way to tell.

You still probably have to patch production if staging isn't hooked up to an MTA (and can't easily be connected to one), but at least you don't have to mail a ton of people.

Confirming that the patch above works (tested using send-test --to)!

Before patch:

To: lvital@twitter.com, fakeemailtest+0@twitter.com, fakeemailtest+1@twitter.com, fakeemailtest+2@twitter.com, fakeemailtest+3@twitter.com, fakeemailtest+4@twitter.com, fakeemailtest+5@twitter.com, fakeemailtest+6@twitter.com, fakeemailtest+7@twitter.com, fakeemailtest+8@twitter.com, fakeemailtest+9@twitter.com, fakeemailtest+10@twitter.com, fakeemailtest+11@twitter.com, fakeemailtest+12@twitter.com, fakeemailtest+13@twitter.com, fakeemailtest+14@twitter.com, fakeemailtest+15@twitter.com, fakeemailtest+16@twitter.com, fakeemailtest+17@twitter.com, fakeemailtest+18@twitter.com, fakeemailtest+19@twitter.com, fakeemailtest+20@twitter.com, fakeemailtest+21@twitter.com, fakeemailtest+22@twitter.com, fakeemailtest+23@twitter.com, fakeemailtest+24@twitter.com, fakeemailtest+25@twitter.com, fakeemailtest+26@twitter.com, fakeemailtest+27@twitter.com, fakeemailtest+28@twitter.com, fakeemailtest+29@twitter.com, fakeemailtest+30@twitter.com, fakeemailtest+31@twitter.com, ".com" <fakeemailtest+32@twitter>, fakeemailtest+33@twitter.com, fakeemailtest+34@twitter.com, fakeemailtest+35@twitter.com, fakeemailtest+36@twitter.com, fakeemailtest+37@twitter.com, fakeemailtest+38@twitter.com, fakeemailtest+39@twitter.com, fakeemailtest+40@twitter.com, fakeemailtest+41@twitter.com, fakeemailtest+42@twitter.com, fakeemailtest+43@twitter.com, fakeemailtest+44@twitter.com, fakeemailtest+45@twitter.com, fakeemailtest+46@twitter.com, fakeemailtest+47@twitter.com, fakeemailtest+48@twitter.com, fakeemailtest+49@twitter.com, fakeemailtest+50@twitter.com, fakeemailtest+51@twitter.com, fakeemailtest+52@twitter.com, fakeemailtest+53@twitter.com, fakeemailtest+54@twitter.com, fakeemailtest+55@twitter.com, fakeemailtest+56@twitter.com, fakeemailtest+57@twitter.com, fakeemailtest+58@twitter.com, fakeemailtest+59@twitter.com, fakeemailtest+60@twitter.com, fakeemailtest+61@twitter.com, fakeemailtest+62@twitter.com, fakeemailtest+63@twitter.com, fakeemailtest+64@twitter.com, fakeemailtest+65@twitter.com, f <akeemailtest+66@twitter.com>, fakeemailtest+67@twitter.com, fakeemailtest+68@twitter.com, fakeemailtest+69@twitter.com, fakeemailtest+70@twitter.com, fakeemailtest+71@twitter.com, fakeemailtest+72@twitter.com, fakeemailtest+73@twitter.com, fakeemailtest+74@twitter.com, fakeemailtest+75@twitter.com, fakeemailtest+76@twitter.com, fakeemailtest+77@twitter.com, fakeemailtest+78@twitter.com, fakeemailtest+79@twitter.com, fakeemailtest+80@twitter.com, fakeemailtest+81@twitter.com, fakeemailtest+82@twitter.com, fakeemailtest+83@twitter.com, fakeemailtest+84@twitter.com, fakeemailtest+85@twitter.com, fakeemailtest+86@twitter.com, fakeemailtest+87@twitter.com, fakeemailtest+88@twitter.com, fakeemailtest+89@twitter.com, fakeemailtest+90@twitter.com, fakeemailtest+91@twitter.com, fakeemailtest+92@twitter.com, fakeemailtest+93@twitter.com, fakeemailtest+94@twitter.com, fakeemailtest+95@twitter.com, fakeemailtest+96@twitter.com, fakeemailtest+97@twitter.com, fakeemailtest+98@twitter.com, fakeemai <ltest+99@twitter.com>, fakeemailtest+100@twitter.com, fakeemailtest+101@twitter.com, fakeemailtest+102@twitter.com, fakeemailtest+103@twitter.com, fakeemailtest+104@twitter.com, fakeemailtest+105@twitter.com, fakeemailtest+106@twitter.com, fakeemailtest+107@twitter.com, fakeemailtest+108@twitter.com, fakeemailtest+109@twitter.com, fakeemailtest+110@twitter.com, fakeemailtest+111@twitter.com, fakeemailtest+112@twitter.com, fakeemailtest+113@twitter.com, fakeemailtest+114@twitter.com, fakeemailtest+115@twitter.com, fakeemailtest+116@twitter.com, fakeemailtest+117@twitter.com, fakeemailtest+118@twitter.com, fakeemailtest+119@twitter.com, fakeemailtest+120@twitter.com, fakeemailtest+121@twitter.com, fakeemailtest+122@twitter.com, fakeemailtest+123@twitter.com, fakeemailtest+124@twitter.com, fakeemailtest+125@twitter.com, fakeemailtest+126@twitter.com, fakeemailtest+127@twitter.com, fakeemailtest+128@twitter.com, fakeemailtest+129@twitter.com, fakeemailtest+130@twitter.com, "fakeemailtest+" <131@twitter.com>, fakeemailtest+132@twitter.com, fakeemailtest+133@twitter.com, fakeemailtest+134@twitter.com, fakeemailtest+135@twitter.com, fakeemailtest+136@twitter.com, fakeemailtest+137@twitter.com, fakeemailtest+138@twitter.com, fakeemailtest+139@twitter.com, fakeemailtest+140@twitter.com, fakeemailtest+141@twitter.com, fakeemailtest+142@twitter.com, fakeemailtest+143@twitter.com, fakeemailtest+144@twitter.com, fakeemailtest+145@twitter.com, fakeemailtest+146@twitter.com, fakeemailtest+147@twitter.com, fakeemailtest+148@twitter.com, fakeemailtest+149@twitter.com, fakeemailtest+150@twitter.com, fakeemailtest+151@twitter.com, fakeemailtest+152@twitter.com, fakeemailtest+153@twitter.com, fakeemailtest+154@twitter.com, fakeemailtest+155@twitter.com, fakeemailtest+156@twitter.com, fakeemailtest+157@twitter.com, fakeemailtest+158@twitter.com, fakeemailtest+159@twitter.com, fakeemailtest+160@twitter.com, fakeemailtest+161@twitter.com, fakeemailtest+162@twitter.com, "witter.com" <fakeemailtest+163@t>, fakeemailtest+164@twitter.com, fakeemailtest+165@twitter.com, fakeemailtest+166@twitter.com, fakeemailtest+167@twitter.com, fakeemailtest+168@twitter.com, fakeemailtest+169@twitter.com, fakeemailtest+170@twitter.com, fakeemailtest+171@twitter.com, fakeemailtest+172@twitter.com, fakeemailtest+173@twitter.com, fakeemailtest+174@twitter.com, fakeemailtest+175@twitter.com, fakeemailtest+176@twitter.com, fakeemailtest+177@twitter.com, fakeemailtest+178@twitter.com, fakeemailtest+179@twitter.com, fakeemailtest+180@twitter.com, fakeemailtest+181@twitter.com, fakeemailtest+182@twitter.com, fakeemailtest+183@twitter.com, fakeemailtest+184@twitter.com, fakeemailtest+185@twitter.com, fakeemailtest+186@twitter.com, fakeemailtest+187@twitter.com, fakeemailtest+188@twitter.com, fakeemailtest+189@twitter.com, fakeemailtest+190@twitter.com, fakeemailtest+191@twitter.com, fakeemailtest+192@twitter.com, fakeemailtest+193@twitter.com, fakeemailtest+194@twitter.com, "r.com" <fakeemailtest+195@twitte>, fakeemailtest+196@twitter.com, fakeemailtest+197@twitter.com, fakeemailtest+198@twitter.com, fakeemailtest+199@twitter.com

After patch:

To: lvital@twitter.com, fakeemailtest+0@twitter.com, fakeemailtest+1@twitter.com, fakeemailtest+2@twitter.com, fakeemailtest+3@twitter.com, fakeemailtest+4@twitter.com, fakeemailtest+5@twitter.com, fakeemailtest+6@twitter.com, fakeemailtest+7@twitter.com, fakeemailtest+8@twitter.com, fakeemailtest+9@twitter.com, fakeemailtest+10@twitter.com, fakeemailtest+11@twitter.com, fakeemailtest+12@twitter.com, fakeemailtest+13@twitter.com, fakeemailtest+14@twitter.com, fakeemailtest+15@twitter.com, fakeemailtest+16@twitter.com, fakeemailtest+17@twitter.com, fakeemailtest+18@twitter.com, fakeemailtest+19@twitter.com, fakeemailtest+20@twitter.com, fakeemailtest+21@twitter.com, fakeemailtest+22@twitter.com, fakeemailtest+23@twitter.com, fakeemailtest+24@twitter.com, fakeemailtest+25@twitter.com, fakeemailtest+26@twitter.com, fakeemailtest+27@twitter.com, fakeemailtest+28@twitter.com, fakeemailtest+29@twitter.com, fakeemailtest+30@twitter.com, fakeemailtest+31@twitter.com, fakeemailtest+32@twitter.com, fakeemailtest+33@twitter.com, fakeemailtest+34@twitter.com, fakeemailtest+35@twitter.com, fakeemailtest+36@twitter.com, fakeemailtest+37@twitter.com, fakeemailtest+38@twitter.com, fakeemailtest+39@twitter.com, fakeemailtest+40@twitter.com, fakeemailtest+41@twitter.com, fakeemailtest+42@twitter.com, fakeemailtest+43@twitter.com, fakeemailtest+44@twitter.com, fakeemailtest+45@twitter.com, fakeemailtest+46@twitter.com, fakeemailtest+47@twitter.com, fakeemailtest+48@twitter.com, fakeemailtest+49@twitter.com, fakeemailtest+50@twitter.com, fakeemailtest+51@twitter.com, fakeemailtest+52@twitter.com, fakeemailtest+53@twitter.com, fakeemailtest+54@twitter.com, fakeemailtest+55@twitter.com, fakeemailtest+56@twitter.com, fakeemailtest+57@twitter.com, fakeemailtest+58@twitter.com, fakeemailtest+59@twitter.com, fakeemailtest+60@twitter.com, fakeemailtest+61@twitter.com, fakeemailtest+62@twitter.com, fakeemailtest+63@twitter.com, fakeemailtest+64@twitter.com, fakeemailtest+65@twitter.com, fakeemailtest+66@twitter.com, fakeemailtest+67@twitter.com, fakeemailtest+68@twitter.com, fakeemailtest+69@twitter.com, fakeemailtest+70@twitter.com, fakeemailtest+71@twitter.com, fakeemailtest+72@twitter.com, fakeemailtest+73@twitter.com, fakeemailtest+74@twitter.com, fakeemailtest+75@twitter.com, fakeemailtest+76@twitter.com, fakeemailtest+77@twitter.com, fakeemailtest+78@twitter.com, fakeemailtest+79@twitter.com, fakeemailtest+80@twitter.com, fakeemailtest+81@twitter.com, fakeemailtest+82@twitter.com, fakeemailtest+83@twitter.com, fakeemailtest+84@twitter.com, fakeemailtest+85@twitter.com, fakeemailtest+86@twitter.com, fakeemailtest+87@twitter.com, fakeemailtest+88@twitter.com, fakeemailtest+89@twitter.com, fakeemailtest+90@twitter.com, fakeemailtest+91@twitter.com, fakeemailtest+92@twitter.com, fakeemailtest+93@twitter.com, fakeemailtest+94@twitter.com, fakeemailtest+95@twitter.com, fakeemailtest+96@twitter.com, fakeemailtest+97@twitter.com, fakeemailtest+98@twitter.com, fakeemailtest+99@twitter.com, fakeemailtest+100@twitter.com, fakeemailtest+101@twitter.com, fakeemailtest+102@twitter.com, fakeemailtest+103@twitter.com, fakeemailtest+104@twitter.com, fakeemailtest+105@twitter.com, fakeemailtest+106@twitter.com, fakeemailtest+107@twitter.com, fakeemailtest+108@twitter.com, fakeemailtest+109@twitter.com, fakeemailtest+110@twitter.com, fakeemailtest+111@twitter.com, fakeemailtest+112@twitter.com, fakeemailtest+113@twitter.com, fakeemailtest+114@twitter.com, fakeemailtest+115@twitter.com, fakeemailtest+116@twitter.com, fakeemailtest+117@twitter.com, fakeemailtest+118@twitter.com, fakeemailtest+119@twitter.com, fakeemailtest+120@twitter.com, fakeemailtest+121@twitter.com, fakeemailtest+122@twitter.com, fakeemailtest+123@twitter.com, fakeemailtest+124@twitter.com, fakeemailtest+125@twitter.com, fakeemailtest+126@twitter.com, fakeemailtest+127@twitter.com, fakeemailtest+128@twitter.com, fakeemailtest+129@twitter.com, fakeemailtest+130@twitter.com, fakeemailtest+131@twitter.com, fakeemailtest+132@twitter.com, fakeemailtest+133@twitter.com, fakeemailtest+134@twitter.com, fakeemailtest+135@twitter.com, fakeemailtest+136@twitter.com, fakeemailtest+137@twitter.com, fakeemailtest+138@twitter.com, fakeemailtest+139@twitter.com, fakeemailtest+140@twitter.com, fakeemailtest+141@twitter.com, fakeemailtest+142@twitter.com, fakeemailtest+143@twitter.com, fakeemailtest+144@twitter.com, fakeemailtest+145@twitter.com, fakeemailtest+146@twitter.com, fakeemailtest+147@twitter.com, fakeemailtest+148@twitter.com, fakeemailtest+149@twitter.com, fakeemailtest+150@twitter.com, fakeemailtest+151@twitter.com, fakeemailtest+152@twitter.com, fakeemailtest+153@twitter.com, fakeemailtest+154@twitter.com, fakeemailtest+155@twitter.com, fakeemailtest+156@twitter.com, fakeemailtest+157@twitter.com, fakeemailtest+158@twitter.com, fakeemailtest+159@twitter.com, fakeemailtest+160@twitter.com, fakeemailtest+161@twitter.com, fakeemailtest+162@twitter.com, fakeemailtest+163@twitter.com, fakeemailtest+164@twitter.com, fakeemailtest+165@twitter.com, fakeemailtest+166@twitter.com, fakeemailtest+167@twitter.com, fakeemailtest+168@twitter.com, fakeemailtest+169@twitter.com, fakeemailtest+170@twitter.com, fakeemailtest+171@twitter.com, fakeemailtest+172@twitter.com, fakeemailtest+173@twitter.com, fakeemailtest+174@twitter.com, fakeemailtest+175@twitter.com, fakeemailtest+176@twitter.com, fakeemailtest+177@twitter.com, fakeemailtest+178@twitter.com, fakeemailtest+179@twitter.com, fakeemailtest+180@twitter.com, fakeemailtest+181@twitter.com, fakeemailtest+182@twitter.com, fakeemailtest+183@twitter.com, fakeemailtest+184@twitter.com, fakeemailtest+185@twitter.com, fakeemailtest+186@twitter.com, fakeemailtest+187@twitter.com, fakeemailtest+188@twitter.com, fakeemailtest+189@twitter.com, fakeemailtest+190@twitter.com, fakeemailtest+191@twitter.com, fakeemailtest+192@twitter.com, fakeemailtest+193@twitter.com, fakeemailtest+194@twitter.com, fakeemailtest+195@twitter.com, fakeemailtest+196@twitter.com, fakeemailtest+197@twitter.com, fakeemailtest+198@twitter.com, fakeemailtest+200@twitter.com

Great, thanks for testing it! We'll upstream that and I'll file something to put a longer-term fix in place.

"witter.com" <fakeemailtest+163@t>

ah yes this is my very favorite email address of all

Great, thanks for testing it! We'll upstream that and I'll file something to put a longer-term fix in place.

"witter.com" <fakeemailtest+163@t>

ah yes this is my very favorite email address of all

ohyou