Page MenuHomePhabricator
Create Task
Maniphest T13115

Handle mail bounces inside Phabricator
Open, LowPublic
Actions

Description

Currently, Phabricator does not maintain its own mail address bounce list.

In many cases, this isn't much of a problem because providers do it for us. Notably, Mailgun and Postmark both maintain a bounce list for you.

However, there are still two broad concerns:

  • Some providers, like SES, don't handle bounces.
  • Because we don't have API support for third-party bounces, there's no way to manage the bounce list from within Phabricator or see that your address has been blacklisted. This particularly affects hosted instances in Phacility, who must interact with an actual human via support if they, e.g., create a new hire's Phabricator account before fully setting up their Gmail mailbox.

In particular:

  • See PHI412. This is approximately a request for SES bounce support.
  • (Via Email) Today (April 2, 2018) a Phacility install reported an issue with their mail provider over the weekend which caused more than a dozen addresses to bounce. I resolved this via script, but it would have been more convenient to resolve via a UI in Phabricator.
  • See PHI641. This is a case where user visibility into blocklists would have improved things.

The list would look something like this:

  • Keep track of first-party bounces (users can add/remove blocked addresses explicitly) and third-party bounces (mirroring the Postmark and Mailgun lists via API) and semi-third-party bounces (bounces reported by SES, which have no corresponding object in the remote system).
  • Webhook support for SES, Mailgun, and Postmark, to learn of new bounces.
  • Deleting a third-party bounce in Phabricator should also delete it in the third-party system.
  • The mailer should decline to deliver to addresses on the bounce list.
  • Possibly, some UI hinting like "hey your address bounced".
  • Possibly, some easy way to review (just from /mail/?) bounces.
  • Possibly, some way way to re-send bounces.
  • MailGun has a separate "Complaints" list. This should probably be a single list on our side but maybe differentiate the two.
  • We should use the terms "bouncelist" or "blocklist" or similar, not "blacklist". Although I think ascribing any real racial malice to the terms "blacklist" and "whitelist" is probably a bit of a stretch and there doesn't seem to be any etymological racism in these terms, they do quite reasonably sound kind of racist in a modern context, and it's easy to avoid them without any loss of clarity.

Related Objects

Event Timeline

epriestley triaged this task as Low priority.Apr 2 2018, 10:53 PM
epriestley created this task.
Herald added a subscriber: eadler. · View Herald TranscriptApr 2 2018, 10:53 PM
epriestley updated the task description. (Show Details)May 4 2018, 6:42 PM
joshuaspence added a subscriber: joshuaspence.Jul 10 2018, 10:36 AM
epriestley added a comment.EditedOct 17 2018, 8:24 PM

See PHI926, where a GSuite-backed install has seen multiple "550 No Such User" bounces to the same address even though the address exists. It's currently unclear if this is an issue in Mailgun, GSuite, the series of tubes between the two, or elsewhere.

Currently, Mailgun's bounce tools are choking on our bounce list. The bounce list is not searchable by partial address (e.g., @examplecompany.com) and the "Export" feature spins for several minutes and fails. We picked up a lot of bounced addresses in the invite spam associated with T13150 earlier this year but I believe the total size of the list is "thousands", not "hundreds of millions", so this seems a little silly.

I'm inclined to at least try to build some of the primitives here to let us interact with the existing Mailgun list over the API. If there was a bin/mail copy-bounce-list-into-phabricator command and then we could use the Phabricator web UI to interact with it, that would be a major step forward over interacting with it via Mailgun's actual tool, although the shorter-term goal is just to pull the list out of Mailgun and check for other @examplecompany.com addresses.

epriestley added a comment.Oct 17 2018, 8:47 PM

To pull the list down (this takes a while since we can only grab 100 results at a time and the API takes a while to respond):

<?php

require_once 'scripts/init/init-script.php';

$key = getenv('MAILGUN_KEY');
if (!$key) {
  throw new Exception('Provide key with MAILGUN_KEY.');
}

$domain = 'phacility.com';

$params = array();
$paging_params = array();

$api_uri = "https://api:{$key}@api.mailgun.net/v3/{$domain}/bounces";
$api_uri = new PhutilURI($api_uri);

$addresses = array();
while (true) {
  $uri = id(clone $api_uri)
    ->setQueryParams($params + $paging_params);

  $future = new HTTPSFuture($uri);

  list($body) = $future->resolvex();

  $body = phutil_json_decode($body);

  $paging = $body['paging'];
  $next = $paging['next'];

  foreach ($body['items'] as $item) {
    $addresses[] = $item;
  }

  $paging_params = id(new PhutilURI($next))->getQueryParams();

  // When we reach the end of the result list, the "next" URI is still
  // populated but does not include a "page=next" parameter.
  if (!isset($paging_params['page'])) {
    break;
  }
}
ivo added a subscriber: ivo.Oct 18 2018, 8:44 AM
renatomassaro awarded a token.Dec 24 2018, 8:06 AM
renatomassaro added a subscriber: renatomassaro.
epriestley mentioned this in T13222: 2018 Week 48-51 Bonus Content.Jan 16 2019, 3:51 PM
epriestley added a comment.Jan 17 2019, 3:55 PM

When the time comes, this should almost certainly also handle contact numbers for non-email media (T920 / D19988).

epriestley mentioned this in D20022: Implement SMS MFA.Jan 23 2019, 7:59 PM
epriestley mentioned this in T13375: Treat "'to'" address failures as permanent in Mailgun adapter.Aug 12 2019, 3:14 PM
Phabricator · Built by Phacility