Currently, secure uses Postmark as a primary route with Mailgun as a backup route (and has for quite a long time), while Phacility production uses Mailgun. I'd like to switch everything to Postmark exclusively.

As part of this, I am going to update Phabricator's documentation to discourage use of Mailgun. This task explains my reasoning.

I've had a couple of slightly rocky experiences with Mailgun. Two older ones with a bit of supporting documentation are:

• (D17831, May 2017) Mailgun made an unannounced breaking API change in the middle of the day, but this was kind of our fault so it's not really a mark against them. Customer support had this to say: "Currently there isnt a public notice for API changes. We apologize for any inconvenience."
• (T13037, January 2018) An attacker gained access to a Mailgun staff account. I was reassured on a call with Josh Odom (Mailgun's CTO), that this did not reflect a failure of internal culture.

This was nowhere near as bad as my experience with SES so I remained fairly comfortable with Mailgun, but I recevied a bizarre unprompted sales-outreach interaction in June 2020 that soured me on Mailgun. Here's a lightly edited version of the exchange, with identifying information redacted and emphasis added:

From: Joe Mailgun Employee
To: Evan Priestley
Subject: Let's take a look

Hi Evan,

I noticed Phacility is consistently sending [a specific large number of] emails or more over the last few months. I’d like to hear about your sending strategy and see how Mailgun can help you see continued success on our platform.

How does Wednesday sound?

Best,
Joe

From: Evan Priestley
To: Joe Mailgun Employee
Subject: Re: Let's take a look

Sorry, I don't take meetings with no agenda.

If you have specific concerns or items in mind, please describe exactly what you'd like to discuss. I'm happy to set up a meeting if we can't settle things via email.

This request is so vague and nonspecific ("Let's take a look"?) that it feels kind of like a low-effort template sales email. If it is, and your primary goal in sending me this email is to get me to purchase more Mailgun services, please do not send me any more emails like this.

Thanks,
Evan

From: Joe Mailgun Employee
To: Evan Priestley
Subject: Let's take a look

Evan,

First, you're right, it is a low effort sales template. It does work.

There wasn't an agenda included, but, the purpose of the proposed call is a real one (albeit not (sic)

Is there a strategy with the transactional emails being sent? There's a delivery rate of only 80%. There are over 30K suppressions still being sent emails.

Is your team familiar with how to remove and ensure the sending volume is clean?

Why doesn't your team use a dedicated IP? At this volume [specific volume number] it would make sense.

I think it would make sense to talk about these things.

How about yourself?

My schedule: [Link to book a meeting]

Joe

From: Evan Priestley
To: Joe Mailgun Employee
Subject: Re: Let's take a look

Is there a strategy with the transactional emails being sent?

Why would there be a strategy with transactional email?

There's a delivery rate of only 80%.

The web console shows a delivery rate of almost 96%. Is the web console wrong? See attached screenshot ("delivery.png").

There are over 30K suppressions still being sent emails. Is your team familiar with how to remove and ensure the sending volume is clean?

See https://secure.phabricator.com/T13115. My understanding is that there is no reason to prioritize this since Mailgun is already managing a suppression list.

Why doesn't your team use a dedicated IP? At this volume [specific volume number] it would make sense.

The web console shows a dedicated IP (see screenshot "dedicated.png").

Mailgun Support confirmed provisioning of a dedicated IP in ticket #299395 on July 4, 2016. The support agent was [specific Mailgun support agent name].

I have been billed for a dedicated IP every month for four years. See attached screenshot of a June 1, 2020 invoice ("invoice.jpg") billing me for a dedicated IP.

Are the web interface, support history, and invoice incorrect? Has Mailgun charged me for a dedicated IP for 4 years without actually giving me a dedicated IP?

(Mail headers show the dedicated IP is functioning correctly, see "headers.png", where the outbound route matches the dedicated IP in the web interface.)

Thanks,
Evan

[Various screenshots substantiating my claims]

From: Joe Mailgun Employee
To: Evan Priestley
Subject: Let's take a look

Evan,

Here's where I'm seeing that delivery rate:

[Screenshot of some other interface showing an 80% delivery rate]

What are the date parameters that you're using?

After the IP request, it doesn't look like it was ever used or followed up on. From that time on, it definitely appears you've been billed since.

Yes, transactional emails can have a strategy. Although it does depend on how critical it is for your users to receive them. Think about password resets, account confirmations, route updates, order updates etc...

Suppression management is always going to stop emails from being delivered. HOwever, if they are not taken out of rotation of the overall sending it'll go to the overall email volume, because they're accepted just not delivered.
What do the suppressions tell you? The recipient domain, [specific customer domain] shows a lot of suppressions. Any ideas why?

Joe

I did not reply.

Based on this exchange, I am concerned that:

• Despite the incident in 2018, Mailgun appears to be giving an excessive level of access to customer data to employees who do not need it in 2020: Joe had access to specific customer domain information, and didn't hesitate to use it purely to try to sell me something. From this, I infer that it is likely routine that sales staff examine customer data without any kind of control or approval. I don't think this is acceptable.
• Mailgun's hiring or training process for sales employees doesn't seem to be very good: Joe didn't seem to understand the Mailgun system. Beyond not needing it, I also don't think it is acceptable for staff who can not consistently demonstrate a high level of competence to have access to customer data.
• I found this whole interaction quite disrespectful, and no longer believe I can trust that the assurances I received from the CTO hold any weight if this sort of interaction is acceptable to Mailgun: if Mailgun is training Joe to waste my time with this deceptive sales nonsense, why should I believe the CTO is above being deceptive when running damage control on a security incident?

In a perfect world, I would have immediately moved away from Mailgun in response to this interaction (that is, 18 months ago in June 2020). In T13037, I said:

I'm satisfied that we aren't violating our commitment to our customers by continuing to use Mailgun as a service provider...

This interaction was so negative to me that it I no longer believe this is true. However, I've had to do a lot of picking my battles over the last couple years and am only getting to fighting this one now.

To counterbalance this with the barest hint of self-awareness, everyone in the non-technical world seems to use this general "let's schedule a mysterious meeting with no agenda, since I'm absolutely sure your time has no value" template -- from shady scammers using addresses like legit.github.customer.list327@gmail.com all the way up to top-tier venture capital (I got one from Andreessen Horowitz back in 2017). Why is this considered acceptable? In what world does some guy I've never heard of from AWS cold-calling me on the account's technical contact number while I'm grocery shopping lead to a sale?