Page MenuHomePhabricator

An attacker gained staff access to Mailgun and was able to read customer API keys
Closed, ResolvedPublic

Description

Mailgun recently disclosed that an attacker gained access to a staff account. The details are a little muddy, but I think this is what happened:

  • An attacker gained staff access to Mailgun.
  • They used it to attack Reddit. (See this reddit thread.)
  • Specifically, they compromised password reset email for /r/btc mods or the /u/tippr Bitcoin Cash bot or some other BTC/BCH-related sort of thing?

Details are pretty light, but I think this doesn't reflect well on Mailgun, since the lack of detail about how the attacker gained access implies internal controls are poor (i.e., the attacker might have just guessed a password). Mailgun's Security Privacy document suggests that they probably just give everyone full access to everything and only require a password ("using an assigned user name and password"), not MFA.

We currently use Mailgun for mail delivery in the Phacility cluster, as we disclose in our Data Use Policy. We also claim:

We will discontinue use of a service provider if we believe their policies are at odds with our own.

I wrote Mailgun this mean email:

Hi,

I have a few questions about Mailgun's security policies as described in this document:

https://www.mailgun.com/security-privacy

The document states:

"Mailgun will restrict the use of administrative access codes for customer accounts to its employees and other agents who need the access codes for the purpose of providing the Services. ... Mailgun personnel who use access codes shall be required to log on using an assigned user name and password."

Can you provide more information about how restrictive the internal access control was prior to late December, and how restrictive it is now? For example, did essentially all technical staff have access to the content of customer email circa December 25th? Do they still? Put another way: was this control meaningful, or did Mailgun implement policy and technical controls in such a way that essentially all technical staff "need" access to customer API keys to provide the service? Has the implementation of this control changed?

Does the second section quoted above mean that Mailgun did not require staff use MFA to access customer data prior to late December? Is Mailgun's policy still that a username and password (without additional controls like MFA) are sufficient for staff to access customer data?

Thanks,
Evan

That said, all the other outbound mail providers we've tried have also had major problems, and Mailgun may still be our least bad option. I don't think this incident is especially damning, the writeup just makes it sound extremely foreseeable.

(I'm slightly sympathetic to Mailgun here because I suspect most customers for these services are delivering enormous volumes of marketing mail, not small volumes of transactional mail, and Mailgun's support workload is probably dominated by customers doing mail marketing with very different expectations from customers delivering transactional/account email.)

Event Timeline

epriestley triaged this task as Normal priority.Jan 5 2018, 8:14 PM
epriestley created this task.

T12677 documents previous general issues with mail providers. Mailgun gets the worst of it there, but just because we've been with them for a while without anything too awful happening.

Among other incidents, SES abruptly terminated service in T12237.

SendGrid doesn't get a mention there but they routinely mangled mail (e.g., T4199).

Ultimately, compromising mail delivery is bad, but isn't quite "everything is on fire", because we support configuring (and requiring) MFA, and getting access to password reset emails doesn't get you access to accounts with MFA. Mailgun can at least deliver mail properly.

Mostly from the HN thread, other possible providers we haven't tried yet include Mandrill, Postmark, and Sparkpost.

Of these, Postmark seems perhaps the most promising and the most focused on transactional email. I think the tension caused by the transactional vs marketing email schism that most providers face may be an underlying force in many of the issues we've run into.

Theoretically, I'd like to try: adding Postmark support, supporting failover in T12677, configuring Postmark as our failover provider, then making them our primary provider if they prove to be able to deliver mail correctly and not terminate our service without notice too often.

Mailgun has yet to respond to me after about three weeks, so I send them a followup.

(They got back to me and we're scheduling a call.)

epriestley claimed this task.

My call with Mailgun was generally reassuring. Based on an uncharitable reading of the January 5th disclosure, my major concern was that they might be starting from a cultural position which was blind to internal actors as threats and everyone just used root / hunter2 written on a sticky note to log in to everything or something like that.

Without going into too much detail, it sounds like there is significant cultural and technical support for staff privilege separation already, and that the issues at hand were largely about balancing policy and practical concerns in the support workflow. (I'll discuss some more details privately.)

I'm satisfied that we aren't violating our commitment to our customers by continuing to use Mailgun as a service provider, although I still plan to pursue the failover changes described in T12677 because they're sound technical changes from an availability/reliability point of view (and our desire to implement them predated this anyway, this just resurfaced the issue). See T13053 for continuing mail-related changes.

I'm satisfied that we aren't violating our commitment to our customers by continuing to use Mailgun as a service provider...

Based on an interaction in June 2020, I no longer believe this is true. See T13669 for context.