Although I'm somewhat confused by this because it came to the address for my personal AWS account.

SES is definitely rejecting mail, though:

[2017-02-09 10:14:09] EXCEPTION: (PhutilProxyException) Error while executing Task ID 2540698. {>} (SimpleEmailServiceException) Unexpected HTTP status while making request to Amazon SES: expected 200, got 400. at [<phabricator>/externals/amazon-ses/ses.php:509]

@chad, just to make sure my ducks are in a row, you aren't using noreply@phabricator.com in association with any marketing mail, right? I don't want to claim "we only send transactional mail" if that isn't true.

I appealed this, but the FAQ says "Due to the way spamtrap sending is reported, it will take a minimum of three weeks before we can confirm that a fix you have put in place has succeeded.".

Of course, the FAQ also says "we will put you on probation and ask you to fix the underlying problem" except in severe cases, and we were not put on probation, so who knows.

I'll see about moving this install to MailGun, I guess, which also abruptly terminated our service without notice, but did so less recently than SES.

Before doing so, I'm going to consider some changes to how we deal with unverified addresses. Currently, for continuity, we will send mail to unverified addresses if they interact with objects or are @mentioned, so you can invite a co-worker and immediately assign them tasks or mention them without mail getting dropped. When auth.require-email-verification is on, I think we should probably change this behavior because it's a tiny nicety for users while dealing with this suspension is an enormous headache for me, personally. Other users will see grey dots next to @mentions anyway and we can make some other minor adjustments to make this more clear.

We also reply to inbound mail in a helpful way, no matter who sent it. I am going to consider a change like: when auth.require-email-verification is on, do not reply to unconfirmed addresses. This makes some scenarios (like replying to mail from the wrong address) potentially much more confusing for users (email gets dropped instead of a helpful reply) but the majority of this automated rejection mail we send is replies to actual spammers trying to sell bugs@phabricator.com cheap prescription drugs. Possibly, I'll just make this the rule in general, regardless of configuration.

In T12237#211025, @epriestley wrote:

Although I'm somewhat confused by this because it came to the address for my personal AWS account.

This is actually expected, since I set this up some time in ~2011 before we incorporated, and it got left behind when secure moved to the corporate account.

The changelog from Mailchimp (opt-in only) is from noreply@phacility.com, but I can't think of anything else off the top of my head

My guess is that this is just something dumb and mostly out of our control, and we fell through the cracks because the email volume we send from this install is tiny. We sent 41,877 messages in January at a cost of $0.00 -- this is too few messages to even qualify for charges, which start once you send 2,000 messages in a single day.

We also sent 0.000003 GB of attachments, which also cost us $0.00.

In theory, at these volumes, I think it should be conceptually fine for us to make a few decisions (like auto-responding to inbound mail, and sending you mail if you get CC'd on a task before you actually verify the address) that are better for users even if they might very occasionally send an email somewhere questionable, which is why I've erred on the side of making things more user-friendly in some of these cases.

But Amazon probably just has a rule like "if an install isn't even paying us, just disable them without probation if we get any complaints at all". The key is also attached to my personal account (<$100/month total) not our corporate account ($money fire/month).

Of course, the cost to us is enormous and I'd gladly pay a bunch of money to not have to deal with this stuff since I'm about to waste who-knows-how-long on this, but I think almost everyone buying email is using it for huge marketing volumes rather than tiny transactional volumes. My earlier experience elsewhere was that there's no "pay a reasonable amount of money per message and it actually gets delivered and you don't have to worry about it" sort of plan anywhere: every provider seems to be competing on price-per-message for installs sending large volumes of marketing stuff.

If we weren't below the charge threshold we'd be paying like $4 for those messages. We'd be happy to pay something like 25x-50x that rate to not have our service terminated abruptly, but maybe there just isn't a position in the marketplace for that sort of provider.

My plan here is:

• drop non-administrative (confirm email, password reset, invite) mail to unconfirmed addresses;
• drop inbound replies to unconfirmed addresses;
• move this install to MailGun;
• (Future?) Add some kind of notification to user accounts like "some email you would have been sent was dropped because you haven't verified your address yet"?
• (Future?) Maybe try to add messaging around inbound mail we drop? I guess?

Possibly, we could allow multiple mail providers to be configured and have MetaMTA transmit across them (at randomly? in master/fallback mode?) and fall back when a provider failed. Then we could have both MailGun and SES configured and survive abrupt termination of service without notice without a service interruption by using the other provider for the three weeks it took to review our tiny volume of purely transactional mail and determine that we are not actually dangerous spammers.

Add some kind of notification to user accounts like "some email you would have been sent was dropped because you haven't verified your address yet"?

This probably has to happen soon (before release cut) because a lot of users (on installs without auth.require-email-verification, which is off by default) who never bothered to click the "yeah that's me" link are going to have mail go missing otherwise. We don't currently have a great channel to surface it on. There's some tentative discussion elsewhere of "sticky notifications" in the notification menu but I'm not sure if that's the best approach, and it's kind of a big thing to build for this fairly small notification.

A sub-issue here is that we're bouncing a lot of email like this, for an exciting product called "Penisole" from reputable domain "bloggay.com":

Your email to Phabricator was not processed, because an error occurred while
trying to handle it:

This email was sent from "Lynn Blackman <xmddmh@duncan.mazdaleads.com>", but
that address is not recognized by Phabricator and does not correspond to any
known user account.

Phabricator is also not configured to allow unknown external users to send
mail to the system using just an email address.

To interact with Phabricator, add this address ("Lynn Blackman
<xmddmh@duncan.mazdaleads.com>") to your account.

Phabricator is misconfigured, the application email 'bugs@phabricator.com' is
set to user '' but that user does not exist.

-- Original Message Body -----------------------------------------------------

If you can't read this email, please view it online http://6url.ru/kavS 

Most Popular Products and Special Deals 
Limited Time Offer 
Hola!   The leading online store  is pleased to offer you  our product list  with delivery service in the United States, Europe and Canada. You can buy anti-acidity, anti-allergy/asthma, antiviral, antifungals, antibiotics, anti-depressant, diabetes medication, herpes medication, antifungals, blood pressure and other various products.  We’ve got discounts at our shop when purchasing Check it Now! (http://6url.ru/jYxo) 
Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle. 
https://aws.amazon.com/support. 

If you no longer wish to receive these emails, simply click on the following link Unsubscribe. (https://aws.amazon.com/support/) 
© 2016 Amazon. All Rights Reserved.

-- Original Message Headers --------------------------------------------------

to: bugs@phabricator.com
from: Lynn Blackman <xmddmh@duncan.mazdaleads.com>
subject: Penisole will never let you down. Best price here!
x-mailgun-incoming: Yes
x-envelope-from: <xmddmh@duncan.mazdaleads.com>
received: (from doltish@localhost)	by nonsensic6.bloggay.com (2.22.0y2/3.54.0/metallic) id y2IUePG3807944;	Thu, 09 Feb 2017 19:18:45 +0300 (CST)
x-message-info: KLNKnJV40qSyjjhJIFIMziBlhBVAed85
date: Thu, 09 Feb 2017 22:23:45 +0600 (CST)
message-id: <183625961706.e3WWkOF99470306@daffodil5.bloggay.com>
mime-version: 1.0
content-type: text/html
content-transfer-encoding: 7bit

This is the notable part:

Phabricator is misconfigured, the application email 'bugs@phabricator.com' is set to user '' but that user does not exist.

That error message is incorrect: the address is properly configured without a default "From" user, with the intent being to reject this mail. This is an easy fix.

• We no longer send normal mail to unverified addresses. Consequently, mail sent between creating an account and that user verifying their address (for example, assigning them tasks) is now dropped.
• We no longer send "sorry, we ignored your mail because we don't recognize the address" replies if we don't recognize the sender, unless phabricator.allow-email-users is set.
  • I would like to eventually (post-Nuance?) remove this option, stop allowing "email users", and move away from "grey users".
• We no longer send a misleading error about "no default user" if an application address is not configured with a default user.
• This install now delivers outbound mail through Mailgun instead of SES.
• The queued backlog appears to have flushed.

Before we cut the release, we need to deal with this:

• If you have auth.require-email-verification off, some of your users may never have verified their email addresses (there was no need to, and no prompt if they missed the verification email for whatever reason). This appears common on this install (e.g., many users in General Chat have grey dots after we turned auth.require-email-verification on). They will no longer receive mail, and will not have any kind of clue in the UI about it.
  • To remedy this, I'd like to add some kind of "We recently discarded mail to you because you haven't verified your primary address." hint for these users when they log in. I'm not yet sure where to put this.
• The grey "unverified user" dot should probably now show even if auth.require-email-verification is off, and mean "this user isn't going to get any email". See also vaguely related T12157.

AWS got back to me with essentially a form response that didn't address any of the particulars of our case.

I suspect think the, uh, "results-oriented" way to navigate this exchange is to just say "yeah we definitely do all that stuff now 100% thanks for bringing this to our attention" and ignore the fact that we have a small collection of obviously-reasonable but technically-not-compliant practices like sending password reset email to unconfirmed addresses if they fill out a CAPTCHA, sending mail to unconfirmed addresses if they are explicitly invited by an administrator, sending mail to unconfirmed addresses if a user signs a Legalpad document on behalf of that address, etc. I'm not thrilled about this, but I have composed this brilliant response by drawing on my tremendous business acumen:

Okay, we fixed the problem we have been discussing. Thanks for bringing this to our attention.

@bcooksley offered some advice elsewhere and there's some downstream motivation from WMF (https://phabricator.wikimedia.org/T157740#3014944) that support developing a way to explicitly "unconfirm" addresses in an administrative way. I plan to build this and tie it into the same UI hints, so affected accounts will see "Some email hasn't been delivered recently because your address is unverified." or "Some email hasn't been delivered recently because your address was disabled by an administrator." and can re-verify to remedy the issue. Hopefully I'll get through this today, although I'm still not totally sure what I'm going to do with the UI.

I imagine that the rest of the process will work like this:

• SES will say "okay, we'll remove the blockade in three weeks if we don't receive more reports" without asking for any details.
• In three weeks, they'll remove the blockade without raising a concern that we haven't sent any mail through SES during that time.
• ¯\_(ツ)_/¯

Perhaps I'm not giving them enough credit, but I think this is more or less how support for a $0.00/month customer has to work in order to scale.

Those seem to have been the magic words:

You recently received a notification informing you that your Amazon SES sending in AWS Region US East (N. Virginia) is no longer suspended.

I think I should just be ethically comfortable with navigating this process like this and giving a procedurally correct answer rather than a factually/technically complete one, but I'm not entirely onboard with it even though there were zero stakes. But perhaps "sneaky business deceit" is a soft skill I should work on developing.

Earlier, I predicted:

In three weeks, they'll remove the blockade without raising a concern that we haven't sent any mail through SES during that time.

But they actually did it in about two and a half:

Congratulations! You addressed the spamtrap sending that put your Amazon SES sending in AWS Region US East (N. Virginia) on probation, and we are very pleased to inform you that, as of this time, your sending is no longer on probation in AWS Region US East (N. Virginia) for sending to spamtraps. You can continue sending normally with Amazon SES.