Starting with repository update:

My impression is that there is a lot of room for optimization in the RepositoryPullLocalDaemon. I've wanted to tackle that for a while because it causes a fair delay between when a commit goes into git and when it actually shows up in Diffusion.

No time like the present.

@epriestley The import has finished, but some commits show as importing still?

I believe I have the first part of this (restructuring the code into a more sensible Overseer > Pool > Daemon sort of thing) working, but it could use more testing. I'm going to see if we have anything else in Daemons that I can fix while I'm here to help me kick the tires a bit.

Some possible approaches here:

There's also a bunch of the usual free-tier shenanigans on this host -- free instances with 1 user importing multiple large repositories -- but that's to be expected.

This is currently the scalability and stability bottleneck for deployments, but should be relatively easy to fix.

(For anyone following along, some additional discussion of clustershell in T2794.)

@epriestley: Have you seen clustershell? It's a massively scaleable management shell with some pretty interesting features. Wikimedia operations are adopting it in place of a lot of dsh and saltstack stuff. From what I have seen it is a really solid framework to build upon.

We're currently at ~82% cluster fullness which makes me slightly uncomfortable, but I'm not going to add more hardware quite yet since I think I can probably get "scale to 0" into production next week. If it's more involved than I think I'll likely put shards 017-020 into production in the next week or so.

Can't the same tricks used to make a single web server handle multiple instances be used for the daemons?

Possible followups from D17344:

Those seem to have been the magic words:

The db001 issue is still an issue, but the shard has managed to stay just under 80% for the last couple of weeks so we're not in any immediate peril. As followups, I think T12217 and T12218 better capture the broader issues here.

I scratched at the surface of T2794, but @chad also bought me a larger monitor so I can put more SSH terminals on it which should buy us another few weeks.

I imagine that the rest of the process will work like this:

AWS got back to me with essentially a form response that didn't address any of the particulars of our case.

• We no longer send normal mail to unverified addresses. Consequently, mail sent between creating an account and that user verifying their address (for example, assigning them tasks) is now dropped.
• We no longer send "sorry, we ignored your mail because we don't recognize the address" replies if we don't recognize the sender, unless phabricator.allow-email-users is set.
  • I would like to eventually (post-Nuance?) remove this option, stop allowing "email users", and move away from "grey users".
• We no longer send a misleading error about "no default user" if an application address is not configured with a default user.
• This install now delivers outbound mail through Mailgun instead of SES.
• The queued backlog appears to have flushed.

A sub-issue here is that we're bouncing a lot of email like this, for an exciting product called "Penisole" from reputable domain "bloggay.com":

Add some kind of notification to user accounts like "some email you would have been sent was dropped because you haven't verified your address yet"?

My plan here is:

My guess is that this is just something dumb and mostly out of our control, and we fell through the cracks because the email volume we send from this install is tiny. We sent 41,877 messages in January at a cost of $0.00 -- this is too few messages to even qualify for charges, which start once you send 2,000 messages in a single day.

The changelog from Mailchimp (opt-in only) is from noreply@phacility.com, but I can't think of anything else off the top of my head

In T12237#211025, @epriestley wrote:

Although I'm somewhat confused by this because it came to the address for my personal AWS account.

I appealed this, but the FAQ says "Due to the way spamtrap sending is reported, it will take a minimum of three weeks before we can confirm that a fix you have put in place has succeeded.".

@chad, just to make sure my ducks are in a row, you aren't using noreply@phabricator.com in association with any marketing mail, right? I don't want to claim "we only send transactional mail" if that isn't true.

SES is definitely rejecting mail, though:

Although I'm somewhat confused by this because it came to the address for my personal AWS account.

Also: I think it's reasonable that if no event has happened in some time (A week? A month?), the Taskmaster would take 15 seconds to react to come back when it's needed (i.e., allow auto-scale to 0 if last task was a long time ago, although I'm not familiar with how the overseer and daemon scaling actually works).

Can't the same tricks used to make a single web server handle multiple instances be used for the daemons?

I deployed that change here (secure.phabricator.com). I'll cherry-pick it and deploy it on admin in a little bit, if nothing catches on explosion fire.

Haha, no worries. Uninstalling Home is pretty unusual.

ugh. sorry

After purging dead instances, db003 and db004 are each <25% utilized on their data volumes.

Purging db001 brought primary volume down to 79.2%, which is technically below the 80% caution threshold. It doesn't need to be fixed immediately, but the pathway forward isn't terribly clear. Some options are:

Instance service windows on db003 and db004 have synchronized and the shards are now purging.

The temporary directory storage pressure on repo016 has an unclear root cause (it's related to importing an SVN repository, but I haven't been able to pinpoint where the issue started). The issue no longer appears to be occurring so I'm going to stop digging until it happens again.

The storage pressure on db001 is primarily because an instance imported a large SVN repository with approximately 60M paths. The instance is a free instance with one user.

These are now in production. I inched some other automation forward but no major changes.

I think we're back in action and I have learned a series of valuable life lessons.

(This host is the replica database for the "Files" application partition, so even if I had to completely burn it to the ground nothing would be lost.)

I foolishly ran apt-get dist-upgrade because I am bad at computer and the motd said I should.

Seems to be working now. ¯\_(ツ)_/¯

For future reference, here's the sequence of problems I hit:

Thansk

I think this is fixed now (config change on the Mailgun upstream side). Let me know if you're still seeing issues.

Test v4

I deployed this, seems to have worked properly in production.

Vaguely related: aphlict exited on notify001 with this error:

Both 32GB volumes have been swapped for 128GB volumes.

These shards are now in service.

Provisioning completed, now deploying.

Starting this now.

I've rotated the certificates on lb, alb, and notify. One minor notify issue that I'll file separately.