All queues have emptied, and repo002 is back in a healthy range again:

rP9329e6a12d9f improved the rate at which the Herald queue is draining by approximately 10x.

I've moved the largest instance from repo002 to a new empty shard, which has put both shards into fairly reasonable range. Queues still aren't draining as quickly as I'd like/expect.

I moved the largest instance from shard 002 to shard 006. This took about 35 minutes, most of which was backup/transfer time. I hit a couple of minor issues, but things appear to have worked properly overall.

I migrated a test instance, which appears to have worked properly. I'm going to run though things again, then move a large instance off repo002.

• Suspend the instance: Admin → Instance → Manage Instance → Suspend Instance
• Stop the daemons:

I've provisioned and deployed a new shard, but not yet opened it for allocations. I'm going to launch and then migrate a test instance next.

I let the queues drain for a bit overnight. One of the large queues is down from ~16K items to ~12K items in the last 8 hours, but I don't want to just let this sit for 24 hours to "resolve" it. I'm going to look at moving instances across shards instead, which is a capability we need to automate eventually anyway.

Looking at improving the raw performance of bin/repository update, one fixable issue is that we run git for-each-ref three times:

I also identified an instance with a bad HTTP hook which had managed to queue up 8,000 tasks in the queue. These already have scaling backoffs and likely didn't contribute a meaningful amount of load, but we could perhaps find a more graceful approach to handling bad webhook URIs during T11330.

I disabled the keys of the user running 30 clones/minute and notified the instance administrator. This appeared to create some headroom; I reverted the daemon pool size limits to the default and we're draining the queues at a somewhat more reasonable pace now.

@swisspol, that's what this refers to ("From T11666"):

I don't know if this is related but since an hour or so, whenever I push I get an error even though the push appear to work (the commits show up in Diffusion):

$ git push
# Push received by "web.phacility.net", forwarding to cluster host.
# Waiting up to 120 second(s) for a cluster write lock...
# Acquired write lock immediately.
# Waiting up to 120 second(s) for a cluster read lock on "repo002.phacility.net"...
# Acquired read lock immediately.
# Device "repo002.phacility.net" is already a cluster leader and does not need to be synchronized.
# Ready to receive on cluster host "repo002.phacility.net".
Counting objects: 56, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (50/50), done.
Writing objects: 100% (56/56), 9.30 MiB | 1.84 MiB/s, done.
Total 56 (delta 32), reused 1 (delta 0)
# Released cluster write lock.
AphrontQueryException: #1364: Field 'messageCount' doesn't have a default value
To ssh://hyper@vault.phacility.com/diffusion/HYPER/hyper.git
   015051e..67b218b  master -> master
error: failed to push some refs to 'ssh://hyper@vault.phacility.com/diffusion/HYPER/hyper.git'

A contributing issue is that one instance appears to be making clone requests for approximately 30 repositories per minute to this shard. This is easy to rate limit, but probably not easy to rate limit in a generalized, not-a-cluster-hack way.

One remaining issue with backoff/cooldown is that mirrors aren't included: a regularly-updated repository with broken mirrors still tries mirroring. This is somewhat more involved to fix, though (see T6131).

Those changes seem stable. Unclear how much they're really helping, but load is looking at least somewhat better. I'm going to dig into things a bit more and see if I can get a better sense of whether we're on the path to recovery or not.

D16574 retains update cooldowns (so we don't immediately update everything after a restart) and sets up for the following change.

Repository retries (failed mirrors, failed fetches) have no real backoff

I could alternatively throttle down the highly active instances (or all instances on this shard) but that's not too great.

• Maybe a good time to launch new shards? Last tier size increase was 10 months ago (late November, 2015) when I fixed the allocation algorithm to be not a huge wasteful pile of garbage that required us to constantly throw hardware at it.
  • Maybe the allocation algorithm should consider active users, not just active instances? I think the 002 shard has the most active users by a hefty margin, just by luck of the draw. But this will tend to be moot at scale.
  • Maybe start moving instances between shards? This seems hard to do without downtime today, and doing all the legwork to reduce downtime through clustering is foreboding.

I don't see anything noticeably broken.

• I nuked the corp and clb devices and DNS, and updated the documentation.
• I aligned reserved instances with actual usage after changes.
• I also added secure002 to plb001 (the phurl.io LB).

I think this is live now. I'm going to complete the rest of the deploy and then button up the AWS stuff, but yell if it's horribly broken or whatever.

I'm just going to merge this into T11076 since this will happen with the rest of the deployment work there. I expect this to be straightforward.

Any test we execute must be distributed with phabricator/ and run on the first page load after an upgrade. That generally means it needs to be small, very fast, and completely conclusive. Compare to the test for Shellshock here, which runs in a few milliseconds, requires 10 lines of code, and is totally conclusive:

In T10832#171473, @epriestley wrote:

Broadly, I lean toward this policy going forward:

• When we can perform an accurate test for the vulnerability in a reasonable amount of time/effort and tell you that you are definitely vulnerable (as with Shellshock), we will continue to do so with an active setup warning.
• When we can not perform such a test (as here), we will publish guidance and note the issue in the changelog, but will not attempt to guess if the installed version may be vulnerable because this test will frequently be confusing/misleading/wrong.
• We can re-evaluate this after T5055, which may give us a wider range of tools for providing more accurate vulnerability notifications.

We appear to be stable here now:

Sorry to hear that @otto.bretz, but glad you're still happy with the service overall. I think this is the longest unplanned downtime we've had since launch and it's definitely not acceptable to us.

Though this downtime came at an unfortunate time for us (we just migrated from bitbucket and in the middle of the workday here in Sweden) I must say that you have dealt with it in a phrofessional way. Thanks.

Thanks, I think that should be fixed now. Not clear if it was related to everything else and I can't immediately come up with a way that it would have been -- I'll keep an eye on it. We've made some changes to the Aphlict server recently and it might have some issues that still need to be worked out, so it could be purely coincidental.

Not sure if its related but I get a message regarding "unresolved setup issue" on our instance[1] now:

Some additional response steps:

From here on, this analysis discusses some private instance data. I'm generally very hesitant to disclose information like this, but the data is non-identifying and the nature of the data is important to understanding the nature of the failure, so I think disclosure is reasonable in this case. I've limited the disclosure to only include information important in understanding the technical aspects of the issue.

confirm noomhq is operational again

As a very mild silver lining, I'm pretty sure this was the actual root cause of the webserver hang, and it's relatively tractable. Here's what appears to have happened:

I think the issue is initially resolved now, is anyone with an affected instance still seeing an issue?

Pretty sure I've figured this out but resolving it will take some finesse.

Pretty sure I've figured this out but resolving it will take some finesse.

Looking up service metadata for some subset of instances is failing, I'm not sure why yet or how large that subset is.

our stuff is still down

Looking into that. There are some additional issues with queued daemon tasks on admin as well that I'm looking into, unsure if those are symptoms of the web issue or not yet.

I connected to the server around 5:55 AM PST and rebooted it after inspecting things, service was restored around 6:02 AM PST.

Events so far:

Thanks man, appreciate it.

Every service on secure is now supported by a redundant version in a separate AWS availability zone. Failover wouldn't be completely clean/automatic, but we could lose either node with zero data loss, and would have a very short path to promoting the remaining node as a full master.

Broadly, I lean toward this policy going forward:

At least for now, maybe I'll remove the setup issue to avoid confusion and just note this in the changelog instead. This is much more likely to be overlooked, but vendors other than Apple seem to have generally taken care of this and using git --version to test for the presence of the vulnerability isn't really meaningful.

safe to ignore if you've updated to the latest version even though it's a lower version number

In other cases (like with Shellshock) we can just test for the vulnerability to see if a binary is vulnerable. Ideally, we'd just perform this test to figure out if you need to upgrade, but I don't think a reasonable test case exists here because we need to create a 2GB pathname and probably can not do that in a reasonable amount of time.

I got hit by that warning today, and while it doesn't bother me because I knew about the vulnerabilities and versions, it could be confusing for others. I've got 2.1.4-2.1+deb8u2 on Debian, which is old but the fixes are backported. Many other distros have older versions with backported fixes as well (as per https://news.ycombinator.com/item?id=11517894). Unfortunately you cannot see that in git --version though.

I think the sequence of action is:

I'm less sure about adding a warning to arc.

• D15695 should fix logged-out users.
• D15696 + D15697 should smooth out security.require-https for intracluster requests.

Ah, yep, thanks.

(I'm just guessing it has something to do with this, because ops)

Logged-out users can't browse diffusion: ERR-INVALID-SESSION: Session key is not present.

This sort of works now. The 002 node is live, just not reachable yet. Some weird stuff I hit:

I'm probably going to break all repositories momentarily. If I do, I'll try to fix them.

[text intensifies]

I haven't been successful locally in getting a GC'd node to reach an appreciable level of memory consumption, and the production servers are currently at ~1.5GB.

I think the ~1.5GB node instance size is also expected.

At least locally, I can't manage to get node to not leak:

We're seeing a lot of connections stick in "R" ("Reading Request") for a long time, but the client is the load balancer and the request hasn't been read yet so it's not clear exactly what's going on with these. I think they may be crawlers, since pretty much every search engine loves indexing everything on this install hundreds of times an hour.

I'm fiddling with LB health checks, so it's my fault if the server vanishes.