This is now live.

Deploying the changes to web now.

I've issued all instances a 24-hour service credit for the disruption. This should be reflected on your next invoice.

Here's the request rate leading up to the rate limiting:

D19653 (above) changes the per-"Host" rate limit to require "X-Forwarded-For" be present in the request. This should exempt ELB requests from these limits.

... in production today as a next step.

This should have the pleasant side effect of letting us drop the goofy hard-coded internal rate limiting IP list.

There are four rate limits, and I don't currently have enough information to figure out which one triggered. The rate limits are:

The remaining work on the audit constants is:

See also PHI862 for a request for arc amend --force.

See PHI860 and T13111. In the future, repository nodes may automatically gc/prune/repack. If they do, it may make sense to sort them to the bottom of the list so traffic is sent to them only if no other nodes are available, in order to minimize the impact that gc/prune/repack have on other activity.

PHI860 is a close variant of this and discusses periodically running git repack. Most concerns around git repack are likely similar to concerns around gc and prune.

D19637 doesn't actually improve this, but it brings us a step closer (since the logic is now in PHP and can be modified more easily to label symbols and add context).

Yes. The diff is a relatively small amount of the work I must do to respond to this issue: install ElasticSearch, configure it in the way you describe, reproduce the issue, write or apply a change, verify it actually fixes the problem, go read through the documentation enough that I'm confident the fix is the best available fix and that we've understood the issue reasonably well.

(Just to set expectations, anything which isn't coming in through Support Pacts may take a very long time for us to get to.)

• D19634 allows reviewers to mark their own inlines as "Done" before they submit them. This addresses the case where you want to leave a non-actionable inline (like "This is much-improved.") but feel like leaving an inline like this is imposing on the author because they "must" check it off.
• D19635 marks the author's own inlines as "Done" by default. They may un-mark them if they prefer (e.g., actually want to leave themselves a TODO). This addresses the case where authors feel like they "must" check off every comment, so they either reply in two passes (one to submit; one to check off all their own replies) or avoid replying.