In reply:

From @durin42, via Mercurial Security:

(Another possibility is to add behavior to arc to control this.)

Maybe. Another possibility is to mark builds as "nonblocking" in some sense. There are some other open requests for defining "advisory" or "noncritical" builds where build failure doesn't cause the Buildable to report a failed result overall. If we end up adding this state to builds anyway, it probably makes sense to include a "Does Not Block Review" flag to the "Does Not Fail Buildable" / "Does Not Block Dependent Builds" / "Failure Is Totally Meaningless" / Whatever flags.

Pretty sure T7704 is the same as this.

PHI181 has an entangling concern (T11372).

Generally, this sort of discussion is better on Discourse (https://discourse.phabricator-community.org) than here.

From T10109, we shouldn't make revisions "Buildable" to Herald if the update is an automatic update in response to a commit being discovered.

Ah, thanks. I think your guess is right. I'll make a note in T2543, should be a one-line fix I think.

After all that stuff:

Additionally, arc diff no longer dispatches ArcanistEventType::TYPE_REVISION_WILLCREATEREVISION if the server is sufficiently modern. (It no longer uses the datastructure this event received.)

See T2543. The flags related to previewing a diff have changed:

This is technically accurate but real awkward for humans:

I pushed secure004 with that patch and it worked fine, although mysql came back up fast enough that we didn't have to wait. Since I don't have a way to actually trigger this condition, I'm going to assume this is resolved until we have evidence otherwise.

This has been stable for about a week now.

This traffic eventually stopped on October 20th.

The pruned (at 0.15 threshold), optimized ngram index for the original affected instance is only ~13GB, which is entirely manageable with the changes to backups, and we haven't run into other issues.

We hit one of more of these (db024) last week so D18725 should fix it.

A very rough version of this is now in master, but only active if phabricator.show-prototypes is on. These are some of the (substantial) remaining issues I'm aware of, and there are probably others:

I'm going to call this tentatively done and leave it open for a week or two to see if there's any fallout.

Something is making a lot of requests

The behavior of mod_reqtimeout is weird in production. Some of this is the LB, but some of it I don't have a good explanation for.

I was able to get web001 to wedge a lot of requests in "R" ("Reading Request") -- note "SS" is "seconds since beginning of most recent request".

I'm also unable to wedge secure. mod_status is working there, but since I can't wedge it it isn't very useful.

I'm going to take a more detailed look at local behavior for this specific request pattern.

This is working somewhat better than before, but I'm still able to wedge apache with enough requests using a pre-patch version of libphutil. I'm going to take a more detailed look at local behavior for this specific request pattern.

It's currently possible for cluster hosts to hit rate limits against admin while doing normal deployment stuff, since they may make a lot of requests to admin very quickly.

I expect I can just reduce the data size to something manageable with the current workflow fairly easily

(I also verified that real multipart/form-data works fine from a browser, by changing a project's profile picture here.)

On the server side, the connection test looks better: secure001 saw 8 HTTP 500's (this might be a bug in the multipart/form-data parser for whatever cURL is doing) and then 135 HTTP 429s, which is more or less exactly what the code is supposed to do. So I think most of the inconclusiveness on what was happening from the client side was curl / nohup stuff.

I used ab to verify the rate limiting code here, and got killed after a bit of abuse with ab -c 10 -n 200 ...:

There's also a bug with arc here. It's supposed to limit simultaneous uploads to 4:

Tangentially related here and to T12611, the traffic volume from the ongoing "attack" in T13003 filled /tmp on admin.phacility.com. I pruned some old logs for now, but the mitigations in T12611 (e.g., separate log volumes) would resolve this properly.

Please file feature requests through the community forum (https://discourse.phabricator-community.org) or a support pact (https://admin.phacility.com/book/phacility/article/support_pacts/).

Our access to superglobals in PhabricatorStartup comes from D6672, and works around an issue where PHP's (fairly insane) sanitization filters are configured.

Would auto_globals_jit help?

I think this is working properly in production, I recall getting at least one notification today.