This is how I pad out my experience points for Hall of Heroes.

I'm going to call this tentatively done and leave it open for a week or two to see if there's any fallout.

Something is making a lot of requests

The behavior of mod_reqtimeout is weird in production. Some of this is the LB, but some of it I don't have a good explanation for.

I was able to get web001 to wedge a lot of requests in "R" ("Reading Request") -- note "SS" is "seconds since beginning of most recent request".

I'm also unable to wedge secure. mod_status is working there, but since I can't wedge it it isn't very useful.

I'm going to take a more detailed look at local behavior for this specific request pattern.

This is working somewhat better than before, but I'm still able to wedge apache with enough requests using a pre-patch version of libphutil. I'm going to take a more detailed look at local behavior for this specific request pattern.

It's currently possible for cluster hosts to hit rate limits against admin while doing normal deployment stuff, since they may make a lot of requests to admin very quickly.

I expect I can just reduce the data size to something manageable with the current workflow fairly easily

(I also verified that real multipart/form-data works fine from a browser, by changing a project's profile picture here.)

On the server die, the connection test looks better: secure001 saw 8 HTTP 500's (this might be a bug in the multipart/form-data parser for whatever cURL is doing) and then 135 HTTP 429s, which is more or less exactly what the code is supposed to do. So I think most of the inconclusiveness on what was happening from the client side was curl / nohup stuff.

I used ab to verify the rate limiting code here, and got killed after a bit of abuse with ab -c 10 -n 200 ...:

• Rewrite explanatory comment.

The previous version of this code has been active in production on secure since D17761, and is currently active, and we haven't seen issues. (You can use Config → Cache Status to review APCu usage. Whichever secure host I just hit is using 2MB, which suggests there's no leak.)

• Rebase on top of a cleaner build.

• Fix unit test path to PhabricatorStartup.php.

There's also a bug with arc here. It's supposed to limit simultaneous uploads to 4:

Tangentially related here and to T12611, the traffic volume from the ongoing "attack" in T13003 filled /tmp on admin.phacility.com. I pruned some old logs for now, but the mitigations in T12611 (e.g., separate log volumes) would resolve this properly.

Yeah, the default behavior for PHP is to write multipart/form-data uploads to tempfiles on disk.

For cases like Animal: zebra we'll end up in state "key" (since we never saw a ; or =), which is okay.