The behavior of mod_reqtimeout is weird in production. Some of this is the LB, but some of it I don't have a good explanation for.

I was able to get web001 to wedge a lot of requests in "R" ("Reading Request") -- note "SS" is "seconds since beginning of most recent request".

I'm also unable to wedge secure. mod_status is working there, but since I can't wedge it it isn't very useful.

I'm going to take a more detailed look at local behavior for this specific request pattern.

This is working somewhat better than before, but I'm still able to wedge apache with enough requests using a pre-patch version of libphutil. I'm going to take a more detailed look at local behavior for this specific request pattern.

It's currently possible for cluster hosts to hit rate limits against admin while doing normal deployment stuff, since they may make a lot of requests to admin very quickly.

I expect I can just reduce the data size to something manageable with the current workflow fairly easily

(I also verified that real multipart/form-data works fine from a browser, by changing a project's profile picture here.)

On the server side, the connection test looks better: secure001 saw 8 HTTP 500's (this might be a bug in the multipart/form-data parser for whatever cURL is doing) and then 135 HTTP 429s, which is more or less exactly what the code is supposed to do. So I think most of the inconclusiveness on what was happening from the client side was curl / nohup stuff.

I used ab to verify the rate limiting code here, and got killed after a bit of abuse with ab -c 10 -n 200 ...:

There's also a bug with arc here. It's supposed to limit simultaneous uploads to 4:

Tangentially related here and to T12611, the traffic volume from the ongoing "attack" in T13003 filled /tmp on admin.phacility.com. I pruned some old logs for now, but the mitigations in T12611 (e.g., separate log volumes) would resolve this properly.

Please file feature requests through the community forum (https://discourse.phabricator-community.org) or a support pact (https://admin.phacility.com/book/phacility/article/support_pacts/).

Our access to superglobals in PhabricatorStartup comes from D6672, and works around an issue where PHP's (fairly insane) sanitization filters are configured.

Would auto_globals_jit help?

I think this is working properly in production, I recall getting at least one notification today.

We should probably update the documentation to recommend --output and --compress, since the error behavior (e.g., in the event of a full disk) is better.

This is still ongoing but it isn't impacting us so I don't plan to do anything else here.

I've deployed the slightly more formal preamble. It no longer has the remote address logging, but here are the addresses captured up to now (first column: request count; second column: address):

A wrench in the gears here is that disabling enable_post_data_reading also disables $_FILES, which we use in about 20 places with vanilla HTTP <input type="file" /> controls. So on top of initialization order issues with $_POST, we also need to build a multipart/form-data parser that can populate a synthetic $_FILES or some similar construct.

I'm going to merge this into T4369, which has some similar discussion and more context.

PHI109 captured a slightly more illuminating error. D18692 has some discussion.

We now have three of these in Support (PHI63, PHI107, PHI128). I think the pathway forward is something like:

This is still ongoing, although we're weathering it without any issues.

Yes, use bin/storage dump --no-indexes. See here for more details:

merely a large number and not an enormous number

Handful of these, too:

I early 500'd these requests in preamble.php and load seems better now:

There are a very large number of originating IP addresses so I'm going to filter this traffic by path rather than by IP address.

On the bin/files download side of things, we currently buffer the whole file into a string and then hit this:

For consistency, I gave the content Cloudfront distribution a 60-second timeout as well.

Pulling stuff from admin.phacility.com via Cloudfront, I get this instead:

In the log above, this download should have worked:

In PHI120, we hit an issue where an instance (whose members seem to be located in Asia) performed 1,600 file uploads at roughly 256KB/sec, apparently putting a substantial amount of pressure on workers for ~12 minutes.

The best place for this kind of question is now your organization's support pact -- @tammybutow or @alexmv should be able to help you get access.

Another thing we can do here which will generally reduce the operational load is to be more selective about how we dump/export tables from MySQL. Broadly, we have three general classes of tables today:

I've pushed the MySQL configuration tweaks here, and manually built the common ngrams table with a 0.15 threshold.

(Basically, a notification is just a copy of a feed story which we deliver directly to an individual at the same time we publish the feed story, so feed and notifications are intertwined.)

Mostly right: each node server process has both a "client" service and an "admin" service on different ports. Everything else is pretty much accurate, except that "admin server -> client server" is generally inside the same process.

I'm also going to increase the db tier innodb_buffer_pool_size from 4GB to 6GB. The hosts have 7.5GB of RAM and 8GB of swap, and are pretty much sitting there with a little under 3GB of RAM mostly unused and no other load.

The thing I'm mostly looking at is how many queries are pushed down to 0 ngrams, i.e. they can't use the ngram index.

I looked at how the last 1,000 queries on this instance would be affected by different "common" thresholds. Note that these charts are all sort of garbage (X axis is nonlinear) because I couldn't figure out how basic spreadsheet software works.

Here are some possible structural changes we can make to the table, using the Maniphest ngrams table on this install as an example dataset (this is roughly 50x smaller than the target dataset):

There's still a big element of mystery here: why did MySQL sustain 80MB/sec of write I/O without daemons running? There "should" have been no queries against the fngrams table, and a large table "should" not require huge I/O volumes if it isn't being used. I don't think it's totally implausible that this was some kind of general thrash/swap/memory management issue in MySQL (the innodb_buffer_pool_size on these hosts if 4GB, and mysql was using all of it) but I would expect "paging-like" activities to cause a large read volume, not a large write volume.

(I'll fix this if you don't get there first, but have used up all my brain energy for today. And thanks for the report!)

Would it be acceptable to change PhutilRemarkupHyperlinkRule to catch this exception and behave as if it were a non-whitelisted protocol? (Happy to draft the patch, just want to check before I do so)