We should probably update the documentation to recommend --output and --compress, since the error behavior (e.g., in the event of a full disk) is better.

This is still ongoing but it isn't impacting us so I don't plan to do anything else here.

I've deployed the slightly more formal preamble. It no longer has the remote address logging, but here are the addresses captured up to now (first column: request count; second column: address):

A wrench in the gears here is that disabling enable_post_data_reading also disables $_FILES, which we use in about 20 places with vanilla HTTP <input type="file" /> controls. So on top of initialization order issues with $_POST, we also need to build a multipart/form-data parser that can populate a synthetic $_FILES or some similar construct.

I'm going to merge this into T4369, which has some similar discussion and more context.

PHI109 captured a slightly more illuminating error. D18692 has some discussion.

We now have three of these in Support (PHI63, PHI107, PHI128). I think the pathway forward is something like:

This is still ongoing, although we're weathering it without any issues.

Yes, use bin/storage dump --no-indexes. See here for more details:

merely a large number and not an enormous number

Handful of these, too:

I early 500'd these requests in preamble.php and load seems better now:

There are a very large number of originating IP addresses so I'm going to filter this traffic by path rather than by IP address.

On the bin/files download side of things, we currently buffer the whole file into a string and then hit this:

For consistency, I gave the content Cloudfront distribution a 60-second timeout as well.

Pulling stuff from admin.phacility.com via Cloudfront, I get this instead:

In the log above, this download should have worked:

In PHI120, we hit an issue where an instance (whose members seem to be located in Asia) performed 1,600 file uploads at roughly 256KB/sec, apparently putting a substantial amount of pressure on workers for ~12 minutes.

The best place for this kind of question is now your organization's support pact -- @tammybutow or @alexmv should be able to help you get access.

Another thing we can do here which will generally reduce the operational load is to be more selective about how we dump/export tables from MySQL. Broadly, we have three general classes of tables today:

I've pushed the MySQL configuration tweaks here, and manually built the common ngrams table with a 0.15 threshold.

(Basically, a notification is just a copy of a feed story which we deliver directly to an individual at the same time we publish the feed story, so feed and notifications are intertwined.)

Mostly right: each node server process has both a "client" service and an "admin" service on different ports. Everything else is pretty much accurate, except that "admin server -> client server" is generally inside the same process.

I'm also going to increase the db tier innodb_buffer_pool_size from 4GB to 6GB. The hosts have 7.5GB of RAM and 8GB of swap, and are pretty much sitting there with a little under 3GB of RAM mostly unused and no other load.

The thing I'm mostly looking at is how many queries are pushed down to 0 ngrams, i.e. they can't use the ngram index.

I looked at how the last 1,000 queries on this instance would be affected by different "common" thresholds. Note that these charts are all sort of garbage (X axis is nonlinear) because I couldn't figure out how basic spreadsheet software works.

Here are some possible structural changes we can make to the table, using the Maniphest ngrams table on this install as an example dataset (this is roughly 50x smaller than the target dataset):

There's still a big element of mystery here: why did MySQL sustain 80MB/sec of write I/O without daemons running? There "should" have been no queries against the fngrams table, and a large table "should" not require huge I/O volumes if it isn't being used. I don't think it's totally implausible that this was some kind of general thrash/swap/memory management issue in MySQL (the innodb_buffer_pool_size on these hosts if 4GB, and mysql was using all of it) but I would expect "paging-like" activities to cause a large read volume, not a large write volume.

(I'll fix this if you don't get there first, but have used up all my brain energy for today. And thanks for the report!)

Would it be acceptable to change PhutilRemarkupHyperlinkRule to catch this exception and behave as if it were a non-whitelisted protocol? (Happy to draft the patch, just want to check before I do so)

Well, it half-works in email, so that's something!

You can do this

They need to be on their own line. I think this rule got added in 2011, shortly after I left FB.

Drive-by mention in PHI91.

It should also work with Mercurial (D18652 and D18653 were my Mercurial tests).

• arc diff --cache (which was a passthrough for arc lint --cache) has been removed.

D18651 is marked as fixing this. The "fix" is very rough and will land in the experimental branch, see T11911 (and perhaps T12996) for followups.

• arc diff --no-diff (an internal workflow) has been removed.
• The internal "async lint" flag has been removed. This was an ancient, Facebook-specific flag for extensions which is obsoleted by Harbormaster.
• arc lint --only-new has been removed. This relied on upstream lint support which was never fully developed.
• arc lint --cache has been removed.
• The arc.lint.cache config option has been removed.
• arc lint --only-changed has been removed. arc now detects whether you want warnings on all lines or only changed lines by examining the other arguments (paths, --everything, --lintall, and --rev).

I think I'm going to land this stuff on the experimental branch since there's also some other related work which is better targeted there, and things are probably going to get worse before they get better.

Can we avoid filling the output buffer in subprocesses by testing if writes to stdout would block? (This should also be a "yes".)