I expect I can just reduce the data size to something manageable with the current workflow fairly easily

(I also verified that real multipart/form-data works fine from a browser, by changing a project's profile picture here.)

On the server side, the connection test looks better: secure001 saw 8 HTTP 500's (this might be a bug in the multipart/form-data parser for whatever cURL is doing) and then 135 HTTP 429s, which is more or less exactly what the code is supposed to do. So I think most of the inconclusiveness on what was happening from the client side was curl / nohup stuff.

I used ab to verify the rate limiting code here, and got killed after a bit of abuse with ab -c 10 -n 200 ...:

There's also a bug with arc here. It's supposed to limit simultaneous uploads to 4:

Tangentially related here and to T12611, the traffic volume from the ongoing "attack" in T13003 filled /tmp on admin.phacility.com. I pruned some old logs for now, but the mitigations in T12611 (e.g., separate log volumes) would resolve this properly.

Oops, I filed it in the wrong place, my apologies!

Please file feature requests through the community forum (https://discourse.phabricator-community.org) or a support pact (https://admin.phacility.com/book/phacility/article/support_pacts/).

Doh!, I was looking at the jshint engine before, not the eslint engine. It looks like phab doesn't have built-in support for eslint yet. But this looks promising: maybe the autofixing eslint reports *is* on an error-by-error basis.

At Facebook we have a custom arcanist lint engine that wraps eslint with a custom (JS) runner that extracts more info about each fix:

Our access to superglobals in PhabricatorStartup comes from D6672, and works around an issue where PHP's (fairly insane) sanitization filters are configured.

Would auto_globals_jit help?

I think this is working properly in production, I recall getting at least one notification today.

We should probably update the documentation to recommend --output and --compress, since the error behavior (e.g., in the event of a full disk) is better.

Thanks for the fix on that. I have my nightly backups running again, however I had to switch from

./bin/storage dump | gzip > file.sql.gz

to

phab/bin/storage dump --output file.sql.gz --compress

This is still ongoing but it isn't impacting us so I don't plan to do anything else here.

I've deployed the slightly more formal preamble. It no longer has the remote address logging, but here are the addresses captured up to now (first column: request count; second column: address):

Thanks for the quick response. I'll try the workaround in the morning.

insomniaqc

When enabled, the SERVER, REQUEST, and ENV variables are created when they're first used (Just In Time) instead of when the script starts.

A wrench in the gears here is that disabling enable_post_data_reading also disables $_FILES, which we use in about 20 places with vanilla HTTP <input type="file" /> controls. So on top of initialization order issues with $_POST, we also need to build a multipart/form-data parser that can populate a synthetic $_FILES or some similar construct.

I'm going to merge this into T4369, which has some similar discussion and more context.

PHI109 captured a slightly more illuminating error. D18692 has some discussion.

We now have three of these in Support (PHI63, PHI107, PHI128). I think the pathway forward is something like:

This is still ongoing, although we're weathering it without any issues.

Yes, use bin/storage dump --no-indexes. See here for more details:

For those of us who don't mind having to wait through a large reindex after restoring from a backup, is there a way to ask the backup process to skip dumping the index tables? (Based on the above I presume they'll still be included)

merely a large number and not an enormous number

Handful of these, too:

I early 500'd these requests in preamble.php and load seems better now:

There are a very large number of originating IP addresses so I'm going to filter this traffic by path rather than by IP address.

On the bin/files download side of things, we currently buffer the whole file into a string and then hit this:

For consistency, I gave the content Cloudfront distribution a 60-second timeout as well.

Pulling stuff from admin.phacility.com via Cloudfront, I get this instead:

In the log above, this download should have worked:

In PHI120, we hit an issue where an instance (whose members seem to be located in Asia) performed 1,600 file uploads at roughly 256KB/sec, apparently putting a substantial amount of pressure on workers for ~12 minutes.

i encountered same problem , but it happened on libphutil/src/xsprintf/xsprintf.php. I resolved this problem by setting max_nesting_level on php xdebug .