A very rough version of this is now in master, but only active if phabricator.show-prototypes is on. These are some of the (substantial) remaining issues I'm aware of, and there are probably others:

I'm going to call this tentatively done and leave it open for a week or two to see if there's any fallout.

Something is making a lot of requests

I'm getting the same #1048: Column 'filename' cannot be null running just regular "arc diff". It used to work and then at some point after updates started failing.

The behavior of mod_reqtimeout is weird in production. Some of this is the LB, but some of it I don't have a good explanation for.

The above did not work for me. The following did:

I was able to get web001 to wedge a lot of requests in "R" ("Reading Request") -- note "SS" is "seconds since beginning of most recent request".

I'm also unable to wedge secure. mod_status is working there, but since I can't wedge it it isn't very useful.

I'm going to take a more detailed look at local behavior for this specific request pattern.

This is working somewhat better than before, but I'm still able to wedge apache with enough requests using a pre-patch version of libphutil. I'm going to take a more detailed look at local behavior for this specific request pattern.

It's currently possible for cluster hosts to hit rate limits against admin while doing normal deployment stuff, since they may make a lot of requests to admin very quickly.

I expect I can just reduce the data size to something manageable with the current workflow fairly easily

(I also verified that real multipart/form-data works fine from a browser, by changing a project's profile picture here.)

On the server side, the connection test looks better: secure001 saw 8 HTTP 500's (this might be a bug in the multipart/form-data parser for whatever cURL is doing) and then 135 HTTP 429s, which is more or less exactly what the code is supposed to do. So I think most of the inconclusiveness on what was happening from the client side was curl / nohup stuff.

I used ab to verify the rate limiting code here, and got killed after a bit of abuse with ab -c 10 -n 200 ...:

There's also a bug with arc here. It's supposed to limit simultaneous uploads to 4:

Tangentially related here and to T12611, the traffic volume from the ongoing "attack" in T13003 filled /tmp on admin.phacility.com. I pruned some old logs for now, but the mitigations in T12611 (e.g., separate log volumes) would resolve this properly.

Oops, I filed it in the wrong place, my apologies!

Please file feature requests through the community forum (https://discourse.phabricator-community.org) or a support pact (https://admin.phacility.com/book/phacility/article/support_pacts/).

Doh!, I was looking at the jshint engine before, not the eslint engine. It looks like phab doesn't have built-in support for eslint yet. But this looks promising: maybe the autofixing eslint reports *is* on an error-by-error basis.

At Facebook we have a custom arcanist lint engine that wraps eslint with a custom (JS) runner that extracts more info about each fix:

Our access to superglobals in PhabricatorStartup comes from D6672, and works around an issue where PHP's (fairly insane) sanitization filters are configured.

Would auto_globals_jit help?

I think this is working properly in production, I recall getting at least one notification today.

We should probably update the documentation to recommend --output and --compress, since the error behavior (e.g., in the event of a full disk) is better.

Thanks for the fix on that. I have my nightly backups running again, however I had to switch from

./bin/storage dump | gzip > file.sql.gz

to

phab/bin/storage dump --output file.sql.gz --compress

This is still ongoing but it isn't impacting us so I don't plan to do anything else here.

I've deployed the slightly more formal preamble. It no longer has the remote address logging, but here are the addresses captured up to now (first column: request count; second column: address):

Thanks for the quick response. I'll try the workaround in the morning.

insomniaqc

When enabled, the SERVER, REQUEST, and ENV variables are created when they're first used (Just In Time) instead of when the script starts.

A wrench in the gears here is that disabling enable_post_data_reading also disables $_FILES, which we use in about 20 places with vanilla HTTP <input type="file" /> controls. So on top of initialization order issues with $_POST, we also need to build a multipart/form-data parser that can populate a synthetic $_FILES or some similar construct.

I'm going to merge this into T4369, which has some similar discussion and more context.

PHI109 captured a slightly more illuminating error. D18692 has some discussion.

We now have three of these in Support (PHI63, PHI107, PHI128). I think the pathway forward is something like:

This is still ongoing, although we're weathering it without any issues.

Yes, use bin/storage dump --no-indexes. See here for more details: