To create resource pressure, I'm now going to try this -- I guess I don't really need the --count flag, but it does make the terminal juggling slightly easier:

The blueprint thing was on the way toward creating allocation pressure, so D21802 allows you to select a blueprint (or a set of possible blueprints) with --blueprint. You can specify an ID or PHID:

That patch is reasonable, and shouldn't break anything as long as the list you provide is a subset of the possible list.

After D21796:

(one orthogonal bug I found is that bin/drydock lease discards any blueprints provided in an attributes JSON)

Grab a test lease on the host with:

Harbormaster needs a "Working Copy" blueprint, not just a "Host" blueprint.

Here's a fairly simple way to reproduce this:

How do modern servers written in C/C++ handle parallelizing requests?

Just for my own notes:

What are the practical limits of multi-system or multi-architecture binaries?

I generated D21794 with a native binary that has no dependency on system PHP (but does depend on system cURL).

Another open question is: before we call posix_isatty(STDOUT), how do we test whether STDOUT is a valid handle?

1. PHP doesn't link cURL statically and doesn't appear (?) to provide any easy way to link it statically.
   1. • Possible solution: build static cURL.
      • Possible solution: link cURL dynamically.
      • Possible solution: replace cURL with mbedTLS + a first-party HTTP client, since we don't care about 99% of what cURL does. Or implement first-party TLS hahaha except I'm half-serious? If HTTP/HTTPS is going to happen from an embedded context on my coffee maker I need a healthy embeddable TLS + HTTP stack anyway.
   2. The STDERR and STDOUT constants are defined by the CLI SAPI, and not present in the embed SAPI.
      • Possible solution: define them in the C wrapper.
      • Possible solution: polyfill them at startup in PHP (I'm not entirely sure this is possible).
      • Possible solution: abstract around them and use php://stderr and php://stdout instead. See also T13556. This is likely desirable anyway.
   3. When arc tries to load PHP code, it needs to read data out of the executable binary in some set of conditions (e.g., "if we miss on disk").
      • Possible solution: hook zend_compile_file(), which seems to be the expected way to approach this. The flow in phar_compile_file() seems similar.
   4. To run unit tests, arc depends on the presence of a php binary on the system.
      • Possible solution: accept that you must also have PHP to run arc's tests (and that system PHP may differ from arc php).
      • Possible solution: also include the PHP CLI wrapper and invoke it when arc is executed as php, providing what is essentially a fallback toolset.
      • Possible solution: provide a "unit test helper" fallback toolset.
      • Possible solution: rewrite the 5 trivial cases where we need this (support/unit/*) in shell script (but: the reason to do these in PHP in the first place was so that they're portable to Windows).

Can we build a single binary with a bunch of data in it (e.g., a picture of a cat) without breaking anything?

See T5479.

See T5479.

See T5479.

I removed Phragment in D21793.

My initial linter implementation didn't actually catch all of these so there were a few more than I expected, but I think I got essentially all of them.

Some general guidelines I'm following here:

I don't want to get embroiled in this too much, but:

Is the purpose of implementing introspection of libraries to allow for building native components that would get picked up and auto-loaded by arcanist?

How can PHP be built statically? How hard is this?

There's nothing particularly useful or actionable here now, so closing it out. (I believe this was the most severe incident Phacility ever experienced while actively maintained.)

This hasn't caused any more problems in like 4 years, so I guess it's kind of whatever.

This is moot given the Phacility wind-down.

This is almost certainly "PhabricatorMetaMTAReceivedMail incorrectly assumes email is valid UTF8", and the solution begins by making all the associated storage binary, but it's hard to imagine that ever happening now.

Difficult to imagine this ever getting fixed in the upstream since I don't have any Android devices.

This only impacts the "Support" application in Phacility and it's hard to imagine building a Doorkeeper bridge for it now.

Discourse got nuked, so just throwing this away until another report somehow surfaces.

This hasn't cropped up again and is presumably resolved.

This isn't really resolved, but almost certainly does not make sense to pursue given the Phacility wind-down.

I believe the above changes resolved this, and it hasn't cropped up again.

I believe the above changes covered everything, and there hasn't been any more action on the motivating customer issues, although this isn't terribly meaningful.

I believe D21466 resolved this, since it hasn't cropped up again even though daemon runtimes between restarts have been very long at various periods in the last year.

It's likely that D21451 didn't catch exhaustively all of these, but the motivating use case is now moot. If further need arises somehow, this strategy can be applied to those cases.

This is resolved and/or mooted.

I believe these were all hunted down.

I think this was resolved. I nuked Discourse anyway so it's effectively resolved until another report shows up.

Almost every host currently in production was provisioned with Piledriver and things have been stable for quite a while, so I'm calling this resolved. See elsewhere for issues with Ubuntu20, mail, etc.

I made this change and deployed it to secure, and this "critical security vulnerability" is now a proper Phabricator 404 page:

Calling this resolved, since it has been in production in the Phacility cluster for some time and worked correctly through relevant hardware changes.

Moved the rest of this to T13640.