That's very likely the same problem, and I think it should be fixed by updating to the current stable (rPf2a7db1 or newer).

I believe we may be hitting either the problem one of the above commit fixes, or suffering from a similar caused as side-effect from it.

...ideally this sort of thing should fail loudly at serialization time...

• The drydock_resource table could use a (status, ...) key to satisfy common/default queries.

After recent changes, only DifferentialRevision and ManiphestTask still have onboard mail keys.

"PhabricatorApplicationTransactionCommentView->setDraft()" has exactly two callers: one in Slowvote...

Resolved by D21824.

Thank you!

In T13588#256950, @cspeckmim wrote:

A user just ran into this exception with Arcanist...

A user just ran into this exception with Arcanist (current master changeset) while running arc diff (using Mercurial) with PHP 8.1. I haven't looked too in depth but it seems similar to other PHP 8.1 issues.

[2022-05-17 20:49:47] EXCEPTION: (RuntimeException) substr(): Passing null to parameter #1 ($string) of type string is deprecated at [<arcanist>/src/error/PhutilErrorHandler.php:261]
arcanist(head=master, ref.master=3cc486d5c156)
  #0 PhutilErrorHandler::handleError(integer, string, string, integer)
  #1 substr(NULL, integer, integer) called at [<arcanist>/src/repository/api/ArcanistRepositoryAPI.php:807]
  #2 ArcanistRepositoryAPI::getDisplayHash(NULL) called at [<arcanist>/src/repository/marker/ArcanistRepositoryMarkerQuery.php:67]
  #3 ArcanistRepositoryMarkerQuery::execute() called at [<arcanist>/src/repository/query/ArcanistRepositoryQuery.php:20]
  #4 ArcanistRepositoryQuery::executeOne() called at [<arcanist>/src/repository/api/ArcanistMercurialAPI.php:988]
  #5 ArcanistMercurialAPI::getActiveBookmark() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:2253]
  #6 ArcanistDiffWorkflow::buildDiffSpecification() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:380]
  #7 ArcanistDiffWorkflow::run() called at [<arcanist>/scripts/arcanist.php:427]
<<< [1] (+8,567) <exec> 8,567,426 us

I am pretty sure that this error message does not exist prior to PHP 8.1, and PHP 7.4 can not possibly emit it. See here for some evidence that this is true -- note that the error message is not present in the script output until PHP 8.1:

Trying to fresh install using PHP 7.4 or PHP 8.1, have the same result when executing:

• ./bin/config set phd.user root

[2022-05-10 02:01:12] EXCEPTION: (RuntimeException) strlen(): Passing null to parameter #1 ($string) of type string is deprecated at [<arcanist>/src/error/PhutilErrorHandler.php:261]
arcanist(head=master, ref.master=fc5b228db537), phabricator(head=master, ref.master=698ada2470b1)
  #0 PhutilErrorHandler::handleError(integer, string, string, integer) called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:128]
  #1 PhabricatorEnv::initializeCommonEnvironment(boolean) called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:75]
  #2 PhabricatorEnv::initializeScriptEnvironment(boolean) called at [<phabricator>/scripts/init/lib.php:26]
  #3 init_phabricator_script(array) called at [<phabricator>/scripts/init/init-setup.php:11]
  #4 require_once(string) called at [<phabricator>/scripts/setup/manage_config.php:5]

There may be additional work here, but presuming this is more or less resolved until evidence to the contrary arises.

I believe D21811 covers this completely.

When this mechanism is removed (by commenting out the logic that cares about the 25% limit), we'd expect Drydock to build 8 resources at a time (limited by number of taskmasters). It actually builds ~1-4...

We may fopen() an additional stdout and/or stderr handle, but do not fclose() it?

I am suspicious that D21794 may have broken something subtle with unix magic, since I'm seeing some hangs out of deployment scripts wrapping daemon management scripts. I think the issue is probably one of:

The outline above isn't quite sufficient because when the active resource list is nonempty, we don't actually reach the "new allocation" logic. Broadly, executeAllocator() is kind of wonky and needs some additional restructuring to cover both the D19762 case ("allocate up to the resource limit before reusing resources") and the normal set of cases. The proper logic is something like:

This issue partially reproduces (consistent with the original report, not immediately consistent with my theorizing about a root cause in PHI2177 -- actually, looks like both parts are right, see below): Drydock builds ~1 working copy per minute serially until it reaches a pool size of 5 resources. Then, it begins allocating 2 simultaneous resources.

What options exist for inter-process communication?

Here's how I'm thinking about overengineering this:

This was resolved as part of T13630. I didn't actually put it in the UI anywhere, but all requests will now originate from 13.56.71.101, and this isn't likely to change given the wind-down of Phacility hosting.

D21801, motivated by T13676, made this more practical.

This is somewhat resolved and neither next steps or motivation are clear any longer, so I'm going to call it done until evidence to the contrary arises.

Perhaps a philosophical question here is: do we care about which repositories are checked out in a working copy resource?

Before, instant reclaim after lease destruction:

To create resource pressure, I'm now going to try this -- I guess I don't really need the --count flag, but it does make the terminal juggling slightly easier:

The blueprint thing was on the way toward creating allocation pressure, so D21802 allows you to select a blueprint (or a set of possible blueprints) with --blueprint. You can specify an ID or PHID:

That patch is reasonable, and shouldn't break anything as long as the list you provide is a subset of the possible list.

fill in the details a bit.