Thank you!

In T13588#256950, @cspeckmim wrote:

A user just ran into this exception with Arcanist...

A user just ran into this exception with Arcanist (current master changeset) while running arc diff (using Mercurial) with PHP 8.1. I haven't looked too in depth but it seems similar to other PHP 8.1 issues.

[2022-05-17 20:49:47] EXCEPTION: (RuntimeException) substr(): Passing null to parameter #1 ($string) of type string is deprecated at [<arcanist>/src/error/PhutilErrorHandler.php:261]
arcanist(head=master, ref.master=3cc486d5c156)
  #0 PhutilErrorHandler::handleError(integer, string, string, integer)
  #1 substr(NULL, integer, integer) called at [<arcanist>/src/repository/api/ArcanistRepositoryAPI.php:807]
  #2 ArcanistRepositoryAPI::getDisplayHash(NULL) called at [<arcanist>/src/repository/marker/ArcanistRepositoryMarkerQuery.php:67]
  #3 ArcanistRepositoryMarkerQuery::execute() called at [<arcanist>/src/repository/query/ArcanistRepositoryQuery.php:20]
  #4 ArcanistRepositoryQuery::executeOne() called at [<arcanist>/src/repository/api/ArcanistMercurialAPI.php:988]
  #5 ArcanistMercurialAPI::getActiveBookmark() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:2253]
  #6 ArcanistDiffWorkflow::buildDiffSpecification() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:380]
  #7 ArcanistDiffWorkflow::run() called at [<arcanist>/scripts/arcanist.php:427]
<<< [1] (+8,567) <exec> 8,567,426 us

I am pretty sure that this error message does not exist prior to PHP 8.1, and PHP 7.4 can not possibly emit it. See here for some evidence that this is true -- note that the error message is not present in the script output until PHP 8.1:

Trying to fresh install using PHP 7.4 or PHP 8.1, have the same result when executing:

• ./bin/config set phd.user root

[2022-05-10 02:01:12] EXCEPTION: (RuntimeException) strlen(): Passing null to parameter #1 ($string) of type string is deprecated at [<arcanist>/src/error/PhutilErrorHandler.php:261]
arcanist(head=master, ref.master=fc5b228db537), phabricator(head=master, ref.master=698ada2470b1)
  #0 PhutilErrorHandler::handleError(integer, string, string, integer) called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:128]
  #1 PhabricatorEnv::initializeCommonEnvironment(boolean) called at [<phabricator>/src/infrastructure/env/PhabricatorEnv.php:75]
  #2 PhabricatorEnv::initializeScriptEnvironment(boolean) called at [<phabricator>/scripts/init/lib.php:26]
  #3 init_phabricator_script(array) called at [<phabricator>/scripts/init/init-setup.php:11]
  #4 require_once(string) called at [<phabricator>/scripts/setup/manage_config.php:5]

It's likely that D21451 didn't catch exhaustively all of these, but the motivating use case is now moot. If further need arises somehow, this strategy can be applied to those cases.

Almost every host currently in production was provisioned with Piledriver and things have been stable for quite a while, so I'm calling this resolved. See elsewhere for issues with Ubuntu20, mail, etc.

Moved the rest of this to T13640.

See also T13646: some very early service calls during initialization currently occur before --trace becomes active. These calls should be visible, possibly by buffering them and then dumping the buffer once --trace comes online.

PHP 8.1 disallows strlen(null).

With max_connections at 1024, the server above still looks fine after ~14 days:

Elsewhere, I compacted Phacility free instances onto a small number of database servers. I attempted to "fill up" these servers, and succeeded, sort of, but maybe too much -- they eventually fall over and need to be rebooted.

PHP 8.1 changed semantics of static variables in methods. See:

See also T13588. Using static::class to work around changes to the scope of static in PHP 8.1 is also possible now that PHP 5.5 is the minimum version, and some uses of get_class($this) might be replaceable. There's no strong motivator for this, it's just a little cleaner.

I put all the database migration stuff everywhere and it appears stable. I'm hooking up Postmark as an outbound pathway now. If I get that working, I'll let it sit for a while and start migrating databases.

Finally, there are other some MySQL version issues which can be avoided with:

The new core/ support for the API is partially deployed; the new services/ support isn't anywhere yet.

See also T13232. Here's a possible build strategy for PHP 8.1 under macOS Mojave. Anyone reasonable should probably give up and use brew instead.

The latest version of Phabricator itself is everywhere.

I'm going to hold it until the weekend and try deploying then if things look calm on my end.

While waiting to deploy db stuff, I was planning to look at pruning dead data out of S3 -- but, on closer examination, the total S3 bill is something like $1/day, so no priority on that whatsoever.

Piledriver also needs to be able to provision database hosts, but these are more-or-less a trivial subset of repository hosts.

• Make InstancesStateQuery use a dictionary when building the database ref information internally.

Piledriver also needs to be able to provision database hosts, but these are more-or-less a trivial subset of repository hosts.

I completed all the repository migrations over the weekend and seemingly haven't run into any issues.

Just for completeness, vault used to be an HAProxy host serving as an SSH load balancer, but this responsibility moved to lb001 once ELBs became able to listen on inbound port 22 and TCP forward, so there is no longer a vault class of machines.

The new provisioning process for repository shards is: