Almost every host currently in production was provisioned with Piledriver and things have been stable for quite a while, so I'm calling this resolved. See elsewhere for issues with Ubuntu20, mail, etc.

Calling this resolved, since it has been in production in the Phacility cluster for some time and worked correctly through relevant hardware changes.

I put all the database migration stuff everywhere and it appears stable. I'm hooking up Postmark as an outbound pathway now. If I get that working, I'll let it sit for a while and start migrating databases.

Finally, there are other some MySQL version issues which can be avoided with:

The new core/ support for the API is partially deployed; the new services/ support isn't anywhere yet.

The latest version of Phabricator itself is everywhere.

I'm going to hold it until the weekend and try deploying then if things look calm on my end.

While waiting to deploy db stuff, I was planning to look at pruning dead data out of S3 -- but, on closer examination, the total S3 bill is something like $1/day, so no priority on that whatsoever.

Piledriver also needs to be able to provision database hosts, but these are more-or-less a trivial subset of repository hosts.

• Make InstancesStateQuery use a dictionary when building the database ref information internally.

Piledriver also needs to be able to provision database hosts, but these are more-or-less a trivial subset of repository hosts.

I completed all the repository migrations over the weekend and seemingly haven't run into any issues.

Just for completeness, vault used to be an HAProxy host serving as an SSH load balancer, but this responsibility moved to lb001 once ELBs became able to listen on inbound port 22 and TCP forward, so there is no longer a vault class of machines.

The new provisioning process for repository shards is:

Piledriver was built before the FutureGraph stuff settled in T11968; it runs into the same general set of sequencing problems and yield would likely be a good approach.

I can't figure out how to delete...

I got rid of everything I could, and nothing appears to be affected.

We have a lot of leftover VPC cruft that I'm going to nuke, notably meta and admin VPCs that (as far as I can tell) have nothing in them, and then a bunch of subnets (meta.private-a, meta.private-b, block-public-222, admin.public-a, admin.public-b, meta.public-a, meta.public-b, block-private-3) and some NGWs etc. I'm like 99% sure this stuff is all leftover from testing years ago and nothing depends on it, but I guess we'll see what happens when I delete all of it.

Here's the last known state of the world from T12816:

See also NAT carryover from T12816, via T13542.

This is very old and I can't reproduce it, and haven't seen any other reports.

The major offender here (services per instance) was fixed by updating caching, and I destroyed all the old services. This is perhaps spiritually continued in T13542.

I think that's pretty much everything. There will be a little followup work in T10883 and maybe T13076 / T13120.

Everything here should pretty much work except:

Before I can get rid of AlmanacDeviceTransaction::TYPE_INTERFACE, we have two meaningful callsites in rSERVICES and one unit test in rSAAS to clean up.

There's a bit of a mess with AlmanacInterface and AlmanacDevice. Currently, AlmanacInterface does not use transactions, and is edited purely as a side effect of INTERFACE transactions applying to AlmanacDevice. I'm going to change how this works so that AlmanacInterface is a normal transactional object and can use the same rules and infrastructure as everything else.

It's also possible to write a custom instances.do-exactly-what-we-need sort of endpoint and generalize later if that seems like a more promising approach.

The other shadow lurking in the water here -- which I think we can mostly avoid -- is that Almanac is mostly a-bit-bare-bones-but-overall-pretty-functional, except that the way properties on Bindings and Services are specified and edited is complete garbage. You more or less just have to magically know which properties are valid, and there's no real support for defaults or nice UI controls or hints about what you can set or suggestions that you're making stuff up and probably typo'd something.

The "most right" way in terms of consistency is to fully convert Binding to EditEngine, then implement almanac.binding.edit which can create/edit bindings. When creating a binding, it would require transactions specifying the service and interface. PhamePostBlogTransaction is sort of an example of this: when you create a new post with phame.post.edit, you must specify a blog transaction.

What's the best way to add API endpoints for resources like bindings? Call it almanac.create_binding and have it take a service and an interface as arguments?