Update Almanac definitions for all instances not on the paired db023 shard.

PHI1566 is resolved narrowly. These cleanup steps still need to happen.

(Updating addresses with bin/host query leaves the service address cache dirty (the "mutable structure cache" via PhabricatorRepository->getAlmanacServiceRefs()) so it should be followed with bin/cache purge --caches general.)

I'll flesh this out more later, but the move away from db123 = repo123 shard pairing, plus bin/host query using mysql makes it difficult to directly query instances using a particular repository service.

Minor issue that should be looked at during service sync arising from improved validation elsewhere:

I'm deploying the new host now. We just crossed a release so I'm going to manually restore it to 72f82abe07 once it comes up (see also T13359). Then, I'll resynchronize instance services for active instances.

Instance termination completed after about 20 minutes and all the volumes detached. Since the original instance can be recycled, I'm going to reattach and restart it, and throw away the replacement host.

I deployed the --sshd-key %k stuff to secure and it looks like that shaved ~1,000-2000ms off the total cost. The ssh-auth cost has dropped to about 200ms:

I saw things hang during deploy and the OpenSSH bug should have been fixed years ago.

I configured ControlMaster on secure.

The migration in PHI1403 seems to have gone through cleanly. This workflow can continue to improve, but it's in relatively good shape now.

Everything here appears to have made it to production cleanly.

Anecdotally from the last time around, gzipping the tarball didn't really do much. Possibly, this might more broadly imply that we'd be better off not compressing repository backups.

I believe I've moved "core/" from "instances.queryinstances" and sequenced all the followup changes properly, now, and that the only remaining piece is glue.

SyncWorkflow also depends on creatorPHID to synchronize the initial administrator account.

The primary Week 34 / Week 35 deployment has completed without apparent issues. I'm going to deploy some followup changes for T13393 later, but it looks like we're out of the woods on the bulk of outbound changes.

Clicking "Pay Now" from landing page fatals in "PhortuneCartCheckoutController.php:104"; "Call to undefined method PhortuneCartCheckoutController::buildCartContentTable()"
Pacts have a bad URI for billing accounts on the "Billing" tab.
Instances also have a bad URI for billing accounts on the "Billing " tab. Maybe the handle is using the wrong URI?
The sync worker is failing on new instance launch for lack of credentials.

The sync worker is failing on new instance launch for lack of credentials.

bin/host restart does not start no-daemon services.

daemon behaviors

Provisioning was once close-ish to automated. Is this close enough to automate?

I think this leaves us with:

Adjacent is the older instances.queryinstances API method. This is still used by service synchronization.

It is also used to cache InstancesManageCapability::CAPABILITY but this can easily just be cached in the request cache instead.

The "Instance" almanac service type can be destroyed.

Can we get rid of the instance-specific services completely now, after changes connected to T11413?

Mostly-promising answers on much of the rest of this:

I'm hoping this is a reasonable excuse to find a way forward [on >2GB downloads] here.

Is the 2GB HTTP stuff in T12907 realistic to fix?

The "Merchant" profile screen is now somewhat inconsistent with the "Account" profile screen, and uses different icons for "Subscriptions" and "Orders".

We're also using inconsistent icons for "Order/Invoice/Receipt/Cart" in different interfaces. I'm going to standardize on this lovely purse, I think:

Stuff to fix with subscriptions:

if this is really an issue we could likely throw a rate limit on the action

This is sort-of-quasi-intentional

It would also be nice to figure out Phortune permissions better here since this is a good opportunity to vet everything.