"Use the API" seemed to work OK. Of those instances, only bastion005 is at all unusual.

I 'm going to get these underway once the deploy finishes.

only bastion005 is at all unusual.

For bastion, I'm going to put a bastion006 in service beside bastion005, swap to it, then tear down bastion005. The bastion hosts have no state, but if we enter a state where we have no bastions up we can't get to anything else in the cluster. Bastions can deploy without other bastions existing (and we've done so four times, RIP bastions 1-4) but it's been a while since they rotated.

I cycled all the hosts except bastion. saux001 needs to be vetted a bit (it handles "Land Revision" from the web UI) but it isn't critical if it needs a bit more work.

There's a minor deadlock on bastion deployment with the current scripts: during deploy, we run deploy-key to copy the deploy key from the bastion to the target host during deployment, so that we don't need to put the entire keystore on normal cluster nodes, and so that we don't need to have the keystore on the control host (staff laptop) outside the cluster.

However, this operation doesn't make sense if we're actually deploying a bastion, since other bastions can't connect to it normally and there may be no other bastions (if we're deploying the first bastion after some kind of failure). For now, I'm just working around this by:

• skipping deploy-key;
• manually putting the deploy key in place in ~/.ssh/id_rsa.

This should be more clever but it's "good" that it can't easily be automated, in the sense that staff laptops should not have the deploy key present at deploy time in any other cases. This might make sense as a separate --deploy-key ... flag which is unique to bin/remote deploy bastion-class operations. I suspect T13076 will lead toward something more structured here.

I turned bastion.phacility.net and bastion-external.phacillity.net into CNAME records and pointed them at the new bastions.

(Technically, bastion007, since bastion006 was already deployed during cluster research.)

I also needed to copy the old master.key from bastion005 to bastion007 in /core/lib/keystore/.

I'm pretty sure we don't actually depend on this key after T12608 and removing it appears to work, but I'm hesitant to break too much stuff all at once.

I think this is all done but want to let things run against bastion007 for a bit before I tear down bastion005.

One more of these just came in for repo003.

Plus: db018.phacility.net, repo001.phacility.net, db024.phacility.net.

Taking care of these now. I expect everything to be pretty routine.

We haven't seen any bastion issues so I stopped bastion005.