• I launched repo025.phacility.net, like other hosts in the tier.
• I added repo025.phacility.net to DNS in Route 53 (172.30.0.115).
• I added a device record (repo025.phacility.net) to Almanac on admin.phacility.com.
• I added a service record (repox025.phacility.net) to Almanac on admin.phacility.com, and bound it to the device.
• I added a device record for the backup volume, rback025.phacility.net.

• I used bin/provision launch --device repo025 to mount and attach EBS volumes.
• I used bin/remote deploy repo025 to deploy the host. This is currently installing all the package junk and then will probably spend a while formatting the swap partition.

While that's running, I closed dbx012 and repox012 to new allocations. The shard status panel correctly reflects repox025 with no paired DB service and the 012 shard as closed:

• Swap stuff finished up.
• I merged master to stable for services/ to pick up bin/services sync --src ... --dst ....
• I merged master to stable for instances/ to pick up bin/instances move ... and related changes.
• core/ just runs master so nothing needed to merge.
• I upgraded repo025, then repo012, then admin001. None of these seemed to hit any issues.

I'm going to use the new "force shard placement" tools to launch a couple of test instances next.

I launched yellow-underneathie.phacility.com with forced services dbx012 and repox025. My goal here is to check for any code I missed which makes the assumption that dbX = repoX. I believe we don't have any of this code, but if we do, this instance should point out whatever problems we have that still need to be cleaned up.

If this works, that points at only the allocation algorithm needing cleanup when we want to separate the db and repo tiers so we can resize them independently.

I read and wrote the repository and verified that the data really ended up on db012 and repo025.

Next, I'm going to launch a similar instance but force placement on to db012 + repo012, write a repository, then move it to repo025.

bin/host restore failed with this error:

$ PHABRICATOR_INSTANCE=admin /core/lib/instances/bin/instances move --instance red-underneathie --to repox025.phacility.net
[2017-06-09 16:58:36] EXCEPTION: (CommandException) Command failed with error #255!
COMMAND
'/core/bin/remote' --internal -u instances -i /core/conf/keystore/instances.key restore repo025.phacility.net --instance 'red-underneathie' --download 'PHID-FILE-qj3g6sya4bptjldhgjhl' --kind 'repository' 

STDOUT
(empty)

STDERR
Usage Exception: Unknown instances: red-underneathie!

This is because red-underneathie is not an instance on repo025, and --instance can only select instances on the current host.

I can come up with two ways we could fix this:

• We could swap the services in the middle, instead of at the end, so the instance was on repo025 by the time restore ran. But I don't like this much because it makes recovering from errors harder.
• We could add a flag like --do-not-restrict-the-instance-query-to-just-instances-on-this-host. That name is silly, but I think this flag is generally reasonable and there exist some other cases where it might be useful (for example, bin/host query doesn't really need to be executed from a host which the instance is bound to, and this flag would let bin/host instances list every instance if you wanted that for some reason).

I think the flag is fairly reasonable so I'm going to look at providing that.

(I missed this locally because all "local" instances always belong to all "local" devices so this situation isn't possible in development.)

I hit two more minor issues:

• bin/host purge-cache --instance X did not work on admin because instance X is not bound to admin. I added --global.
• --global did not work on admin when executed on the host directly because admin is a special tier which has a hard-coded binding to the admin instance. Instead, query the global instance pool with --global.

As a sort of aside, --global --instance X is really slow right now (about 10s), and will only get slower in the future: it loads every instance page by page over Conduit, then does the remaining filtering by name locally. But it gets the right result, so this is reasonable enough for the moment. I may try to tune this before moving a lot of instances.

With those changes, bin/instances move completed successfully.

I pulled and pushed the test repository and verified the new data went to repo025, not repo12. Almanac also synchronized properly. So it looks like this more or less works.

I'm going to go run some errands and probably grab lunch, then settle in and start moving disabled/suspended instances.

I'm moving all the disabled/suspended instances now.

A verrrry minor thing is that because we always transfer a repository backup (even if it's empty), a migration always generates a repository directory on the far end. We could maaaybe have restore remove the directory if it's empty after the restore finishes to tidy this up just a touch.

The disabled/suspended instances finished up with no apparent issues, so I expect to move the live instances tomorrow.

Deploying, then picking this up again.

Instances are moving now.

🌴 🐫 🐫 🐫 🌴 🐫 🐫 🐫

The move completed without errors.

The total amount of data on repo025 is appreciably smaller than the total amount of data on repo012, but this appears to be because of the git clone --mirror we use to create repository copies during backup/migration: it only clones reachable objects, so it's effectively like running git gc in each repository first. This completely accounts for the size difference as far as I can tell: I spot-checked several repositories and they have the same number of reachable commits and refs, and the repo012 version of the repository is reduced to the repo025 size by running git gc.

(Currently, we never run git gc automatically, so it is not surprising that repositories would collect some garbage.)

I've forced blue-underneathie.phacility.com to allocate on the dead shard and am setting up some repository data for it now.

nice!