I moved everything except one test instance off in T12817, and we appear to be ready for this host to die.

No one has complained that all their data is gone so I'm going to assume the best here.

nice!

The move completed without errors.

Instances are moving now.

Deploying, then picking this up again.

I'm going to call this "resolved" since I migrated instances in T12817 successfully. There's followup work, but T12798 is probably a better point of consolidation for future work now than this task is.

The disabled/suspended instances finished up with no apparent issues, so I expect to move the live instances tomorrow.

A verrrry minor thing is that because we always transfer a repository backup (even if it's empty), a migration always generates a repository directory on the far end. We could maaaybe have restore remove the directory if it's empty after the restore finishes to tidy this up just a touch.

I'm moving all the disabled/suspended instances now.

With those changes, bin/instances move completed successfully.

As a sort of aside, --global --instance X is really slow right now (about 10s), and will only get slower in the future: it loads every instance page by page over Conduit, then does the remaining filtering by name locally. But it gets the right result, so this is reasonable enough for the moment. I may try to tune this before moving a lot of instances.

I hit two more minor issues:

bin/host restore failed with this error:

I read and wrote the repository and verified that the data really ended up on db012 and repo025.

I launched yellow-underneathie.phacility.com with forced services dbx012 and repox025. My goal here is to check for any code I missed which makes the assumption that dbX = repoX. I believe we don't have any of this code, but if we do, this instance should point out whatever problems we have that still need to be cleaned up.

• Swap stuff finished up.
• I merged master to stable for services/ to pick up bin/services sync --src ... --dst ....
• I merged master to stable for instances/ to pick up bin/instances move ... and related changes.
• core/ just runs master so nothing needed to merge.
• I upgraded repo025, then repo012, then admin001. None of these seemed to hit any issues.

While that's running, I closed dbx012 and repox012 to new allocations. The shard status panel correctly reflects repox025 with no paired DB service and the 012 shard as closed:

• I launched repo025.phacility.net, like other hosts in the tier.
• I added repo025.phacility.net to DNS in Route 53 (172.30.0.115).
• I added a device record (repo025.phacility.net) to Almanac on admin.phacility.com.
• I added a service record (repox025.phacility.net) to Almanac on admin.phacility.com, and bound it to the device.
• I added a device record for the backup volume, rback025.phacility.net.

I think the next steps are:

I cleaned up all that stuff and it doesn't look like I broke the world.

I think this means we don't need to migrate every host: the problem with swapping block001 over to be NAT'd was probably that the NAT gateway itself couldn't reach the public internet. When we changed the 0.0.0.0/0 rule to nat-..., that blocked the NAT gateway itself from getting out.

• I launched a new NAT gateway into the public subnet (block001); nat-0d470fc54d846a993.
• I reconfigured the private subnet route table to point at the new gateway instead of the old one.
• I can now reach the public internet from the host on the private subnet:

I think the problem is that the NAT gateway is currently in the private001 subnet, but needs to be in the public subnet (the NAT gateway itself must have a route to the public internet).

• I stopped the host, and got kicked off as expected.
• I started the host.
• Problem: This failed to magically fix anything, and I still can't connect to the public internet.

• I built a new route table, following the main route table.
• I replaced the target for the 0.0.0.0/0 rule with nat-... instead of igw-....
• Under "Endpoints", I added the new route table to the S3 endpoint so the private subnet can access S3.
  • (This wasn't immediately reflected in the "Routes" UI, but it's not important for now so I'm going to assume it's a propagation issue until I need it.)
  • (It showed up about a minute later.)
• I changed private001.phacility.net to use the new route table, with nat-... instead of igw-....
• I can still connect to the test host from the bastion.
• Problem: I still can't connect to the public internet from the test host.

• We launched a new private subnet (private001.phacility.net) yesterday.
• I launched a new instance (nat-test001.phacility.net) into this subnet, with no public IP.
• I can connect to the instance from the bastion host (bastion005), which is good.
• Problem: I can not connect to the public internet from the test host.

Conveniently, bin/restore already exists and does much of the above, although it doesn't have a --download flag yet and its current behavior is to abort if data already exists.

Actually, from the logs above I didn't even do it manually and just made it up for no reason.

Thinking about this a little more, there's really no reason to do any of the rebinding stuff until after we move the data. I did that earlier when moving manually, largely for convenience -- but we don't need to sequence things like that when automating, particularly if we don't need bin/storage to work on the new database.

I think we have a generally tricky situation on the database stuff here. The general algorithm this process is using so far is:

(None of this impacts anything outside the ops realm in any way, except that some of the admin/ops-flavored UIs are changing a little.)

There will be a final written exam on June 19th.

Just to clarify, it's ok that I'm not following, nor understand any of this, correct?

Oh. My. God.

(Slightly related: I killed the test-nat instance this morning since it doesn't help actually test the NAT).

Er, that was rCORE085cd7e4 but I typed D instead of T.

I inched us forward on getting EC2 and Almanac to agree with one another, but need T12414 to make much more headway:

Some of the syntax has changed slightly from the above, but I think the overall plan remains basically correct.

phage remote status --pools db,repo and such work now, using Conduit credentials from the bastion to query admin. We can swap that stuff later if we want to switch to EC2.

As far as pulling this state from EC2, it's really easy to fetch the list of hosts that have a given tag: http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html

Come up with a plan for where phage --pool repo is going to get the host list from.

Another adjacent piece of work here is automating provisioning (e.g., through autoscale groups or bin/provision) but I think we should mostly leave that for the future too since we're theoretically bringing up only one new host (or maybe two, natXXX + repoXXX). Many of the other changes here give us tools toward that anyway, and it would probably be better to pursue that as part of a future change that involves launching a larger number of new hosts (e.g., chassis swap on the whole tier).

One broad issue is that admin.phacility.com sees a generally high, sustained request rate of almanac.service.search calls (up to dozens per second). I think the uneven performance I've sometimes observed is just load-related, and that this load is sometimes high enough to max out the Apache workers and cause queue stalls.

Attempt to improve the performance of the almanac.service.search call

The next actions I plan to take here are:

GC has caught up to Conduit and the conduit_methodcalllog table is down to 160MB from ~1GB.

I think the Leylines change didn't actually work because we make the call via the LB so the request appears to coming from outside the cluster.

Deploying Leylines + GC changes to admin now.

I used bin/files migrate to move some old files which were stored in MySQL before S3 was configured to S3. The file blob table is now down to 160MB from 477MB.

A big part of this appears to be that dumping the huge Leylines table locks it, so requests can't record their own Leylines data:

Specifically, I'd like to make these changes:

The Aphlict logs should probably get consolidated here too, they're set up a little weird and don't currently rotate during normal deployments.

I'm going to declassify this too since there's nothing private.

Also javelinjs.com currently has bogus DNS, and there's probably no value in continuing to support it for the moment. I'll keep the bills paid on the domain in case we eventually revive Javelin as a standalone framework, but we could safely drop it from config.