Page MenuHomePhabricator

Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware
Closed, ResolvedPublic

Description

secure.phabricator.com is currently two hosts, secure001.phacility.net and secure003.phacility.net. Previously, secure also had two more hosts (002, 004), primarily to put database partitioning into production. See some discussion in T13654.

001 is a master for all databases, except file; 003 is a master for file.

I want to merge them, then move to new hardware. I think this process will work:

  1. Put the service into readonly mode.
  2. Dump just the files databases from 003.
  3. Load the files databases onto 001.
  4. Drop 003 and readonly, restart service.
  5. Stop 003.

Then this becomes a standard sort of migration, except it probably all has to be pretty manual since secure has a bunch of special cases.

Step (2) needs some changes to bin/storage dump to allow it to dump a subset of databases.

Event Timeline

epriestley created this task.

I'm putting secure into read-only mode now, with the intent of completing steps 1-5 above.

Merging 003 into 001 worked fine with a few expected tricks (e.g., when secure is in read-only mode, you can't push a change to take it out of read-only mode, since pushing is a write). Next up is launching a modern m4.large secure-pool host and then migrating the data.

I brought up the new host and pointed the slb001 load balancer at it. The database is still on the old host, and the new host doesn't have repositories yet, but the basics seem to be working.

I'm going to put secure back into read-only mode now and move the databases to the new host.

Databases are moved and secure is out of read-only mode. I'm going to adjust repository configuration, then I should be able to tear down secure001.

The aphlict/notify stuff still needs to be tweaked. I think the snlb + slb setup can be merged into a single slb with "TCP (Secure)" forwarding now.

I just swapped configs over without merging the LBs, since it wasn't immediately obvious to me what the Application vs Classic state of the world is and swapping was good enough.