Page MenuHomePhabricator

Migrating repository between storage hosts in a cluster
Closed, ResolvedPublic

Description

In our high availability setup, we currently have a single storage machine. We want to spin up a newer box using a new configuration, and want to migrate the repositories from the old storage machine to the new storage machine, without causing data to be re-imported, and without any downtime.

I assume this is being done already in the Phacility cluster on some level when repositories get really large, but I'm not particularly sure how to perform this migration.

Our plan in the event that it's not possible to perform such a migration is to detach the EBS volume containing the repositories and attach it to the new instance, but obviously this carries a much higher risk as we won't know if the new storage machine is fully working until we attempt to do this an update the Almanac records to point to the new IP.

Event Timeline

eadler added a project: Restricted Project.Mar 28 2016, 8:23 PM
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Mar 28 2016, 8:29 PM
eadler edited projects, added Clusters; removed Restricted Project.May 13 2016, 6:05 PM
epriestley claimed this task.
epriestley added a subscriber: epriestley.

I assume this is being done already in the Phacility cluster on some level when repositories get really large, but I'm not particularly sure how to perform this migration.

We just accept a few minutes of downtime. This operation is rare and the downtime is short.

This operation can be performed seamlessly like this:

  • Clusterize the repository.
  • Bring up the new host and add it to the cluster.
  • Wait for the cluster to synchronize.
  • Mark the old host as non-writable.
  • Remove the old host.

Not necessarily applicable in the general case, but see also T13393.