Page MenuHomePhabricator

Adding Hardware
Updated 2,665 Days AgoPublic

This document explains how to add new hardware to the Phacility Cluster.


To add hardware to the cluster:

  • Log in to the AWS console using your IAM role.
  • Launch a new instance into the VPC.
  • Add a DNS entry for the device in Route 53.
  • Launch and attach any required EBS storage.
  • Using the CLI Tools, deploy the device.
  • If the device receives traffic from load balancers, add the host to the load balancer pool.
  • If the device hosts a service, create a new Almanac device entry and bind it to new or existing services.

This process is not as automated as it could be, but we an automate it as the need arises. The more complicated and error-prone parts are automated.

Launching Instances

Before launching instances, consider these cases:

adminDo not launch multiple admin devices without making software changes to support this first.
bastionRequires master key. Use --master. Copy master key to /core/conf/keys/master.key after deployment.
notifyThese need software changes to scale up.
secureRequires software changes to scale up.
vaultThese are tricky to launch properly.

Launch instances using these settings:

DeviceAMITypeSecurity GroupKeyVPC

Adding DNS

For all devices, add a DNS entry in Route 53 with the internal (VPC) device IP address. This should be an "A" record, similar to the other device records. The UI defaults are generally correct.

Set the name of the device in the EC2 console to be the same as the DNS entry you selected.

For example, if you launch a new web instance, you might call it (you should use the next number which doesn't have a DNS entry yet, of course).

EBS Storage

These types of devices have persistent EBS storage:

DeviceStorage Types
adminadata, abak
dbddata, dbak
repordata, rbak, rlog
securesdata, srepo, sbak, slog

If you're launching a device which uses one or more kinds of persistent EBS storage, you need to create and attach EBS volumes to the device. Use these settings for each storage type:

StorageSizeAttach To

For example, if you are brining up a db device, create an adata volume and attach it to /dev/sdh. Then create an abak volume and attach it to /dev/sdf.

Name any volumes you create with names, for example

Storage devices should have the same number as the hosts they are attached to. For example, db007 should use ddata007 and dbak007.

Volumes which will have backups written to them need device entries in Almanac on These entries are used to store backup logs.


Using the CLI tools in rCORE, deploy the device (for example, web007):

core/ $ ./bin/remote deploy web007

The deploy workflow automatically runs the upgrade and restart workflows.

Add Devices to Load Balancers

If you've launched a device which is part of a load balancer pool, add it to the appropriate load balancer:

DeviceAdd ToNotes
adminalbShould you really be launching these?
weblbShould also be added to vault but that's muddy.
notifynlbNo process for putting multiple nlb nodes into production yet.

Other devices do not need to be added to load balancer pools.

Update Almanac Services

If you've launched a device which provides a cluster service, create a device record for it in Almanac on, then attach it to an existing Almanac Service (to add or replace a device in a service pool) or create a new Almanac Service for it (to add a new service pool).

Create device records for any storage devices you created, too.

All services have only one host in them today, so this is largely future work.

Last Author
Last Edited
Nov 13 2016, 12:53 AM

Event Timeline

epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)