Page MenuHomePhabricator

Adding Hardware
Updated 2,714 Days AgoPublic

This document explains how to add new hardware to the Phacility Cluster.

Overview

To add hardware to the cluster:

  • Log in to the AWS console using your IAM role.
  • Launch a new instance into the VPC.
  • Add a phacility.net DNS entry for the device in Route 53.
  • Launch and attach any required EBS storage.
  • Using the CLI Tools, deploy the device.
  • If the device receives traffic from load balancers, add the host to the load balancer pool.
  • If the device hosts a service, create a new Almanac device entry and bind it to new or existing services.

This process is not as automated as it could be, but we an automate it as the need arises. The more complicated and error-prone parts are automated.

Launching Instances

Before launching instances, consider these cases:

DeviceConsiderations
adminDo not launch multiple admin devices without making software changes to support this first.
bastionRequires master key. Use --master. Copy master key to /core/conf/keys/master.key after deployment.
notifyThese need software changes to scale up.
secureRequires software changes to scale up.
vaultThese are tricky to launch properly.

Launch instances using these settings:

DeviceAMITypeSecurity GroupKeyVPC
adminami-5c120b19m3.largeinstance.adminphacility-masterblock001.phacility.net
auxami-5c120b19m3.largeinstance.auxphacility-masterblock001.phacility.net
bastionami-5c120b19m3.largeinstance.bastionphacility-masterblock001.phacility.net
dbami-5c120b19m3.largeinstance.dbphacility-masterblock001.phacility.net
notifyami-5c120b19m3.largeinstance.notifyphacility-masterblock001.phacility.net
repoami-5c120b19m3.largeinstance.repophacility-masterblock001.phacility.net
secureami-5c120b19m3.largeinstance.securephacility-masterblock001.phacility.net
sauxami-5c120b19m3.largeinstance.sauxphacility-master block001.sbuild.phacility.net
sbuildami-5c120b19m3.largeinstance.sbuildphacility-master block001.sbuild.phacility.net
vaultami-5c120b19m3.largeinstance.vaultphacility-masterblock001.phacility.net
webami-5c120b19m3.largeinstance.webphacility-masterblock001.phacility.net

Adding DNS

For all devices, add a phacility.net DNS entry in Route 53 with the internal (VPC) device IP address. This should be an "A" record, similar to the other device records. The UI defaults are generally correct.

Set the name of the device in the EC2 console to be the same as the DNS entry you selected.

For example, if you launch a new web instance, you might call it web007.phacility.net (you should use the next number which doesn't have a DNS entry yet, of course).

EBS Storage

These types of devices have persistent EBS storage:

DeviceStorage Types
adminadata, abak
auxauxdata
dbddata, dbak
repordata, rbak, rlog
securesdata, srepo, sbak, slog
sauxsauxdata
sbuildsbuilddata

If you're launching a device which uses one or more kinds of persistent EBS storage, you need to create and attach EBS volumes to the device. Use these settings for each storage type:

StorageSizeAttach To
abak64GB/dev/sdf
adata64GB/dev/sdh
auxdata64GB/dev/sdh
dbak128GB/dev/sdf
ddata256GB/dev/sdh
rbak256GB/dev/sdf
rdata64GB/dev/sdg
rlog32GB/dev/sdj
sbak64GB/dev/sdf
sauxdata64GB/dev/sdh
sbuilddata64GB/dev/sdh
sdata128GB/dev/sdh
slog32GB/dev/sdj
srepo32GB/dev/sdg
swap-/dev/sdi

For example, if you are brining up a db device, create an adata volume and attach it to /dev/sdh. Then create an abak volume and attach it to /dev/sdf.

Name any volumes you create with phacility.net names, for example dbak007.phacility.net.

Storage devices should have the same number as the hosts they are attached to. For example, db007 should use ddata007 and dbak007.

Volumes which will have backups written to them need device entries in Almanac on admin.phacility.com. These entries are used to store backup logs.

Deploy

Using the CLI tools in rCORE, deploy the device (for example, web007):

core/ $ ./bin/remote deploy web007

The deploy workflow automatically runs the upgrade and restart workflows.

Add Devices to Load Balancers

If you've launched a device which is part of a load balancer pool, add it to the appropriate load balancer:

DeviceAdd ToNotes
adminalbShould you really be launching these?
weblbShould also be added to vault but that's muddy.
notifynlbNo process for putting multiple nlb nodes into production yet.

Other devices do not need to be added to load balancer pools.

Update Almanac Services

If you've launched a device which provides a cluster service, create a device record for it in Almanac on admin.phacility.com, then attach it to an existing Almanac Service (to add or replace a device in a service pool) or create a new Almanac Service for it (to add a new service pool).

Create device records for any storage devices you created, too.

All services have only one host in them today, so this is largely future work.

Last Author
epriestley
Last Edited
Nov 13 2016, 12:53 AM

Event Timeline

epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)
epriestley edited the content of this document. (Show Details)