Page MenuHomePhabricator

Replace vault001 with an "ELB-VPC" load balancer
Closed, ResolvedPublic


We currently run haproxy on a vault001 host, which purely serves as a port 22 TCP load balancer for the web tier.

From T12978, modern "ELB-VPC" LBs can balance TCP traffic on port 22, so this doesn't need to be a separate host. When I originally set this stuff up, port 22 either wasn't an option or I failed to identify how to configure it.

Since the pool behind is the same as the pool behind * and will be for the foreseeable future, this can actually just be folded into the lb ELB. So the plan is probably:

  • Add a 22 -> 2223 to lb001.
  • Locally, hostfile DNS to whatever is resolving to (ELB external address).
  • Test that it works.
  • Point DNS for at the lb001 ELB.
  • Test that it works.
  • Decommission the vault001 host, the vault role, and nuke all the HAProxy config and install operations from core/. None of this is stateful so it doesn't need any special care.

Event Timeline

+ @amckinley

I'm going to take a stab at this now since I think it's non-disruptive and straightforward.

  • I opened up 22 -> 2223 on lb001.
  • I allowed external 22 in the security group.
  • I hard-coded my hostfile and cloned successfully:
$ grep vault /etc/hosts

$ git clone ssh://
Cloning into 'pohems'...
# Fetch received by "", forwarding to cluster host.
# Waiting up to 120 second(s) for a cluster read lock on ""...
# Acquired read lock immediately.
# Device "" is already a cluster leader and does not need to be synchronized.
# Cleared to fetch on cluster host "".
remote: Counting objects: 16836, done.
remote: Compressing objects: 100% (312/312), done.
remote: Total 16836 (delta 483), reused 16836 (delta 483)
Receiving objects: 100% (16836/16836), 724.82 KiB | 1.18 MiB/s, done.
Resolving deltas: 100% (483/483), done.
Checking out files: 100% (16024/16024), done.

I'm going to:

  • Swap DNS.
  • Leave vault001 around for now in case issues come up and since DNS propagation isn't instantaneous.
  • Probably decommission it tomorrow during PhabOpsConf2017.

Swap DNS.

I've made this change, so now points at (via Route53 alias magic).

epriestley claimed this task.

vault002 is dead. Long live lb001.