Page MenuHomePhabricator
Create Task
Maniphest T12983

Replace vault001 with an "ELB-VPC" load balancer
Closed, ResolvedPublic
Actions

Description

We currently run haproxy on a vault001 host, which purely serves as a port 22 TCP load balancer for the web tier.

From T12978, modern "ELB-VPC" LBs can balance TCP traffic on port 22, so this doesn't need to be a separate host. When I originally set this stuff up, port 22 either wasn't an option or I failed to identify how to configure it.

Since the pool behind vault.phacility.com is the same as the pool behind *.phacility.com and will be for the foreseeable future, this can actually just be folded into the lb ELB. So the plan is probably:

  • Add a 22 -> 2223 to lb001.
  • Locally, hostfile vault.phacility.com DNS to whatever anything.phacility.com is resolving to (ELB external address).
  • Test that it works.
  • Point DNS for vault.phacility.com at the lb001 ELB.
  • Test that it works.
  • Decommission the vault001 host, the vault role, and nuke all the HAProxy config and install operations from core/. None of this is stateful so it doesn't need any special care.

Related Objects
Search...

Event Timeline

epriestley created this task.Sep 14 2017, 5:07 PM
Herald added subscribers: chad, eadler. · View Herald TranscriptSep 14 2017, 5:07 PM
epriestley added a subscriber: amckinley.Sep 15 2017, 4:26 PM

+ @amckinley

I'm going to take a stab at this now since I think it's non-disruptive and straightforward.

epriestley added a comment.Sep 15 2017, 4:38 PM
  • I opened up 22 -> 2223 on lb001.
  • I allowed external 22 in the security group.
  • I hard-coded my hostfile and cloned successfully:
$ grep vault /etc/hosts
52.8.162.129 vault.phacility.com

$ git clone ssh://meta@vault.phacility.com/source/pohems.git
Cloning into 'pohems'...
# Fetch received by "web.phacility.net", forwarding to cluster host.
# Waiting up to 120 second(s) for a cluster read lock on "repo001.phacility.net"...
# Acquired read lock immediately.
# Device "repo001.phacility.net" is already a cluster leader and does not need to be synchronized.
# Cleared to fetch on cluster host "repo001.phacility.net".
remote: Counting objects: 16836, done.
remote: Compressing objects: 100% (312/312), done.
remote: Total 16836 (delta 483), reused 16836 (delta 483)
Receiving objects: 100% (16836/16836), 724.82 KiB | 1.18 MiB/s, done.
Resolving deltas: 100% (483/483), done.
Checking out files: 100% (16024/16024), done.

I'm going to:

  • Swap DNS.
  • Leave vault001 around for now in case issues come up and since DNS propagation isn't instantaneous.
  • Probably decommission it tomorrow during PhabOpsConf2017.
epriestley added a comment.Sep 15 2017, 4:40 PM

Swap DNS.

I've made this change, so vault.phacility.com now points at lb001.phacility.net (via Route53 alias magic).

arend.danielek added a subscriber: arend.danielek.Sep 15 2017, 10:37 PM
epriestley mentioned this in T12978: Phacility Cluster Maintenance: 2017 Week 37.Sep 16 2017, 4:28 PM
epriestley added a parent task: T12978: Phacility Cluster Maintenance: 2017 Week 37.
epriestley closed this task as Resolved.Sep 16 2017, 7:10 PM
epriestley claimed this task.

vault002 is dead. Long live lb001.

Phabricator · Built by Phacility