• We launched a new private subnet (private001.phacility.net) yesterday.
• I launched a new instance (nat-test001.phacility.net) into this subnet, with no public IP.
• I can connect to the instance from the bastion host (bastion005), which is good.
• Problem: I can not connect to the public internet from the test host.

I suspect this is because the private subnet is still using the block001 route table, which uses an IGW. I'm building a new route table to see if that fixes things.

• I built a new route table, following the main route table.
• I replaced the target for the 0.0.0.0/0 rule with nat-... instead of igw-....
• Under "Endpoints", I added the new route table to the S3 endpoint so the private subnet can access S3.
  • (This wasn't immediately reflected in the "Routes" UI, but it's not important for now so I'm going to assume it's a propagation issue until I need it.)
  • (It showed up about a minute later.)
• I changed private001.phacility.net to use the new route table, with nat-... instead of igw-....
• I can still connect to the test host from the bastion.
• Problem: I still can't connect to the public internet from the test host.

I'm going to try just relaunching the test host since I suspect this might be a route caching issue or something like that and I don't know enough about CLI routing commands to tell, but I do know how to turn it off and on again. I also can't, e.g., install traceroute since I can't reach the package server on the public internet.

• I stopped the host, and got kicked off as expected.
• I started the host.
• Problem: This failed to magically fix anything, and I still can't connect to the public internet.

I'm out of obvious buttons to click so I'm going to read the NAT Gateway documentation.

I think the problem is that the NAT gateway is currently in the private001 subnet, but needs to be in the public subnet (the NAT gateway itself must have a route to the public internet).

To ensure that your NAT gateway can access the Internet, the route table associated with the subnet in which your NAT gateway resides must include a route that points Internet traffic to an Internet gateway.
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html

See also this diagram -- note that the NAT gateway has been drawn inside the public subnet:

I'm going to try launching a new NAT gateway in the public subnet, then changing the route table to use that gateway instead.

• I launched a new NAT gateway into the public subnet (block001); nat-0d470fc54d846a993.
• I reconfigured the private subnet route table to point at the new gateway instead of the old one.
• I can now reach the public internet from the host on the private subnet:

ubuntu@ip-172-30-2-166:~$ curl https://ifconfig.co
13.56.71.101

• This is the correct IP address too (the elastic IP bound to the NAT gateway).
• Since the other NAT gateway in the private subnet appears to be useless, I deleted it (nat-094a2518c0c363c5d).

I think this means we don't need to migrate every host: the problem with swapping block001 over to be NAT'd was probably that the NAT gateway itself couldn't reach the public internet. When we changed the 0.0.0.0/0 rule to nat-..., that blocked the NAT gateway itself from getting out.

So I think we could do this, instead:

• Launch a new public subnet.
• Move external-facing servers (bastion, vault, notify) to that subnet. Or could do the alb swap for notify first and move just bastion and vault, I think.
• Also launch a NAT gateway into that subnet (not the existing subnet which we want to make private).
• Give the public subnet a route table with an IGW to make it public.
• When ready to flip the switch, change the main route table rule for "0.0.0.0/0": IGW → NAT in Public Subnet.

The worst case if that doesn't work is that we fall back to the "move everything" plan and moved a couple of simple services we didn't really need to move, but I'm pretty sure it will work since I effectively just did it on the private001 subnet (that is, there was nothing making it inherently "private").

As this pertains to T12798, I'm inclined to launch repo025 into the existing subnet and plan on trying the "public + move vault/bastion + swap to NAT" plan at a later date. It seems like there probably aren't too many dependencies/interactions between the two changes.

Since I don't plan to launch repo025 into a separate subnet, I'm going to clean up these resources:

• The NAT gateway I just launched.
• The new route table I built for the subnet.
• The subnet itself.
• The nat-test001 host.

I cleaned up all that stuff and it doesn't look like I broke the world.

It also seems possible that this reduced-complexity approach might work:

• Launch a new public subnet.
• Put ONLY the NAT gateway in it.
• Point block001 at NAT instead of IGW.
• Maybe everything magically works with no additional changes, since inbound traffic still just hits the IPv4 public IPs?

But even if this did work, I think it's worth the effort to separate the subnets -- we don't get anything super concrete out of it, but feels much cleaner from a general security/isolation viewpoint.

I think it makes sense in a private cluster world too, where private clusters are probably (?) on dedicated subnets, not necessarily dedicated VPCs, and we want to block traffic between the shared/instanced cluster and private clusters.