They should fix themselves once the instance is suspended.

Just keeping an eye on this: growth on these shards is slow and both are still below the 80% caution threshold, so I'm punting this for another week.

Only db001 and db002 are anywhere close to having problems with this and they still have a large amount of headroom, and we can bin/storage optimize at least some amount of additional headroom into existence, so this isn't "we're about to fall off a cliff" urgent or anything, but would be nice to tackle sooner than later.

I think the minimum size is 2 hosts, in separate AZs, in one region. e.g., everything1 in us-west-1a and everything2 in us-west-1c, so that assumption is sound.

Are we going to let customers start with a single EC2 instance, or require them to have at least some form of HA? My plan is to always create the VPC/subnet infrastructure assuming that the customer will have a second AZ (and ALBs for example require you to listen on at least two different subnets).

I'll look at the pricing breakdown and I might do some perf testing just to see if there's any real difference.

If we're using ELBs for this (which it looks like we may be able to, per above?) I don't think (?) we can do any path-based stuff.

22280->22280 for aphlict (or we could possibly use path-based routing on the ALB instead of opening up a 2nd port?)

We should enumerate all the ports we're going to open up so we can do the security groups sooner rather than later. My list:

Cool, that all sounds like it's roughly what I expected. Thanks!

Skipping questions where you have the right answer:

This is very minor, but blue-underneathie is still active even though repo012 is dead, which is causing the queue to spawn impossible backup jobs that sit there retrying forever. They should fix themselves once the instance is suspended. I'll likely pursue the remaining cleanup here shortly since we haven't seen any issues in a few weeks.

Ah, that makes sense. Everything else sounds entirely reasonable to me on the "VPC per customer" plan.

In T12847#229598, @epriestley wrote:

bastion host per subnet

Is "subnet" a typo or am I thinking of the wrong subnets? Not 4 bastions per customer, right? Don't the bastions have to be in the customer VPCs?

bastion host per subnet

So having done a bunch more reading/thinking about private cluster isolation, I'm starting to lean towards "one VPC per customer per region" instead of "one subnet per customer per region".

In T12847#229548, @joshuaspence wrote:

AWS doesn't have a single device which can both listen for TCP on 22 (only ELB) and terminate SSL for websockets (only ALB).

That's not correct. We are routing our websocket traffic for Aphlict through an ELB that has been setup using SSL instead of TCP as the protocol. In Terraform, it looks like this:

It looks like the price difference is pretty large -- $45/month for a m4.large reserved instance vs $80/month for a m4.large dedicated -- plus ~$1,500/month/region for having at least one dedicated instance. It would be easy to start using dedicated hosts in the future, right? Just an issue of converting existing hosts if we run into issues?

We should also make a decision about whether or not we want to use "dedicated" AWS instances: https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/

Another correction, Puppet doesn't //need/ a dedicated server. You can run Puppet in a standalone (agent) mode, if you are willing to somehow get your Puppet manifests onto each host that you wish to provision.

In T12856#229564, @joshuaspence wrote:

In T12856#227621, @amckinley wrote:

Terraform Review
Pros:

• Written in Python, which everyone knows is superior to Ruby in every way

Huh? Terraform is written in Golang, not Python.

In T12856#227621, @amckinley wrote:

Terraform Review
Pros:

• Written in Python, which everyone knows is superior to Ruby in every way

FWIW, I am actively working on a Puppet module for Phabricator, see https://forge.puppet.com/joshuaspence/phabricator.

AWS doesn't have a single device which can both listen for TCP on 22 (only ELB) and terminate SSL for websockets (only ALB).

One thing I should add is that your different application tiers should be different stacks: i.e. DNS config should be one stack, web boxes another stack, DB another etc.

@hach-que thanks for the suggestion. I've used CloudFormation with some success previously. My biggest problem is occasionally having a stack just get "stuck" while deploying, and taking forever to timeout on a failure. I've also seen very ambiguous error messages when some random resource fails to deploy, but these are all just anecdotes. I'm working on a demo CF config to give it a try.

In my personal experience, CloudFormation is vastly better than Ansible/Chef/insertflavourofthemonthtoolhere. Those other tools all require some specialised syntax, meanwhile CF is just JSON and you can use the CloudFormation designer to get the JSON for any AWS resource that you're not sure how to describe with JSON.

Also we aren't the only ones to be surprised by the AWS NAT junk:

It looks like Terraform probably can add a listener without destroying the LB (since it's a separate resource type? But the "changing the size of a volume destroys the instance" case suggests maybe it doesn't work like this, if that's how that works), but if you want to enable access logging on your LBs hope you didn't need 'em for anything important? I could probably just install it and figure this out / nuke production forever.

If PHP just injects that information into the templates during generation, we might not need to feed anything back into Almanac.

Maybe we should just use CloudFormation ¯\_(ツ)_/¯

First, a disclaimer that I'm not by any means wedded to terraform. From playing with the other tools, terraform looked like the clearest pathway towards my goal of "minimizing AWS UI clicks", so I decided to see what the terraform config for a private cluster would look like and play with the tool. See example terraform configs here: P2063 P2064. I agree with the general thrust of your comment, which I interpret as "this all sounds terrifying and we probably shouldn't do it", but I wanted to at least get a feeling for the tool and build a proof of concept.

Terraform also seems to have terraform taint (mark a resource for destruction) but no, e.g., terraform freeze (mark a resource for preservation). This is sort of philosophically bewildering to me. Is the modern approach to ops just like "data isn't important"? Is everyone just really, really careful when they type commands into their fully-automatic resource destruction rifles loaded with high-explosive foot-seeking shells?

How does the weekly deployment to all clusters work?

(Both adjust and doc/symbol documentation affect an entire service tier rather than particular servers, too -- we only need to generate docs or make schema adjustments once, no matter how many hosts are in the secure or admin tiers.)

My recollection is a little fuzzy, but I think admin may have once had a hacky crontab for backups that I replaced with a normal backup trigger at some point when the infrastructure matured a little.

And here's the official AWS docs for connecting VPCs using OpenSWAN: https://aws.amazon.com/articles/5472675506466066

So it occurs to me that we could have one "shared" public subnet that contains: 1 shared Internet Gateway, 1 shared NAT Gateway, and then per-install VPN nodes. We should be able to setup routing tables for the private subnets that point to that install's VPN node. Since routing is enforced at the network layer (routing tables on actual EC2 nodes are bogus), that would enforce traffic separation. In theory, installs could saturate the IGW or NAT gateway, but I'm pretty sure those resources should autoscale since they're managed by EC2. A NAT gateway with no traffic going through it is $32/month, and as far as I can tell, an Internet Gateway is free, but eventually we'd be saving some money.

Another AWS gotcha learned the hard way: nodes in a "private" subnet can still be part of an "internet-facing" ELB, but the trick is that you have to attach the ELB to the "public" subnet that contains the IGW and the NAT Gateway, and the nodes in the "private" subnet need a route to the NAT Gateway.

Note that we can not send SSH traffic over an ALB -- they only speak HTTP/HTTPS. However, we can send it over v1.5 VPC ELB, apparently (v1 classic ELBs do not let you listen on 22).

On secure, I could only find a crontab for the ubuntu user, which is as follows:

0 6 * * * /core/bin/host backup
0 7 * * * /core/bin/host prune --force
0 8 * * * /core/conf/util/generate-documentation
0 9 * * * /core/conf/util/generate-symbols

Having played with a few more of these tools and thinking about the problem, I'm starting to lean towards "use Terraform or CloudFormation for orchestrating AWS infrastructure, and leave the provisioning stuff in rCORE as-is". The basic flow for provisioning private instances could be:

You probably don't want to put much value in advice from passing Internet strangers, but... Having used Ansible for configuration management for a while, it certainly has a "worse is better" feeling to it. You folks seem like you might be more of a "better is better" outfit. Contrast, for example, your approach to upstreaming user-written linters versus their approach to upstreaming user-written modules.

I think we should have our crontabs in version control regardless of whether or not we add tmpreaper to them, so I'll make a task for that.

If you want to move forward with that:

This should do the trick. It runs off atime by default. We could just set the time period to several days if we wanted to. Alternatively, if the filenames for extremely long-running jobs are predictable, there's a --protect '<shell_pattern>' argument we could use to avoid cleaning up those files.

Every repo host is equally affected, so I'd like to deploy crontabs as part of the regular deployment process if we use them as part of the approach here. That would require first codifying a handful of custom crontabs, including one on secure which regenerates documentation daily on only one host. This codification should happen anyway eventually, but it's a little bit of work, and wasted effort if we're switching to Chef/Salt/Ansible/etc soon anyway.

Should we just add a crontab entry to clean /tmp to paper this over until we get it fixed for real?

Here's another clue, from the relevant host's error log:

In T12856#227623, @rabahmeradi wrote:

Just out of curiosity, why Salt is not a candidate? I think it is very comparable to the others.

Just out of curiosity, why Salt is not a candidate? I think it is very comparable to the others.

Terraform Review
Pros:

• Simple declarative syntax
• Intended to be invoked first with a dry run, which shows in detail all the resources that will be rebuilt or altered
• Very focused on "immutable" infrastructure. Instead of manipulating resources in place, prefers to spin up new resources with the new config and spin down the old ones. For the stateful hosts, we might need to be clever about mounting and unmounting EBS volumes when instances get rebuilt.

• I filed T12854 for final cleanup.
• I stopped repo012.phacility.net.
• The repo tier now has a discontinuity. --timeout will "fix" it but this should be fixed properly, except that we're presumably throwing away Phage in favor of Chef/Puppet/Ansible in T12847 which will moot this.

I started the instance, then used bin/remote deploy to deploy it. Everything came back up cleanly with no additional steps. Things were back online after about 10 minutes (maybe a bit less).

AWS stopped the instance; I'm starting it again now.

Another possibility is that since we need only a very small number of connections between subnets (database replication and VCS over SSH only, I think), we could enumerate everything and then tunnel/forward them with something like stunnel/ssh/haproxy rather than a full-blown VPN. That also feels like kind of a mess, but maybe not as much of a mess as doing a full VPN setup. It would probably look something like the OpenSWAN diagram, except the OpenSWAN boxes would be haproxy/stunnel boxes or whatever instead that just forwarded a specific set of ports between devices:

what network level safeguard

If we do VPC-level VPNs, what network level safeguard prevents web001.evil-attacker.us-west.phacility.com from connecting directly to db001-master.wholesome-good-company.us-east.phacilty.com? Subnet ACLs can no longer enforce this, right? All the traffic arriving on the east coast will originate from the local VPN endpoint, not from the corresponding west-coast subnet? Or am I misunderstanding how this works?

To back up a bit, how much human touching are you envisioning for bring up a new private cluster? In my head I was thinking we'd at least have to manually deal with requesting the cert, but it looks like There's An API For That, so I think it's at least possible to make the whole thing work magically. It would be something like filling out the form triggers the creation of a new terraform (or whatever) config which then gets deployed.

That we have VPC <-> VPC VPNs instead of subnet <-> subnet VPNs?

I'm a bit out of my depth here -- what's the alternative you're suggesting? That we have VPC <-> VPC VPNs instead of subnet <-> subnet VPNs? What, specifically, is the "just use TLS" plan?

Okie dokie, I'll start evaluating.

All done.

AWS support case ID is 2243430271.

I think the worst possible outcome is that we throw it all away for some reason we don't currently forsee, you learn a lot more about how the cluster works in the process, and I can answer "Why aren't you using Chef/Ansible/Puppet?" when we make the next hire.

I think we just have two very different models of where operational risk lies and how to hedge against it -- the risks you associate with first-party tooling are risks I associate with third-party tooling. We can't really objectively prove that either model is better because failures are rare in either case. I think we're worse off trying to compromise the two models than picking either one and sticking to it.

I'm concerned that this may turn a 2-month project into a 6-month project if we write something ourselves, or a 12-month project if we use Ansible, Chef or Puppet.

I'll also suggest that we should implement some kind of "infrastructure as code" tool

Today, we read config from admin.phacility.com on each request. This is generally good for shared clusters, although T12646 discusses some issues.

I'll also suggest that we should implement some kind of "infrastructure as code" tool before going too far down the road of setting this stuff up. I'd really like to get to the point where no one ever has to push buttons in the AWS console unless something is really broken. I don't have super strong feelings about Ansible vs chef vs puppet, but I'd recommend against CFEngine and SaltStack.

Instances should probably still have short display names (like "epriestley" corresponding to epriestley.phacility.com) for tooling consistency and, at least today, for mail. It's annoying/inconvenient if we can never call this instance "epriestley" and always have to call it "phabricator.epriestley.com". After T11413, they will also have possibly-different internal names, perhaps inst-abc or PHID-INST-abcdfe.

Each private cluster should be on its own subnet. If/when we let installs run custom code on an instance and it's a security nightmare and roots the box, no other customer should be vulnerable.

At least, not until we launch Phacility Apps for Domains and rebrand it to PSuite a few years later.

(We can't ever host bugs@epriestley.com directly anyway -- that always has to forward -- and that's probably what installs would want if they care about any of this.)

no lol

As far as requesting certs goes, our "Request Certificate" button in the AWS console will work correctly by sending an email to the domain's owner:

The main domain name, phabricator.epriestley.com, must be pointed at the ALB.

SSL Certificates and Load Balancers

I think there are broadly two halves to this which we can think about mostly separately: