Just out of curiosity, why Salt is not a candidate? I think it is very comparable to the others.

Terraform Review
Pros:

• Simple declarative syntax
• Intended to be invoked first with a dry run, which shows in detail all the resources that will be rebuilt or altered
• Very focused on "immutable" infrastructure. Instead of manipulating resources in place, prefers to spin up new resources with the new config and spin down the old ones. For the stateful hosts, we might need to be clever about mounting and unmounting EBS volumes when instances get rebuilt.

Ahhh, that makes more sense. I parsed that as saying "we could migrate the code" instead of the saved objects. If you want to add the glue code now and then assign it back to me after, I'll work on the migration in my Copious Free Time™.

Oh, sorry, maybe I wasn't clear. My plan is this:

If you're happy long-term with accepting all three types, I'm ok with that as a permanent fix. I'm planning on working through this week, just with weird/inconsistent hours.

Yes, we could migrate these instead.

Are saved queries/EditEngine forms things we could write migrations for? That might be even more brittle (for example, what to do about saved queries for priorities that no longer exist on the install), but accepting three different priority specifications of two different types seems a little too Postel's Principle for me.

I think this is a general problem with all JSON config types. The code for this is a bit messy and mixes "builtin" and "custom" types, but should probably just be rebuilt to not have all that weird magic.

This came from D18028, when I changed the Include stories about projects I am a member of. checkbox to be a CheckboxesField. I think the issue is that existing saved queries may have the old value for the field stored (true), instead of the new value (array('self')). These queries would be impossible to construct in the UI now.

I'll see if I can smooth this over a little before installs figure it out.

After some additional poking, there are a couple of other compatibility issues here:

• I filed T12854 for final cleanup.
• I stopped repo012.phacility.net.
• The repo tier now has a discontinuity. --timeout will "fix" it but this should be fixed properly, except that we're presumably throwing away Phage in favor of Chef/Puppet/Ansible in T12847 which will moot this.

I started the instance, then used bin/remote deploy to deploy it. Everything came back up cleanly with no additional steps. Things were back online after about 10 minutes (maybe a bit less).

AWS stopped the instance; I'm starting it again now.

It was restricted as a prototype for testing, I don't believe there was an originating task, maybe T8250. If you liked it, let us know!

Please remember to update Phabricator before filing bug reports. This UI no longer exists.

My guess though is the Mac your using is a laptop without a mouse. We didn't display the bar on "touch" interfaces, which MacOS + Trackpad registers as.

Please remember to update Phabricator before filing bug reports. This UI no longer exists.

haha

I was going to wait with updating this task until I'm finished but I'm stuck on GoCD side for some time. The extension for Phabricator is 99% ready so anyone interested may give it a try:
https://github.com/kszatan/harbormaster-gocd-buildstep
It's enough to be able to schedule a pipeline in GoCD but nothing more. This issue hampers my progress with notifying Harbormaster about pipeline status. I'll update this task when all is working as planned.

done! cheers

I didn't see this bug report earlier and was about to...start sprinkling error_log() everywhere to see what was going wrong. (What better way to spend Friday afternoon, right?)

Another possibility is that since we need only a very small number of connections between subnets (database replication and VCS over SSH only, I think), we could enumerate everything and then tunnel/forward them with something like stunnel/ssh/haproxy rather than a full-blown VPN. That also feels like kind of a mess, but maybe not as much of a mess as doing a full VPN setup. It would probably look something like the OpenSWAN diagram, except the OpenSWAN boxes would be haproxy/stunnel boxes or whatever instead that just forwarded a specific set of ports between devices:

what network level safeguard

If we do VPC-level VPNs, what network level safeguard prevents web001.evil-attacker.us-west.phacility.com from connecting directly to db001-master.wholesome-good-company.us-east.phacilty.com? Subnet ACLs can no longer enforce this, right? All the traffic arriving on the east coast will originate from the local VPN endpoint, not from the corresponding west-coast subnet? Or am I misunderstanding how this works?

To back up a bit, how much human touching are you envisioning for bring up a new private cluster? In my head I was thinking we'd at least have to manually deal with requesting the cert, but it looks like There's An API For That, so I think it's at least possible to make the whole thing work magically. It would be something like filling out the form triggers the creation of a new terraform (or whatever) config which then gets deployed.

That we have VPC <-> VPC VPNs instead of subnet <-> subnet VPNs?

Milestones works great for this, but you still have to manually create the workboard (just going into workboard and clicking create - milestones show up).

We also technically do something like this in the production cluster today with bin/services sync, which (among other things) synchronizes Almanac services from the admin tier. The script currently just cheats and forces them to allocate with the same PHIDs, so we know local service X is the same as remote service X because they have the same PHID. I'm not sure we'd necessarily change this if we had a real "utilities" API but it's definitely a bit hacky as written today.

I'm a bit out of my depth here -- what's the alternative you're suggesting? That we have VPC <-> VPC VPNs instead of subnet <-> subnet VPNs? What, specifically, is the "just use TLS" plan?

That does sound powerful. +1

Okie dokie, I'll start evaluating.

All done.

AWS support case ID is 2243430271.

I think the worst possible outcome is that we throw it all away for some reason we don't currently forsee, you learn a lot more about how the cluster works in the process, and I can answer "Why aren't you using Chef/Ansible/Puppet?" when we make the next hire.

I think we just have two very different models of where operational risk lies and how to hedge against it -- the risks you associate with first-party tooling are risks I associate with third-party tooling. We can't really objectively prove that either model is better because failures are rare in either case. I think we're worse off trying to compromise the two models than picking either one and sticking to it.

I'm concerned that this may turn a 2-month project into a 6-month project if we write something ourselves, or a 12-month project if we use Ansible, Chef or Puppet.

I'll also suggest that we should implement some kind of "infrastructure as code" tool

Today, we read config from admin.phacility.com on each request. This is generally good for shared clusters, although T12646 discusses some issues.

I'll also suggest that we should implement some kind of "infrastructure as code" tool before going too far down the road of setting this stuff up. I'd really like to get to the point where no one ever has to push buttons in the AWS console unless something is really broken. I don't have super strong feelings about Ansible vs chef vs puppet, but I'd recommend against CFEngine and SaltStack.

Instances should probably still have short display names (like "epriestley" corresponding to epriestley.phacility.com) for tooling consistency and, at least today, for mail. It's annoying/inconvenient if we can never call this instance "epriestley" and always have to call it "phabricator.epriestley.com". After T11413, they will also have possibly-different internal names, perhaps inst-abc or PHID-INST-abcdfe.

Each private cluster should be on its own subnet. If/when we let installs run custom code on an instance and it's a security nightmare and roots the box, no other customer should be vulnerable.

At least, not until we launch Phacility Apps for Domains and rebrand it to PSuite a few years later.

(We can't ever host bugs@epriestley.com directly anyway -- that always has to forward -- and that's probably what installs would want if they care about any of this.)

no lol

As far as requesting certs goes, our "Request Certificate" button in the AWS console will work correctly by sending an email to the domain's owner:

The main domain name, phabricator.epriestley.com, must be pointed at the ALB.

8a:7a:a7:de:52:8e:eb:7a:de:9b:73:c4:9f:88:13:9d

@chad, @epriestley - I think this is a reasonable question.

SSL Certificates and Load Balancers

What is the problem? What is wrong/not working?

I think there are broadly two halves to this which we can think about mostly separately:

I'm not sure I understand something:

I am running into this issue too and am on commits:

Maybe another capability here is just letting callers put a random bag of properties on any object. I don't reaaaaally have use cases for that but it "seems powerful" if we're building all of this already at some point.

Okay thanks for your answer. So if i used audit.Query to get (for example) audit statuses etc.. I have to wait that diffusion.commit.search implements all functionnalities?

No.

Is it possible to unfreeze audit.query if diffusion.commit.search is not returning enough informations to replace audit.Query?