Here's another clue, from the relevant host's error log:

Yeah, I'm pretty pessimistic about the security of SMS (or voice) as a factor. See also T7639 and T6549#97130. The article identifies these vectors:

I'm not really sold on this use case, but I think the result is unintuitive/buggy enough to fix anyway.

Oh, sorry, maybe I wasn't clear. My plan is this:

Yes, we could migrate these instead.

I think this is a general problem with all JSON config types. The code for this is a bit messy and mixes "builtin" and "custom" types, but should probably just be rebuilt to not have all that weird magic.

This came from D18028, when I changed the Include stories about projects I am a member of. checkbox to be a CheckboxesField. I think the issue is that existing saved queries may have the old value for the field stored (true), instead of the new value (array('self')). These queries would be impossible to construct in the UI now.

I'll see if I can smooth this over a little before installs figure it out.

After some additional poking, there are a couple of other compatibility issues here:

• I filed T12854 for final cleanup.
• I stopped repo012.phacility.net.
• The repo tier now has a discontinuity. --timeout will "fix" it but this should be fixed properly, except that we're presumably throwing away Phage in favor of Chef/Puppet/Ansible in T12847 which will moot this.

I started the instance, then used bin/remote deploy to deploy it. Everything came back up cleanly with no additional steps. Things were back online after about 10 minutes (maybe a bit less).

AWS stopped the instance; I'm starting it again now.

Another possibility is that since we need only a very small number of connections between subnets (database replication and VCS over SSH only, I think), we could enumerate everything and then tunnel/forward them with something like stunnel/ssh/haproxy rather than a full-blown VPN. That also feels like kind of a mess, but maybe not as much of a mess as doing a full VPN setup. It would probably look something like the OpenSWAN diagram, except the OpenSWAN boxes would be haproxy/stunnel boxes or whatever instead that just forwarded a specific set of ports between devices:

If we do VPC-level VPNs, what network level safeguard prevents web001.evil-attacker.us-west.phacility.com from connecting directly to db001-master.wholesome-good-company.us-east.phacilty.com? Subnet ACLs can no longer enforce this, right? All the traffic arriving on the east coast will originate from the local VPN endpoint, not from the corresponding west-coast subnet? Or am I misunderstanding how this works?

We also technically do something like this in the production cluster today with bin/services sync, which (among other things) synchronizes Almanac services from the admin tier. The script currently just cheats and forces them to allocate with the same PHIDs, so we know local service X is the same as remote service X because they have the same PHID. I'm not sure we'd necessarily change this if we had a real "utilities" API but it's definitely a bit hacky as written today.

I'm a bit out of my depth here -- what's the alternative you're suggesting? That we have VPC <-> VPC VPNs instead of subnet <-> subnet VPNs? What, specifically, is the "just use TLS" plan?

I think the worst possible outcome is that we throw it all away for some reason we don't currently forsee, you learn a lot more about how the cluster works in the process, and I can answer "Why aren't you using Chef/Ansible/Puppet?" when we make the next hire.

I think we just have two very different models of where operational risk lies and how to hedge against it -- the risks you associate with first-party tooling are risks I associate with third-party tooling. We can't really objectively prove that either model is better because failures are rare in either case. I think we're worse off trying to compromise the two models than picking either one and sticking to it.

I'll also suggest that we should implement some kind of "infrastructure as code" tool

Today, we read config from admin.phacility.com on each request. This is generally good for shared clusters, although T12646 discusses some issues.

Instances should probably still have short display names (like "epriestley" corresponding to epriestley.phacility.com) for tooling consistency and, at least today, for mail. It's annoying/inconvenient if we can never call this instance "epriestley" and always have to call it "phabricator.epriestley.com". After T11413, they will also have possibly-different internal names, perhaps inst-abc or PHID-INST-abcdfe.

Each private cluster should be on its own subnet. If/when we let installs run custom code on an instance and it's a security nightmare and roots the box, no other customer should be vulnerable.

At least, not until we launch Phacility Apps for Domains and rebrand it to PSuite a few years later.

(We can't ever host bugs@epriestley.com directly anyway -- that always has to forward -- and that's probably what installs would want if they care about any of this.)

no lol

The main domain name, phabricator.epriestley.com, must be pointed at the ALB.

8a:7a:a7:de:52:8e:eb:7a:de:9b:73:c4:9f:88:13:9d

SSL Certificates and Load Balancers

I think there are broadly two halves to this which we can think about mostly separately:

Maybe another capability here is just letting callers put a random bag of properties on any object. I don't reaaaaally have use cases for that but it "seems powerful" if we're building all of this already at some point.

No.

I think this is worth acting on (by changing MySQL sandboxing or config sandboxing or some combination of things) if:

Oh, I don't think anyone should ever be running unit tests with actual production credentials. You should assume that arc unit will immediately escape all sandboxing and destroy all the data it can touch, then run BENCHMARK() for an hour and then crash the server. But you should assume arc unit will do this even if it is "supposed" to be Totally Seriously Sandboxed 100% into SQLite.

I'm not particularly strongly opposed to adding more options here, I just want to avoid the dropdown becoming this:

For the compliance case, you should already be able to trigger this stronger rule ("always add the package as a reviewer, even if an owner authored it") with Herald, by writing this rule:

I can't really come up with a problem that would prevent, even theoretically -- do you have a specific scenario in mind offhand where some kind of MySQL configuration could mask a test failure locally?

This is semi-intended, because we have to pull a lot of config (like mysql.host) from your local install for things to make sense: we can't create test fixtures if we don't know how to connect to the database.

The use case in this task's description does not make sense to me.

Oh, this doesn't isolate things because they're on different databases, and thus we establish different connections. The daemon insert does not happen inside a transaction.

We actually do this ourselves in the "Locations" technical interview scenario: the script tries to create a set of tasks, or update them if they already exist. It currently uses title-based search to accomplish this, but could more reliably use "utility labels" if they existed.

We do something similar ourselves for technical interviews, but I just related all the tasks by embedding them in the description of a central task:

I think this use case is a reasonable one.

Yeah -- the workflow from the Maniphest side dumps all the values into inputs in the body and does a POST, I believe.

@swisspol hit a variant of this with 1,172 tasks bumping up against an Apache URL limit in the Phacility cluster when editing from a workboard column.

I used the current version, but there should be no meaningful changes to this behavior between your version and HEAD.

I can't reproduce this. Here's what I did:

How did you generate this diff?

I moved everything except one test instance off in T12817, and we appear to be ready for this host to die.

I believe this should now be resolved in HEAD of master. Let us know if it doesn't work for you or if you're still seeing issues after upgrading. Thanks for the report!

No one has complained that all their data is gone so I'm going to assume the best here.

This is not a configuration problem, and you can't fix it by changing configuration. D18119 should fix it.

Presuming resolved by T12757 (May 25).