My expectation is that this action can never appear -- this is the pre-Drydock hacky temporary action -- so something troubling is afoot.

See also T10005 (bulk editors for all applications) and T6722.

I think a minimal reproduction case which is typical of this example is:

Ah, alright. Yeah, behavior is:

Oh, maybe I misunderstood. I thought you wanted an easier way to test changes in a development environment, but it sounds like you actually want end users to immediately get the new images instead of needing to wait 7 days for them?

A slightly more aggressive fix would be to make these have a 5 second TTL in development or something (or hash the file modification time into the builtin key? But that's a bit tricky...) so you'd never have to purge the cache, but I think this arises rarely enough that it isn't too valuable.

Strictly speaking, you still have to do a "manual purge" (with bin/cache ...) after D18146, but I assume you just didn't want to have to fish around in the DB.

Also not sure how to "retire" ones from Projects I don't intend to replace (low value).

If you want to move forward with that:

Great! Please mail support at support@phacility.com and identify which instance you're paying us for, then we're happy to help you. We won't help you here, since this isn't the right way for customers to get support.

Every repo host is equally affected, so I'd like to deploy crontabs as part of the regular deployment process if we use them as part of the approach here. That would require first codifying a handful of custom crontabs, including one on secure which regenerates documentation daily on only one host. This codification should happen anyway eventually, but it's a little bit of work, and wasted effort if we're switching to Chef/Salt/Ansible/etc soon anyway.

If you're a Phacility customer, please mail support at support@phacility.com (and identify your instance).

Why is git diff --no-index being used instead of and diff, and why is --no-prefix being used?

Here's another clue, from the relevant host's error log:

Yeah, I'm pretty pessimistic about the security of SMS (or voice) as a factor. See also T7639 and T6549#97130. The article identifies these vectors:

I'm not really sold on this use case, but I think the result is unintuitive/buggy enough to fix anyway.

Oh, sorry, maybe I wasn't clear. My plan is this:

Yes, we could migrate these instead.

I think this is a general problem with all JSON config types. The code for this is a bit messy and mixes "builtin" and "custom" types, but should probably just be rebuilt to not have all that weird magic.

This came from D18028, when I changed the Include stories about projects I am a member of. checkbox to be a CheckboxesField. I think the issue is that existing saved queries may have the old value for the field stored (true), instead of the new value (array('self')). These queries would be impossible to construct in the UI now.

I'll see if I can smooth this over a little before installs figure it out.

After some additional poking, there are a couple of other compatibility issues here:

• I filed T12854 for final cleanup.
• I stopped repo012.phacility.net.
• The repo tier now has a discontinuity. --timeout will "fix" it but this should be fixed properly, except that we're presumably throwing away Phage in favor of Chef/Puppet/Ansible in T12847 which will moot this.

I started the instance, then used bin/remote deploy to deploy it. Everything came back up cleanly with no additional steps. Things were back online after about 10 minutes (maybe a bit less).

AWS stopped the instance; I'm starting it again now.

Another possibility is that since we need only a very small number of connections between subnets (database replication and VCS over SSH only, I think), we could enumerate everything and then tunnel/forward them with something like stunnel/ssh/haproxy rather than a full-blown VPN. That also feels like kind of a mess, but maybe not as much of a mess as doing a full VPN setup. It would probably look something like the OpenSWAN diagram, except the OpenSWAN boxes would be haproxy/stunnel boxes or whatever instead that just forwarded a specific set of ports between devices:

If we do VPC-level VPNs, what network level safeguard prevents web001.evil-attacker.us-west.phacility.com from connecting directly to db001-master.wholesome-good-company.us-east.phacilty.com? Subnet ACLs can no longer enforce this, right? All the traffic arriving on the east coast will originate from the local VPN endpoint, not from the corresponding west-coast subnet? Or am I misunderstanding how this works?

We also technically do something like this in the production cluster today with bin/services sync, which (among other things) synchronizes Almanac services from the admin tier. The script currently just cheats and forces them to allocate with the same PHIDs, so we know local service X is the same as remote service X because they have the same PHID. I'm not sure we'd necessarily change this if we had a real "utilities" API but it's definitely a bit hacky as written today.

I'm a bit out of my depth here -- what's the alternative you're suggesting? That we have VPC <-> VPC VPNs instead of subnet <-> subnet VPNs? What, specifically, is the "just use TLS" plan?

I think the worst possible outcome is that we throw it all away for some reason we don't currently forsee, you learn a lot more about how the cluster works in the process, and I can answer "Why aren't you using Chef/Ansible/Puppet?" when we make the next hire.

I think we just have two very different models of where operational risk lies and how to hedge against it -- the risks you associate with first-party tooling are risks I associate with third-party tooling. We can't really objectively prove that either model is better because failures are rare in either case. I think we're worse off trying to compromise the two models than picking either one and sticking to it.

I'll also suggest that we should implement some kind of "infrastructure as code" tool

Today, we read config from admin.phacility.com on each request. This is generally good for shared clusters, although T12646 discusses some issues.

Instances should probably still have short display names (like "epriestley" corresponding to epriestley.phacility.com) for tooling consistency and, at least today, for mail. It's annoying/inconvenient if we can never call this instance "epriestley" and always have to call it "phabricator.epriestley.com". After T11413, they will also have possibly-different internal names, perhaps inst-abc or PHID-INST-abcdfe.

Each private cluster should be on its own subnet. If/when we let installs run custom code on an instance and it's a security nightmare and roots the box, no other customer should be vulnerable.

At least, not until we launch Phacility Apps for Domains and rebrand it to PSuite a few years later.

(We can't ever host bugs@epriestley.com directly anyway -- that always has to forward -- and that's probably what installs would want if they care about any of this.)

no lol

The main domain name, phabricator.epriestley.com, must be pointed at the ALB.

8a:7a:a7:de:52:8e:eb:7a:de:9b:73:c4:9f:88:13:9d

SSL Certificates and Load Balancers

I think there are broadly two halves to this which we can think about mostly separately:

Maybe another capability here is just letting callers put a random bag of properties on any object. I don't reaaaaally have use cases for that but it "seems powerful" if we're building all of this already at some point.

No.

I think this is worth acting on (by changing MySQL sandboxing or config sandboxing or some combination of things) if:

Oh, I don't think anyone should ever be running unit tests with actual production credentials. You should assume that arc unit will immediately escape all sandboxing and destroy all the data it can touch, then run BENCHMARK() for an hour and then crash the server. But you should assume arc unit will do this even if it is "supposed" to be Totally Seriously Sandboxed 100% into SQLite.

I'm not particularly strongly opposed to adding more options here, I just want to avoid the dropdown becoming this:

For the compliance case, you should already be able to trigger this stronger rule ("always add the package as a reviewer, even if an owner authored it") with Herald, by writing this rule:

I can't really come up with a problem that would prevent, even theoretically -- do you have a specific scenario in mind offhand where some kind of MySQL configuration could mask a test failure locally?

This is semi-intended, because we have to pull a lot of config (like mysql.host) from your local install for things to make sense: we can't create test fixtures if we don't know how to connect to the database.

The use case in this task's description does not make sense to me.

Oh, this doesn't isolate things because they're on different databases, and thus we establish different connections. The daemon insert does not happen inside a transaction.

We actually do this ourselves in the "Locations" technical interview scenario: the script tries to create a set of tasks, or update them if they already exist. It currently uses title-based search to accomplish this, but could more reliably use "utility labels" if they existed.

We do something similar ourselves for technical interviews, but I just related all the tasks by embedding them in the description of a central task: