See T12996 for spiritual followup.

This is now in stable, and deployed here and to the Phacility production cluster.

Modern flow is:

Some followup SHOW INDEXES from the case in PHI47 provided output which didn't have any evidence of key cardinality issues, so it seems less likely that this is a cardinality problem.

An unusual case here is Herald rules which take the "Send me an email" action, and act "only the first time". Their current behavior will be to activate when the draft is created, have their mail killed later in the pipeline when the Editor declines to generate any, and then never fire again.

I deployed this to secure without issue, so it looks like LOCK TABLES ... is promising as a general tool for migrations.

We actually already have bin/remote stop, I just run it so rarely that I forgot about it.

Next chapter: T12989: Phacility Deployment: 2017 Week 38

vault002 is dead. Long live lb001.

This is now deployed on this server, we'll see how it holds up. It won't promote to stable until next week, and I'd discourage anyone from picking it up right away.

This has promoted to stable. See T12974 for upgrade guidance. See T12985 for one followup change. This may still have some bugs (or it may scale very differently on real-world workloads and need substantial additional work) but I think we can handle those separately.

Swap DNS.

• I opened up 22 -> 2223 on lb001.
• I allowed external 22 in the security group.
• I hard-coded my hostfile and cloned successfully:

I'm going to take a stab at this now since I think it's non-disruptive and straightforward.

It's also possible to write a custom instances.do-exactly-what-we-need sort of endpoint and generalize later if that seems like a more promising approach.

The other shadow lurking in the water here -- which I think we can mostly avoid -- is that Almanac is mostly a-bit-bare-bones-but-overall-pretty-functional, except that the way properties on Bindings and Services are specified and edited is complete garbage. You more or less just have to magically know which properties are valid, and there's no real support for defaults or nice UI controls or hints about what you can set or suggestions that you're making stuff up and probably typo'd something.

The "most right" way in terms of consistency is to fully convert Binding to EditEngine, then implement almanac.binding.edit which can create/edit bindings. When creating a binding, it would require transactions specifying the service and interface. PhamePostBlogTransaction is sort of an example of this: when you create a new post with phame.post.edit, you must specify a blog transaction.

Yeah, this is hard coded here:

In Mercurial, can a branch have some open heads and some closed heads?

This seems like a legacy data issue that's probably unique to this install. The only way I could reproduce it locally was:

See also PHI68.

Oh, yes, sorry, looked at the wrong tab.

I'm going to presume this is resolved by D18598 -- I updated T11767 with more plans and context for smoothing over the behavior, but I think it will no longer be disastrously bad. I'll follow up in PHI36 PHI55.

This is also affecting other persistent sticky notifications, notably "Read-Only Mode" and "High Security Mode". I need "Read-Only" mode for PHI36 so I'll take a look at this.

In T12974#231875, @saggid wrote:

Very intresting update. Now we can search in projects code, as i see? Thanks for this supir dupir project guys)

(@amckinley, if you have time to pick this up it's one of the more tackle-able tasks in the support queue.)

Both bak volumes are now swapped. The old volumes are detached as dbak001.phacility.net-old and dbak002.phacility.net-old. I'll delete them after the deployment on Saturday if no issues arise before then.

I haven't gotten any emails yet so I may have to go muck with the WHOIS stuff and make sure there's a valid email somewhere -- phurl.io itself has no mail or MX records.

Upgrade dbak001 to 128GB (from 64GB).
Upgrade dbak002 to 128GB (from 64GB).

Swap notify001 to notify002 in the LB.

I think this is basically "node is bananas" and our AMI is Ubuntu 14 which ships with "Node for DOS".

I'm bumping into T12171 when bringing up the new host. I'm going to take another stab at figuring out what's going on there because the workaround I found in that task is ridiculous.

Switch all SSL to AWS.

Test that moving SSL termination to nlb001 works.

Oh, sorry, misread -- that makes more sense. phurl.io is the only one we serve anything from ourselves right now, and I think the only one we have plans to serve anything from.

Let's make sure all the SSL is swapped first just so we don't get into trouble if we make it halfway, run into issues, and something expires, but that's be helpful once SSL is in the clear. I'm not sure if nlb was the last case of SSL terminating somewhere other than LBs or not, but I think there's one left that I just don't remember offhand.

Replace notify001, which is scheduled for AWS downtime on September 20th.