I decommissioned the old .org host. I'll give the old secure host another week or so since we did hit a couple minor config things where it was useful to double-check that there was a config change.

I've moved the blog and stopped traffic to the old secure host. I'll decommission both hosts in a few days if nothing crops up.

Document everything.

Authorize trusted external keys.

Get phabot running again.

• I (probably?) corrected the HTTPS issue by splitting the certificate and certificate chain files.
• I allowed the host to serve blog.phacility.com, javelinjs.com, and phabricator.org.
• Javelin and org are working fine so I updated DNS. I'll decommission the existing .org host.
• blog has some HTTP vs HTTPS stuff that I think I'm going to clean up in Phame itself.

Actually, looking at https://sslcheck.globalsign.com/en_GB/sslcheck?host=secure.phabricator.com#52.8.228.58-srv-reg provides some better insight:

Yeah I'm seeing SSL errors as well. Not sure if it helps but there are some items in yellow on https://www.ssllabs.com/ssltest/analyze.html?d=secure.phabricator.com

I had the same thing as @ftdysa but its fine now...

Huh, well I don't think I'm crazy. I'll take a screenshot if it happens again.

It's a single host and I didn't touch anything in the last couple hours.

Green now, is this behind an lb or could a server be misconfigured?

Currently on phone so I'm a bit limited but secure.phabricator.com is showing a red lock and the "Your connection is not private warning".

Can you be more specific? I can't reproduce this.

Unsure if it is due to this but chrome is complaining about your ssl cert.

I think I'm going to bring .org onto this host for simplicity/consistency, too. Putting it on corp is messy because of SSL, but putting it here for now should be straightforward.

Oh, I need to hook up diviner / symbols, too. I'm probably just going to cron those. So still-remaining stuff I'm aware of is:

I manually installed xhprof on the host since it needs some special setup steps that aren't trivial to integrate into the deployment tools, and isn't particularly important to have available.

Backups work but won't be run automatically by cluster administration services, although they are reported. I'm just going to cron them for now.

Stuff I'm still tracking:

• I configured HTTP -> HTTPS in a standard way (on this host, it involves some legacy oddness).
• I added host SSH keys and switched the tier to use them.
• I verified external connectivity and reasonable-seeming repo connectivity.
• I created secure001.phacility.net and sbak001.phacility.net devices on Admin, so generating routine backups will depend on the cluster being available. This seems acceptable as a dependency.
• I added run.repo to the host attributes to get repo backups working and verified repo backups get written.
• I dumped all the data and moved it over.
• I renamespace'd the database to secure and loaded it, then made config adjustments to use this as the storage namespace.
• I unpacked the repos into /core/repo/secure, then made a config adjustment for this.
• I allocated a new elastic IP and swapped DNS over (we still have javelinjs on the old box, plus we can't reuse a non-VPC elastic IP in a VPC).

I've completed the cluster deployment and am resuming this now.

I think this is kind-of working now. I'm going to take a break for today and pick it back up tomorrow since I'm pretty close to bringing the data over, which might take a while, and I'm sort of committed to finishing once I start it.

• I created a new instance.secure security group. I removed port 25 (SMTP), since we now use Mailgun for inbound mail. I removed port 843 (Flash sockets) since we no longer use them. I changed administrative SSH from 222 to 2222 for consistency with the vault tier. This leaves us with 22, 80, 443, 2222, and 22280.
• I launched a secure001 host and added a secure001 DNS entry for it in Route 53.
• I created and attached sbak001, srepo001 and sdata001 volumes. This is a little overkill but should work well with the tooling.
• I created a new "secure" host role in rCORE44a038f, and deployed the bastion to get it to recognize the role.
• I used bin/remote deploy secure001 --port 22 to do an initial deploy.
• I added secure.conf and the missing sshd library in rCORE95a18f3.
• I added a bunch of config files and stuff in rCORE78c6c9a. This got SSH moved to 2222 correctly, so --port 22 is no longer required (like vault, this tier does need it for initial deploys).
• Kept fiddling with config (see diffs), thing is sort of deploying properly now, at least.

I'm starting this now.

Stuff that went poorly:

• I set instances.open to 0 on the 001 services and to 1 on the 002 services.
• I created a new instance and verified it came up on the 002 services.
• I verified that the test-on-002 instance still worked. Verified that a known 001-instance (meta) still works.

• I connected to the test instance, enabled Darkconsole, verified it was hitting the right DB, etc.
• I created, cloned, and pushed a repository.
• I ran host backup explicitly on both machines. This wanted device entries for the volumes for logging. I created them; this completed smoothly afterward.
• I verified that the backups appeared in the web UI log, that the repository was on the correct host, etc.

AppArmor uses file modification time heuristics to decide whether it can read a cache or not. This heuristic guesses the wrong result in the deployment case. I added -T to stop it from using the cache in rCORE8371c05b. Instance upgrade/sync operations now work correctly.

• I created Almanac service entries for repox002 and dbx002, following the 001 templates. Notably, repo bindings need a protocol specified. I didn't specify "instances.open" on these yet.
• I created a new instance and verified it came up on repox001 and dbx001 still, since the new services aren't open.
• I manually edited the instance to move it to the 002 services, then tried to upgrade the database. This failed.

• I created Almanac device entries for these hosts, following db001 and repo001. Both deploys went cleanly once the device entries were created.
• I used remote ssh <host> to connect to the devices and verified connectivity, volume attachment, deployed libraries, branches, that mysql works on db002, iptables rules, and that nothing appeared otherwise funky or broken. I didn't catch anything suspicious, except that remote deploy doesn't initialize or mount the backup volume: we do that when we actually perform backups. It probably should, just to catch misconfiguration/errors earlier. For now, I'll do this manually later on. I filed T8688 to follow up.

When creating device entries, I discovered AlmanacServiceQuery has a bug where name is not returned in the paging map. This prevents paging when an install has more than 100 service entries.

When a new host comes up with no instances, we don't handle it cleanly at the tail end of deployment.
This cleared MySQL deployment and just left us with the "no instances" issue on both hosts.

We don't actually install MySQL onto hosts in the db role.

• I launched repo002 and db002 into the cluster.
• I created .phacility.net DNS entries for the hosts.
• I created ddata002, dbak002, rdata002 and rbak002 volumes and attached them according to Adding Cluster Hardware.
• I used bin/remote deploy <host> to deploy each host. Both hosts deployed mostly cleanly.

I'm starting this now. This should ideally be routine, but I expect there will be a few hiccups.

For communication, my plan today would just be to use IRC/email/phonecalls since we have a small team, run operations mostly in the open, and are all regularly available on these other channels, and the likely course of action is "wait for AWS to come back up". I think that's generally reasonable until we have a larger operations team and more operational responses available, so I'm not particularly concerned about Conpherence/Maniphest on this instance being down.

This is great. When / ever do you plan to be able to survive the whole AWS datacenter going out/

See subtasks for details, but I've:

ops woo~~

I, uh, guess this happened? Or I didn't do math? Or Amazon pranked us? I was running some errands and didn't see the thing go down, but I guess I'll take it.

I math.

If I did the math right, this is happening in about two hours (1PM PST). I'll do a backup before the downtime but don't plan to move the instance (I think it would take long enough that it's not worth it), so we'll probably be offline from about 1PM - 3PM.