MaxRequestWorkers 24 appears to be creating some issues. This host only serves about 5 requests/second, but I can hit delays with concurrency of 10 (e.g., ab -n100 -c10):

The Aphlict logs don't reveal anything useful, and could use some attention: they're heavily geared toward setup/debugging, not maintenance/operations.

I've deployed a MaxRequestWorkers 24 change to this server and wasn't able to get it to burst over 24 workers with ab, so that seems to be working. I'll keep an eye on it for the rest of the week, and the 48 limit will go out to the cluster later this week if no problems turn up.

To prevent this, I plan to:

Here's the smoking gun from /var/log/syslog:

This is now fully configured.

c( ⁰ 〰 ⁰ )੭

(•ω•)

I requested SES limits of about 2X what we currently need and AWS gave us about 10X, so things look good so far on that front.

I did a chunk of this work as part of T10629 and we're in better shape on accepting SSL-terminated traffic now.

I've probably updated the notification server certificate, too?

CDN stuff is maybe-sorted.

Something on the CDN side has grown deeply unhappy, looking into it.

Seems OK. Swapping DNS shortly.

This appears to have deployed cleanly. I'm going to test notifications and SSH, and then I'll rebind DNS if they look good.

I'm going to deploy this host with rSERVICES + cluster awareness so it can accept SSL-terminated traffic. If I got something wrong, we may be down for a few minutes.

This is a little bit tricky because notifications use the same hostname, we don't have a wildcard certificate, and we can't (apparently) terminate SSL on the load balancer and pass it through with websockets.

The upstream provider gave me a not-quite-English canned response:

I've issued all instances a 24-hour credit for the service disruption. This will be reflected on the next invoice you receive.

Our original upstream provider enabled the account again, also without explanation. Outbound queues are flushing now, but will take some time to completely de-queue because of backoff behavior in the queue.

I renewed the certificate and launched a new VPC load balancer, which can forward port 22 traffic:

We use secure.phabricator.com for both HTTP/HTTPS and SSH traffic and ELB can't currently forward port 22 -- except the docs now say EC2-VPC LBs can? So I need to figure out what's going on with that.

Test instance workflow looks good, and general monitoring also looks clean. I'll keep an eye on things but I think we're in the clear.

Also, the backup changes from last week (rCOREbf0c93a46f8b) seem to have worked correctly. The instances that were hitting problems and delays have written backups exactly on schedule every day since then.

Daemons look good -- we accumulated a bit of a queue of backup tasks while the tier was out of sync, but these have now almost completely flushed on their own.

I've deployed the remainder of the repo tier, and the auxiliary service tiers. I believe deployment is essentially complete.

repo001 seems to have gone smoothly and daemons/repositories are now restored, I'm deploying the remainder of the tier.

db tier is deployed, now deploying repo001.

I've upgraded db001 and sorted out host instances and service sync flows, which seem OK now. I'm going to upgrade the rest of the db tier.

web is sort of back up, but needs the new schema to actually make any progress, so this isn't conclusive. I'm deploying the db tier now to get the new schema.

• I stopped web.
• I deployed admin.
• I manually removed now-invalid configuration from instances and from Alamanac properties.
• I spot-checked instance/Almanac service relationships.
• I spot-checked Almanac service properties.
• I manually invoked almanac.service.search to look for issues, but everything looks good.

I've stopped the daemons. I'm going to stop the web tier, deploy admin, fix instance configuration, then deploy the web tier. I expect this to be the riskiest part of the upgrade.

I'm getting this underway now. I'm going to preemptively stop all daemons, which will pause repository imports, outgoing mail, etc. They'll be restarted at the end of the process.

There was a minor config issue with db005 but I sorted that out; no users were affected.

The 005 nodes are in production. I'm not going to bother with web002 for now but I'll probably do the other half of that the next time I do ops stuff.

I've brought up the 005 nodes. I also brought up web002 while I was in there, although we don't appear to be near capacity on web001 yet.

Couple of very minor hiccups with notify not having a completely clean deploy since it doesn't get pushed/expanded very often and was a bit out of date, but should be fixed now.

I've updated lb, alb and clb and the respective domains are now serving updated certificates.

• I closed the 003 nodes to new allocations and opened the 004 nodes.
• I created a new test instance and verified it allocated on the 004 nodes.
• I vetted backups, login, web, repo import, and daemon behavior on the new 004 nodes.
• I also pruned some older test instances off of the 003 nodes and cycled them to give us a little more general headroom on repo003. Load should stabilize over time as more test instances suspend over time.

I brought these nodes up normally, they're deploying now. I'll swap them over and vet things once that finishes up.

fyi libvirt takes just a little over a minute to spin up a Linux build machine using copy-on-write disks (this includes waiting for SSH to come up and connect)

Anyway it is not for now !!

In T9431#137264, @epriestley wrote:

Some day you'll be able to spin up a container or host per build and tear it down afterward, but doing so creates a performance/efficiency tradeoff because you have to build and tear down more state per build. I suspect this tradeoff is undesirable for this project even if the code existed to make it (because host startup / teardown would likely overwhelm build cost).

This tier now exists and rCORE knows how to deploy hosts inside it.

This is now up, and rCORE knows how to deploy instances in this tier.

Some day you'll be able to spin up a container or host per build and tear it down afterward, but doing so creates a performance/efficiency tradeoff because you have to build and tear down more state per build. I suspect this tradeoff is undesirable for this project even if the code existed to make it (because host startup / teardown would likely overwhelm build cost).

Although builds will be isolated from services, they won't be isolated from one another. That is, anyone who can run a build can interfere with anything else happening on the same machine.

Corp + Org will also move here after CORGI so this will generally make a good deal of sense as things consolidate.

These things have been fixed:

• I closed the 002 services and opened the 003 services.
• I created a new instance, verified it allocated on 003, connected to it, pushed/pulled, ran backups, etc.

• I launched repo003 and db003 in the cluster.
• I created .phacility.net DNS entires for the hosts.
• I created ddata, dbak, rdata, rbak, and rlog volumes for the hosts and attached them.
• I created device entries for the hosts and *bak volumes.
• I created service entries for the new repox and dbx services.
• I used bin/remote deploy to deploy the hosts.

This seems to have completed without incident.

I've added support for dedicated log volumes (see commit above). Only repo class hosts currently have much use for them, although I'd expect to deploy them more widely eventually.

D13939 will fix T8672, the most immediate cause of high log volumes, but I'm going to hold it until after we cut the release since it also raises the urgency of potential existing errors.

I'm planning to take these corrective actions:

Some contributing issues:

I cycled the host to trigger compaction and bring everything back up, which seems to have resolved the initial issue.

I've decommissioned the old secure.phabricator.com host.

I pushed a change to record instance out-of-service dates (D13561) and stop running backups for out-of-service instances after about 48 hours (D13563). I promoted these changes to rSAAS stable and deployed admin. I manually updated all "Suspended" and "Disabled" instances to have an out-of-service date of July 1, 2015.

I stopped storage upgrade from running on out-of-service instances (rCOREe9f998b109c1) and upgraded db002 and repo002 to verify this change. We don't need to keep out-of-service schemata up to date, and this reduces the ongoing cost of out-of-service instances.

I no-op'd the debian-start script and deployed db002 with the new changes (rCORE51d7570f5ddf) to verify it. I'm not going to restart db001 for the moment, since bringing it down is more disruptive and I only need confidence that the change is correct.

I've restarted the tier with a ~50% buffer pool configuration (4GB of 7.5GB) and 8GB swap (rCORE471d3a7acee1) and things seem to be holding stable. I'll pursue the other changes now.

Ubuntu also ships with some debian-sys-maint junk which runs huge, slow queries to check for problematic table data on every restart, which we should likely disable.

While I was writing this up, the host ran out of RAM and stopped working, so this is probably a memory management issue.

Some immediate observations:

I don't want to install it on cluster hosts, see T7785.

Need Graphviz install too, pls. Test Page