Yeah, that should work fine. To promote, you'd update the VIP stuff and Phabricator could remain in the dark.

Ah, I see. In my world master promotion independently of the web tier. The web hosts connect over a VIP which always reaches the current master. We can also connect over a VIP to reach the slaves but don't currently use this.

I think the answer is yes:

will this task support setting a separate read-only username and password for database connections? This is useful in our environment to access the slaves.

This is now deployed and appears to be working.

I believe D11874 covers the last of this for Git.

I'm merging this into T10751, which is a cleaner followup without two years of outdated history. The goals remain the same.

Thanks Everyone!

The more modern, slightly more concise process is:

I haven't been successful locally in getting a GC'd node to reach an appreciable level of memory consumption, and the production servers are currently at ~1.5GB.

I think the ~1.5GB node instance size is also expected.

At least locally, I can't manage to get node to not leak:

We're seeing a lot of connections stick in "R" ("Reading Request") for a long time, but the client is the load balancer and the request hasn't been read yet so it's not clear exactly what's going on with these. I think they may be crawlers, since pretty much every search engine loves indexing everything on this install hundreds of times an hour.

I'm fiddling with LB health checks, so it's my fault if the server vanishes.

MaxRequestWorkers 24 appears to be creating some issues. This host only serves about 5 requests/second, but I can hit delays with concurrency of 10 (e.g., ab -n100 -c10):

The Aphlict logs don't reveal anything useful, and could use some attention: they're heavily geared toward setup/debugging, not maintenance/operations.

I've deployed a MaxRequestWorkers 24 change to this server and wasn't able to get it to burst over 24 workers with ab, so that seems to be working. I'll keep an eye on it for the rest of the week, and the 48 limit will go out to the cluster later this week if no problems turn up.

To prevent this, I plan to:

Here's the smoking gun from /var/log/syslog:

c( ⁰ 〰 ⁰ )੭

(•ω•)

I requested SES limits of about 2X what we currently need and AWS gave us about 10X, so things look good so far on that front.

I've probably updated the notification server certificate, too?

CDN stuff is maybe-sorted.

Something on the CDN side has grown deeply unhappy, looking into it.

Seems OK. Swapping DNS shortly.

This appears to have deployed cleanly. I'm going to test notifications and SSH, and then I'll rebind DNS if they look good.

I'm going to deploy this host with rSERVICES + cluster awareness so it can accept SSL-terminated traffic. If I got something wrong, we may be down for a few minutes.

This is a little bit tricky because notifications use the same hostname, we don't have a wildcard certificate, and we can't (apparently) terminate SSL on the load balancer and pass it through with websockets.

The upstream provider gave me a not-quite-English canned response:

I've issued all instances a 24-hour credit for the service disruption. This will be reflected on the next invoice you receive.

Our original upstream provider enabled the account again, also without explanation. Outbound queues are flushing now, but will take some time to completely de-queue because of backoff behavior in the queue.

I renewed the certificate and launched a new VPC load balancer, which can forward port 22 traffic:

We use secure.phabricator.com for both HTTP/HTTPS and SSH traffic and ELB can't currently forward port 22 -- except the docs now say EC2-VPC LBs can? So I need to figure out what's going on with that.

Drydock and Almanac are now available in the cluster. Here's a rough guide to configuring them -- I'll turn this into something formal once things work a little more smoothly.

Quick update here ahead of deployment tomorrow:

@epriestley : sorry, I should have searched more thoughtfully.

@tycho.tatitscheff See T8656 or Planning.

Is support of ansi color possible in this harbormaster iteration ? I mean not only parse the ansi code but also generate a markup based on the color.
As far as I remenber it wasn't the case.