Page MenuHomePhabricator
Feed Advanced Search

Apr 20 2022

epriestley closed T10847: 30GB Phacility instance caused a series of cascading failures which left web services unreachable as Resolved.

There's nothing particularly useful or actionable here now, so closing it out. (I believe this was the most severe incident Phacility ever experienced while actively maintained.)

Apr 20 2022, 10:43 PM · Ops, Phacility
epriestley closed T12610: Audit behavior of LB healthchecks against *.phacility.com and secure.phabricator.com as Wontfix.

This hasn't caused any more problems in like 4 years, so I guess it's kind of whatever.

Apr 20 2022, 10:30 PM · Ops, Phacility

Dec 19 2021

epriestley closed T12847: A Pathway Towards Private Clusters as Wontfix.

After T13630:

Dec 19 2021, 8:39 PM · Plans, Ops, Phacility

Feb 26 2021

epriestley placed T12611: Write Phabricator HTTP and SSH logs in the production cluster up for grabs.

Enable the SSH and HTTP application logs on the web, repo and admin tiers.

Feb 26 2021, 10:57 PM · Phacility, Ops
epriestley added a revision to T12611: Write Phabricator HTTP and SSH logs in the production cluster: Restricted Differential Revision.
Feb 26 2021, 10:49 PM · Phacility, Ops
epriestley added a revision to T12611: Write Phabricator HTTP and SSH logs in the production cluster: Restricted Differential Revision.
Feb 26 2021, 10:48 PM · Phacility, Ops

May 26 2020

epriestley closed T13076: Plans: Phacility cluster caching, renaming, and rebalance/compaction as Resolved.

Continued in T13542.

May 26 2020, 8:13 PM · Plans, Ops, Infrastructure, Phacility
epriestley closed T12801: Simplify Almanac services in the Phacility production cluster as Resolved.

The major offender here (services per instance) was fixed by updating caching, and I destroyed all the old services. This is perhaps spiritually continued in T13542.

May 26 2020, 8:11 PM · Almanac, Ops, Phacility
epriestley closed T12999: Replace cluster magnetic volumes with SSD volumes as Resolved.

Continued in T13542.

May 26 2020, 8:10 PM · Phacility, Ops
epriestley closed T12856: Evaluate various "infrastructure-as-code" products as Resolved.

Continued in T13542. I wrote a Terraform/CloudFormation-style service in PHP over the last couple of days.

May 26 2020, 8:07 PM · Ops, Phacility
epriestley closed T12816: Setup NAT for the primary Phacility cluster as Resolved.

Continued in T13542.

May 26 2020, 8:03 PM · Ops, Phacility

Feb 3 2020

epriestley closed T13483: (2020 Week 5) Restart AWS host db001 (i-3a9b99f2) as Resolved.

Both of these hosts restarted cleanly.

Feb 3 2020, 2:19 PM · Phacility, Ops
epriestley added a comment to T13483: (2020 Week 5) Restart AWS host db001 (i-3a9b99f2).

AWS is also rebooting web007.

Feb 3 2020, 1:22 PM · Phacility, Ops

Jan 30 2020

epriestley triaged T13483: (2020 Week 5) Restart AWS host db001 (i-3a9b99f2) as Normal priority.
Jan 30 2020, 6:36 PM · Phacility, Ops

Jan 15 2020

epriestley closed T13477: (2020 Week 3) Restart AWS host db025 (i-05bc80634586ef7a0) as Resolved.

This went through cleanly.

Jan 15 2020, 4:19 PM · Ops, Phacility
epriestley triaged T13477: (2020 Week 3) Restart AWS host db025 (i-05bc80634586ef7a0) as Low priority.
Jan 15 2020, 2:36 AM · Ops, Phacility

Nov 26 2019

epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

there is no way to bin/host query against the set of instances using a particular repository shard service

Nov 26 2019, 12:14 AM · Phacility, Ops
epriestley added a revision to T13466: AWS instance termination may fail/hang indefinitely: Restricted Differential Revision.
Nov 26 2019, 12:09 AM · Phacility, Ops

Nov 25 2019

epriestley added a revision to T13466: AWS instance termination may fail/hang indefinitely: Restricted Differential Revision.
Nov 25 2019, 11:57 PM · Phacility, Ops
epriestley added a revision to T13466: AWS instance termination may fail/hang indefinitely: Restricted Differential Revision.
Nov 25 2019, 10:46 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

Update Almanac definitions for all instances not on the paired db023 shard.

Nov 25 2019, 5:01 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

PHI1566 is resolved narrowly. These cleanup steps still need to happen.

Nov 25 2019, 4:42 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

(Updating addresses with bin/host query leaves the service address cache dirty (the "mutable structure cache" via PhabricatorRepository->getAlmanacServiceRefs()) so it should be followed with bin/cache purge --caches general.)

Nov 25 2019, 4:29 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

I'll flesh this out more later, but the move away from db123 = repo123 shard pairing, plus bin/host query using mysql makes it difficult to directly query instances using a particular repository service.

Nov 25 2019, 4:15 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

Minor issue that should be looked at during service sync arising from improved validation elsewhere:

Nov 25 2019, 4:06 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

I'm deploying the new host now. We just crossed a release so I'm going to manually restore it to 72f82abe07 once it comes up (see also T13359). Then, I'll resynchronize instance services for active instances.

Nov 25 2019, 3:57 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

Instance termination completed after about 20 minutes and all the volumes detached. Since the original instance can be recycled, I'm going to reattach and restart it, and throw away the replacement host.

Nov 25 2019, 3:51 PM · Phacility, Ops
epriestley added a project to T13466: AWS instance termination may fail/hang indefinitely: Phacility.
Nov 25 2019, 3:48 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

Normal volume detachment is just spinning, which isn't exactly surprising. I'm going to give it a few minutes and then force detachment.

Nov 25 2019, 3:47 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

To deal with this narrowly, I'm going to:

Nov 25 2019, 3:44 PM · Phacility, Ops
epriestley triaged T13466: AWS instance termination may fail/hang indefinitely as Low priority.
Nov 25 2019, 3:42 PM · Phacility, Ops

Aug 11 2019

epriestley added a revision to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting: Restricted Differential Revision.
Aug 11 2019, 4:19 PM · Ops, Restricted Project, Phacility
epriestley added a revision to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting: Restricted Differential Revision.
Aug 11 2019, 4:10 PM · Ops, Restricted Project, Phacility

Aug 1 2019

epriestley triaged T13359: Phacility deploy workflow should not conflate versions-for-deployment with "latest stable release" as Low priority.
Aug 1 2019, 5:48 PM · Phacility, Ops

Jul 30 2019

epriestley added a revision to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting: Restricted Differential Revision.
Jul 30 2019, 6:20 PM · Ops, Restricted Project, Phacility
epriestley added a revision to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting: Restricted Differential Revision.
Jul 30 2019, 6:18 PM · Ops, Restricted Project, Phacility

Jul 24 2019

epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

Adjusting log_warnings = 2 in production (to get connection aborts into the error log) is also possibly desirable, although the background level of connection abortions (general network flakiness, server restarts during deploy, wait_timeout on very long-running demons?) may be high enough that this is more noise than signal.

Jul 24 2019, 2:03 PM · Ops, Restricted Project, Phacility

Jul 23 2019

epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

We could also consider these things:

Jul 23 2019, 1:10 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

So actual actionable stuff here is:

Jul 23 2019, 1:04 PM · Ops, Restricted Project, Phacility
epriestley renamed T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting from During MySQL import, server may "2006 MySQL server has gone away" when row data size is large relative to "innodb_log_file_size" (?) to During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.
Jul 23 2019, 12:59 PM · Ops, Restricted Project, Phacility

Jul 22 2019

epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

Bumping max_allowed_packet to 1G in the server config resolved things. The export process then spent a long time doing a bin/files migration (which could use a progress bar, maybe) and is now doing a dump (which could too, although I'm less sure of how we'd build one).

Jul 22 2019, 11:52 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.
190722 18:55:55 [Warning] Aborted connection 6 to db: '<instance>_differential' user: 'root' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
Jul 22 2019, 6:59 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

I adjusted innodb_log_file_size to 1GB and attempted the import again, but ran into the same issue.

Jul 22 2019, 6:16 PM · Ops, Restricted Project, Phacility
epriestley renamed T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting from During MySQL import, server may "2006 MySQL server has gone away" when row data size is large relative to "innodb_log_file_size" to During MySQL import, server may "2006 MySQL server has gone away" when row data size is large relative to "innodb_log_file_size" (?).
Jul 22 2019, 6:13 PM · Ops, Restricted Project, Phacility
epriestley added a revision to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting: D20677: Provide import/restore guidance for "max_allowed_packet" and "innodb_log_file_size".
Jul 22 2019, 5:05 PM · Ops, Restricted Project, Phacility
epriestley added a revision to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting: Restricted Differential Revision.
Jul 22 2019, 4:38 PM · Ops, Restricted Project, Phacility
epriestley renamed T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting from During MySQL import, server may "go away" on large dumps? to During MySQL import, server may "2006 MySQL server has gone away" when row data size is large relative to "innodb_log_file_size".
Jul 22 2019, 4:27 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

The "age of the last checkpoint" error appears to primarily implicate innodb_log_file_size, which is currently set to the default value (5MB):

Jul 22 2019, 4:26 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

I'll also double check wait_timeout and interactive_timeout...

Jul 22 2019, 4:01 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

Aha! The MySQL error log actually appears to have something useful:

Jul 22 2019, 3:55 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

Run it with source ...;

Jul 22 2019, 3:54 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

Unzip the dump before running it.

Jul 22 2019, 3:45 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

Look at the unzipped dump and see if line 13935 is bad in some obvious way.

Jul 22 2019, 3:40 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

(Whatever the resolution is here might also motivate tailoring our restore/import instructions, since this error is pretty opaque and the next steps aren't obvious.)

Jul 22 2019, 3:21 PM · Ops, Restricted Project, Phacility
epriestley added a comment to T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting.

(Internally, see also PHI1329.)

Jul 22 2019, 3:18 PM · Ops, Restricted Project, Phacility
epriestley triaged T13347: During MySQL import, server may "2006 MySQL server has gone away" when "max_allowed_packet" server setting is too small, despite client setting as Low priority.
Jul 22 2019, 3:18 PM · Ops, Restricted Project, Phacility

Jul 17 2019

epriestley moved T10696: Aphlict slowly grows to use an uncomfortably large amount of memory if the disk is full from vNext to Far Future on the Aphlict board.
Jul 17 2019, 6:30 PM · Aphlict, Phacility, Ops

Mar 1 2019

epriestley added a revision to T13121: Remove "-q" from SSH commands executed by `bin/remote` and similar cluster commands: Restricted Differential Revision.
Mar 1 2019, 5:53 PM · Phacility, Ops
epriestley added a revision to T13121: Remove "-q" from SSH commands executed by `bin/remote` and similar cluster commands: D20240: Use "LogLevel=ERROR" to try to improve "ssh" hostkey behavior without doing anything extreme/hacky.
Mar 1 2019, 5:52 PM · Phacility, Ops
epriestley added a comment to T13121: Remove "-q" from SSH commands executed by `bin/remote` and similar cluster commands.

Another possible approach is to use -o LogLevel=ERROR. This gets us into trouble if there are useful INFO messages other than "permanently added X to list of known hosts", but presumably all the important stuff is rasied at ERROR or better.

Mar 1 2019, 5:43 PM · Phacility, Ops
epriestley added a comment to T13121: Remove "-q" from SSH commands executed by `bin/remote` and similar cluster commands.

For Drydock/Harbormaster, the only real way forward I see here is:

Mar 1 2019, 5:38 PM · Phacility, Ops

Jan 30 2019

epriestley added a comment to T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile.

Success! D20046 worked to fix the "profiler not sticking across form posts" issue on secure. 🐈

Jan 30 2019, 7:56 PM · XHProf, Ops, Phacility

Jan 28 2019

epriestley added a revision to T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile: D20046: Read "$_POST" before hooking the profiler, and remove "aphront.default-application-configuration-class".
Jan 28 2019, 7:09 PM · XHProf, Ops, Phacility
epriestley added a comment to T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile.

Yeah, I think the issue is:

Jan 28 2019, 6:41 PM · XHProf, Ops, Phacility
epriestley added a comment to T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile.

The "keep the profiler on across form submissions" code isn't working on secure.phabricator.com, even though it works locally and __profile__=page appears on the "Request" tab.

Jan 28 2019, 4:58 PM · XHProf, Ops, Phacility

Nov 3 2018

epriestley closed T13207: Cycle More AWS Hosts (October 2018) as Resolved.

I think everything here is now fully cycled, synchronized, and cleaned up.

Nov 3 2018, 6:44 PM · Phacility, Ops
epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

Taking care of these now. I expect everything to be pretty routine.

Nov 3 2018, 6:23 PM · Phacility, Ops

Oct 22 2018

epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

Plus: db018.phacility.net, repo001.phacility.net, db024.phacility.net.

Oct 22 2018, 8:46 PM · Phacility, Ops

Oct 19 2018

epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

One more of these just came in for repo003.

Oct 19 2018, 8:56 PM · Phacility, Ops

Oct 8 2018

epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

I think this is all done but want to let things run against bastion007 for a bit before I tear down bastion005.

Oct 8 2018, 3:43 PM · Phacility, Ops
epriestley added a revision to T13207: Cycle More AWS Hosts (October 2018): Restricted Differential Revision.
Oct 8 2018, 3:41 PM · Phacility, Ops
epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

I also needed to copy the old master.key from bastion005 to bastion007 in /core/lib/keystore/.

Oct 8 2018, 3:40 PM · Phacility, Ops
epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

I turned bastion.phacility.net and bastion-external.phacillity.net into CNAME records and pointed them at the new bastions.

Oct 8 2018, 3:35 PM · Phacility, Ops
epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

There's a minor deadlock on bastion deployment with the current scripts: during deploy, we run deploy-key to copy the deploy key from the bastion to the target host during deployment, so that we don't need to put the entire keystore on normal cluster nodes, and so that we don't need to have the keystore on the control host (staff laptop) outside the cluster.

Oct 8 2018, 3:31 PM · Phacility, Ops

Oct 6 2018

epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

I cycled all the hosts except bastion. saux001 needs to be vetted a bit (it handles "Land Revision" from the web UI) but it isn't critical if it needs a bit more work.

Oct 6 2018, 3:38 PM · Phacility, Ops
epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

I 'm going to get these underway once the deploy finishes.

Oct 6 2018, 2:45 PM · Phacility, Ops

Oct 1 2018

epriestley added a revision to T13207: Cycle More AWS Hosts (October 2018): Restricted Differential Revision.
Oct 1 2018, 4:37 PM · Phacility, Ops
epriestley added a comment to T13207: Cycle More AWS Hosts (October 2018).

"Use the API" seemed to work OK. Of those instances, only bastion005 is at all unusual.

Oct 1 2018, 4:35 PM · Phacility, Ops
epriestley updated the task description for T13207: Cycle More AWS Hosts (October 2018).
Oct 1 2018, 4:34 PM · Phacility, Ops
epriestley triaged T13207: Cycle More AWS Hosts (October 2018) as Normal priority.
Oct 1 2018, 3:59 PM · Phacility, Ops

Sep 11 2018

epriestley closed T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes as Resolved.

This is now live.

Sep 11 2018, 12:52 AM · Ops, Phacility
epriestley added a comment to T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.

Deploying the changes to web now.

Sep 11 2018, 12:48 AM · Ops, Phacility

Sep 10 2018

epriestley added a comment to T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.

I've issued all instances a 24-hour service credit for the disruption. This should be reflected on your next invoice.

Sep 10 2018, 9:17 PM · Ops, Phacility
epriestley added a comment to T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.

Here's the request rate leading up to the rate limiting:

Sep 10 2018, 9:11 PM · Ops, Phacility
epriestley added a comment to T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.

D19653 (above) changes the per-"Host" rate limit to require "X-Forwarded-For" be present in the request. This should exempt ELB requests from these limits.

Sep 10 2018, 8:55 PM · Ops, Phacility
epriestley added a comment to T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.
...
[Mon Sep 10 20:48:43.928021 2018] [:error] [pid 21570] [client 172.30.0.171:16516] Array\n(\n    [f] => \n    [h] => 172.30.0.60\n)\n
...
Sep 10 2018, 8:51 PM · Ops, Phacility
epriestley added a comment to T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.

... in production today as a next step.

Sep 10 2018, 8:46 PM · Ops, Phacility
epriestley added a comment to T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.

This should have the pleasant side effect of letting us drop the goofy hard-coded internal rate limiting IP list.

Sep 10 2018, 8:44 PM · Ops, Phacility
epriestley added a comment to T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.

There are four rate limits, and I don't currently have enough information to figure out which one triggered. The rate limits are:

Sep 10 2018, 8:41 PM · Ops, Phacility
epriestley triaged T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes as High priority.
Sep 10 2018, 8:25 PM · Ops, Phacility

Sep 5 2018

epriestley moved T13076: Plans: Phacility cluster caching, renaming, and rebalance/compaction from Tentative to Soon on the Plans board.
Sep 5 2018, 1:34 PM · Plans, Ops, Infrastructure, Phacility

Aug 25 2018

epriestley closed T13183: AWS is rebooting instances in late August 2018 as Resolved.

That one seemed straightforward.

Aug 25 2018, 1:57 PM · Ops, Phacility
epriestley added a comment to T13183: AWS is rebooting instances in late August 2018.

Doing admin001 now.

Aug 25 2018, 1:52 PM · Ops, Phacility
epriestley added a comment to T13183: AWS is rebooting instances in late August 2018.

Kicking secure001 now.

Aug 25 2018, 1:52 PM · Ops, Phacility
epriestley added a comment to T13183: AWS is rebooting instances in late August 2018.

(It not being covered is covered by T12879.)

Aug 25 2018, 1:38 PM · Ops, Phacility
epriestley added a comment to T13183: AWS is rebooting instances in late August 2018.

I think the only thing on secure or admin which isn't properly covered by deploy automation is the crontab on secure001:

Aug 25 2018, 1:37 PM · Ops, Phacility
epriestley added a comment to T13183: AWS is rebooting instances in late August 2018.

I'm going to do admin001 and secure001 today.

Aug 25 2018, 1:19 PM · Ops, Phacility

Aug 18 2018

epriestley added a comment to T13183: AWS is rebooting instances in late August 2018.

Think I got through the easy ones without any issues. I suspect admin and secure may be a little more involved so I'm going to leave the cat in the bag for the moment.

Aug 18 2018, 9:21 PM · Ops, Phacility
epriestley updated the task description for T13183: AWS is rebooting instances in late August 2018.
Aug 18 2018, 9:10 PM · Ops, Phacility
epriestley updated the task description for T13183: AWS is rebooting instances in late August 2018.
Aug 18 2018, 9:04 PM · Ops, Phacility