Page MenuHomePhabricator
Create Task
Maniphest T13180

AWS is rebooting sbuild001.phacility.net on August 15, 2018
Closed, ResolvedPublic
Actions

Description

One or more of your Amazon EC2 instances is scheduled for maintenance on 2018-08-15 for 2 hours starting at 2018-08-15 08:00:00 UTC. During this time, the following instances in the us-west-1 region will be unavailable and then rebooted:

i-6b8bffab

This is sbuild001.phacility.net.

This box isn't critical (it runs builds for secure) so we can just stop and start it at any time, or let AWS knock it off the grid.

Related Objects

Event Timeline

epriestley triaged this task as Low priority.Aug 1 2018, 1:09 PM
epriestley created this task.
Herald added a subscriber: eadler. · View Herald TranscriptAug 1 2018, 1:09 PM
epriestley assigned this task to amckinley.Aug 8 2018, 3:41 PM
epriestley added a subscriber: amckinley.

@amckinley, do you want to take this one whenever you get a chance? This seemed to work the last time:

  • Stop in AWS console.
  • Start in AWS console.
  • bin/remote deploy sbuild001

You can test things are working by restarting any build in Harbormaster, I think, and verify that the "upcoming events" thing in AWS no longer shows impending doom for the host.

This just runs Harbormaster builds here, so nothing else will be impacted if there are issues (or we don't get to it before AWS kills it).

amckinley added a comment.Aug 8 2018, 4:40 PM

Sure, will do this afternoon.

amckinley added a comment.Aug 8 2018, 10:58 PM

Doing this now.

amckinley closed this task as Resolved.Aug 9 2018, 12:04 AM

This was more fun than expected. Notes for future historians:

  • This host lives in a separate VPC for isolation reasons, and is therefore accessed from the bastion via the DNS name sbuild001-external.phacility.net.
  • The route53 DNS entry for sbuild001-external.phacility.net needs to be updated with the EC2 instance's new public IP in order to bin/remote deploy sbuild001.
  • The Alamanc record for sbuild001 needs to be updated with the new public IP as well in order for builds to actually work.
epriestley mentioned this in T13121: Remove "-q" from SSH commands executed by `bin/remote` and similar cluster commands.Aug 9 2018, 12:05 AM
epriestley awarded a token.Aug 9 2018, 12:07 AM
Phabricator · Built by Phacility