See PHI1315, where an instance restarted without any apparent notification. This happened once before previously, although I'm not immediately able to dig up the issue.
Some user on Reddit suggests this is routine, and/or fabricates an imaginary story about it being routine: https://www.reddit.com/r/aws/comments/ax944r/is_it_possible_that_amazon_itself_rebooted_my_ec2/ehsgb7y/
...although other users in that thread suggest this isn't expected:
Whenever aws schedules an ec2 for migration/reboot due to underlying hardware/software issues you will receive usually an email with the notice and also you will have a notification in the console.
I checked these places for any kind of "host is restarting because X" information:
- AWS email.
- AWS notifications bell in the menu.
- CloudTrail logs.
- last reboot
No luck finding any actual cause for the restart.
Prior to these recent cases, all restarts have been scheduled/announced. However, now that this has happened more than once, it seems like something we should handle better.
The major issues are:
- After restart, volumes don't automatically remount.
- After restart, services don't automatically start.
These can be remedied by running bin/host upgrade on restart. However, this also upgrades application libraries (e.g., Phabricator), which we don't want. The shortest path to a reasonable fix is probably:
- Add a flag like bin/host upgrade --keep-deployed-version to disable the git pull steps.
- Run this on host startup.
- Kick some hosts to make sure it works.
Ideally, this would also include a flag like --and-complain to generate a notification/alert.