If you uninstall "Home", the user menu no longer appears. This makes it very difficult to log out or access your settings on admin.phacility.com.
Description
Revisions and Commits
Related Objects
Event Timeline
I'm going to move forward with the deploy since this shouldn't impact actual instances, the stuff in this menu isn't super-critical, and workarounds (albeit inconvenient ones) exist to access the items.
I'll clean this up and hotfix it to admin.phacility.com once the rest of the deployment is live.
The db001, db003 and db004 shards have also hit 80% full on their primary volumes. I'm purging dead instances from db001 now. db003 and db004 have never been purged and need to have instance out-of-service dates synchronized first. This is currently underway for db004.
repo016 had a nearly-full temporary volume. I nuked all the temporary data (large diffs) but need to identify and fix the root cause.
The storage pressure on db001 is primarily because an instance imported a large SVN repository with approximately 60M paths. The instance is a free instance with one user.
The temporary directory storage pressure on repo016 has an unclear root cause (it's related to importing an SVN repository, but I haven't been able to pinpoint where the issue started). The issue no longer appears to be occurring so I'm going to stop digging until it happens again.
Instance service windows on db003 and db004 have synchronized and the shards are now purging.
Purging db001 brought primary volume down to 79.2%, which is technically below the 80% caution threshold. It doesn't need to be fixed immediately, but the pathway forward isn't terribly clear. Some options are:
- write some "you need to log in every 30 days to keep free instances around" stuff, deploy that, then wait another ~90 days and hope the user abandons the instance and nothing else big hits the shard (we could close the shard to reduce the amount of hope required);
- increase the volume size (causes downtime for all other instances, procedure isn't robust/routine yet);
- move just the problem instance (causes instance downtime; procedure isn't robust/routine, T11670);
- push something in the next week which magically reduces the size of the repository_pathchange table (not likely).
After purging dead instances, db003 and db004 are each <25% utilized on their data volumes.
I deployed that change here (secure.phabricator.com). I'll cherry-pick it and deploy it on admin in a little bit, if nothing catches on explosion fire.