Page MenuHomePhabricator

When Home is uninstalled, the user menu disappears / 2017 Week 3 Deployment Log
Closed, ResolvedPublic

Description

If you uninstall "Home", the user menu no longer appears. This makes it very difficult to log out or access your settings on admin.phacility.com.

Event Timeline

I'm going to move forward with the deploy since this shouldn't impact actual instances, the stuff in this menu isn't super-critical, and workarounds (albeit inconvenient ones) exist to access the items.

I'll clean this up and hotfix it to admin.phacility.com once the rest of the deployment is live.

The db001, db003 and db004 shards have also hit 80% full on their primary volumes. I'm purging dead instances from db001 now. db003 and db004 have never been purged and need to have instance out-of-service dates synchronized first. This is currently underway for db004.

repo016 had a nearly-full temporary volume. I nuked all the temporary data (large diffs) but need to identify and fix the root cause.

epriestley renamed this task from When Home is uninstalled, the user menu disappears to When Home is uninstalled, the user menu disappears / 2017 Week 3 Deployment Log.Jan 21 2017, 2:58 PM
epriestley added a project: Ops.
epriestley added a project: Phacility.

The storage pressure on db001 is primarily because an instance imported a large SVN repository with approximately 60M paths. The instance is a free instance with one user.

The temporary directory storage pressure on repo016 has an unclear root cause (it's related to importing an SVN repository, but I haven't been able to pinpoint where the issue started). The issue no longer appears to be occurring so I'm going to stop digging until it happens again.

Instance service windows on db003 and db004 have synchronized and the shards are now purging.

Purging db001 brought primary volume down to 79.2%, which is technically below the 80% caution threshold. It doesn't need to be fixed immediately, but the pathway forward isn't terribly clear. Some options are:

  • write some "you need to log in every 30 days to keep free instances around" stuff, deploy that, then wait another ~90 days and hope the user abandons the instance and nothing else big hits the shard (we could close the shard to reduce the amount of hope required);
  • increase the volume size (causes downtime for all other instances, procedure isn't robust/routine yet);
  • move just the problem instance (causes instance downtime; procedure isn't robust/routine, T11670);
  • push something in the next week which magically reduces the size of the repository_pathchange table (not likely).

After purging dead instances, db003 and db004 are each <25% utilized on their data volumes.

Haha, no worries. Uninstalling Home is pretty unusual.

I deployed that change here (secure.phabricator.com). I'll cherry-pick it and deploy it on admin in a little bit, if nothing catches on explosion fire.

epriestley claimed this task.

The db001 issue is still an issue, but the shard has managed to stay just under 80% for the last couple of weeks so we're not in any immediate peril. As followups, I think T12217 and T12218 better capture the broader issues here.