Page MenuHomePhabricator
Create Task
Maniphest T9823

Expand Phacility cluster repo and db tiers (add nodes: 005)
Closed, ResolvedPublic
Actions

Description

See T9581 for previous expansion. The 004 nodes are pretty much full.

Revisions and Commits

Restricted Differential RevisionRestricted Diffusion Commit
Restricted Differential RevisionRestricted Diffusion Commit
Restricted Differential RevisionRestricted Diffusion Commit

Related Objects

Event Timeline

epriestley created this task.Nov 21 2015, 5:45 PM
epriestley claimed this task.
epriestley raised the priority of this task from to Normal.
epriestley updated the task description. (Show Details)
epriestley added projects: Ops, Phacility.
epriestley added subscribers: chad, epriestley.
epriestley added a comment.Nov 21 2015, 6:28 PM

I've brought up the 005 nodes. I also brought up web002 while I was in there, although we don't appear to be near capacity on web001 yet.

I haven't put any of these nodes in service yet, I want to try fixing T9291 first and using this as a test case for that.

epriestley added a revision: Restricted Differential Revision.Nov 21 2015, 8:12 PM
epriestley added a commit: Restricted Diffusion Commit.Nov 21 2015, 8:25 PM
epriestley added a revision: Restricted Differential Revision.Nov 21 2015, 8:29 PM
epriestley added a commit: Restricted Diffusion Commit.Nov 21 2015, 8:30 PM
epriestley added a commit: Restricted Diffusion Commit.
epriestley closed this task as Resolved.Nov 21 2015, 8:42 PM

The 005 nodes are in production. I'm not going to bother with web002 for now but I'll probably do the other half of that the next time I do ops stuff.

I reclaimed ~164% of a shard in unused capacity by improving the allocator, and the 005 nodes represent 100% of a shard, so I think we should have ~6-8 weeks of runway before I need to add more hardware.

epriestley added a comment.Nov 21 2015, 8:56 PM

There was a minor config issue with db005 but I sorted that out; no users were affected.

epriestley mentioned this in T10847: 30GB Phacility instance caused a series of cascading failures which left web services unreachable.Apr 21 2016, 6:53 PM
Phabricator · Built by Phacility