Page MenuHomePhabricator
Create Task
Maniphest T9581

Expand Phacility cluster repo and db tiers (add nodes: 004)
Closed, ResolvedPublic
Actions

Description

See T9283 for the previous adventure. See T9580 for immediate rationale, but we're pretty full here anyway, we've just been able to stretch things a bit more this time because we introduced test instances and those shut down after 48 hours instead of 45 days.

Related Objects
Search...

Event Timeline

epriestley created this task.Oct 15 2015, 6:38 PM
epriestley raised the priority of this task from to Normal.
epriestley updated the task description. (Show Details)
epriestley added projects: Phacility, Ops.
epriestley added subscribers: epriestley, chad.
epriestley added a comment.Oct 15 2015, 6:58 PM

I brought these nodes up normally, they're deploying now. I'll swap them over and vet things once that finishes up.

Smooth sailing so far, looks like the kinks got worked out pretty well by the last time around.

epriestley closed this task as Resolved.Oct 15 2015, 7:17 PM
epriestley claimed this task.
  • I closed the 003 nodes to new allocations and opened the 004 nodes.
  • I created a new test instance and verified it allocated on the 004 nodes.
  • I vetted backups, login, web, repo import, and daemon behavior on the new 004 nodes.
  • I also pruned some older test instances off of the 003 nodes and cycled them to give us a little more general headroom on repo003. Load should stabilize over time as more test instances suspend over time.

So I think we're in good shape. I'll keep an eye on things over the next few hours.

epriestley mentioned this in T9580: Slow POST/PUTs.Oct 15 2015, 7:43 PM
epriestley mentioned this in T9823: Expand Phacility cluster repo and db tiers (add nodes: 005).Nov 21 2015, 5:45 PM
Phabricator · Built by Phacility