Page MenuHomePhabricator
Create Task
Maniphest T12331

Taskmasters on `admin.phacility.com` are not scaling up correctly
Closed, ResolvedPublic
Actions

Description

After the recent deploy, the admin taskmasters don't appear to be scaling up properly. This sometimes leads to queue backlogs, particularly when a large backup task blocks progress.

Revisions and Commits

rPHU libphutil
D17433rPHU76983f49b179 (stable) Fix an issue where daemons can stick in a pool
D17433rPHUc58d4fed2658 Fix an issue where daemons can stick in a pool
D17432rPHU1f45b7c36b53 (stable) Add more autoscale pool logging to daemons in verbose mode
D17432rPHU7463809e4bb8 Add more autoscale pool logging to daemons in verbose mode
rP Phabricator
D17431rP216f6be11ece (stable) Add "--pool" and "--duration" flags to daemon CLI tools
D17431rP90ec21f9996e Add "--pool" and "--duration" flags to daemon CLI tools

Related Objects

Event Timeline

epriestley created this task.Feb 28 2017, 12:06 PM
Herald added subscribers: chad, eadler. · View Herald TranscriptFeb 28 2017, 12:06 PM
epriestley added a revision: D17431: Add "--pool" and "--duration" flags to daemon CLI tools.Feb 28 2017, 3:31 PM
epriestley added a revision: D17432: Add more autoscale pool logging to daemons in verbose mode.Feb 28 2017, 3:37 PM
epriestley added a comment.Feb 28 2017, 3:43 PM

I'm picking those to stable and sending them to production now.

epriestley added a commit: rPHU7463809e4bb8: Add more autoscale pool logging to daemons in verbose mode.Feb 28 2017, 3:43 PM
epriestley added a commit: rP90ec21f9996e: Add "--pool" and "--duration" flags to daemon CLI tools.
epriestley added a commit: rPHU1f45b7c36b53: (stable) Add more autoscale pool logging to daemons in verbose mode.
epriestley added a commit: rP216f6be11ece: (stable) Add "--pool" and "--duration" flags to daemon CLI tools.
epriestley added a revision: D17433: Fix an issue where daemons can stick in a pool.Feb 28 2017, 4:10 PM
epriestley added a comment.Feb 28 2017, 4:14 PM

I'm picking D17433 now and sending it to admin.

epriestley added a commit: rPHUc58d4fed2658: Fix an issue where daemons can stick in a pool.Feb 28 2017, 4:15 PM
epriestley added a commit: rPHU76983f49b179: (stable) Fix an issue where daemons can stick in a pool.
epriestley added a comment.Feb 28 2017, 4:21 PM

The pool appears to be scaling up and down properly now. I'm going to deploy the fix to the secure and repo tiers since it may affect all instances.

epriestley closed this task as Resolved.Feb 28 2017, 4:37 PM
epriestley claimed this task.

I've deployed everything so I believe this is now resolved. I'll keep an eye on it going forward.

Phabricator · Built by Phacility