Page MenuHomePhabricator

Destroying a repository does not remove the commit worker tasks from the queue
Closed, WontfixPublic


This basically exhausts the queue because the commit worker tasks continuously fail forever and nothing else can ever be processed.

Event Timeline

hach-que assigned this task to epriestley.
hach-que raised the priority of this task from to Needs Triage.
hach-que updated the task description. (Show Details)
hach-que added a subscriber: hach-que.

It looks like they don't actually fail forever, but with 20k commit tasks in the queue it still takes ages to go through each of them only to discover that they're no longer relevant.

If you're not at HEAD, there was a fix for large queues recently (see T6615).

It's expected that you'll get through ~100/sec of these per taskmaster (i.e., this is the actual rate of completion I realized with bin/worker flood + bin/phd debug taskmaster during D10895), and thus clear 20K in roughly 1 minute with 4 taskmasters. If you're seeing a dramatically slower rate than that at HEAD, we should fix whatever that underlying issue is, not try to dequeue things. See also T3554.

Similar tasks at HEAD (4 taskmasters on my laptop):

Screen_Shot_2014-12-01_at_2.19.49_PM.png (143×1 px, 22 KB)

We had the fix for large queues applied.

They've cleared themselves overnight, but I don't think we were seeing times as long as 2us (we certainly weren't getting through 370k in 15 minutes). It was probably closer to about 100 tasks every 2 minutes.