This basically exhausts the queue because the commit worker tasks continuously fail forever and nothing else can ever be processed.
It looks like they don't actually fail forever, but with 20k commit tasks in the queue it still takes ages to go through each of them only to discover that they're no longer relevant.
If you're not at HEAD, there was a fix for large queues recently (see T6615).
It's expected that you'll get through ~100/sec of these per taskmaster (i.e., this is the actual rate of completion I realized with bin/worker flood + bin/phd debug taskmaster during D10895), and thus clear 20K in roughly 1 minute with 4 taskmasters. If you're seeing a dramatically slower rate than that at HEAD, we should fix whatever that underlying issue is, not try to dequeue things. See also T3554.
We had the fix for large queues applied.
They've cleared themselves overnight, but I don't think we were seeing times as long as 2us (we certainly weren't getting through 370k in 15 minutes). It was probably closer to about 100 tasks every 2 minutes.