I needed to cancel 5 million queued worker tasks and the act of
canceling them via the cli was using 30gigs of memory without making
any progress. Adding a limit argument makes the cli usable for
deleting queued workers in batches of a reasonable size
(e.g. 100,000 at a time)
Details
Details
- Reviewers
epriestley - Group Reviewers
Blessed Reviewers
tested on wikimedia's fork in production. Without --limit I saw
a huge memory usage without making any progress. With --limit 100000
progress was possible.
Diff Detail
Diff Detail
- Repository
- rP Phabricator
- Branch
- master
- Lint
Lint Passed - Unit
Tests Passed - Build Status
Buildable 20422 Build 27734: Run Core Tests Build 27733: arc lint + arc unit
Event Timeline
Comment Actions
I think we should rewrite this to use the recent PhabricatorQueryIterator iterator (so it "just works") instead of having an explicit limit. That will be a slightly more involved change since we can't do some of these tests directly anymore. We also want to really return two iterators (active + archived) that act like one iterator, which we can do with MultipleIterator (but, PHP 5.3.0+) or by writing a copy of that which works with PHP 5.2.3, but that's even more involved. It might effectively be necessary anyway to get the setServerTime() call working transparently.
This is kind of a lot of weird stuff, so maybe just throw this in Maniphest with a pointer here and I'll figure out how to get PhabricatorIteratingMultipleIteratorIteratorForIterators or whatever actually fits the bill hooked up?