Page MenuHomePhabricator

Queue tasks from importing repositories at a lower priority level
Closed, ResolvedPublic

Description

@jmeador had the clever idea to queue tasks from initially importing repositories at a lower priority level. This is likely straightforward and should improve queue behavior overall by better aligning it with user expectations and desires.

A vaguely related issue is that all repository import tasks have the same priority level, which is also the default priority level. Because discovery is faster than import and the secondary ordering is task ID (i.e., the queue is roughly FIFO, as one would expect a queue to be), this tends to mean that we queue 1M message tasks, do all of those, they yield 1M change tasks, we do all those, then we do 1M owners tasks, then we do 1M herald tasks, then we're finally done.

Likewise, if someone pushes 1K commits we import all of them before starting builds for any of them.

Both of these behaviors are probably not in good alignment with user expectation or desire. We could improve this by:

  • Adding a slightly-above-default level like PRIORITY_FINISH_WHAT_YOU_STARTED. Harbormaster builds and followup tasks from normal imports would run at this priority level so that we'd tend to finish task pipelines completely before starting new ones.
  • Moving initial tasks queued from importing repositories to a new PRIORITY_INITIAL_IMPORT with very low priority -- this might just reuse the existing PRIORITY_IMPORT, which is currently only used for search index rebuilds.
  • Adding a new PRIORITY_FINISH_WHAT_YOU_STARTED_INITIALLY_IMPORTING priority level slightly above the PRIORITY_INITIAL_IMPORT level so that we work through the import pipeline on a commit-by-commit basis instead of a task class by task class basis.

These changes are likely not too complicated, since there isn't any meaningful complexity associated with introducing new priorities.