Page MenuHomePhabricator

When a large number of commits are pushed to a repository, import them at lower priority
Closed, ResolvedPublic

Description

See PHI1373. Particularly for remote-upstream-tracking repositories, some pushes are effectively "synchronize" operations and may append thousands of commits to a local repository.

This can stall commit imports for other repositories until the synchronize finishes.

Ideally, we'd probably like to process commits from different repositories in round-robin order: each repository with some commits to import gets to import one commit, then we repeat until everything is imported.

Given how the discovery process and queue work, we don't have great tools for approximating a fair/round-robin ordering.

You can currently work around this with a bin/repository mark-imported --mark-not-imported dance, but this requires operator intervention.

What we can reasonably do to move closer to round-robin without operator intervention is dump these big pushes down to PRIORITY_IMPORT, creating a two-tier system where small pushes process normally and big pushes process at low priority, even if they do not belong to importing repositories.

We already have this two-tier system, but we currently put imports in the low-priority tier only during initial import of a repository.

Related Objects