Page MenuHomePhabricator

Testing for changed refs in repositories with >65K refs misses ref cache
Closed, ResolvedPublic

Description

See PHI1979. During the "discovery" step of import, while testing to see if any ref positions are unknown, a repository with ~200K refs queries them into cache. The cache only holds 65K entries, so the first 65K populate the cache and can be tested quickly, while the other ~130K hit the database one at a time. That is, this should require 4 queries, and actually requires ~130,001 queries.

This can likely be remedied easily by testing refs in blocks that are no larger than the maximum size of the cache.

Revisions and Commits

Restricted Differential Revision
Restricted Differential Revision

Related Objects

Event Timeline

epriestley triaged this task as Normal priority.Jan 25 2021, 4:56 PM
epriestley created this task.

I updated the description, but the relevant workflow is during the "discovery" step, not the "refs" step. The "refs" step uses the RefCursor table and doesn't interact with the commit cache.

epriestley added a revision: Restricted Differential Revision.Jan 26 2021, 7:10 PM
epriestley added a commit: Restricted Diffusion Commit.Jan 26 2021, 8:27 PM
epriestley added a commit: Restricted Diffusion Commit.Jan 26 2021, 8:31 PM

This at least resolved the obvious badness in the case of PHI1979.