Page MenuHomePhabricator

Introduce ref cursors for repository parsing
ClosedPublic

Authored by epriestley on Jan 16 2014, 8:13 PM.
Tags
None
Referenced Files
F14065414: D7982.id18061.diff
Tue, Nov 19, 4:35 AM
F14065405: D7982.id18059.diff
Tue, Nov 19, 4:32 AM
F14054788: D7982.diff
Sat, Nov 16, 4:50 AM
F14042292: D7982.diff
Tue, Nov 12, 2:12 AM
F14028481: D7982.id18086.diff
Fri, Nov 8, 1:38 PM
F14025956: D7982.diff
Thu, Nov 7, 9:31 PM
F14020461: D7982.id18086.diff
Wed, Nov 6, 12:36 AM
F14020460: D7982.id18061.diff
Wed, Nov 6, 12:36 AM
Subscribers

Details

Summary

Ref T4327. I want to make change parsing testable; one thing which is blocking this is that the Git discovery process is still part of PullLocal daemon instead of being part of DiscoveryEngine. The unit test stuff which I want to use for change parsing relies on DiscoveryEngine to discover repositories during unit tests.

The major reason git discovery isn't part of DiscoveryEngine is that it relies on the messy "autoclose" logic, which we never implemented for Mercurial. Generally, I don't like how autoclose was implemented: it's complicated and gross and too hard to figure out and extend.

Instead, I want to do something more similar to what we do for pushes, which is cleaner overall. Basically this means remembering the old branch heads from the last time we parsed a repository, and figuring out what's new by comparing the old and new branch heads. This should give us several advantages:

  • It should be simpler to understand than the autoclose stuff, which is pretty mind-numbing, at least for me.
  • It will let us satisfy branch and tag queries cheaply (from the database) instead of having to go to the repository. We could also satisfy some ref-resolve queries from the database.
  • It should be easier to extend to Mercurial.

This implements the basics -- pretty much a table to store the cursors, which we update only for Git for now.

Test Plan
  • Ran migration.
  • Ran bin/repository discover X --trace --verbose on various repositories with branches and tags, before and after modifying pushes.
  • Pushed commits to a git repo.
  • Looked at database tables.

Diff Detail

Branch
xparser2
Lint
Lint Passed
Unit
Tests Passed

Event Timeline

epriestley updated this revision to Unknown Object (????).Jan 16 2014, 8:17 PM
  • Also remove ref cursors when deleting a repository.

I'm going to separate this out more and make it work with Mercurial, which changes a lot of the code.

epriestley updated this revision to Unknown Object (????).Jan 16 2014, 9:49 PM
  • Separate this out more, into a separate refs step.
  • Make it work with Mercurial, including multiple branch heads.
  • Add bin/repository refs to run it in isolation for testing/debugging.

Test plan:

  • Ran refs on SVN, Git and Mercurial repositories.
  • Ran discover on SVN, Git and Mercurial repositories.