Page MenuHomePhabricator

Split the GitHub import cursor into separate repository and issues event importers
ClosedPublic

Authored by epriestley on Mar 9 2016, 1:39 PM.
Tags
None
Referenced Files
F13254401: D15446.diff
Sat, May 25, 3:23 AM
F13247733: D15446.diff
Thu, May 23, 10:01 PM
F13233674: D15446.diff
Tue, May 21, 2:29 AM
F13229905: D15446.diff
Mon, May 20, 7:37 PM
F13211839: D15446.diff
Fri, May 17, 6:08 AM
F13206105: D15446.id37228.diff
Wed, May 15, 5:13 AM
F13206094: D15446.id37237.diff
Wed, May 15, 5:06 AM
F13206093: D15446.id.diff
Wed, May 15, 5:06 AM
Subscribers
None

Details

Summary

Ref T10538. The primary GitHub event activity stream does not report minor events (labels, milestones, etc).

GitHub has a second, similar activity stream which does report these events (the "Issues Events API").

Use two separate cursors: one consumes the primary stream; the second consumes the events stream.

One possible issue with this is that we may write events in a different order than they occurred, so GitHub shows "comment, label, close" but we show "comment, close, label" or similar. This is probably OK because the secondary API doesn't seem to have any very important events (e.g., it's probably fine if label changes are out-of-order), but we can conceivably put some buffer stage in between the two if it's an issue.

Test Plan

Screen Shot 2016-03-09 at 5.33.44 AM.png (1×1 px, 305 KB)

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

epriestley retitled this revision from to Split the GitHub import cursor into separate repository and issues event importers.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.
chad edited edge metadata.
This revision is now accepted and ready to land.Mar 9 2016, 3:20 PM
This revision was automatically updated to reflect the committed changes.