There's some level of general interest in this, so I figured I'd centralize the rough level-of-effort estimates I've been throwing around.

There are some large pieces that aren't technical blockers which we can leave out of scope for now, depending on needs:

• Virtual Refs (T8093). This makes handoff (sending changes to a build/CI system) nicer (basically, more automatic/invisible) for users.
  • If you don't host repositories on Phabricator, this won't do anything for you and you don't need it.
  • If you do use hosted repositories, but you're fine with side-band handoff for now (see T8238), you don't need this yet. If you want nicer handoff later later, we could build it later.
  • If you do use hosted repositories, but you want to use arc patch handoff or some other kind of rickety-cart-based handoff, you don't need this.
  • If you definitely want nice handoff now, we need it now.
  • This is basically just polish and it's fairly involved so I suspect no one cares for now.
• Full Result Reporting (Part of T8097). This lets engineers review all test results in Phabricator, without going to an external tool, instead of just seeing pass/fail in Phabricator and clicking through.
  • If you're happy with having users click through to an external tool like Jenkins for now to get test details, and don't plan to CI lint yet/ever, you don't need this for now.
  • Instead, we'd just build enough of this to let you link to an external tool's result page.

All told, I think this roughly breaks out like this:

I'm pretty sure no one cares about T8093 right now since the tradeoffs for the side-band stuff in T8238 are pretty reasonable/moderate, so this is something like 10-15 hours of work, depending on how much reporting you care about having right now vs later.

Development Log

Total Hours: 0

Available Now

• (After Merge/Push) Old lint/unit/build results are reviewable in Differential.

Discussion

Feeble start on this, but getting the UI split went cleanly.

Next Up

The next major thing I want to do is have arc push lint/unit results into Harbormaster, and fall back to the current mechanism only if it fails.

Rough steps here are:

Build Plans for arc: We need a build to attach results. We need a build plan in order to create a build. We don't currently have one we can use, since the user may never have touched Harbormaster when they first run arc.

I think I'm going to add a specialKey to build plans and create "real" arc lint and arc unit plans implicitly. They will be immutable and implicit but have real PHIDs and IDs. I think this is the shortest path to everything just working properly.

Third-party tests in a similar situation which run proactively for some reason (I expect this to be very rare) can just hard-code plan PHIDs, but I don't want users to have to do a bunch of error-prone setup steps just to keep arc unit working. The alternatives (like virtual PHIDs and virtual objects) seem much messier and I think the UI/UX complexity of this approach will be very low ("it just works like you expect it to") and the internal complexity will be fairly low.

APIs: I'm going to build out modern-enough versions of the required core APIs, then special APIs for publishing results. This may derail slightly or significantly into T7715 but I want whatever I build now to survive the coming API apocalypse mostly unscathed. I expect this to be largely straightforward, the new SearchFields abstraction which supports this feels pretty clean to me and I think the boundaries are in a good place.

Result Storage: I need to plan this more specifically. I'm not sure if logs, test results, and unit results are all flavors of the same object, or if logs are different but test and unit results are the same thing, or if all three are different. I lean toward all three being different (and "artifacts" being a fourth, also different category, and build status being a fifth, different sort of result), I just want to think about this some more because I think the "0, 1, or N" rule is often a pretty good one, and "5" isn't any of those numbers. I think there are almost certainly strong use cases for each result/output type being a separate, distinct data type, though.

(There's also a sort of nebulous 6th result type, which is "data/properties", particularly links to external result set(s). I doubt these are a 6th distinct type -- maybe they're just special artifacts?)

Once I convince myself that "5" or "6" or something in this realm is a good value for "1", I'll build the actual storage and connect it to the API, then put UI on it.

The visible effect will be that arc reports results into Harbormaster instead of Differential (using generic APIs which are equally available to other build software), and both Harbormaster and Differential display this information in a reasonable way, and we're generally well-positioned to start making Harbormaster do more interesting things than sleep for 30 seconds.

This stuff is still behind Spaces in queue, so progress will be slow for a bit.

Development Log

Total Hours: 3

Available Now

• Nothing user-facing yet.

Discussion

Slow week in general, but I'm pretty happy with the actual code here. D13345 adds "autoplans", which feel pretty reasonable as a way to solve the "where does arc actually put results?" problem. These roughly follow the outline described above.

Next Up

Connecting arc to autoplans from the CLI, and the UI to Harbormaster. I expect this to be straightforward and resolve T8095.

Development Log

Total Hours: 5

Available Now

• Nothing user-facing.

Discussion

This stuff basically all works now but I want to clean it up a bit more before landing it.

Next Up

Finishing this up so we can cut over to Harbormaster ownership.