Page MenuHomePhabricator

Harbormaster v(-2)
ClosedPublic

Authored by epriestley on Oct 20 2013, 3:50 PM.
Tags
None
Referenced Files
F12849171: D7368.id.diff
Fri, Mar 29, 4:23 AM
Unknown Object (File)
Tue, Mar 26, 8:00 AM
Unknown Object (File)
Tue, Mar 26, 8:00 AM
Unknown Object (File)
Tue, Mar 26, 8:00 AM
Unknown Object (File)
Fri, Mar 22, 3:53 PM
Unknown Object (File)
Sat, Mar 16, 7:26 PM
Unknown Object (File)
Tue, Mar 12, 12:55 AM
Unknown Object (File)
Tue, Mar 12, 12:55 AM
Tokens
"Pterodactyl" token, awarded by Firehed."Love" token, awarded by dctrwatson."Mountain of Wealth" token, awarded by Unknown Object (User)."Baby Tequila" token, awarded by chad."Baby Tequila" token, awarded by asherkin."Orange Medal" token, awarded by seporaitis.

Details

Reviewers
btrahan
Maniphest Tasks
T1049: Implement Harbormaster
Commits
Restricted Diffusion Commit
rPb5a009337f8e: Harbormaster v(-2)
Summary

Ref T1049. I don't really want to sink too much time into this right now, but a seemingly reasonable architecture came to me in a dream. Here's a high-level overview of how things fit together:

  • "Build": In Harbormaster, "build" means any process we want to run against a working copy. It might actually be building an executable, but it might also be running lint, running unit tests, generating documentation, generating symbols, running a deploy, setting up a sandcastle, etc.
  • HarbormasterBuildable: A "buildable" is some piece of code which build operations can run on. Generally, this is either a Differential diff or a Diffusion commit. The Buildable class just wraps those objects and provides a layer of abstraction. Currently, you can manually create a buildable from a commit. In the future, this will be done automatically.
  • HarbormasterBuildStep: A "build step" is an individual build operation, like "run lint", "run unit", "build docs", etc. The step defines how to perform the operation (for example, "run unit tests by executing 'arc unit'"). In this diff, this barely exists.
  • HarbormasterBuildPlan: This glues together build steps into groups or sequences. For example, you might want to "run unit", and then "deploy" if the tests pass. You can create a build plan which says "run step "unit tests", then run step "deploy" on success" or whatever. In the future, these will also contain triggers/conditions ("Automatically run this build plan against every commit") and probably be able to define failure actions ("If this plan fails, send someone an email"). Because build plans will run commands, only administrators can manage them.
  • HarbormasterBuild: This is the concrete result of running a BuildPlan against a Buildable. It tracks the build status and collects results, so you can see if the build is running/successful/failed. A Buildable may have several Builds, because you can execute more than one BuildPlan against it. For example, you might have a "documentation" build plan which you run continuously against HEAD, but a "unit" build plan which you want to run against every commit.
  • HarbormasterBuildTarget: This is the concrete result of running a BuildStep against a Buildable. These are children of Build. A step might be able to produce multiple targets, but generally this is something like "Unit Tests" or "Lint" and has an overall status, so you can see at a glance that unit tests were fine but lint had some issues.
  • HarbormasterBuildItem: An optional subitem for a target. For lint, this might be an individual file. For unit tests, an individual test. For normal builds, an executable. For deploys, a server. For documentation generation, there might just not be subitems.
  • HarbormasterBuildLog: Provides extra information, like command/execution transcripts. This is where stdout/stderr will get dumped, and general details and other messages.
  • HarbormasterBuildArtifact: Stores side effects or results from build steps. For example, something which builds a binary might put the binary in "Files" and then put its PHID here. Unit tests might put coverage information here. Generally, any build step which produces some high-level output object can use this table to record its existence.

This diff implements almost nothing and does nothing useful, but puts most of these object relationships in place. The two major things you can't easily do with these objects are:

  1. Run arbitrary cron jobs. Jenkins does this, but it feels tacked on and I don't know of anyone using it for that. We could create fake Buildables to get a similar effect, but if we need to do this I'd rather do it elsewhere in general. Build and cron/service/monitoring feel like pretty different problems to me.
  2. Run parameterized/matrix steps (maybe?). Bamboo has this plan/stage/task/job breakdown where a build step can generate a zillion actual jobs, like "build client on x86", "build server on x86", "build client on ARM", "build server on ARM", etc. We can sort of do this by having a Step map to multiple Targets, but I haven't really thought about it too much and it may end up being not-great. I'd guess we have like an 80% chance of getting a clean implementation if/when we get there. I suspect no one actually needs this, or when they do they'll just implement a custom Step and it can be parameterized at that level. I'm not too worried about this overall.

The major difference between this and Jenkins/Bamboo/TravisCI is that all three of those are plan-centric: the primary object in the system is a build plan, and the dashboard shows you all your build plans and the current status. I don't think this is the right model. One disadvantage is that you basically end up with top-level messaging that says "Trunk is broken", not "Trunk was broken by commit af32f392f". Harbormaster is buildable-centric: the primary object in the system is stuff you can run build operations against (commits/branches/revisions), and actual build plans are secondary. The main view will be "recent commits on this branch, and whether they're good or not" -- which I think is what's most important in a larger/more complex product -- not the pass/fail status of all jobs. This also makes it easier and more natural to integrate with Differential and Diffusion, which both care about the overall status of the commit/revision, not the current status of jobs.

Test Plan

Poked around, but this doesn't really do anything yet.

Diff Detail

Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

More concisely:

  • BuildPlan Description of what to do to code and when ("Run unit tests against every commit in rE 'master', and email alincoln if it fails.")
    • BuildStep Individual part of a build plan (like "run tests" or "build docs").
  • Buildable Commit or revision. Primary Harbormaster object.
    • Build Progress or result of running a BuildPlan against a Buildable ("Ran unit tests and deployed." or "Unit tests failed" or "Documentation building").
      • BuildTarget Progress or result of running a BuildStep against a Buildable ("Ran unit tests.").
        • BuildItem Progress or result of fine-grained details of a target ("Unit test TestXYZ passed", "Lint for file Y").
          • BuildLog Raw console output, etc.
    • Artifact A useful side effect of a build (coverage, binary, library, etc.)

I was a little scared of this diff but the code is straight forward.

Architecture wise, I dig it. My thoughts are

  • Are these terms overloaded at all for folks who compile a bunch? Ideally this would map nicely to how people are thinking about builds in general and I think it does, but wanted to double check. I don't want to fight some existing lexicon if we can avoid it.
  • The architecture allows this but I think folks will want to run fairly arbitrary code here. I just say this because while I agree cron / service monitoring / etc don't belong here, I think people will write pretty interesting things here.
  • On that note, this feels like maybe these build plans should be created via herald?
    • eg you'd make a rule like for new differential criteria, in project foo, compile, deploy to test tier, etc?
    • "buildable" could easily become other things that aren't code then maybe. this is probably a con of this approach.
    • already have some flexibility on condition(s) which could determine if buildplan(s) should run
  • Or maybe its part of a project / repository?
    • eg you'd configure your project / repository to do various steps for new diffs / commits
    • firmly locks this into code changes
    • would probably need some Herald-like conditions
      • "if intern, test the crap out of this and don't deploy"

Are these terms overloaded at all for folks who compile a bunch? Ideally this would map nicely to how people are thinking about builds in general and I think it does, but wanted to double check. I don't want to fight some existing lexicon if we can avoid it.

I think it's pretty good, and similar to terminology used by other products in the space.

  • TravisCI uses "Build" the same way this does, has "Build History", etc, although it doesn't have some of the other concepts.
  • Jenkins uses "Build" the same way this does, has "Build History", etc. It uses the term "Job" instead of "BuildPlan". It has a popular plugin which uses the term "Artifact" the same way this does. It doesn't have most of the other concepts, or I they don't have obvious names in the UI ("Log" looks like it's called "Console", maybe). It has "Workspaces", which are similar to "Buildables", but not really the same because of differences in approach.
  • Bamboo uses "Build" in the same way this does. I think it uses "Job" instead of "BuildPlan", "Stage" instead of "BuildStep" (not quite a 1:1 mapping). It has "Logs" and "Artifacts". It seems like tests are first-class instead of being abstracted as "Items".

So I think the overlap is pretty good and the terms should be familiar to people coming from other software in this space.

The architecture allows this but I think folks will want to run fairly arbitrary code here. I just say this because while I agree cron / service monitoring / etc don't belong here, I think people will write pretty interesting things here.

As long as they're running the code in response to a diff or commit coming into existence, we should be fine. If the code is running in response to other events I think we should collect those requirements and build a different service for "random stuff you want to run", or just say "use cron" / whatever.

On that note, this feels like maybe these build plans should be created via herald?

Yeah, I've been kind of thinking about that. My thinking so far is that the ruleset will be different enough that they don't overlap much (for instance, I can't think of too many cases where you'd want to run build stages only for authors or certain files), but you'll often want to run continuously (e.g., when this completes, run it again if there's a new HEAD), which isn't easily expressible in Herald. But we might end up closer together here than I'm currently thinking.

Or maybe its part of a project / repository?

In many cases, I think the same build plan will want to be run on a lot of different repositories/projects, which is why I haven't locked them together more tightly. I do think "repository" will probably be part of the rules we need. Something like this would probably get 90% of the use cases:

Run on [continuously | on every commit] in these repositories:
  Repository: (rX, branch master), (rY, branch develop), (rSVN, path trunk/master/)

If that's more like 99% than 90% than maybe that's all we need and you just do the rest in your build script. If that's more like 60% than 90% then maybe we bring Herald into the action more.

Oh, and there would need to be vaguely similar rules for Differential, which I haven't thought about too much.

We probably could additionally have a Herald action "Execute build plan", so maybe that's how we deal with the other 40%? That actually seems pretty reasonable to me. Which I guess is what you're saying. So, good job, I agree we should do that.