Page MenuHomePhabricator

Implement Harbormaster
Closed, ResolvedPublic

Subscribers
Tokens
"Mountain of Wealth" token, awarded by johnny-bit."Like" token, awarded by salehe."Like" token, awarded by michaeloa.
Assigned To
Authored By
seporaitis, Mar 26 2012

Description

I am thinking of writing a daemon and an interface for a simple build/continuous integration tool.

By 'simple' I mean:

Terminology:

  1. Job - a named list of user defined commands to be executed on a code from source code repository. Additionaly, Job can have subscribers - users who receive notifications about failures.
  2. Each Job execution is a build, that can be successful or not successful in both cases it would also have an output log attached to it.

UI:

  1. A list of Jobs page, where you can see the status of latest build;
  2. Clicking on a Job presents you with a Build list of that Job.
  3. "Create/Edit Job" - visible/used only by administrators (editing is also limited to administrators).
  4. "Subscribe to Job" - is possible for all users (like in Project view in Maniphest).

Daemon:

  1. Export some branch to a temporary directory (git archive 'master' | tar -x -C /tmp/<build-dir>)
  2. Execute arbitrary number of user commands, stop on first failure and save build output.
  3. Send a notification for subscribers.

The idea/rationale is: people at my work expressed they would like to have tools integrated into single interface rather than bunch of different urls to remember.

I don't have any code, thus I ask for triage - would you find this application worth merging into main project? And if yes, would you have any advice of how to proceed? In the sence that it is not a trivial change and should I create a single differential revision for all this, split this task into smaller ones, etc.? Any other helpful advice is ofcourse very apreciated too.

P.S. A name I am thinking for this application should add coolness too: "Phusion", which by definition means combining some pieces into a single entity; and fits the Phabricator naming convention. :-)

Details

Differential Revisions
D13651: [harbormaster/core] Allow build variables to be optional by prefixing them with '?'
D13639: Prevent 'Start or Wait for Build Plan' from running disabled build plans
D13635: [harbormaster/core] Implement parameterized build plans
D13625: [harbormaster/dashboard] Add panel for monitoring repository build status
D13506: [harbormaster/working-copies/v0] Implement lease working copy build steps
D12783: Implement leasing working copies from Drydock
D10556: [drydock/core] Support different shells in Harbormaster / Drydock
D10292: Allow users to restart builds at any time
D9113: Implement a very rough version of saving lint from Harbormaster
D8786: Allow Harbormaster HTTP steps to pass credentials
D8788: Allow Harbormaster build targets to wait for messages
D8791: Add a "Create build step" transaction to Harbormaster
D8792: Allow tasks to yield to other tasks
D8794: Add a rough `harbormaster.querybuildables` Conduit API method
D8795: Add "harbormaster.querybuilds" Conduit API
D8796: Drop nonsense buildStatus field from Buildable
D8799: Give Buildables a status, populate it, and return it over Conduit
D8800: Add `activeDiffPHID` to differential.query
D8801: Show build results in `arc land`
D8802: Implement PhabricatorApplicationTransactionInterface in Differential
D8803: Record build success or failure on buildable objects
D8609: Send emails to subscribers when builds pass or fail
D7771: Fix an issue in Harbormaster when a buildable has no repository
D7750: Allow users to choose what types of buildables they want to publish fragments for
D7745: Implement "Wait for Previous Builds" build step
D7715: Implement explicit build step ordering in Harbormaster
D7710: Remove `-t -t` from Drydock SSH interface
D7708: Make "Edit Build Plan" more resiliant so old build configurations can be deleted
D7707: Migrate "Run Command" to use Drydock hosts
D7706: Implement support for leasing from Drydock hosts in Harbormaster
D7703: Introduce HarbormasterBuildTarget to snapshot build steps through a build
D7697: Update preallocated hosts to use Passphrase credentials
D7695: Validate resource attributes for preallocated hosts before executing leases
D7582: Implement "Upload Artifact" build step
D7546: Allow users to create buildables from diffs
D7544: Render build status on revisions and commits
D7540: Added `build.id` variable and other improvements / fixes
D7537: Fix content being missed in the build logs
D7539: Put back the stronger variable replacement
D7521: Implemented support for build logs
D7519: Implement "Run Remote Command" build step implementation.
D7511: Remove build step implementations from Phabricator.
D7510: Move build step implementations to Arcanist and started work on build agents.
D7503: Implement "Run Local Command" build step implementation.
D7502: Make the buildables list a little nicer.
D7501: Use herald to trigger builds of revisions and commits.
D7500: Implement interface for adding, editing and deleting build steps on plans.
D7499: Add build step implementation infrastructure and sleep build step.
D7498: Implement basic Harbormaster daemon and start builds.
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Commits
D11870 / rPd5dc4588fce4: [harbormaster/abort-builds] Support aborting builds in Harbormaster
D10870 / rP254b394f2c2b: Associate Harbormaster build target with leases
D10362 / rPb163883d88c6: Hide "Warning: Permanently added ... to the list of known hosts." in…
D10336 / rPa26c6147f537: Prevent artifact key collision when builds are restarted
D10293 / rP2a4a30044b3c: Set the working directory when providing SSH / SFTP interfaces
D10174 / rPefc82c727bb4: Measure how long build targets take in Harbormaster
D10173 / rP3785f8113eba: Allow build steps to create URI artifacts
D10172 / rPbc116d7e02b4: Change "Stop" to "Pause" in Harbormaster build UI
D10093 / rPdfa9b27a949c: Use tabs on build targets and allow build steps to have a description
D10092 / rP298a30e647a8: Hide build target messages if there are no messages for the target
D9816 / rPa3d50118e117: Allow users to specify names of build steps
D9618 / rPf7f866445681: Move build variables into HarbormasterBuildableInterface
D8813 / rP9889892e5b76: Actually squelch Harbormaster "test passed" mail
D8608 / rPfc3b5ddce6a5: Prevent buildable list in Harbormaster from breaking when container or…
D8607 / rPe8e12910a70f: Implement a "credential" standard custom field
D8606 / rPacfc3c3e5d84: Add an "instructions" key to custom fields
D8605 / rP6e3c17e6f9fa: Don't create invalid build steps while adding them
D8604 / rPd6b937ca2796: Allow external systems to send messages to build targets
D8603 / rP25f91567a76b: Make various minor Harbormaster UI improvements
D8602 / rPcec8d1073150: Rename concrete Harbormaster step implementations
D8601 / rP281f06e2810a: Rename "BuildStepImplementation" to "HarbormasterBuildStepImplementation"
D8600 / rPa246c85c6b51: Use ApplicationTransactions and CustomField to implement build steps
D8599 / rP72337dedaff9: Make Harbormaster input and output artifacts more explicit
D8588 / rP5b74fa0a756c: Make all build steps support variables
D8587 / rP150a3adf2c78: Minor UI improvements for Harbormaster
D7899 / rP46139bd1f6a2: Allow entire buildables to restart/stop/resume
D7898 / rPb952b6f61973: Improve restart/stop/resume UI
D7897 / rP6dc582f459f2: Make HarbormasterBuild a better source of truth about…
D7892 / rP1786093c6e41: Replace "Cancel Build" with "Stop", "Resume" and "Restart"
D7890 / rP4d5e8a149a7d: Split Harbormaster workers apart so build steps can run in parallel
D7824 / rPac19c558225c: Formalize "manual" buildables in Harbormaster
D7825 / rPaad6b57c36f8: Add `bin/harbormaster` to make builds easier to debug
D7823 / rP4a56e26a6767: Allow buildables to be queried by container and underlying buildable
D7822 / rP60288edf80cb: Allow Harbormaster build plans to be disabled
D7746 / rP0ed281d25e44: Make taskmaster consumption of failed tasks more FIFO-ey
D7541 / rP56c65e33ef1b: Add a "Make an HTTP Request" build step
D7538 / rPHU1ffb66347246: Add `urisprintf()` for encoded printing of URIs
D7513 / rP4f0f95f7b587: Assign PHIDs to all diffs
D7512 / rP377742c23af0: Get some level of meaningful status information into Harbormaster buildable list
D7368 / rPb5a009337f8e: Harbormaster v(-2)

Related Objects

StatusAssignedTask
Resolvedepriestley
Resolvedepriestley
ResolvedNone
Wontfixepriestley
OpenNone
DuplicateNone
Resolvedepriestley
Resolvedepriestley
Wontfixhach-que
Resolvedhach-que
DuplicateNone
Resolvedepriestley
DuplicateNone
Resolvedhach-que
ResolvedNone
ResolvedNone
Duplicatehach-que
OpenNone
InvalidNone
Resolvedhach-que
Resolvedepriestley
OpenNone
ResolvedNone
DuplicateNone
Resolvedhach-que
Wontfixepriestley
Wontfixepriestley
Wontfixepriestley
ResolvedNone
Resolvedepriestley
OpenNone
OpenNone
OpenNone
Resolvedepriestley
OpenNone
Resolvedepriestley
Resolvedbtrahan
OpenNone
Resolvedhach-que
OpenNone
Resolvedepriestley
Wontfixepriestley

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
hach-que closed subtask Restricted Maniphest Task as Resolved.Aug 11 2014, 11:15 PM
mbeck removed a subscriber: mbeck.Aug 12 2014, 7:08 AM
gpaumier removed a subscriber: gpaumier.Aug 19 2014, 7:48 AM
bartus added a subscriber: bartus.Aug 19 2014, 5:13 PM
scfc added a subscriber: scfc.Sep 14 2014, 8:46 PM
J5lx added a subscriber: J5lx.Nov 4 2014, 5:00 PM
hvr added a subscriber: hvr.Nov 6 2014, 7:15 AM
epriestley changed the visibility from "All Users" to "Public (No Login Required)".Nov 22 2014, 3:24 PM
mrjoops added a subscriber: mrjoops.Dec 1 2014, 9:25 AM

Hi

Will it be possible to define the location of the temporary directory? You are mentioning /tmp/.... in the description, which might be good in most but not all situations. On my servers for example, I tend to mount /tmp and /var/tmp with noexec,nosuid,nodev which is considered a good security practise for various reasons (a lot of articles about that on the web).

The problem then is, that commands cannot be executed inside /tmp, so I'd need to define a custom location for builds.
Any thoughts on that?

Thanks.

The build locations are configured in Drydock on the host blueprints you have configured.

Unknown Object (User) added a subscriber: Unknown Object (User).Jan 22 2015, 7:27 AM
pere added a subscriber: pere.Jan 25 2015, 1:04 PM
johnny-bit added a subscriber: johnny-bit.
witrin added a subscriber: witrin.Feb 14 2015, 10:54 AM
nickz added a subscriber: nickz.Apr 21 2015, 11:03 PM
eadler added a subscriber: eadler.Apr 24 2015, 12:13 AM
eadler removed a subscriber: eadler.
pere removed a subscriber: pere.May 31 2015, 11:16 AM
tiguchi added a subscriber: tiguchi.Jul 5 2015, 6:44 AM
joshma added a subscriber: joshma.Jul 22 2015, 1:41 AM
epriestley closed this task as Resolved.Aug 8 2015, 7:14 PM

Harbormaster is now unprototyped. See T8089 for more focused/modern discussion.

qgil removed a subscriber: qgil.Sep 21 2015, 9:33 PM
jpoehls removed a subscriber: jpoehls.Sep 22 2015, 1:31 PM
urzds added a subscriber: urzds.Jul 12 2017, 11:14 AM