Thanks for your reply, the detailed description really helps. The build system works now, and even though the "Lease Working Copy" step doesn't really make sense for us, I'm really glad everything works :-).

"Lease Host" is not currently expected to work at all. It hasn't been updated for any of the changes, including authorizations. "Lease Working Copy" (under "Drydock") should.

(Yes, I know that Drydock is a prototype... But I figured it's better to file an issue that will be closed as invalid than to not file one at all?)

Thanks for reopening @tycho.tatitscheff. Before I document anything officially I'd like to wait for @epriestley's answer on T9671#142468, to make sure I'm not Doing It Wrong™ :-).

Reopened since the main concern was more the installation of T9669 and theire is still pending question.

@marceldegraaf : i did the first part (almanac + drydock almanac based host blueprint) of how to configure drydock here : Q179: How do you configure an Almanac based Drydock resource but not found time to get the second part (drydock working copy blueprint + harbormaster + herald). Say me if you wanna details the last part in a ponder question or shall I do it later when i have time.

Thanks, I set "Also Clone" to a random repository and that solves the exception I pasted. I got my builds working again. For future generations: this is how I configured Almanac/Drydock/Harbormaster now:

Basically you need to extra set the repo to be cloned. The UI is kinda misleading and the exception is not handled correctly when allow_phid is null

marceldegraf see T9669: Harbormaster working copy build step shouldn't crash when user don't set any also clone repository

Alright, I've managed to upgrade Phabricator and fix the storage schemata issues. I've also reconfigured Almanac, Drydock, and Harbormaster to reflect the latest changes regarding blueprint authorizations. My Harbormaster build step is authorized on the Drydock blueprint, so that looks good.

Upgrading Phabricator to the latest HEAD, and running ./bin/storage upgrade results in this error:

Drydock is a prototype and all of the Harbormaster build steps which interact with it are prototypes. Much of this code has been substantially rewritten in the last two weeks. See:

You'll want to update to HEAD if you need support. Please please please.

Thanks, I'll read those. However, I'd still like to know why Drydock leases suddenly stopped working, and why I can't cancel them to free up the daemons, and there's no way to cancel or remove the build tasks from the daemons, too. They just keep retrying forever.

You can find info on authorizations here: https://secure.phabricator.com/book/phabricator/article/drydock_blueprints/ and here: https://secure.phabricator.com/book/phabricator/article/drydock_security/

I have, a week or so ago. However, around October 12 "Blueprint Authorizations" were added to Drydock and those changes broke our entire build setup. There was no documentation on how to work with these Authorizations at the time (or I couldn't find it), so I rolled back to the last commit before those changes.

That commit is from October 8th, have you tried updating to HEAD?

Doing

diff --git a/src/applications/harbormaster/step/HarbormasterLeaseWorkingCopyBuildStepImplementation.php b/src/applications/harbormaster/step/HarbormasterLeaseWorkingCopyBuildStepImplementation.php
index 3c4d18f..9e20b6c 100644
--- a/src/applications/harbormaster/step/HarbormasterLeaseWorkingCopyBuildStepImplementation.php
+++ b/src/applications/harbormaster/step/HarbormasterLeaseWorkingCopyBuildStepImplementation.php
@@ -43,7 +43,9 @@ final class HarbormasterLeaseWorkingCopyBuildStepImplementation

If you agree can I submit a fix :

• same as rP43569d4e278c365a1feb9497acfa4ce336a8262c for first part
• if needed (maybe first part solve all later on by adding buildable.repository if no repository are selected) a modification of UI that say that clone instead of also clone and make it required.

Basically I see two problems:

• the fact that null isn't a normal argument and should be and empty array : it thus for me seems very similar with T9631: Don't crash when users fail to specify additional working copies and should be easy to fix
• the fact that it is not working while all seem's to be set correctly (that is the reason I posted all this (let me know if you need more). It seem's to be a reall problem too but it is probably just me being dumb !

I'm going to close this out as I think we're generally in good shape here and we're now hosting builds and doing server-side lands in the upstream, and don't anticipate much more Drydock-specific work in this iteration.

@epriestley : is making /var/drydock non hard written or just documented part of this ??

if you have a many-minutes-long build process anyway maybe shaving startup time from 2 minutes to 2 seconds doesn't really matter

I'm generally imagining we'll get something like this in the Blueprint configuration:

Progress here:

It's just this thing, basically:

Can I make it look better, or something?

I'm broadly pretty happy with this after using it for a bit, at least for use cases we have today. It does feel a little bit labor intensive, but not overwhelmingly so, and all the stuff is pretty well linked up so once you get the hang of it you're almost always only a click away from wherever you want to be to do the next thing. It's also reassuring to be able to go to a blueprint and check what might be using it at a glance.

Yeah, this is still feeling pretty reasonable to me after thinking about it for a day or two.

After giving this some more thought, the "I have to update lots of build plans" issue might be negated when Harbormaster supports embedding build plans (not just calling them). When that happens I could define a Lease Windows Host build plan with a Lease Host step with the right configuration, and then just ended that Lease Windows Host build plan where needed.

Protobuild is written in C#; it could abstract the build command to call, but I've avoided doing that because I don't want to hide this invocation. Too many build systems wrap commands which then wrap other commands and I always find it painful to pull them apart again to find out what's going on. So instead, Protobuild just generates the input solution files which can then be opened in any IDE or any build tool on the given platform.

Just a question from an observer of this conversation; @hach-que - wouldn't it be possible for Protobuild itself to use a script written in a widely available scripting language, like PHP/Ruby/Python/Bash which does the platform detection, thus allowing it to have pretty much the same build command for all platforms? Or perhaps even wrapping it in Make and require MinGW/Cygwin on Windows?
This is touching on what Google is doing with Bazel and what Facebook is doing with Buck, but taking it even further and having the same tool for all teams and technologies as well.

Yeah, that makes sense. Neither Travis nor CircleCI appear to support Windows at all, nor does Codeship, nor did ship.io before shutting down. And all of them just build off GitHub. So I think these are all very low-complexity build systems from the Harbormaster / Drydock / Jenkins / Bamboo perspective, but it makes sense that a lot of consumer web software doesn't need more than "build master off GitHub on linux".

I think largely the issue with that is that in this case the build system isn't controlled by me or open source. On Windows, the Microsoft-controlled MSBuild tool is used to build code, and on Linux / Mac the open source tools are used to build code. We can't replace those tools with something universal without also sacrificing the ability to open projects in the IDE. Protobuild (the project generator) goes a long way towards generating different project formats for different platforms, but it doesn't abstract the different build tool paths or command invocations.

Broadly, since I'd expect humans to sometimes run all the things that Harbormaster runs (to test, debug, profile, extend, etc., them), I'm assuming that this per-platform complexity should usually be pushed into projects so you can give humans one build process instead of separate per-platform build processes.

Like if I wanted to build MonoGame on Windows as a user, from the command-line (which you would almost never do, you'd use the IDE), I would run the same steps as Harbormaster.

Harbormaster also does Protobuild.exe --generate, but it has to run a different command line build tool after that depending on the host platform. Normally the graphical IDE just hides away that command invocation from a regular user.

So both the ".sln" file and Harbormaster have essentially separate copies of the same platform-to-build-steps ruleset? Why isn't it "Protobuild.exe --generate" for Harbormaster, too?

Most humans would load up MonoGame in their installed IDE, and wouldn't need to worry about the command-line tools to do that build. For humans it's "Protobuild.exe --generate" (it assumes current host platform, but users can pass "iOS" as an argument to target mobile platforms for example), and then opening the .sln file in whatever the default IDE is.

If you were telling a human how to do the MonoGame builds, you'd pretty much point them at a wiki document with like 10 different sections, each of which had different steps and build commands on it and discussion about platform specifics? And they'd go through 10 similar-looking but ultimately separate build processes?

So here's like a snapshot of a few build plans:

Can you walk me through why you have so many build plans? I understand that you might need one per, say, technology stack or team, but I don't understand why you'd ever have 100 unless you had a very large number of teams/projects, in which case I'd imagine it would be very rare to want to reconfigure any meaningful fraction of them at once.

@hach-que, are there cases I'm missing where double approval would pose an unreasonably high maintenance burden in your use cases?

We are now building all of the repositories, and all revisions submitted by members of Community.

I'm aware this is a prototype so i don't expect support at all. However if the new workflow is quite what is expected to be done, is it that hard to give some hints ? Like what is done here !
Anyway I will try to find myself.

https://secure.phabricator.com/book/phabricator/article/prototypes/

Couple of issues I've seen so far:

Allocate, but don't setActivateWhenAllocated(). You'll get a callback to activateResource() later. Check for an IP. If you have one, set it on the resource and call activateResource() to finish activation. If you don't have one yet, throw a yield and you'll get another call later. Repeat until you get an IP. Does that sound approximately reasonable?

@epriestley I just tried to start reimplementing some of my patches on top of HEAD and I've run into a problem:

I think I'm not explicitly going to include anything to deal with this in v1 since the v1 resource types "probably won't" break for no reason, but I think we likely do need to have more formal tools for it soon. I roughly imagine: