(Although another benefit of T6139 is being able to surface build logs in emails that are sent out, so developers don't have to dig for the answer)

I don't really mind how it's implemented; it's mainly that if another task is required to set the reason for failure, then what happens when *that* fails or is otherwise not run (because the error is too fatal for the script or application that's being run to continue). Harbormaster right now doesn't have build steps that run on failure, so there's no post-build steps that can be guaranteed to run to assess the failure result.

While I'm at it, I'll add that the lack of this feature is a significant cause for developers ignoring build failures. Recently we had builds failing for at least over a week without anyone paying attention because there was no distinguishment between "failed to clone because of some network issue" and "the database scripts are broken and someone needs to really fix this now". It's now at the point where developers just ignore builds because it doesn't tell them what's wrong, and nothing happens about broken builds until a build engineer realizes and emails the rest of the development floor.

T6139 is also a request for this (I'll leave it up to @epriestley to decide which way to merge it).

Looks like I need to update to HEAD, since the Conduit call works here.

Right now we're working around it by having another build plan called "Post results to Phabricator" which has phid and state parameters. This build plan then runs the following PowerShell on a Windows build agent to post the result:

Is there any documentation on configuring Jenkins for commit builds (instead of differential builds)?

You mean from a Harbormaster Lease Working Copy perspective or a Land Revision perspective (when it comes to nonsensical UI).

@epriestley What is upstream's opinion on this? Is it something that can be prioritized or have an diff submitted and reviewed against it?

We have patched around this issue locally with:

Yes, it was because the SSH agent wasn't running (on Windows). I would have preferred to have it error and not submitted the diff, rather than having to do "arc diff" a second time to get the diff pushed to the staging repository.

This would make flags actually useful. I think the last time I used them was years ago, and I've just been adding comments to tasks whenever I need to add personal reminders.

@epriestley Is there any capacity to get this Prioritized?

I managed to temporarily workaround this issue by upping the limits on the blueprints, but obviously that is not ideal if the machines can't sustain that number of working copies in parallel.

I think it might be possible to cause a deadlock with limited leases even if the Expected Behaviour is achieved, approximately by devising a build plan like this:

Yes, that is expected behaviour. Only failed Harbormaster builds generate Feed transactions.

No, this works out-of-the-box for me.

I definitely see Harbormaster events in Feed queries, so I'm not sure what this task is asking for:

The sprint extension is not developed by this upstream. and is not supported by this upstream. You should contact the extension author instead.

Right, but that's because to me master being incorrect is a symptom of all branches being incorrect, which is what the task title and reproduction steps demonstrate.

This tasks describes a deficit in being able to fairly schedule builds, that is not having builds of one project consume all of the available Drydock resources. The response here (and in general it feels like most Drydock-related tasks) was roughly "add more resources". I don't believe "add more resources" is the appropriate or necessarily viable solution to every problem here.

You don't need to examine the remote branches beyond git fetch, which I'm assuming has already been done when the working copy is fetched. You just need to map the local branches to the remote ones (you can actually just do this cp .git/refs/remotes/origin/* .git/refs/heads).

In T10027#150524, @epriestley wrote:

I don't know of any scenario in which physical hardware or renting virtual machines is cheap?

I am building Harbormaster and Drydock for commercial companies, where the cost of hardware is small compared to the cost of personnel. I believe this to be true for essentially every commercial company. It is true even for us, and we don't pay salaries! We pay more for health insurance than for all development and production hardware combined. Hardware is incredibly cheap compared to personnel -- even personnel who work for free.

I'm not actually after a "current branch"; I know that there's potentially unlimited branches for the current commit. What I'm interested in is actually having the branches from the remote fetched down as the local branch names, so that "master" locally is the same as "origin/master" and "abcdef" is the same as "origin/abcdef", or to put it another way, as if the repository had been freshly cloned with "git clone".

Hmm, are you expecting that Harbormaster builds will only ever run for a few minutes then? The text in the Drydock documentation seems to indicate that you don't expect people to run builds that take hours.

Anyway, if the intent is that the limit property shouldn't used and it's usage isn't supported, then perhaps it should be removed?

I don't believe so, https://secure.phabricator.com/T4209#146794 clearly states that until that patch lands, daemons and repos must run on the same host.

Well my point is that I need some equivalent of "Lease Working Copy" that just provides an empty folder to work in.

I don't know of any scenario in which physical hardware or renting virtual machines is cheap? That seems like it excludes Drydock from being suitable for literally any system where you need to run commands on machines :/

I don't know what reliable reproduction steps you expect here. Drydock doesn't really give me any indication as to what it's doing or why it's doing nothing. The screenshots clearly indicate the issue at hand, and the accompanying text indicates what I did to reproduce the issue on my install.

Right, try stopping the pull daemons on the other tiers and leave only one running on the repository machine and see if that resolves your issue.

It means you don't need to run the taskmaster daemons on the same machines as the repositories because it can just make a Conduit call instead. You no longer need to run separate daemon tiers or use the --no-discovery flag.

You probably need to apply D11874, which fixes this issue.

Right now all of my build plans have "Wait for Previous Commits" so that builds for one project don't block builds for other projects due to Drydock limits being hit.

My concern is that the regression tests will consume all of the Drydock leases, so other builds that need working copies on the host will stall until all of the regression tests have finished running.

So I managed to workaround this issue. First I reset all the attributes of the blueprint with:

I just restarted the first build and looked at the lease that was trying to be obtained:

This is what I see in the daemon page (there are two builds trying to get a lease on this working copy blueprint):

Apparently even new builds that are trying to use this blueprint are now stalling.

How would I exclude tasks for projects that are not visible then? Or do I have to tag every single task with like an "Open Source" project and then make it filter based on that (this is undesirable because now every task just has a project on there so it can be queried on)?

In case the case of a limit being set, then yes, I expect it to fail immediately (and I expect them to fail until the resources drop below the limit).

I would expect 4 of the builds to fail as they can't allocate more resources.

No, I'd suggest that Drydock fail (and clean up resources used by failed builds immediately) instead of deadlocking.

If you drive without a seatbelt, it will be easier to escape from your vehicle when it becomes submerged in the ocean.

I believe there's another deadlock issue here as well though, say 10 builds start with the following plan:

T9875 is different to this. This discusses leases being deadlocked, whereas T9875 is there not being enough storage space on the host to hold copies of all working copies + build artifacts at once.

@epriestley I sure hope there's a badge for whoever creates T10000...

@epriestley I hit this again. These are some stats from SSHing into the machine:

Actually it looks like msysgit finally released Git 2.x for Windows, which is supposed to have fixed the race condition that was present on Windows in Git (that was supposedly the cause of the issue). When I return I'll get everyone to update in the hope that fixes the issue.

I believe this change will negatively affect the workflows we have at our workplace (when I return there I'll know for sure). Basically, we advise everyone to land with arc land --hold due to issues where interactions with msysGit and arc will cause commits to revert part of the branch (I can't find the task, it was filed ages and ages ago). So not being able to just check the landed commit to make sure it's okay and then hitting push will be a regression in the workflow that people are currently using...

whooop

(consistent edit endpoints in Conduit and custom task forms are what I'm most looking forward to)

Presumably this will be addressed when upstream implements running or embedding build plans.

We'll keep this patched locally while-ever we keep encountering the issue.

So you have a machine full of unused working copies, and Drydock is failing every build instead of reusing them?

This happens after upgrading Phabricator and having a build step implementation no longer exist; when the build plan runs you get an unexpected error. Getting the unexpected error isn't what's wrong - but that the buildable doesn't get marked as failed is incorrect.

What happens if a new configuration / request comes in that doesn't match any of the available 16 working copies?