Didn't see that ticket but just read over this and that would be what I would need. I think I am also seeing a very rare edge case as the specific issue I am seeing every day or so is that git clean -ffxd fails with warning: failed to remove /something but every time I go in to look around after everything looks completely normal and then running git clean -ffxd runs successfully.

T8153 possibly also related (detect someone broke the resource and destroy it so it gets recreated).

New APIs are going out tomorrow morning (Saturday, February 27). I expect to unprototype Almanac shortly after that, provided things go relatively smoothly, so it will become available the following week (Saturday, March 5).

When I put var_dump() in other parts of the code, I can see the output from /var/log/apache/error.log, but just can't see any output for the ones I mentioned above.

I think var_dump() will normally be hard to observe, since it will go to stdout but that isn't connected to anything by default so it will just vanish (we don't write stdout to the daemon log). My guess is that the code is running, you just aren't seeing the var_dump() output.

It looks like "git push" in DrydockLandRepositoryOperation.php doesn't get executed.
I put some var_dump as the following, and click "Land Revision", but no related log printed out.

Oh, this is probably actually $operation->setWorkingCopyVCSError(...);

To fix this:

Quick summary here:

Okay, I am able to expand further by adding more requests. Thanks for the explaining!

(And, obviously, all of this could be way more clear in the UI.)

Sorry, this actually is expected, if the resource pool is totally empty to begin with. Here's why:

Not at this time. This is far afield from our roadmap and we've seen little interest in it outside of your use case.

This looks much better! Thanks!

@epriestley Is there any capacity to get this Prioritized?

I don't want the error messages to just disappear since I worry that might be confusing, but I've added an option to dismiss errors once you're done with them in D14886:

This works for me now! Thanks!

Right, but that's because to me master being incorrect is a symptom of all branches being incorrect, which is what the task title and reproduction steps demonstrate.

That's entirely reasonable, but completely unclear from the task title, reproduction steps, description, or other context here. Here's an example of how you could have filed this task instead to make the root issue more clear:

You don't need to examine the remote branches beyond git fetch, which I'm assuming has already been done when the working copy is fetched. You just need to map the local branches to the remote ones (you can actually just do this cp .git/refs/remotes/origin/* .git/refs/heads).

That would needlessly slow down working copy construction in the common case, by adding extra unnecessary steps. Have your build tools examine remote branches instead of local branches, or do this synchronization yourself at the beginning of your build, or wait for T9492 and use "one-time" working copies in exchange for inefficient working copy construction.

I'm not actually after a "current branch"; I know that there's potentially unlimited branches for the current commit. What I'm interested in is actually having the branches from the remote fetched down as the local branch names, so that "master" locally is the same as "origin/master" and "abcdef" is the same as "origin/abcdef", or to put it another way, as if the repository had been freshly cloned with "git clone".

There is no such thing as a "current branch" in this sense in the general case. Harbormaster builds commits, and a commit may be on anything between zero and infinity branches, and may be the head of any equal or smaller number of branches.

Well my point is that I need some equivalent of "Lease Working Copy" that just provides an empty folder to work in.

I don't know what reliable reproduction steps you expect here. Drydock doesn't really give me any indication as to what it's doing or why it's doing nothing. The screenshots clearly indicate the issue at hand, and the accompanying text indicates what I did to reproduce the issue on my install.

This report does not include reproduction steps that we can use to reproduce the problem. See Contributing Bug Reports.

It doesn't make sense for "Lease Working Copy" to not provide a working copy.

You can probably set it to "0" to achieve the same effect, but this is a bug and I can reproduce it locally.

So I managed to workaround this issue. First I reset all the attributes of the blueprint with:

I just restarted the first build and looked at the lease that was trying to be obtained:

This is what I see in the daemon page (there are two builds trying to get a lease on this working copy blueprint):

Apparently even new builds that are trying to use this blueprint are now stalling.

Thanks @epriestley! I'll try it out.

@nickz I believe both issues are now fixed at HEAD. Let me know if you run into anything else.

Okay. You can implement the behavior you want like this:

In case the case of a limit being set, then yes, I expect it to fail immediately (and I expect them to fail until the resources drop below the limit).

I would expect 4 of the builds to fail as they can't allocate more resources.

I would expect 4 of the builds to fail as they can't allocate more resources.

How would you propose Drydock detect that it should fail if the situation is a legitimate deadlock that can not be alleviated by cleaning up resources?

No, I'd suggest that Drydock fail (and clean up resources used by failed builds immediately) instead of deadlocking.

So you'd suggest that we encourage Drydock users to configure limits in excess of available resources ("not wear a seatbelt") to improve error behavior when they configure a deadlock ("fall in the ocean")? If so, wouldn't it be better to simply remove the limits entirely ("uninstall seatbelts from cars") so that they can't accidentally configure a limit with bad behavior?

If you drive without a seatbelt, it will be easier to escape from your vehicle when it becomes submerged in the ocean.

I've reproduced the first issue with the lease breaking but not being released. Here's what I did:

If you drive without a seatbelt, it will be easier to escape from your vehicle when it becomes submerged in the ocean.

I believe there's another deadlock issue here as well though, say 10 builds start with the following plan:

For example, suppose we have perfect resource reclamation and reuse algorithms which always instantly destroy a resource the moment it is no longer in use.

The recommended solution for T9875 is to set a limit lower than "available disk space / size of largest repository", which reduces it to this deadlock issue.