T9875 is different to this. This discusses leases being deadlocked, whereas T9875 is there not being enough storage space on the host to hold copies of all working copies + build artifacts at once.

This does not describe a bug with the upstream. We do not have the resources to investigate or fix problems which begin with "apply <large amount of custom code>". See also T5447.

@epriestley I hit this again. These are some stats from SSHing into the machine:

I can't reproduce this issue by following the steps in the report:

So you have a machine full of unused working copies, and Drydock is failing every build instead of reusing them?

What happens if a new configuration / request comes in that doesn't match any of the available 16 working copies?

still won't get cleaned up and instead Drydock will just fail to allocate.

It's running out of disk space because Drydock isn't cleaning up the working copies and they're consuming all the disk space.

So you have a machine full of unused working copies, and Drydock is failing every build instead of reusing them?

Because if I don't clean it up, then every build from then on will fail until I fix it?

When this happens, there'll be hundreds of builds all running at the same time, which can result in hundreds of working copies for the same repository.

fyi my Phabricator instance currently has 80 working copy resources open

This usually happens when rebasing an upstream repository after a significant amount of time (like six months). When this happens, there'll be hundreds of builds all running at the same time, which can result in hundreds of working copies for the same repository.

Because I have a lot of repositories, and there are multiple copies of each repository (because things are running in parallel), I will run out of disk space if I have multiple full copies of every repository in Phabricator kept on the build hosts for all time.

I'll need to at some point go and clean these up.

• I believe username association works correctly.
• Assigning an email is already mandatory when creating a bot:

I don't plan Almanac to ever autoscale. Autoscaling would be via a new, non-Almanac EC2 blueprint (or similar blueprints, which permit creation of new host resources). I'm just going to merge this into T5544.

fyi I ended up implementing Protobuild.exe --build and a bunch of other functionality (like support for build scripts), which negated the need for having separate build plans for each platform

Can you point out which parts of Contributing Bug Reports, User Guide: Prototype Applications and Support Resources are unclear to you with respect to this report so we can improve the documentation?

So I guess this bug is neither a known issue nor do you want me to provide more details since it's a prototype app?

Drydock is also a prototype, see User Guide: Prototype Applications.

We always need a fully formed bug report. See Contributing Bug Reports.

This obscurity is currently intentional, for policy/security reasons. I don't want to leak anything sensitive into the relatively-accessible web UI.

I'm going to repurpose this as "get that data in the database so we're better prepared if this happens again", but feel free to follow up if you go on a hunting expedition or hit the issue in the meantime.

Based on my memory, I think it's possible that exception is the cause of the issue; when it occurs, the exception ends the current worker task with a permanent failure, and the task won't be revisited. Thus the build remains pending because the worker that would have performed the next update failed to do so.

My next step would be to put more crumbs in the UI, but we're missing a little data right now. Basically, I'd do this:

We can leave this at "reduced expected incidence rate to < 1/3k" and I'll come back here if it happens again if you don't have any more obvious ideas.

Oh I lied, checked the old logs, this error was present the last two times it occurred, but not present when it happened this evening.

This perfectly recreates the symptoms, except it generates an error in the daemon log:

Specifically, if you put like a sleep(10) directly after applyPlan() I would expect that to trigger the race ~100% of the time as long as the queue isn't too full. You could see if the behavior is similar to the behavior you've observed.

Yeah, I think that can race. Here's an example -- open two connections to MySQL, mysql1> and mysql2>. In the first window do this:

DAO's prefixed CISubmitQueue are in their own database. There has never been any problems with referential integrity / never seen anything in the database that would indicate we'd written records incorrectly. But you may be right about the tasks being written before the plan is written. I suspect that would cause errors though, I can try to reproduce that first in order to rule out our specifics being the problem.

Let me post the whole block, it didn't seem relevant:

Oh, you removed openTransaction(), so that's probably not the culprit.

It's vaguely possible that openTransaction() is the culprit. applyPlan() queues a daemon task, and I could imagine the race might be:

@epriestley I have a reasonable implementation of this working on the new Drydock architecture. I want to pursue this before EC2 so we can ensure that all of the locking mechanisms are working correctly before we use the new Drydock on something that potentially costs money (and in the event that Drydock goes haywire, may cost thousands of dollars). KVM offers a safe alternative, because if Drydock goes haywire, the worst that will happen is that it won't be able to allocate new VMs on the KVM host due to out of memory errors.

In T9760#144113, @epriestley wrote:

Right, Almanac does not support autoscaling.

Right, Almanac does not support autoscaling. We'll have an EC2 blueprint (and similar) in the future which can manage autoscaling more simply. Almanac is the ground level "I have a list of IPs" version of things.

bin/almanac register is just a shortcut for "put the private half of a key in the right place on disk, and the public half on the specified Almanac device". Maybe a clearer name would be bin/almanac install-keys.

Right okay, but I can't autoscale the Drydock Pool right? Because I need to manually create new devices through the UI.

This just means adding a device interface?

In T9760#144107, @epriestley wrote:

• Bind it to your pool to actually put it in service.

So what does ./bin/almanac register actually do?

Right. You just need to:

Ok thanks. So I don't need to register my build agent with ./bin/almanac register?

Don't do steps (3) or (4)! Those are useless at best, and dangerous at worst.

This is now meaningfully achievable through Drydock as a repository operation, but we've seen very little interest in it.

Does committing as botusername <anything@anything> definitely not work?

D14368 should fix the issue where you're required to set a dummy "Also Clone" repository.