I think everything here is now fully cycled, synchronized, and cleaned up.

Taking care of these now. I expect everything to be pretty routine.

P2110 has a very rough cut of this.

Sorry! That reproduces cleanly for me too. I must have made a mistake following the reproduction steps or overlooked something. D19768 should fix this.

The immediate issue now appears fixed and it seems likely that the root cause is also fixed. We should still hunt down the bad cache behavior in the presence of wonky permissions.

This cropped up again and the permissions error also resurfaced, even though I previously nuked the directory.

To reproduce this issue, just need to Click on Default Column -> View as Query then leaving without Saving the query.

See PHI916. T11908 should move forward to the "migrate 700 callsites" stage, at least.

See PHI755, which could use verification around WorkingCopy blueprint selection.

D19762 adds a "supplemental allocation" behavior, which basically lets blueprints say "I want to grow the pool instead of allowing this otherwise valid lease acquisition".

After that, both hosts will have resources and jobs will allocate randomly, which should be good enough.

I believe you can work around this today by disabling the binding to host "A" in Almanac, running one job (which will be forced to allocate on host "B"), then re-enabling the binding. After that, both hosts will have resources and jobs will allocate randomly, which should be good enough. This is exceptionally cumbersome and ridiculous, of course (and it's possible that it doesn't even work).

A specific subcase here is when the binding to an Almanac host has been disabled. We should possibly test this during Interface construction, treat it as a failure, then recover from it.

Some of this touches Drydock but I'm just kicking it off the board since this mostly turned into a Harbormaster task.

I believe D16594 should implement this, one way or another, unless I'm misunderstanding the request.

Complicating this: there is no drydock.lease.search call upstream. So you're probably running some variation of D16594? But that already has ownerPHIDs.

goodbye pup

See PHI943. We should verify the modern behavior of PhabricatorGlobalLock in the presence of KILL <id> on the GET_LOCK() connection. I believe it is clear and explicit after D19702, which made the catch (...) more narrowly scoped.

My theory on the Drydock behavior is that we're getting into an "A, A, A" state. A lease comes in that needs a "B", so it releases a resource.

Next, I hit T13179 again. Yeesh. I rebased on my local hack-around for now.

From previous work, I already have functional Host and Working Copy blueprints locally so I'm likely punting on the Hoax/EC2 issue.

I'm happy to make these changes myself, or you mentioned wanting to contribute a patch?

I'm hoping to do a relatively small iteration here. The primary goal is to investigate PHI570, which reports some sketchy behavior. I generally want to unblock adopters of Drydock and fix critical weirdness but avoid improving it too much (i.e., avoid a real iteration on T13073) until we can clear other stuff out of the queue.