I think everything here is now fully cycled, synchronized, and cleaned up.

Taking care of these now. I expect everything to be pretty routine.

P2110 has a very rough cut of this.

Sorry! That reproduces cleanly for me too. I must have made a mistake following the reproduction steps or overlooked something. D19768 should fix this.

The immediate issue now appears fixed and it seems likely that the root cause is also fixed. We should still hunt down the bad cache behavior in the presence of wonky permissions.

This cropped up again and the permissions error also resurfaced, even though I previously nuked the directory.

In T13208#242142, @epriestley wrote:

To reproduce this issue, just need to Click on Default Column -> View as Query then leaving without Saving the query.

These reproduction steps don't work for me, and don't make sense given how the software works. If you believe you have a set of reproduction steps, can you walk through them step-by-step in more detail so I can reproduce the issue?

If anyone can come up with reproduction steps which actually work, this will probably take me about 3 minutes to fix. The only thing blocking this is that I have no idea how to reproduce it.

To reproduce this issue, just need to Click on Default Column -> View as Query then leaving without Saving the query.

See PHI916. T11908 should move forward to the "migrate 700 callsites" stage, at least.

See PHI755, which could use verification around WorkingCopy blueprint selection.

D19762 adds a "supplemental allocation" behavior, which basically lets blueprints say "I want to grow the pool instead of allowing this otherwise valid lease acquisition".

After that, both hosts will have resources and jobs will allocate randomly, which should be good enough.

I believe you can work around this today by disabling the binding to host "A" in Almanac, running one job (which will be forced to allocate on host "B"), then re-enabling the binding. After that, both hosts will have resources and jobs will allocate randomly, which should be good enough. This is exceptionally cumbersome and ridiculous, of course (and it's possible that it doesn't even work).

A specific subcase here is when the binding to an Almanac host has been disabled. We should possibly test this during Interface construction, treat it as a failure, then recover from it.

Some of this touches Drydock but I'm just kicking it off the board since this mostly turned into a Harbormaster task.

I believe D16594 should implement this, one way or another, unless I'm misunderstanding the request.

Complicating this: there is no drydock.lease.search call upstream. So you're probably running some variation of D16594? But that already has ownerPHIDs.

goodbye pup

See PHI943. We should verify the modern behavior of PhabricatorGlobalLock in the presence of KILL <id> on the GET_LOCK() connection. I believe it is clear and explicit after D19702, which made the catch (...) more narrowly scoped.