I'm getting these kinds of scenarios happening for working copies that are configured to source host leases from a non-instant resource pool (i.e. not Almanac):
Naturally this causes the build to fail.
However the host lease is actually fulfilled and activates successfully:
This only happens most of time, and happens extremely frequently under high load. I have to restart these builds again until they work.
I can't seem to figure out a reason why the lease is released, and no information is given for what triggers it?
To reproduce:
- Apply D14583, or presumably make Almanac host allocator yield + activate instead of doing activateOnAllocate for resources / leases.
- Do a bunch of working copy leases and observe them fail.