Page MenuHomePhabricator

Working copy leases are randomly released when drawing on a non-instant host resource pool
Closed, InvalidPublic

Description

I'm getting these kinds of scenarios happening for working copies that are configured to source host leases from a non-instant resource pool (i.e. not Almanac):

pasted_file (668×825 px, 72 KB)

Naturally this causes the build to fail.

However the host lease is actually fulfilled and activates successfully:

pasted_file (884×958 px, 136 KB)

This only happens most of time, and happens extremely frequently under high load. I have to restart these builds again until they work.

I can't seem to figure out a reason why the lease is released, and no information is given for what triggers it?

To reproduce:

  • Apply D14583, or presumably make Almanac host allocator yield + activate instead of doing activateOnAllocate for resources / leases.
  • Do a bunch of working copy leases and observe them fail.

Event Timeline

hach-que raised the priority of this task from to Needs Triage.
hach-que updated the task description. (Show Details)
hach-que changed the edit policy from "All Users" to "Custom Policy".
hach-que added projects: Bug Report, Drydock.
hach-que added a subscriber: hach-que.
epriestley claimed this task.
epriestley added a subscriber: epriestley.

This does not describe a bug with the upstream. We do not have the resources to investigate or fix problems which begin with "apply <large amount of custom code>". See also T5447.