Page MenuHomePhabricator

Make Drydock reclaim unused resources when it reaches a resource limit
ClosedPublic

Authored by epriestley on Dec 18 2015, 5:59 PM.
Tags
None
Referenced Files
Unknown Object (File)
Tue, May 7, 7:52 AM
Unknown Object (File)
Fri, May 3, 7:31 AM
Unknown Object (File)
Mon, Apr 29, 4:50 PM
Unknown Object (File)
Sat, Apr 27, 8:37 AM
Unknown Object (File)
Thu, Apr 25, 1:35 AM
Unknown Object (File)
Thu, Apr 11, 9:14 AM
Unknown Object (File)
Mon, Apr 8, 7:59 PM
Unknown Object (File)
Apr 7 2024, 10:13 PM
Subscribers
None

Details

Summary

Fixes T9994. Currently, when Drydock can't allocate a new resource because some limit has been reached, it waits patiently for a resource to become available.

It is possible that no resource will ever become available. Particularly with "Working Copy" resources, the new lease may want a copy of rB, but the resource may already be maxed out on rA.

Right now, no process exists to automatically reclaim the unused rA.

When we encounter this situation, try to reclaim one of the other resources if it is just sitting there unused.

Specifically:

  • Add a "reclaim" command which means "release this resource if it is completely unused".
  • Add a bin/drydock reclaim to send this command to every active resource.
  • When we try to acquire a resource and can't, but only because of some kind of limit / utilization problem, try to release an unused resource to free up some room.
Test Plan
  • Set "Working Copy" resource limit to 1.
  • Ran "Test Configuration" in rA, which worked.
  • Ran "Test Configuration" in rB, which hung forever.
  • Applied patch.
  • Ran "Test Configuration" in rB, saw it reclaim the rA resource, use the slot, then succeed.
  • Ran "Test Configuration" in rA again, saw it grab the slot back.
  • Ran bin/drydock reclaim and saw it reclaim a bunch of old orphaned resources.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

The build failed because of conflicts between my local working copy and resources on this host (I'm using the same server for both), I'll straighten that out in a sec.

chad edited edge metadata.
This revision is now accepted and ready to land.Dec 18 2015, 7:25 PM
This revision was automatically updated to reflect the committed changes.