Page MenuHomePhabricator

Make Drydock reclaim unused resources when it reaches a resource limit
ClosedPublic

Authored by epriestley on Dec 18 2015, 5:59 PM.
Tags
None
Referenced Files
F14049905: D14819.diff
Thu, Nov 14, 3:05 PM
F14036257: D14819.diff
Sun, Nov 10, 9:23 AM
F14021777: D14819.diff
Wed, Nov 6, 12:03 PM
F14002592: D14819.id35833.diff
Fri, Oct 25, 8:21 PM
F13996435: D14819.id35829.diff
Wed, Oct 23, 9:08 PM
F13986557: D14819.diff
Mon, Oct 21, 3:51 AM
F13985511: D14819.id.diff
Sun, Oct 20, 9:02 PM
Unknown Object (File)
Oct 10 2024, 1:23 PM
Subscribers
None

Details

Summary

Fixes T9994. Currently, when Drydock can't allocate a new resource because some limit has been reached, it waits patiently for a resource to become available.

It is possible that no resource will ever become available. Particularly with "Working Copy" resources, the new lease may want a copy of rB, but the resource may already be maxed out on rA.

Right now, no process exists to automatically reclaim the unused rA.

When we encounter this situation, try to reclaim one of the other resources if it is just sitting there unused.

Specifically:

  • Add a "reclaim" command which means "release this resource if it is completely unused".
  • Add a bin/drydock reclaim to send this command to every active resource.
  • When we try to acquire a resource and can't, but only because of some kind of limit / utilization problem, try to release an unused resource to free up some room.
Test Plan
  • Set "Working Copy" resource limit to 1.
  • Ran "Test Configuration" in rA, which worked.
  • Ran "Test Configuration" in rB, which hung forever.
  • Applied patch.
  • Ran "Test Configuration" in rB, saw it reclaim the rA resource, use the slot, then succeed.
  • Ran "Test Configuration" in rA again, saw it grab the slot back.
  • Ran bin/drydock reclaim and saw it reclaim a bunch of old orphaned resources.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

The build failed because of conflicts between my local working copy and resources on this host (I'm using the same server for both), I'll straighten that out in a sec.

chad edited edge metadata.
This revision is now accepted and ready to land.Dec 18 2015, 7:25 PM
This revision was automatically updated to reflect the committed changes.