HomePhabricator

Make Drydock reclaim unused resources when it reaches a resource limit

Description

Make Drydock reclaim unused resources when it reaches a resource limit

Summary:
Fixes T9994. Currently, when Drydock can't allocate a new resource because some limit has been reached, it waits patiently for a resource to become available.

It is possible that no resource will ever become available. Particularly with "Working Copy" resources, the new lease may want a copy of rB, but the resource may already be maxed out on rA.

Right now, no process exists to automatically reclaim the unused rA.

When we encounter this situation, try to reclaim one of the other resources if it is just sitting there unused.

Specifically:

  • Add a "reclaim" command which means "release this resource if it is completely unused".
  • Add a bin/drydock reclaim to send this command to every active resource.
  • When we try to acquire a resource and can't, but only because of some kind of limit / utilization problem, try to release an unused resource to free up some room.

Test Plan:

  • Set "Working Copy" resource limit to 1.
  • Ran "Test Configuration" in rA, which worked.
  • Ran "Test Configuration" in rB, which hung forever.
  • Applied patch.
  • Ran "Test Configuration" in rB, saw it reclaim the rA resource, use the slot, then succeed.
  • Ran "Test Configuration" in rA again, saw it grab the slot back.
  • Ran bin/drydock reclaim and saw it reclaim a bunch of old orphaned resources.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9994

Differential Revision: https://secure.phabricator.com/D14819