Lock resources briefly while acquiring leases on them to prevent acquiring near-death resources
Summary:
Depends on D19078. Ref T13073. Currently, there is a narrow window where we can acquire a resource after a reclaim has started against it.
To prevent this, briefly lock resources before acquiring them and make sure they're still good. If a resource isn't good, throw the lease back in the pool.
Test Plan:
This is tricky. You need:
- Hoax blueprint with limits and a rule where leases of a given "flavor" can only be satisfied by resources of the same flavor.
- Reduce the 3-minute "wait before resources can be released" to 3 seconds.
- Limit Hoaxes to 1.
- Allocate one "cherry" flavored Hoax and release the lease.
- Add a sleep(15) to releaseResource() in DrydockResourceUpdateWorker, after the canReclaimResource() check, with a print.
Now:
- Run bin/phd debug task in two windows.
- Run bin/drydock lease --type host --attributes flavor=banana in a third window.
- This will start to reclaim the existing "cherry" resource. Once one of the phd windows prints the "RECLAIMING" message run bin/drydock lease --type host --attributes flavor=cherry in a fourth window.
- Before patch: the "cherry" lease acquired immediately, then was released and destroyed moments later.
- After patch: the "cherry" lease yields.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13073
Differential Revision: https://secure.phabricator.com/D19080