Page MenuHomePhabricator

Lock resources briefly while acquiring leases on them to prevent acquiring near-death resources
ClosedPublic

Authored by epriestley on Feb 13 2018, 7:11 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Dec 22, 1:49 AM
Unknown Object (File)
Thu, Dec 19, 9:27 PM
Unknown Object (File)
Sat, Dec 14, 4:22 PM
Unknown Object (File)
Fri, Dec 6, 5:41 PM
Unknown Object (File)
Thu, Nov 28, 9:26 AM
Unknown Object (File)
Wed, Nov 27, 4:12 PM
Unknown Object (File)
Nov 19 2024, 9:18 PM
Unknown Object (File)
Nov 15 2024, 11:42 PM
Subscribers
Restricted Owners Package
Tokens
"Mountain of Wealth" token, awarded by timhirsh.

Details

Summary

Depends on D19078. Ref T13073. Currently, there is a narrow window where we can acquire a resource after a reclaim has started against it.

To prevent this, briefly lock resources before acquiring them and make sure they're still good. If a resource isn't good, throw the lease back in the pool.

Test Plan

This is tricky. You need:

  • Hoax blueprint with limits and a rule where leases of a given "flavor" can only be satisfied by resources of the same flavor.
  • Reduce the 3-minute "wait before resources can be released" to 3 seconds.
  • Limit Hoaxes to 1.
  • Allocate one "cherry" flavored Hoax and release the lease.
  • Add a sleep(15) to releaseResource() in DrydockResourceUpdateWorker, after the canReclaimResource() check, with a print.

Now:

  • Run bin/phd debug task in two windows.
  • Run bin/drydock lease --type host --attributes flavor=banana in a third window.
  • This will start to reclaim the existing "cherry" resource. Once one of the phd windows prints the "RECLAIMING" message run bin/drydock lease --type host --attributes flavor=cherry in a fourth window.
  • Before patch: the "cherry" lease acquired immediately, then was released and destroyed moments later.
  • After patch: the "cherry" lease yields.

Diff Detail

Repository
rP Phabricator
Branch
dry10
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 19527
Build 26428: Run Core Tests
Build 26427: arc lint + arc unit

Event Timeline

Owners added a subscriber: Restricted Owners Package.Feb 13 2018, 7:11 PM
epriestley retitled this revision from Lock resources breifly while acquiring leases on them to prevent acquiring near-death resources to Lock resources briefly while acquiring leases on them to prevent acquiring near-death resources.Feb 13 2018, 7:14 PM
This revision was not accepted when it landed; it landed in state Needs Review.Feb 13 2018, 9:22 PM
This revision was automatically updated to reflect the committed changes.