Page MenuHomePhabricator

Prevent allocator race condition where all resources are in "ALLOCATING" status
AbandonedPublic

Authored by hach-que on Sep 18 2014, 1:33 AM.
Tags
None
Referenced Files
F14475406: D10511.diff
Fri, Dec 27, 6:49 PM
Unknown Object (File)
Thu, Dec 26, 6:16 PM
Unknown Object (File)
Fri, Nov 29, 1:34 AM
Unknown Object (File)
Nov 25 2024, 7:55 AM
Unknown Object (File)
Nov 25 2024, 7:54 AM
Unknown Object (File)
Nov 20 2024, 11:33 PM
Unknown Object (File)
Nov 17 2024, 9:21 AM
Unknown Object (File)
Oct 20 2024, 10:35 PM

Details

Reviewers
epriestley
Group Reviewers
Blessed Reviewers
Maniphest Tasks
T2015: Implement Drydock
Summary

Ref T2015. This prevents a scenario where 50 lease requests come in, and there are currently no active resources for a blueprint. When this happens, the workers allocate resources and place them into the "ALLOCATING" status, but they then leave the lock (and leave it up to allocateResource to move it to "PENDING").

This means that when the 6th worker gains the lock (for a configuration where the maximum resources is set to 5), it may be looking at a scenario like this:

  • Resource 1: ALLOCATING
  • Resource 2: ALLOCATING
  • Resource 3: ALLOCATING
  • Resource 4: ALLOCATING
  • Resource 5: ALLOCATING

It can't lease against "ALLOCATING" resources, so it attempts to allocate a new resource, but this fails because the number of resources is at the maximum (and this count does take into account "ALLOCATING" resources).

Test Plan

Tested that at least existing allocations work, but I'll have to put this into a production system with high lease usage before I'm confident the issue is resolved.

Diff Detail

Repository
rP Phabricator
Branch
allocator-race
Lint
Lint Passed
SeverityLocationCodeMessage
Advicesrc/applications/drydock/worker/DrydockAllocatorWorker.php:319XHP16TODO Comment
Unit
No Test Coverage
Build Status
Buildable 2565
Build 2569: [Placeholder Plan] Wait for 30 Seconds

Event Timeline

hach-que retitled this revision from to Prevent allocator race condition where all resources are in "ALLOCATING" status.
hach-que updated this object.
hach-que edited the test plan for this revision. (Show Details)
hach-que added a reviewer: epriestley.
epriestley edited edge metadata.

Depends on multiple other changes with substantial feedback.

This revision now requires changes to proceed.Aug 8 2015, 6:20 PM