Change Details

Because of the new locking mechanisms around Drydock allocation, this occurs much more frequently that it probably previously did. I thought about just adding the "bespoke" status like the comment said, but then I think we run into this scenario: * Build A and build B both start requesting leases. * Build A obtains the allocator lock when determining how to grab the lease. * Build B is waiting. * Build A exits the lock determining that it needs to allocate a resource, and has one sitting in STATUS_ALLOCATING. * Build A moves the resource into STATUS_PENDING, which now allows other things to take a lease on it. * Build B takes a lease on the Pending resource while Build A is still finishing setup (i.e. waiting for EC2 to fully start). * Build A moves the resource into STATUS_OPEN. * Build A attempts to allocate a lease, but the blueprint only allows one lease per resource, so it loses the allocation race. The "obvious" solution here is to disable taking leases on pending resources, but this results in a less desirable situation where if the blueprint allows more than 1 lease per resource, it will allocate more resources instead of leasing against pending resources (and resources generally equate to actual expenditure, whereas leases do not). I wonder if the right solution here is just to do: ``` $lease->setStatus(DrydockLeaseStatus::STATUS_ACQUIRING); $lease->setResourceID($resource->getID()); ``` right after we move the resource into STATUS_ALLOCATING, and before the allocator exits the lock?

Because of the new locking mechanisms around Drydock allocation, this occurs much more frequently that it probably previously did. I thought about just adding the "bespoke" status like the comment said, but then I think we run into this scenario: * Build A and build B both start requesting leases. * Build A obtains the allocator lock when determining how to grab the lease. * Build B is waiting. * Build A exits the lock determining that it needs to allocate a resource, and has one sitting in STATUS_ALLOCATING. * Build A moves the resource into STATUS_PENDING, which now allows other things to take a lease on it. * Build B takes a lease on the Pending resource while Build A is still finishing setup (i.e. waiting for EC2 to fully start). * Build A moves the resource into STATUS_OPEN. * Build A attempts to allocate a lease, but the blueprint only allows one lease per resource, so it loses the allocation race. The "obvious" solution here is to disable taking leases on pending resources, but this results in a less desirable situation where if the blueprint allows more than 1 lease per resource, it will allocate more resources instead of leasing against pending resources (and resources generally equate to actual expenditure, whereas leases do not). I wonder if the right solution here is just to do: ``` $lease->setStatus(DrydockLeaseStatus::STATUS_ACQUIRING); $lease->setResourceID($resource->getID()); $lease->save(); ``` right after we move the resource into STATUS_ALLOCATING, and before the allocator exits the lock? This should prevent Build B taking a lease against the pending resource, since there will already be a lease against it when it goes to check the lease count.

Because of the new locking mechanisms around Drydock allocation, this occurs much more frequently that it probably previously did. I thought about just adding the "bespoke" status like the comment said, but then I think we run into this scenario: * Build A and build B both start requesting leases. * Build A obtains the allocator lock when determining how to grab the lease. * Build B is waiting. * Build A exits the lock determining that it needs to allocate a resource, and has one sitting in STATUS_ALLOCATING. * Build A moves the resource into STATUS_PENDING, which now allows other things to take a lease on it. * Build B takes a lease on the Pending resource while Build A is still finishing setup (i.e. waiting for EC2 to fully start). * Build A moves the resource into STATUS_OPEN. * Build A attempts to allocate a lease, but the blueprint only allows one lease per resource, so it loses the allocation race. The "obvious" solution here is to disable taking leases on pending resources, but this results in a less desirable situation where if the blueprint allows more than 1 lease per resource, it will allocate more resources instead of leasing against pending resources (and resources generally equate to actual expenditure, whereas leases do not). I wonder if the right solution here is just to do: ``` $lease->setStatus(DrydockLeaseStatus::STATUS_ACQUIRING); $lease->setResourceID($resource->getID()); $lease->save(); ``` right after we move the resource into STATUS_ALLOCATING, and before the allocator exits the lock? This should prevent Build B taking a lease against the pending resource, since there will already be a lease against it when it goes to check the lease count.