We're currently running into the following scenario with Drydock:
- A build starts and grabs a lease, which allocates an EC2 instance.
- At some point, someone restarts the daemons while the builds are still running.
- The lease never gets released because the daemon that was using it is terminated.
There is currently no way to track a lease back to the thing that obtained it, and no way to determine whether or not that thing still requires the lease.
I'm proposing that we add an ownershipPHID to leases, where the target object implements LeaseOwnerInterface or something of that nature. This provides a method stillRequiresLease which is used to evaluate whether or not a lease is still required, based on the status of the object.
In the above scenario, the build would be the owner; and the build would be in a failed status, so it knows that it can release the lease (and we perhaps just run this logic inside the GC daemon?)