Running Phabricator at commit `3bccb0d`. For some reason today our Harbormaster builds got stuck, and cannot be resumed or aborted:
- The "All Buildables" list in Harbormaster has about 20 builds that are all in the "Building" state, stuck on the "Lease Host" build step. Aborting a build puts the "Lease Host" step in the "Pausing" state, but it never gets out of that.
- The "Active Leases" list in Drydock has the same amount of leases in a "Pending" state. Releasing a lease puts it in the "Releasing" state, and adds a "release" command to the "Commands" list, but it never gets out of this state to be actually released.
A consequence of these stuck builds is that all our 16 daemons are stuck on `HarbormasterTargetWorker` tasks, with a growing queue of pending tasks that will never be processed, including commit parsers and search indexers. This basically renders Phabricator useless to us, until we can clear up those stuck builds or remove the `HarbormasterTargetWorker` tasks from the daemons so they can process tasks again.
Can you advise me on:
1. Where to find more info/logging about why the pending Drydock leases don't get through, and why they can't be released?
1. How to force-remove the stuck builds (and the pending Drydock leases) so our daemons can continue working?
Thanks!