Currently we have a build target that is executing forever, due to some incorrect logic in the script that it's executing. Unfortunately the lease that the taskmaster holds on the task only lasts for 24 hours, so when the lease expires, another taskmaster picks up the task and starts executing the target again. In our case, this script is responsible for launching EC2 instances, so every 24 hours we get another set of instances launched.
I think we need some way of the Harbormaster target worker to continually extend the lease expiry while ever the target is progressing; i.e. the "Run Command" build step would extend the expiry as it runs through this loop: https://secure.phabricator.com/diffusion/P/browse/master/src/applications/harbormaster/step/HarbormasterCommandBuildStepImplementation.php;04ee853cecf45ffb9ff7cb144d1225d5588b3982$69. This would ensure that the task expires if the taskmaster crashes, but not while ever the command is still running.