Page MenuHomePhabricator

Harbormaster targets execute multiple times if they run for longer than 24 hours
Open, Needs TriagePublic

Description

Currently we have a build target that is executing forever, due to some incorrect logic in the script that it's executing. Unfortunately the lease that the taskmaster holds on the task only lasts for 24 hours, so when the lease expires, another taskmaster picks up the task and starts executing the target again. In our case, this script is responsible for launching EC2 instances, so every 24 hours we get another set of instances launched.

I think we need some way of the Harbormaster target worker to continually extend the lease expiry while ever the target is progressing; i.e. the "Run Command" build step would extend the expiry as it runs through this loop: https://secure.phabricator.com/diffusion/P/browse/master/src/applications/harbormaster/step/HarbormasterCommandBuildStepImplementation.php;04ee853cecf45ffb9ff7cb144d1225d5588b3982$69. This would ensure that the task expires if the taskmaster crashes, but not while ever the command is still running.

Event Timeline

hach-que assigned this task to epriestley.
hach-que raised the priority of this task from to Needs Triage.
hach-que updated the task description. (Show Details)
hach-que added a project: Harbormaster.
hach-que added a subscriber: hach-que.
eadler added a project: Restricted Project.Aug 5 2016, 4:54 PM