HomePhabricator

Provide a workflow to restart Harbormaster builds

Description

Provide a workflow to restart Harbormaster builds

Summary:
Ref T10867 for original use case. This workflow provides a plausible way for administrators to stop the daemons when performing upgrades or maintenance, then bring those daemons back up without resulting in the failure of builds that were running at the time.

On our organization's phab install, builds are running 24/7. The majority of these builds last for at least several minutes, and contain build steps which fail if interrupted and then resumed, as happens when turning daemons on and off.

Instead of allowing these build steps to resume execution as normal, this workflow will instruct active builds to restart their entire build process instead of just resuming whichever step they were on.

Test Plan:
contrived a build plan which would fail if resumed partway through:

  • lease a working copy
  • command touch restart_{build.id}
  • command test -e restart_{build.id} && rm restart_{build.id} && sleep 60

followed old procedure:

  • run a few of these builds manually
  • ./bin/phd stop
  • ./bin/phd start
  • saw the builds fail

followed new procedure:

  • run a few of these builds manually
  • ./bin/phd stop
  • ./bin/harbormaster restart --active
  • ./bin/phd start
  • saw the builds pass

Reviewers: epriestley, Blessed Reviewers

Reviewed By: epriestley, Blessed Reviewers

Subscribers: Korvin, epriestley

Maniphest Tasks: T10867

Differential Revision: https://secure.phabricator.com/D16485