Page MenuHomePhabricator

In Harbormaster, release artifacts as soon as no waiting/running build steps will use them
ClosedPublic

Authored by epriestley on Jun 17 2016, 9:15 PM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Apr 8, 12:18 PM
Unknown Object (File)
Sat, Apr 6, 12:45 PM
Unknown Object (File)
Fri, Apr 5, 7:53 PM
Unknown Object (File)
Thu, Apr 4, 6:59 PM
Unknown Object (File)
Tue, Apr 2, 12:55 PM
Unknown Object (File)
Mon, Mar 25, 6:36 PM
Unknown Object (File)
Mon, Mar 25, 4:30 PM
Unknown Object (File)
Mon, Mar 25, 4:15 PM
Subscribers
None

Details

Summary

Ref T11153. If you have a build plan like this:

  • Lease machine A.
  • Lease machine B.
  • Run client-tests on machine A.
  • Run server-tests on machine B.

...and we get machine A quickly, then finish the tests, we currently do not release machine A until the whole plan finishes.

In the best case, this wastes resources (something else could be using that machine for a while).

In a worse case, this wastes a lot of resources (if machine B is slow to acquire, or the server tests are much slower than the client tests, machine A will get tied up for a really long time).

In the absolute worst case, this might deadlock things.

Instead, release artifacts as soon as no waiting/running steps take them as inputs. In this case, we'd release machine A as soon as we finished running the client tests.

In the case where machines A and B are resources of the same type, this should prevent deadlocks. In all cases, this should improve build throughput at least somewhat.

Test Plan

I wrote this build plan which runs a "fast" step (10 seconds) and a "slow" step (120 seconds):

Screen Shot 2016-06-17 at 2.13.01 PM.png (1×1 px, 178 KB)

Before the patch, running this build plan held the lease on the "fast" machine for the full 120 seconds, then released both leases at the same time at the very end.

After this patch, I ran this plan and observed the "fast" lease get released after 10 seconds, while the "slow" lease was held for the full 120.

(Also added some var_dump() into things to sanity check the logic; it appeared correct.)

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

epriestley retitled this revision from to In Harbormaster, release artifacts as soon as no waiting/running build steps will use them.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.
chad edited edge metadata.
This revision is now accepted and ready to land.Jun 17 2016, 11:03 PM
This revision was automatically updated to reflect the committed changes.