Working Copy Caches: I'd like to see a broader need for this before bringing it upstream. My expectation is that working copy resources themselves will provide adequate caching for most workloads, and this runs into complex constraint problems in T8671.

I have a pretty strong use case for working copy caches.

Basically working copies themselves don't cache anything; every build will clone a new working copy from the internet, with a single lease on each working copy (because multiple lease don't make sense). This means every build performs Git clone from the Internet.

We have build plans that clone repositories which have individual submodules that are >400MB to clone. The repository size as a whole is even larger than this (probably around 600MB+). Cloning this much data from the Internet means that builds take about 10 minutes or so just to get started, because of the data transfer rate. With working copy caches, this drops down to about 30 seconds, which is much more reasonable.

Basically working copies themselves don't cache anything; every build will clone a new working copy from the internet, with a single lease on each working copy (because multiple lease don't make sense). This means every build performs Git clone from the Internet.

This isn't how WorkingCopy resources behave at HEAD. The initial resource allocation will clone a new working copy, but we don't throw it away when we're done. Leases just do fetch + reset when acquired, so the same clone is used over and over again as long as the resource is alive.

Is there a reason this can't work with your build process?

Does it run git clean -xdff on the working copy at HEAD to ensure a clean working state?

Also how does HEAD behave if there are 5 builds running of the same repository in parallel? Do we end up with 5 working copies permanently on that machine, or do they expire over time (my concern here is disk space usage with a large number of repositories)?

Does it run git clean -xdff on the working copy at HEAD to ensure a clean working state?

Yes (well, d and f -- we should likely add x):

https://secure.phabricator.com/diffusion/P/browse/master/src/applications/drydock/blueprint/DrydockWorkingCopyBlueprintImplementation.php;24845c70b918789be5309f88ed3f6455f5f29748$160-162

Do we end up with 5 working copies permanently on that machine, or do they expire over time (my concern here is disk space usage with a large number of repositories)?

Currently, 5 permanent copies. Resources don't expire right now, but will be able to soon (T6569). Leases can already expire, resources just need the same mechanism copied over.

The cost of disk is so small that I'd expect that not expiring these generally costs a few dollars per year, but we'll be able to expire them if this is more of a concern in some environments.

It's mostly about running into "out of disk space" scenarios than storage costs, since when the disk runs out of space, typically everything breaks in a horrible fashion.

Yes (well, d and f -- we should likely add x):

You need -f twice to force the removal of submodules (hence -ff).