Alright, let's tackle this. At the very least it'll give us a good idea of what's working in Drydock and what's not.

The general use cases that Drydock needs to provide for in the region of build machines is:

1. A preset list of build machines that the user has defined (e.g. preallocated hosts).
2. Allocating out new hosts using various blueprints (AWS, etc.)
3. Finding a host with a suitable set of capabilities.
4. Allocating out a working directory on hosts of various types.
5. Invoking commands remotely on the host.
6. Cleaning up the working directory when the build finishes, regardless of whether the build succeeds or fails.
7. Cleaning up and terminating the host resource if it's not a preallocated resource.

Looking at how Drydock behaves for allocating a working copy, it seems like it'll handle the AWS scenario quite well, but we'll need some sort of interface in Harbormaster for defining preallocated build machines (wherein Harbormaster will provide a blueprint and use that to create a specific resource).

We also need some mechanism to ensure that requesting a working copy uses a particular host; I'm reasonably confident this can just be an attribute on the working copy lease (host.lease=4) when requesting it.

If this all sounds good, I think we should proceed like so:

1. Define DrydockPreallocatedHostBlueprint. This should only answer to leases with an attribute of preallocated=yes and trust the attributes to provide information about the resource it allocates. I'm reasonably confident that this should allocate out resources with a type of host. It can then provide the command interface based on the attributes of the resource. The resources it allocates should also have attributes preallocated=yes and remote=yes (in contrast with DrydockLocalHostBlueprint which will only provide where remote=no).
2. Add a UI in Harbormaster to list preallocated host resources and create / edit new ones. This will basically query drydock resources where they are of the host type with attribute preallocated=yes. This UI will allow the user to define the SSH connection details as well as the target machine type (windows, mac or linux).
3. Add a host lease PHID column and a working copy lease PHID column to the HarbormasterBuild object. Each build has a lease on the host and a lease for a working copy on that host.
4. Update the DrydockWorkingCopyBlueprint implementation to accept an attribute of host.lease to force it to allocate on a particular host.
5. Update the Harbormaster worker to request a lease for the host and working copy. It'll first request a host lease for the particular build type (windows, mac or linux) with attributes remote=yes (but leave preallocated unspecified to allow for dynamically allocated hosts). Once it has a lease on the host, it'll request a lease on a working copy where host.lease is equal to the ID of the lease it just got. The reason for requesting the lease on the host explicitly is that we need to be sure of the target machine's type (because we will be executing build commands on it).
6. Update HarbormasterBuildPlan to add a machine type column where the values can either be windows, mac or linux. We'll make this a drop-down field in the UI, but we could change it to a text field if there's demand for it later on.
7. Introduce DrydockRemoteCommandBuildStepImplementation; a variant of RemoteCommandBuildStepImplementation that instead uses the command interface on the host lease to run the command in the working directory provided by the working copy lease. It seems DrydockCommandInterface already supports everything we'll need (it returns an ExecFuture) so this should be reasonably trivial.
8. Ensure that build plans configured for Windows and Linux target machines execute and lease correctly, and that DrydockRemoteCommandBuildStepImplementation executes the builds correctly.
9. If UploadArtifactBuildStepImplementation is present in upstream, also create a version of that that uses the host and working copy leases. We should probably add a transfer interface used to transfer files independent of how the host resource is provided. This would even allow us to switch between SFTP and SCP on varying host types (e.g. differences between Windows and Linux).
10. Once the dust settles and this all looks like it's working, drop the RemoteCommandBuildStepImplementation and UploadArtifactBuildStepImplementation build steps. Remove the harbormaster.temporary.hosts.whitelist configuration option.
11. Celebrate that Drydock is now being used.