Our actual internal use case is as follows:

• We're constructing a service which continually runs all of our unit tests, basically:
  • git pull origin master
  • record what sha we're at
  • run the build
  • run all of the tests, this is not equivalent to arc unit --everything and runs every conceivable test
  • eventually this may get fancy and use more than one machine in order to run all of the tests for every sha
• We record the results of these test runs in this external service
• Our "regular" unit test invocations (eg, per-diff in Harbormaster) can compare their results against the relatively pristine results this service is generating in order to get a better idea of whether or not a failure was actually their fault or a test is flaky
• Since the whole point of this service is to improve consistency / confidence in test results that end users see, it's really important that we use identically managed environments (servers leased by Drydock) to run the tests

Do you have any use cases which are not approximately some flavor of:

$ arc unit --on-a-remote-server

I'm more hesitant to pursue this stuff upstream in the near term if most use cases would be mostly obsoleted by a more powerful arc unit.

arc unit is already close to being able to push HEAD to a staging area and then start a build on it. It couldn't currently do this entirely or figure out what build to start and I'm sure there's some weird edge cases and stuff, but these seem pretty easy to glue together conceptually -- and this workflow can report build results to the web UI, doesn't require lease authorization dances, doesn't require use of arc release, etc., so it seems like a better long-term approach to --on-a-remote-server than just shipping the building blocks, maybe?

Well one facet of this is that remote servers tend to be significantly more powerful than developer machines, and so since I've implemented this downstream, I have basically taken to not building locally anymore.

This is an excellent tool for us organizationally to start weaning ourselves off of local machines and moving onto managed environments.

(not building locally anymore, eg, any time I want to do anything with a built binary at all, i'm actually doing it with a binary on a remote server because I can get the build results faster)

Evan, do you think there is a route forward with this that would be feasible for me to implement upstream?

I'm comfortable bringing this stuff upstream today:

• General support for authorizing users on blueprints and having them own leases (this may be zero actual code in terms of overlap between your workflows and what's missing in the upstream).
• Support for querying authorizations via Conduit (D16592, already accepted).
• Support for querying leases via Conduit (D16594), except I want to drop attributes for now, per discussion there. Exposing what you need via an attachment should be a reasonable way to progress.
• Support for querying blueprints via Conduit (D16593, already accepted).
• Some kind of drydock-ssh-exec --lease 123 -- ls -alh sort of SSH workflow, which:
  • Checks if that lease exists, you own it, and it exposes an SSH interface.
  • If it does, proxies via the interface and runs whatever arbitrary commands you provided, broadly acting like you'd just SSH'd directly to the host.
  • No diff for this yet, you presumably have this locally in some form though.

I want to hold off on this stuff upstream for now:

• Requesting/creating authorizations.
• Requesting/creating leases.
• Destroying leases.
• An arc workflow for gluing all this stuff together.

The blockers there are:

Authorizations: Major concern is that this would be a nonstandar endpoint today. We should make these transactional, then we can expose a standard endpoint. I can walk you through this. Once they're transactional, I'm comfortable with bringing drydock.authorization.edit upstream.

Requesting/Creating Leases: I feel like Drydock needs to evolve further before I'm ready to commit to an approach on this. This API is necessarily nonstandard and I don't think we're on firm enough footing to confidently define what it should look like (D16565 has some discussion).

Destroying Leases: Probably OK with this coming upstream as drydock.lease.sendcommand or similar. Feels low-priority to upstream and like we might build it differently later, though.

arc workflow: Needs all the other pieces upstream, and I want to wait for arc work.

So I think the pathway is:

• Commandeer and land D16592, D16593, D16594. Some of these have minor feedback.
• Upstream whatever you're doing locally to make SSH work.
• I'll sketch out the pathway for making authorizations transactional, then do that and upstream it and add a drydock.authorization.edit standard endpoint.

That's probably all the good stuff to upstream for now, and then we can come back to this after Harbormaster/Drydock get their next real iteration and we're on more solid footing generally.

I think making authorizations transactional isn't too much work. They already have PHIDs, and aren't edited in very many places.

Basically, overall approach will be something like:

• Add an Editor.
• Add storage for the Transactions.
• Add a TransactionQuery.
• Make DrydockAuthorization implement ApplicationTransactionInterface.
• Add modular transactions (TransacitonType + individual transaction classes) for all the edits that take place (just creation [object phid + blueprint PHID?] and status [approval/rejection]?).
• Find all the places that mutate authorizations and replace them with Editor invocations.
• You probably need to define an EditEngine so we can generate a Conduit method out of it, but could do the initial conversion without this.

D16675 is a recent change to possibly serve as a quasi-reference.

You should be able to do this very gradually. In particular, a good place to start is probably the existing "Approve" / "Decline" actions in DrydockAuthorizationAuthorizeController. Change those to apply a transaction instead of mutating the object directly. Once that works, add an edit history timeline to the detail page in DrydockAuthorizationViewController. Once that works, expand to the places that create authorizations. Finally, implement the Conduit endpoints for querying/editing.

Well, just Conduit for editing -- querying is already in D16592.