Page MenuHomePhabricator

Support for private build systems and public phabricators
Closed, WontfixPublic

Description

Context

I would like to create a Build Plan that can interact with a private build system via humans. The caveat is that our Phabricator instance is hosted on a public server with no access to the private build system.

The workflow I have in mind:

arc diff

# At some point in the future, someone with access to the private build system runs
internal_ci

The tentative idea is for internal_ci to initiate the continuous integration tooling.
Upon completion of the continuous integration, the tooling would call harbormaster.sendmessage to update the status of the build for the relevant diff.

The problem

sendmessage requires a buildTargetPHID value.

As best as I can tell, this value is only sent via the HTTP request as part of the Build Step. Our internal system can't receive this request, so we'd need to be able to fetch the value when we're ready to update the build.
Invocations to harbormaster.querybuilds do not return this value.
I could not find another API that would return the target.phid value seen in the "Variables" tab under /harbormaster/build/###/

The feature request

Returning the target phid in harbormaster.querybuilds would allow me to build this integration.

Assumptions

Based on T9456 I'm assuming the workflow of "some private build system" is not a priority, so I can understand if this won't be built for some time. If that is the case โ€” and the feature request seems sound โ€” I'm happy to contribute a patch.

I am also assuming that the work required to build this feature is relatively minor.

I think it would be reasonable to assume that most companies using this sort of workflow likely have an instance of Phabricator running on the local network that can therefor talk to the internal build system. This would allow the internal system to be invoked by harbormaster. My particular situation may be a bit odd in that sense.

Event Timeline

You can sort of do this like this:

  • Subclass HarbormasterBuildAutoplan to create a internal_ci autoplan, like HarbormasterBuildArcanistAutoplan.
  • Subclass HarbormasterBuildStepImplementation to create one or more targets for the plan, like HarbormasterArcUnitBuildStepImplementation.
  • Now, use harbormaster.queryautotargets to query the new autoplan target PHIDs (this will generate them if they do not yet exist).

This is what arc itself does. Basically, the flow supports "pushing" builds into Phabricator, and works like this:

  • internal_ci asks for the target PHIDs for some revision/commit/etc for particular named "autoplans", which are pre-arranged builds you plan to push into the system.
  • Phabricator generates them if they do not yet exist, or looks them up if they already do, then returns them.

Does that sound like it would work?

Without using this mechanism, no targetPHIDs will exist, so a harbormaster.querybuilds or harbormaster.querytargets which could return targets wouldn't help: the results would be empty (well, they might include the arc autoplans, but ideally you shouldn't just pile your data on top of those). You could get around this by having Herald run some "do nothing and wait forever" build plan, but I think this is generally less clean.

(We could also let you push builds into the system without any of this autoplan stuff, but I generally think this workflow is undesirable as an end state (e.g., Harbormaster can never restart, pause, resume, etc., builds it can't control) and it's nice for all builds to have real plans and real targets behind them.)

There's some sort-of-related discussion in T9529.

You could also conceivably have Herald genuinely trigger "do nothing, wait forever" targets, and just send these to some public server which internal_ci queries, so you kind of queue them up in a holding area and then flush the queue with internal_ci. This public/private manual-bridging-by-humans seems a little weird to me so I'm not totally sure what's reasonable or not, though.

(There's also no technical reason we can't implement a modern harbormaster.target.search, and implementing it isn't very much work, I'm just not sure it will solve your problem.)

Ah yes I forgot to mention that I have set up a Herald rule that kicks off the build plan.

This public/private manual-bridging-by-humans seems a little weird to me.

Agreed as well. This work is somewhat speculative so I may dabble on our own instance in the meantime based on your suggestions to see if we even like it.

Thank you! Feel free to close this issue in the meantime (doesn't look like I can).

I added you to Community so you should have access now.

featherless claimed this task.

Marking wontfix in the sense that I'll take on this work rather than upstream.