This adds a build step in Harbormaster for publishing file artifacts as fragments in Phragment.
Details
Details
- Reviewers
epriestley - Group Reviewers
Blessed Reviewers - Maniphest Tasks
- T4205: Implement Phragment
- Commits
- Restricted Diffusion Commit
rP8fd256a1fd69: Implement step for publishing files as fragments to Phragment in Harbormaster
Created a build plan with the following steps:
- Lease Host
- Upload Artifact
- Publish Fragment
Ran the build plan against a buildable and saw the fragment get created in Phragment. Ran the plan again and saw the fragment get updated with a new version. Modified the file that got uploaded and ran the plan again, checked the history of the fragment, and saw the differences represented as a Diff-Match-Patch patch.
Diff Detail
Diff Detail
- Lint
Lint Skipped - Unit
Tests Skipped
Event Timeline
Comment Actions
An issue I see here is in figuring out which Version corresponds to which Buildable. In particular:
- A minor thought is that you might just want to know this, in general. We could write an edge or some other type of metadata to satisfy this.
- A more serious issue is that there is no guarantee builds run in order. We might finish a new build before an older one, and then we'll end up with Versions that are out of order.
Some thoughts about resolving this:
- We could solve this by having Builds that run only on HEAD, and never run multiple copies at once. We'll need these sorts of builds for other things (like symbol indexing or documentation generation). However, this limits the utility of Fragment to some degree.
- Alternatively, we could solve this by having Builds that run only in order. Both of these are bad for "run the unit tests", but OK for "build executables".
- We could solve this with some sort of logical clock, so the fragment doesn't publish if it's older. This is basically an alternative to putting a logical clock on builds. I think it's worse than clocking builds.
- In some use cases, we could publish to a variably-named fragment, e.g. path/to/thing/B139.
A logical clock on builds could itself be a build step called "Wait for Earlier Builds to Finish" or similar. This seems like it might be the cleanest approach for at least some use cases, but is sort of hard to figure out. I don't like that the default behavior would be "break build pipelines with subtle race conditions, occasionally".
Do you have thoughts on this? Does this not actually matter, maybe?
Comment Actions
I think that it matters enough that we need to consider it. I think the best option is a "Wait for previous builds to finish" build step:
- This allows us to ensure that fragments get published in order.
- It allows people who are publishing to other, external sources to also ensure they get published in order.
- It allows people who have build processes that can't be parallellised to prevent that from happening as well.
Comment Actions
I'll send through another diff later for a "Wait until previous builds have finished" build step, but that doesn't block this diff.