Resolves T9124. This adds Conduit methods for logging to Harbormaster targets.
Details
Details
- Reviewers
epriestley - Group Reviewers
Blessed Reviewers - Maniphest Tasks
- T5822: Implement garbage collection / automatic archiving for Harbormaster logs
T9124: Support uploading build log data via the Harbormaster API
june-linux-laptop:~/Projects/Phabricator/phabricator> echo '{"buildTargetPHID":"PHID-HMBT-5pcgxksmgpnn4ufvz42r", "logSource": "external", "logType": "test"}' | arc call-conduit --conduit-uri=http://phabricator.local/ harbormaster.startlog
{"error":null,"errorMessage":null,"response":{"buildLogPHID":"PHID-HMCL-hpurf6jm5qmceguznopw","startTimestamp":1439339686}}
june-linux-laptop:~/Projects/Phabricator/phabricator> echo '{"buildLogPHID":"PHID-HMCL-hpurf6jm5qmceguznopw", "data": "Hello World"}' | arc call-conduit --conduit-uri=http://phabricator.local/ harbormaster.appendlog
{"error":null,"errorMessage":null,"response":null}
june-linux-laptop:~/Projects/Phabricator/phabricator> echo '{"buildLogPHID":"PHID-HMCL-hpurf6jm5qmceguznopw", "startTimestamp": 1439339686}' | arc call-conduit --conduit-uri=http://phabricator.local/ harbormaster.finalizelog
{"error":null,"errorMessage":null,"response":null}Diff Detail
Diff Detail
- Repository
- rP Phabricator
- Branch
- build-log-api
- Lint
Lint Passed - Unit
No Test Coverage - Build Status
Buildable 7600 Build 8240: [Placeholder Plan] Wait for 30 Seconds Build 8239: arc lint + arc unit
Event Timeline
Comment Actions
I believe I just want to add a "logs" parameter to harbormaster.sendmessage. Create/append/finalize should almost certainly be entirely implicit.
Even internally, these should likely be implicit. I recall that the PHP API isn't very easy to use, either. I probably want to fix the internal API first.
I want to resolve T5822 first. At least one install has a use case where they need to retain build metadata (e.g., "we did actually run all the tests") for a long period of time, which makes sense to me. With T6139, this likely implies log-level GC policies.
I want to plan T6139 in more detail first. It interacts here via a possible new designation of a log class ("summary" vs "normal").
I probably want to plan T8656 in more detail first. It interacts here via a possible new designation of a log format ("text" vs "ansi" vs "remarkup", etc).
I want to plan T8659 in more detail first. There are several ways forward and I'm not yet sure how I want to designate these artifacts as magic. It is possible that no way is satisfactory and that links to external systems are better represented as something like a "link" class of logs.
I may want to plan sub-builds and build vectorization first. These interact with multiple artifacts of the same name, which requires some behavior after T8659.
Comment Actions
I don't believe the API can be simpler than this because we need to support streaming of logs. In order to support streaming of log data, we need to separate creation of a log and appending new data to it.
In addition, it's important to know if the log is currently live (having data appended to it) so that we can monitor for new data in real-time for the logs that are actually changing. This means we also need an API call to indicate log streaming has finished (this also gives us an opportunity to compress the data if we want).
Regarding other task impact:
I don't believe T5822 impacts this, as that task relates to migrating log data out of MySQL and into file storage when it gets old enough; not deleting actual build logs.
These methods are flagged as unstable, and I expect parameters to be added to them as T6319 and T8656 need (though I'd also expect them to be optional parameters).
I don't think T8659 impacts logs either; that has more to do with showing link artifacts to external build systems (which can be done as a URI artifact). Mirroring build lots from an external system would just use the internal API as-is, and I'd find it pretty weird if the lot storage system was used to store individual URIs (instead of a URI artifact).