Page MenuHomePhabricator

Implement methods for logging to Harbormaster via Conduit
AbandonedPublic

Authored by hach-que on Aug 12 2015, 12:38 AM.
Tags
None
Referenced Files
F14012840: D13874.id.diff
Fri, Nov 1, 7:46 PM
F13994542: D13874.diff
Wed, Oct 23, 6:52 AM
Unknown Object (File)
Fri, Oct 11, 10:25 PM
Unknown Object (File)
Oct 3 2024, 8:48 AM
Unknown Object (File)
Oct 1 2024, 5:39 PM
Unknown Object (File)
Sep 5 2024, 3:38 PM
Unknown Object (File)
Sep 5 2024, 3:38 PM
Unknown Object (File)
Sep 2 2024, 2:06 AM
Subscribers

Details

Summary

Resolves T9124. This adds Conduit methods for logging to Harbormaster targets.

Test Plan
june-linux-laptop:~/Projects/Phabricator/phabricator> echo '{"buildTargetPHID":"PHID-HMBT-5pcgxksmgpnn4ufvz42r", "logSource": "external", "logType": "test"}' | arc call-conduit --conduit-uri=http://phabricator.local/ harbormaster.startlog
{"error":null,"errorMessage":null,"response":{"buildLogPHID":"PHID-HMCL-hpurf6jm5qmceguznopw","startTimestamp":1439339686}}
june-linux-laptop:~/Projects/Phabricator/phabricator> echo '{"buildLogPHID":"PHID-HMCL-hpurf6jm5qmceguznopw", "data": "Hello World"}' | arc call-conduit --conduit-uri=http://phabricator.local/ harbormaster.appendlog
{"error":null,"errorMessage":null,"response":null}
june-linux-laptop:~/Projects/Phabricator/phabricator> echo '{"buildLogPHID":"PHID-HMCL-hpurf6jm5qmceguznopw", "startTimestamp": 1439339686}' | arc call-conduit --conduit-uri=http://phabricator.local/ harbormaster.finalizelog
{"error":null,"errorMessage":null,"response":null}

pasted_file (348×632 px, 27 KB)

Diff Detail

Repository
rP Phabricator
Branch
build-log-api
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 7600
Build 8240: [Placeholder Plan] Wait for 30 Seconds
Build 8239: arc lint + arc unit

Event Timeline

hach-que retitled this revision from to Implement methods for logging to Harbormaster via Conduit.
hach-que updated this object.
hach-que edited the test plan for this revision. (Show Details)
hach-que added a reviewer: epriestley.
hach-que edited edge metadata.
epriestley edited edge metadata.

I believe I just want to add a "logs" parameter to harbormaster.sendmessage. Create/append/finalize should almost certainly be entirely implicit.

Even internally, these should likely be implicit. I recall that the PHP API isn't very easy to use, either. I probably want to fix the internal API first.

I want to resolve T5822 first. At least one install has a use case where they need to retain build metadata (e.g., "we did actually run all the tests") for a long period of time, which makes sense to me. With T6139, this likely implies log-level GC policies.

I want to plan T6139 in more detail first. It interacts here via a possible new designation of a log class ("summary" vs "normal").

I probably want to plan T8656 in more detail first. It interacts here via a possible new designation of a log format ("text" vs "ansi" vs "remarkup", etc).

I want to plan T8659 in more detail first. There are several ways forward and I'm not yet sure how I want to designate these artifacts as magic. It is possible that no way is satisfactory and that links to external systems are better represented as something like a "link" class of logs.

I may want to plan sub-builds and build vectorization first. These interact with multiple artifacts of the same name, which requires some behavior after T8659.

This revision now requires changes to proceed.Aug 12 2015, 2:33 PM

I don't believe the API can be simpler than this because we need to support streaming of logs. In order to support streaming of log data, we need to separate creation of a log and appending new data to it.

In addition, it's important to know if the log is currently live (having data appended to it) so that we can monitor for new data in real-time for the logs that are actually changing. This means we also need an API call to indicate log streaming has finished (this also gives us an opportunity to compress the data if we want).

Regarding other task impact:

I don't believe T5822 impacts this, as that task relates to migrating log data out of MySQL and into file storage when it gets old enough; not deleting actual build logs.

These methods are flagged as unstable, and I expect parameters to be added to them as T6319 and T8656 need (though I'd also expect them to be optional parameters).

I don't think T8659 impacts logs either; that has more to do with showing link artifacts to external build systems (which can be done as a URI artifact). Mirroring build lots from an external system would just use the internal API as-is, and I'd find it pretty weird if the lot storage system was used to store individual URIs (instead of a URI artifact).