Page MenuHomePhabricator

Make a line-linkable view for individual build logs
Closed, ResolvedPublic


It's not super uncommon for us to have massive amounts of output logged for any given build. By "massive" I mean our median build log size is 125kb, and our largest build log to date is 31MB. p99 is sitting somewhere around 1MB.

When attempting to talk about builds and the log output, we usually just link people to the build page and give them a line number along with the step that issued it, leaving them to the awesome task of scrolling through tens of thousands or hundreds of thousands of lines of log output to find what we're talking about.

I'd propose just making it possible to page through log output on a separate display somewhere, but more importantly just being able to link to a specific line in a specific log would be helpful. Typically for requests like this I just go add my own pages to our install, but I wasn't able to cook up a good way to deal with the log output in a line-by line manner given the current software API that exposes build logs.

Event Timeline

This seems reasonably easy to tackle. Roughly:

  • Modify ShellLogView.php so that it can render out the line numbers as links. Some method like setLineNumberLink that has a URL passed in like https://..../harbormaster/build/12345/log/<build log ID>/%d#B<build log ID>L%d should work fine, and that allows the ShellLogView to substitute the %d with the actual line number using the standard formatting functions.
  • Modify HarbormasterBuildViewController.php to check for build log ID and build line numbers from the URL, and if so, adjust the start / limit values of that logic so that the focused line appear in the build log.

It kind of surprises me that the build view logic just loads the entire log, and only cuts the content at the PHP level - that seems pretty inefficient and it could do something smarter. But for now, I think the above two changes are likely to achieve the immediate outcome of making the logs linkable, without tying us down to later if we want to adjust the Harbormaster URI (if build logs eventually move onto their own page, we can just change the routes, which is the reason for picking the /log/ format instead of ?log=).

@epriestley, thoughts?

@hach-que, that plan sounds good to me.

There's are a lot of changes I'd like to make to this UI element (many of which have tasks elsewhere), but I think all of those steps are fully compatible with everything I'm thinking about.

Specifically, offhand:

  • (T9516) Use the same copy/paste line number stuff that Paste, Diffusion, etc use.
  • (T10179) Make formatting smarter. For example, the "Run CircleCI Build" plan produces logs that we know to be HTTP headers (which we could format as a table) and (almost always) a JSON blob (which we could format as human-readable JSON).
  • (T8656) Make one of the smart formatters "ANSI".
  • I'd like to style this like other stuff (Paste, Diffusion) and let users pick dark themes globally with T5701 if they prefer.
  • Logic for loading the data is clunky, and fetches much more data than it needs.
  • Clicking pager links affects all build logs on the page, this should probably be JS-ish at some point.
  • Pager logic often feels clunky to me. For example, if a log is exactly 21 lines long, we still show pager controls. If a log is 27 lines long, we should probably show the whole thing. "Show more" would be more consistent with how other pagers work, etc.
  • (T6139) Not sure if this is really the right way to frame it, but we often have information about how useful a build log is and can do a better job of organizing them (e.g., successful build logs are usually uninteresting and could be folded by default).
eadler added a project: Restricted Project.Aug 5 2016, 4:45 PM
epriestley claimed this task.

Resolved by D19163 etc., see T13088.