Page MenuHomePhabricator

When showing a small piece of a Harbormaster build log, load a small piece of data instead of the entire log
ClosedPublic

Authored by epriestley on Feb 28 2018, 2:22 PM.
Tags
None
Referenced Files
F13088711: D19149.diff
Thu, Apr 25, 1:32 AM
Unknown Object (File)
Fri, Apr 19, 9:37 AM
Unknown Object (File)
Fri, Apr 19, 2:50 AM
Unknown Object (File)
Fri, Apr 19, 2:50 AM
Unknown Object (File)
Fri, Apr 19, 2:50 AM
Unknown Object (File)
Wed, Apr 17, 1:55 AM
Unknown Object (File)
Thu, Apr 11, 9:05 AM
Unknown Object (File)
Thu, Apr 11, 4:07 AM
Subscribers
Restricted Owners Package

Details

Summary

Depends on D19148. Ref T13088. The new rendering always executes range requests for data it needs, and we can satisfy these requests by loading the smallest number of chunks which span that range.

Test Plan

Piped 50,000 lines of Apache log into Harbormaster, viewed it in the new UI, got sensible rendering times and a reasonable amount of data actually going over the wire.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Owners added a subscriber: Restricted Owners Package.Feb 28 2018, 2:22 PM
This revision was not accepted when it landed; it landed in state Needs Review.Feb 28 2018, 8:32 PM
This revision was automatically updated to reflect the committed changes.
jcox added inline comments.
resources/sql/autopatches/20180228.log.01.offset.sql
5

Is there a reason that this is two statements instead of one? ie:

ALTER TABLE {$NAMESPACE}_harbormaster.harbormaster_buildlogchunk
  ADD headOffset BIGINT UNSIGNED NOT NULL,
  ADD tailOffset BIGINT UNSIGNED NOT NULL;

As-is, this patch rebuilds the table twice, which takes a very long time on our install (~3hrs each). It's not relevant for this case, but I imagine that T13088 will possibly lead to more alterations to the chunks table. If so, it'd be preferable to us if those changes were batched as much as possible. I believe that the above is valid in ~all flavors of mysql phab supports.

Oh: no, they should probably be one statement.

In most cases, statements should be separated into different migrations so that bin/storage can resume where it left off if any statement fails. If a single file contains two statements, the first statement may apply but the second statement may fail, and then the patch may fail when trying to re-apply the first statement. Separating them generally avoids this. This particularly affects ALTER TABLE (very unlikely to fail) followed by UPDATE (occasionally hits some kind of weird data issue).

In this case, the statements apply to the same table and have the same type, and could reasonably be combined into a single statement. This just doesn't matter most of the time (most tables aren't large) and is the opposite of the best way to do most groups of statements (separate them fully).

Also I feel like maybe ALTER TABLE didn't allow multiple alterations when I was a small child first learning MySQL? But even this incredibly old copy of the manual says that it worked so I probably made that up.