This looks good to me, although there are a bunch of features I think we should consider planning for now, before anyone has a gig of data in this table. Some of this is a bit YAGNI, but I think reasonable use cases exist for almost all of it:

• I think we should assume each build step can have more than one thing which generates logs, and add logSource (or similar), which is a short-ish human-readable string describing where the log came from (for example, the command which was run, the API endpoint which was requested, etc). For build steps which run several commands in sequence or in parallel, this will let us distinguish between the different log sources.
• I think we should assume each log source can generate multiple logs, and add a logType (or similar), which is a short-ish human-readable-but-kind-of-conventional string describing exactly what the log captures. Examples would be "return" (return code), "stdout", "stderr", "file:side_effect.log", "full" (full command / full request?), "http-headers", "http-body", etc.

I think we should also consider these fields:

• duration: Nullable field recording the command duration. This wouldn't always be populated, but could be populated on well-known logs by convention. For example, every step could add a "full" log which populates "duration", and the convention could be that the UI pulls just these logs to provide an overview. I think having this information in detail could be helpful for complex builds. It will also let us determine how much overhead Harbormaster itself is adding, and give us a heads up if there's a problem.
• encoding: Field allowing us to compress the actual log data for the row and set a flag here so we know to decompress it later. This is sort of a weak optimization since it's maybe like 4x in the best case, but logs should generally compress well and I don't anticipate needing to search through them except maybe in very local scopes ("in this build"). This is probably premature, too, but if we wait until the logs are unwieldy it will be more painful because everyone will have to migrate.
• live: This could be overlaid on encoding since a live log will always be plain text, but it might be good to have a flag to tell the UI that the log is live (so it can show an indicator, and Ajax-refresh it) vs complete. I'd lean toward having live, text, and gzipped values in encoding. This is a little fishy but I can't come up with any case where a live log would not be plain text.
• Store text as binary, not UTF-8: If we're going to compress logs, we have to do this. Even if we don't, this might be a good idea? It makes it easier to compress them later, and logs aren't necessarily UTF-8 and we might run into weird issues with terminal junk and ascii art if we store them as utf-8. Since the field is LONGTEXT, there's no risk of truncating in the middle of a character. This does mean we need to be a little more careful when displaying the logs.
• size: I don't think this is really that useful, but it's easy and it might be nice to have a field recording the byte size of the log. Particularly, a case I anticipate is being able to show "This log is very large. Only the first 1MB of 615MB is shown." without pulling all the data over the wire, although this gets trickier with compression. I'm not sure if we can easily store a large compressed value such that SELECT SUBSTR(0, 1024 * 1024) will be decompressible, although it should be possible in theory. There are various ways we could work around this to get good behavior. Generally, writing size is easy and gives us more options for dealing with edge cases involving unreasonably large data.

For the special case of stdout and stderr, I think they should be stored separately (for example, some commands produce a machine-readable stdout and it's believable that this will eventually be available as a build artifact or something like that, maybe, or some output will produce a variable for some other build step)? To interleave them for display, we could write a third log with sequence information, which just looked like "OEOEOEOOOOEEOE", maybe (indicating which buffer to take each line from in order to produce a combined result)? (Or it could just have all of the data in the right sequence, although that's a little less efficient, since it would be in both the "combined" and "stdout" or "stderr" logs.) This would be optional, and the UI could use it to interleave the logs if it was available. Otherwise, it could show them separately (if a build step can't produce this information for some reason). This is kind of messy, but the whole stdout/stderr thing is kind of messy in general. I'm loathe to just mush them together in the database since I think being able to separate them is sometimes valuable.

For the append issue, I haven't seen the CONCAT problem before. I'll see if I can reproduce it once it's more practical to write logs from build steps. One approach might be to add a finalize() method, which would let append() start buffering and doing fewer writes. I'm not sure if this would fix CONCAT, but it would probably reduce overhead by a fair bit. finalize() could flush the buffer and, in the future, compress the log itself and mark it as compressed in the encoding field and clear the live flag.

(I do generally agree that one row per log transcript is a better approach than one row per log line.)

Here's how I'm leaning for this diff:

• Add logSource.
• Add logType.
• Add duration.
• Add encoding, for now always "text".
• Make logData field binary, and utf8-sanitize it on output.
• Separate stdout and stderr.
• Add size and populate it.

Then, somewhat soonish:

• Add finalize() for writing.
• Add return logs.
• Add full command logs.
• Populate duration on full command logs.
• Add the sequence log for stdout/stderr.
• Figure out garbage collection policies for these.

Eventually;

• Add "live" encoding flag.
• Add Ajax update.
• Add UI logic for enormous logs.
• Add gzipping encoding on finalize.

Does that sound reasonableish? It's kind of a lot of architecture for log files, but this table is probably going to get big, which will make changing it harder in the future without making users do long migrations. I'm not completely convinced we need most of this, but if we think we probably will we likely save a bit of effort by adding the fields now, even if we don't get around to using them for a while.

Cancel

For "Cancel", I have a mild leaning toward making it a separate field -- see inlines.

UI View

I'd like to maybe unify the UI a bit (we have some logs in Daemons too, and SourceCodeView or whatever might be generalizable) at some point, but ShellLogView is entirely reasonable for the moment.