Currently the log chunk table gets very large, very quickly in Harbormaster (especially for builds with lots of output). The GC should take old content from this table and compress / archive it somewhere. This means viewing old builds will take longer to load logs, but new builds should be very quick to look at (verses all builds being slow to look at because the table is so huge).
Revisions and Commits
- Mentioned In
- T13257: Plans: Unbeta New Harbormaster Log UI
T13088: Plans: Harbormaster UI usability and interconnectedness
T11402: Garbage collect and/or compress/archive harbormaster build unit messages
D13874: Implement methods for logging to Harbormaster via Conduit
- Mentioned Here
- T10457: Improve Harbomaster/Drydock UI/UX before release
T9494: Standardize GarbageCollector TTLs
T5821: Implement "Run Arc Unit" build step and support showing unit test results inside Harbormaster
Placing this back for triage, but I think this should probably be of a reasonably high priority on the Harbormaster roadmap, simply because of the impact it can have in migration and schema upgrade times.
T9494 discusses a GC TTL change I'd like to make before putting real GC here.
We can also probably get chunk compression into HEAD before too long since it's pretty simple and should be dramatic on most build logs.
Archiving old logs into Files I'm less excited about and it's probably a v-Far-Future sort of thing.
Current plan here is:
- Compress log chunks in-place.
- Add a GC to move the data to Files (default = 30 days).
- Add a second GC to destroy them (default = 180 days). This GC can be disabled if you want to retain them indefinitely.