This table can get extremely unwieldy if left untended, especially when you do things like report coverage data for your builds.
FWIW we're writing ~36 million rows every 30 days, which takes up ~28 gigs. I've been manually clearing the table with a script that finds the lowest id in the table and then deletes all of the results that belong to the same target. Haven't seen any detrimental effects from that yet.
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Open | None | T11402 Garbage collect and/or compress/archive harbormaster build unit messages | ||
| Open | None | T10635 Loading differential revision slow when lots of unit test messages exist | ||
| Open | None | T9365 Harbormaster results are hard to aggregate | ||
| Open | None | T5822 Implement garbage collection / automatic archiving for Harbormaster logs | ||
| Open | None | T9124 Support uploading build log data via the Harbormaster API | ||
| Resolved | epriestley | T5955 Refactor Conduit auth to be stateless, token-based, and support wire encodings | ||
| Resolved | btrahan | T5603 Implement a separate "Can Leave Project" policy | ||
| Resolved | None | T9494 Standardize GarbageCollector TTLs |
I'd like to sort out T9365 / T10635 (better aggregation and/or query options for unit/coverage data) first, since that likely impacts how we GC this data. Ideally, we'd leave an aggregated stub behind (e.g., pass/failure count) rather than nuking the data completely.
Although T5822 discusses different data, we should probably plan the pathway through log collection at the same time. I think this became somewhat less urgent after D15380, which added compression, but we still can't grow forever.
I'd like to leave behind a "Results were GC'd on X/Y/Z" message when we GC these so it's clear what's going on.
Putting this in the "BuildUnitMessage" table makes it hard to GC.
Putting this in the "BuildTarget" table is a bit complicated, since we must acquire a lock to modify that table. The code would likely look like: send the target a "gc yourself" message; process the GC in the daemons. This is a lot of work, a lot of effort for a GC flow, and it also isn't trivial to stop the GC from sending a given target multiple "gc yourself" messages.
I'm just going to add a new "Stub/Receipt" table with this information. This may or may not eventually become a cache for things on T9365.