Page MenuHomePhabricator

Garbage collect and/or compress/archive harbormaster build unit messages
Open, Needs TriagePublic


This table can get extremely unwieldy if left untended, especially when you do things like report coverage data for your builds.

FWIW we're writing ~36 million rows every 30 days, which takes up ~28 gigs. I've been manually clearing the table with a script that finds the lowest id in the table and then deletes all of the results that belong to the same target. Haven't seen any detrimental effects from that yet.

Event Timeline

Yeah, unit messages are stored in a different table.

(ノಠ益ಠ)ノ彡 sboן ʇsǝʇ ʇıun

eadler added a project: Restricted Project.Aug 5 2016, 4:45 PM

I'd like to sort out T9365 / T10635 (better aggregation and/or query options for unit/coverage data) first, since that likely impacts how we GC this data. Ideally, we'd leave an aggregated stub behind (e.g., pass/failure count) rather than nuking the data completely.

Although T5822 discusses different data, we should probably plan the pathway through log collection at the same time. I think this became somewhat less urgent after D15380, which added compression, but we still can't grow forever.

I'd like to leave behind a "Results were GC'd on X/Y/Z" message when we GC these so it's clear what's going on.

Putting this in the "BuildUnitMessage" table makes it hard to GC.

Putting this in the "BuildTarget" table is a bit complicated, since we must acquire a lock to modify that table. The code would likely look like: send the target a "gc yourself" message; process the GC in the daemons. This is a lot of work, a lot of effort for a GC flow, and it also isn't trivial to stop the GC from sending a given target multiple "gc yourself" messages.

I'm just going to add a new "Stub/Receipt" table with this information. This may or may not eventually become a cache for things on T9365.