Only db001 and db002 are anywhere close to having problems with this and they still have a large amount of headroom, and we can bin/storage optimize at least some amount of additional headroom into existence, so this isn't "we're about to fall off a cliff" urgent or anything, but would be nice to tackle sooner than later.

Just keeping an eye on this: growth on these shards is slow and both are still below the 80% caution threshold, so I'm punting this for another week.

Looking at the actual data, I'm less sure this is actually a good strategy. Here's the data for this install, considering the production configuration of storage.mysql-engine.max-size as 65535:

mysql> select IF(LENGTH(data) > 65535, 'S3', 'MySQL') Engine, SUM(LENGTH(data)) Size, COUNT(*) N FROM differential_hunk_modern GROUP BY Engine;
+--------+-----------+--------+
| Engine | Size      | N      |
+--------+-----------+--------+
| MySQL  | 838006438 | 285338 |
| S3     | 222007657 |   2458 |
+--------+-----------+--------+
2 rows in set (19.52 sec)

That is, if we archive all hunks on this install, only 0.8% of them (2,458 of 287,796) are actually large enough to go to S3. The hunks that go to S3 are the biggest ones, so this 0.8% of hunks represents about 20% of all data stored in hunks, but we would still put 80% of data back in MySQL, just in Files instead of Differential. This is probably worse on the balance.

So I'm currently thinking:

• Land D18584 (support for serving hunks from Files, plus fixes for destroying diffs), but it serves mostly as a tool for dealing with exceptional cases ("we accidentally put a 1GB hunk in the database and can't just delete it for complicated reasons").
• Don't pursue D18585 (automatically moving old hunks to Files) since the actual savings under production settings are fairly small, and I don't think this use case really motivates changing production settings.

To deal with shard fullness, I'll just schedule maintenance in this week's upgrade cycle and expand the volumes with some shard downtime. This should be relatively brief and happen off-hours so I don't think it will be particularly disruptive.

We could possibly revisit this in the future, but given that the rows are numerous and small I think there's less opportunity here than I'd originally suspected.