Some immediate observations:

• The instance which was doing backups is a suspended test instance. This specific issue could be resolved immediately by adding the ability to destroy instances. The two largest instances on this hardware are both long-disabled, destroyable test instances.
• innodb_buffer_pool_size got bumped down informally and then back up again formally, but may be too high. I think I'm going to drop it to ~50%-66% of the host's RAM (it's about 85% right now).
• The root cause of the 3 -> 29 minute jump isn't clear to me. Obviously, 3 minutes is a lot better than 29 minutes. top / ps / free / CloudWatch didn't report the host running out of anything, and CPU utilization was low. The EBS volumes also didn't report an unusual amount of utilization. There was less memory used during the backup which makes me suspect this is some kind of complex buffer pool size interaction.
• In the long term, we should use replicas and dump backups off them so main database performance is unaffected during a backup, but this requires significant work on T4209.
• We could nice the backups, but they weren't actually limited by CPU so I suspect this wouldn't help.

So my plans are:

• Adjust the buffer pool size back down to something in the 50%-66% range.
• Add the ability to destroy instances.
• Destroy old instances.

I think the actual extent of "destroy" we need is just "stop backups" -- keeping the data itself around costs us roughly nothing -- so I'll probably just tailor that.

Possibly I can just stop backups on suspended/disabled instances if they've been offline for ~48 hours -- we won't destroy the old backups just because we stop the new ones. So that's probably simpler, and just requires a new...

While I was writing this up, the host ran out of RAM and stopped working, so this is probably a memory management issue.

I'm dropping the pool size and giving these hosts swap now.

Ubuntu also ships with some debian-sys-maint junk which runs huge, slow queries to check for problematic table data on every restart, which we should likely disable.

I've restarted the tier with a ~50% buffer pool configuration (4GB of 7.5GB) and 8GB swap (rCORE471d3a7acee1) and things seem to be holding stable. I'll pursue the other changes now.

I no-op'd the debian-start script and deployed db002 with the new changes (rCORE51d7570f5ddf) to verify it. I'm not going to restart db001 for the moment, since bringing it down is more disruptive and I only need confidence that the change is correct.

I stopped storage upgrade from running on out-of-service instances (rCOREe9f998b109c1) and upgraded db002 and repo002 to verify this change. We don't need to keep out-of-service schemata up to date, and this reduces the ongoing cost of out-of-service instances.