Yeah, this is a combination of a bulk editor bug (which lets you delete task titles) and legacy compatibility code for transaction title rendering (which treats null -> anything transactions as "created this thing.").

I'm leaning toward:

Hmm, I can't reproduce the issue. Here's pre-comment:

• Source view rendering in Paste has a missing background color.

I think that's pretty much everything. There will be a little followup work in T10883 and maybe T13076 / T13120.

Everything here should pretty much work except:

I think all the T13076 stuff is blocked on me for now unless you're feeling especially ambitious about trying to get bin/host download working on 2GB+ files (T12907 / D19011). That might be a bit of a mess of a task though since I think I have a lot of secret cURL knowledge from over the years that is only somewhat-documented in HTTPSFuture.

Backups should run continuously (starting 12 hours after the instance launches, then every 24 hours after that):

Before I can get rid of AlmanacDeviceTransaction::TYPE_INTERFACE, we have two meaningful callsites in rSERVICES and one unit test in rSAAS to clean up.

There's a bit of a mess with AlmanacInterface and AlmanacDevice. Currently, AlmanacInterface does not use transactions, and is edited purely as a side effect of INTERFACE transactions applying to AlmanacDevice. I'm going to change how this works so that AlmanacInterface is a normal transactional object and can use the same rules and infrastructure as everything else.

I think "run weekly with the deploy" already works pretty well for normal log rotation with logs on local disk, and in a perfect world logs would be buffering on disk only briefly before being sent over the network to dedicated log service so we wouldn't need cron there either.

yikes

Yeah, I think "solve it forever" is probably "first-party Sentry-like application", this just seems like a problem that should have a 15-minute "solve it for now" solution in the form of something that gives us "a thing that looks like a disk but never fills up" that will hold for a year or two until we build "Phentry".

I believe there are zero or nearly-zero actionable errors in the log today and that if we had a perfect Sentry-like system it wouldn't actually help us identify or fix any errors -- we'd just ignore it after a week like we currently ignore all the other production error logs.

The "logs fill up disks" problem is more "sudden bursts of unusual error/activity that cause log rates to increase 1000x+ fill up disks", not "logs don't rotate / normal log volumes fill up disks". Most logs do already rotate today, I think with the exception of the aphlict log being a bit of a weird case, maybe.

phage tail <some-instance-name> <some-log-type>

Possibly something dumb like "mount an EFS volume on /mnt/logs/ on every host" is another AWS-only approach. That feels really dumb but maybe it's only somewhat dumb?

I'm fine with CloudWatch logs, I didn't realize it had a logging thing. I'm currently not terribly warm on investing a ton in CloudWatch alarms or log analysis, but if we can get logs reliably streaming into some storage service so they can't fill up the disk I think that's quite valuable on its own. We can always replace that with something we control directly in the future if we want to invest more here. I can't imagine ever really caring about old logs no matter what happens in the future.

In the modern era: I think we generally understand what private clusters will look like now, but I'd like to take a much more iterative approach to getting there than we have in the past. I had this concern above (circa June 2017):

Multiple installs would like API access to edit Almanac bindings.

This feels a bit like a return to the era of "epriestley design", but I think it's an improvement overall: