Hmmm, then maybe the narrowest way to solve this is to just setup disk space alarms. That's a thing we'd want independent of how we eventually build the One True Logging System. The CloudWatch agent collects disk stats out of the box, and takes a list of mountpoints to track. Details here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
All Stories
Apr 10 2018
Before I can get rid of AlmanacDeviceTransaction::TYPE_INTERFACE, we have two meaningful callsites in rSERVICES and one unit test in rSAAS to clean up.
There's a bit of a mess with AlmanacInterface and AlmanacDevice. Currently, AlmanacInterface does not use transactions, and is edited purely as a side effect of INTERFACE transactions applying to AlmanacDevice. I'm going to change how this works so that AlmanacInterface is a normal transactional object and can use the same rules and infrastructure as everything else.
I think "run weekly with the deploy" already works pretty well for normal log rotation with logs on local disk, and in a perfect world logs would be buffering on disk only briefly before being sent over the network to dedicated log service so we wouldn't need cron there either.
i feel unethical
Apr 9 2018
How often does the size directive in a logrotate script get evaluated?
Maybe vertical-align: top? Or display: inline-flex?
yikes
FYI Chrome/Windows after the change (extra whitespace added?)
Yeah, I think "solve it forever" is probably "first-party Sentry-like application", this just seems like a problem that should have a 15-minute "solve it for now" solution in the form of something that gives us "a thing that looks like a disk but never fills up" that will hold for a year or two until we build "Phentry".
That makes sense. I was coming at it more from the direction of “if we’re going to touch logs, lets solve it forever”. How about we make everything log locally through syslog and use logrotate to avoid filling disks? Then we can come back and point syslog at something more elaborate later without having to touch all the things that currently generate logs.
I believe there are zero or nearly-zero actionable errors in the log today and that if we had a perfect Sentry-like system it wouldn't actually help us identify or fix any errors -- we'd just ignore it after a week like we currently ignore all the other production error logs.
There's some flavor of bin/remote ssh <some-host> -- tail -f <some-log_location>.
The "logs fill up disks" problem is more "sudden bursts of unusual error/activity that cause log rates to increase 1000x+ fill up disks", not "logs don't rotate / normal log volumes fill up disks". Most logs do already rotate today, I think with the exception of the aphlict log being a bit of a weird case, maybe.
phage tail <some-instance-name> <some-log-type>
But I don't think this is too valuable if it doesn't solve the "logs fill up disks" problem -- does the agent rotate/reap the logs?
In my mind, the only real problem here is...
Possibly something dumb like "mount an EFS volume on /mnt/logs/ on every host" is another AWS-only approach. That feels really dumb but maybe it's only somewhat dumb?
I'm fine with CloudWatch logs, I didn't realize it had a logging thing. I'm currently not terribly warm on investing a ton in CloudWatch alarms or log analysis, but if we can get logs reliably streaming into some storage service so they can't fill up the disk I think that's quite valuable on its own. We can always replace that with something we control directly in the future if we want to invest more here. I can't imagine ever really caring about old logs no matter what happens in the future.
I definitely don't want to send logs to a third-party service.
In the modern era: I think we generally understand what private clusters will look like now, but I'd like to take a much more iterative approach to getting there than we have in the past. I had this concern above (circa June 2017):
Multiple installs would like API access to edit Almanac bindings.
This feels a bit like a return to the era of "epriestley design", but I think it's an improvement overall:
Still needs work:
Apr 8 2018
I don't currently plan to pursue this. We haven't seen interest from other installs and I think the upstream behavior is reasonably good most of the time and not a huge issue even when we get it wrong (you can still review the file without doing anything, and can get highlighting with a few clicks -- plus, the file and page render much faster).
- The secret $1-2,7-9 syntax for highlighting multiple noncontiguous blocks of lines in Diffusion is no longer supported.
Apr 7 2018
im super ethical https://github.com/PoE-Profile/poe-profile/pull/20
A related attack is a bare whatever.patch file which writes to .git/config or .hg/hgconfig or whatever.
In Safari we get this marvel:
