Add a utility class for getting system memory information
ClosedPublic
Actions

Authored by epriestley on Feb 23 2015, 6:08 PM.

Details

Reviewers

btrahan

Maniphest Tasks

T7352: Improve daemon scalability in the cluster

Commits

rPHU77f0eda5b427: Add a utility class for getting system memory information

Summary

Ref T7352. I want to implement the hard memory barrier described in T7352#98018 to prevent all pools from scaling upward and throwing a machine into a death spiral.

To do this, I need to be able to get at least a rough picture of free system memory. This is a little murky, but we only need to have an approximate picture of it so we can cut daemons off sometime before everything goes to swap.

Add a class that queries the system and gives us some usable information back in a standard format.

Test Plan

Added and executed unit tests for vm_stat (os x) and /proc/meminfo (linux).
Ran method locally, got reasonable output.

Diff Detail

Repository

rPHU libphutil

Branch

daemon6

Lint

Lint Passed

Unit

Tests Passed

Build Status

Buildable 4616
Build 4630: [Placeholder Plan] Wait for 30 Seconds

Event Timeline

epriestley updated this revision to Diff 28575.Feb 23 2015, 6:08 PM

epriestley retitled this revision from to Add a utility class for getting system memory information.

epriestley updated this object.

epriestley edited the test plan for this revision. (Show Details)

epriestley added a reviewer: btrahan.

epriestley added a task: T7352: Improve daemon scalability in the cluster.

Herald added a subscriber: epriestley. · View Herald TranscriptFeb 23 2015, 6:08 PM

Fix comment.

On this memory detection stuff, if certain OS flavors are particularly bad, maybe we should not allow for autoscaling on those flavors? Not even sure how murky things can get, but I would want to avoid users having problems with this feature, especially since I think the autoscaling thing isn't useful for smaller (most) installations. Just waxing here I guess...

This revision is now accepted and ready to land.Feb 23 2015, 6:14 PM

Yeah, current state of the world is:

It's OK if we underreport free space, since it will just mean the daemons never autoscale up (or don't scale up as much), which is fine.
On OSX, we may underreport free space relative to Linux, I think (?). Specifically, I don't have a way to identify and subtract disk cache like we can on Linux.

So, right now, this really just means that OSX may not scale up as much as Linux would, but that should be fine. No one (almost no one?) is running a production server on OSX anyway, and they could theoretically set the memory reserve to 0 if they were and really wanted aggressive autoscaling.

I didn't want to put a ton of daemon-specific details in the comments here so I left it sort of vague, but we only get into trouble if some system dramatically overreports free space.

I'm also probably just going to try/catch this test in the daemons (so we don't break on, say, FreeBSD) and default the reserved memory to 0 (so normal installs don't do this check anyway), since this concern is only a problem on instanced clusters.

Closed by commit rPHU77f0eda5b427: Add a utility class for getting system memory information (authored by epriestley, committed by epriestley). · Explain WhyFeb 24 2015, 3:52 PM

This revision was automatically updated to reflect the committed changes.