Merge GC daemon into Trigger daemon
ClosedPublic
Actions

Authored by epriestley on Feb 24 2015, 2:41 AM.

Details

Reviewers

btrahan

Maniphest Tasks

T7352: Improve daemon scalability in the cluster

Commits

Restricted Diffusion Commit
rPa3518e19a565: Merge GC daemon into Trigger daemon

Summary

Fixes T7352. This reduces the memory footprint for instances by combining these two similar daemons into one daemon which handles the responsibilities of both.

The fit isn't 100% perfect here but it's pretty close, and the GC daemon is fairly trivial.

Test Plan

Adjusted all the numbers to small numbers (5 second sleep, 120 second GC length).
Added a ton of logging.
Started trigger daemon.
- Saw it run a GC cycle.
- Saw it reschedule another cycle after 120 seconds (adjusted down from 4 hours).
Reverted all the logging/small numbers.
Ran bin/phd start, saw stable trigger daemon running.
Grepped for removed daemon class name.

Diff Detail

Repository

rP Phabricator

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

epriestley updated this revision to Diff 28594.Feb 24 2015, 2:41 AM

epriestley retitled this revision from to Merge GC daemon into Trigger daemon.

epriestley updated this object.

epriestley edited the test plan for this revision. (Show Details)

epriestley added a reviewer: btrahan.

epriestley added a task: T7352: Improve daemon scalability in the cluster.

Herald added a subscriber: epriestley. · View Herald TranscriptFeb 24 2015, 2:41 AM

What's the rough footprint reduction?

This revision is now accepted and ready to land.Feb 24 2015, 5:00 PM

The base weight of a PHP process is about 25MB, which is most of the footprint here. Before any of these changes, the minimum footprint per instance was:

1 Garbage
1 Trigger
1 PullLocal
2 TaskMaster
5 Overseer

So a total of 10 processes, or a minimum of 250MB per instance. I was able to save:

4 Overseers (by using one overseer for all daemons)
1 Taskmaster (by autoscaling taskmaster pools)
1 Garbage (by merging into Trigger, here).

So the minimum is now 4 processes (Trigger, PullLocal, Task, Overseer) or about 100MB. This change, specifically, took us from about 125MB to about 100MB. Overall, I increased the number of instances we can put in the same amount of memory by 2.5X.

The repo hosts (where daemons live) are currently m3.large instances with 7.5GB of RAM, of which the daemon will have access to maybe 6GB. So we can put about 50-ish mostly-idle instances on a box now instead of 20-ish.

Presumably we won't have very many instances with only 1-2 users who never log in or do anything, so this isn't really that big of a deal, but it gives us a lot more headroom.