Change Details

After T7352, daemons are organized into "pools" which can autoscale up and down, so each instance can, for example, run 1 taskmaster by default and scale up to 4 when there's work to be done. This allowed us to get about 100 instances onto each `repo` host. Now that we have free instances, scalability is again bottlenecked by daemon memory pressure. The most-available fix we can apply is to allow pools to scale down below 1 process, to 0, so they don't need to be using any resources while asleep. In theory, this gives us about 4x headroom by stopping the Taskmaster, PullLocal, and Trigger processes and just leaving the Overseer rrunning (although, realistically, we can't use all of that since we need some free memory for actual work, but 2x-3x is likely safe). The infrastructure changes out of T7352 work, but they aren't especially clean. In particular, the Overseer has some awkward responsibilities, not everything is really a pool, autoscaling does some magic, there's a lot of "dictionary of keys" stuff instead of "actual object" stuff, and so on. I plan to clean this up first (let the Overseer have a list of Pools, not a list of DaemonHandles, then make the Pools deal with the daemon/autoscale stuff), then give daemons tools to entomb themselves.

After T7352, daemons are organized into "pools" which can autoscale up and down, so each instance can, for example, run 1 taskmaster by default and scale up to 4 when there's work to be done. This allowed us to get about 100 instances onto each `repo` host. Now that we have free instances, scalability is again bottlenecked by daemon memory pressure. The most-available fix we can apply is to allow pools to scale down below 1 process, to 0, so they don't need to be using any resources while asleep. In theory, this gives us about 4x headroom by stopping the Taskmaster, PullLocal, and Trigger processes and just leaving the Overseer running (although, realistically, we can't use all of that since we need some free memory for actual work, but 2x-3x is likely safe). The infrastructure changes out of T7352 work, but they aren't especially clean. In particular, the Overseer has some awkward responsibilities, not everything is really a pool, autoscaling does some magic, there's a lot of "dictionary of keys" stuff instead of "actual object" stuff, and so on. I plan to clean this up first (let the Overseer have a list of Pools, not a list of DaemonHandles, then make the Pools deal with the daemon/autoscale stuff), then give daemons tools to entomb themselves.

After T7352, daemons are organized into "pools" which can autoscale up and down, so each instance can, for example, run 1 taskmaster by default and scale up to 4 when there's work to be done. This allowed us to get about 100 instances onto each `repo` host. Now that we have free instances, scalability is again bottlenecked by daemon memory pressure. The most-available fix we can apply is to allow pools to scale down below 1 process, to 0, so they don't need to be using any resources while asleep. In theory, this gives us about 4x headroom by stopping the Taskmaster, PullLocal, and Trigger processes and just leaving the Overseer rrunning (although, realistically, we can't use all of that since we need some free memory for actual work, but 2x-3x is likely safe). The infrastructure changes out of T7352 work, but they aren't especially clean. In particular, the Overseer has some awkward responsibilities, not everything is really a pool, autoscaling does some magic, there's a lot of "dictionary of keys" stuff instead of "actual object" stuff, and so on. I plan to clean this up first (let the Overseer have a list of Pools, not a list of DaemonHandles, then make the Pools deal with the daemon/autoscale stuff), then give daemons tools to entomb themselves.