Currently, daemons potentially cycle connections very aggressively while under load. After a connection is closed, the outbound port usually can't be reused for about 60 seconds, so sufficiently aggressive cycling can exhaust outbound ports.
A simple partial mitigation is to skip cycling while loaded:
diff --git a/src/infrastructure/daemon/PhabricatorDaemon.php b/src/infrastructure/daemon/PhabricatorDaemon.php index 349e0f0..aadef5a 100644 --- a/src/infrastructure/daemon/PhabricatorDaemon.php +++ b/src/infrastructure/daemon/PhabricatorDaemon.php @@ -11,7 +11,9 @@ abstract class PhabricatorDaemon extends PhutilDaemon { } protected function willSleep($duration) { - LiskDAO::closeAllConnections(); + if ($duration) { + LiskDAO::closeAllConnections(); + } return; }
There should be a number of other similar adjustments to make behavior here a bit more reasonable. We'll still do a connection per application (at least until after T11044) but should be keeping things under ~100 connections per daemon until then.