Page MenuHomePhabricator

Fix some issues with empty daemon autoscale pools
ClosedPublic

Authored by epriestley on Jun 13 2016, 8:49 PM.
Tags
None
Referenced Files
F14388303: D16110.diff
Sat, Dec 21, 4:04 PM
Unknown Object (File)
Thu, Dec 19, 9:05 PM
Unknown Object (File)
Thu, Dec 5, 3:25 AM
Unknown Object (File)
Tue, Nov 26, 6:03 AM
Unknown Object (File)
Oct 19 2024, 9:54 AM
Unknown Object (File)
Oct 19 2024, 12:38 AM
Unknown Object (File)
Oct 18 2024, 4:40 AM
Unknown Object (File)
Sep 12 2024, 8:33 AM
Subscribers
None

Details

Summary

Ref T10811. I'm not sure this causes any real problems, but I caught a couple of errors in the production logs in this vein:

[13-Jun-2016 14:26:55 UTC] [2016-06-13 14:26:55] ERROR 8: Undefined index:  at [/core/lib/libphutil/src/daemon/PhutilDaemonOverseer.php:302]
[13-Jun-2016 14:26:55 UTC] arcanist(head=stable, ref.master=29839e8c72c5, ref.stable=7b0aac5c6f31), libcore(), phabricator(head=stable, ref.stable=b26ecb189e48), phutil(head=stable, ref.master=992abe4a420c, ref.stable=52748950bb36), services(head=stable, ref.master=5b51b63027b7, ref.stable=66cf59af4580)
[13-Jun-2016 14:26:55 UTC]   #0 PhutilDaemonOverseer::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:224]
[13-Jun-2016 14:26:55 UTC]   #1 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:20]
[13-Jun-2016 14:26:55 UTC] [2016-06-13 14:26:55] EXCEPTION: (InvalidArgumentException) Argument 1 passed to PhutilDaemonOverseer::getAutoscaleProperty() must be an instance of PhutilDaemonHandle, null given, called in /core/lib/libphutil/src/daemon/PhutilDaemonOverseer.php on line 303 and defined at [<phutil>/src/error/PhutilErrorHandler.php:200]
[13-Jun-2016 14:26:55 UTC] arcanist(head=stable, ref.master=29839e8c72c5, ref.stable=7b0aac5c6f31), libcore(), phabricator(head=stable, ref.stable=b26ecb189e48), phutil(head=stable, ref.master=992abe4a420c, ref.stable=52748950bb36), services(head=stable, ref.master=5b51b63027b7, ref.stable=66cf59af4580)
[13-Jun-2016 14:26:55 UTC]   #0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:278]
[13-Jun-2016 14:26:55 UTC]   #1 PhutilDaemonOverseer::getAutoscaleProperty(NULL, string, integer) called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:303]
[13-Jun-2016 14:26:55 UTC]   #2 PhutilDaemonOverseer::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:224]
[13-Jun-2016 14:26:55 UTC]   #3 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:20]

These are caused by attempting to update empty autoscale pools. Currently, autoscale pools read configuration from the first daemon in the pool, but obviously that doesn't work if the pool is empty.

This could occur when shutting down or possibly while reloading daemons.

Also, add a diagnostic message bout pool autoscaling. This might be a little too chatty, but maybe useful for some other issues.

Test Plan
  • A reliable way to hit this was bin/phd debug task --autoscale, then ^C it. It no longer errors.
  • I used bin/worker flood to artificially create a lot of work, and saw the pool autoscale up and down properly.
  • I launched daemons normally (no autoscale).
  • I faked daemons as busy and saw the pool autoscale to full.
  • I changed config and saw daemons automatically reload properly.

Diff Detail

Repository
rPHU libphutil
Branch
daemon1
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 12637
Build 16050: Run Core Tests
Build 16049: arc lint + arc unit

Event Timeline

epriestley retitled this revision from to Fix some issues with empty daemon autoscale pools.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.
chad edited edge metadata.
This revision is now accepted and ready to land.Jun 13 2016, 8:50 PM
This revision was automatically updated to reflect the committed changes.