Page MenuHomePhabricator
Create Task
Maniphest T13182

Something in the Ubuntu environment changed, affecting node / "ws"
Closed, ResolvedPublic
Actions

Description

After phage remote deploy --hosts secure001-4 --limit 1, all four hosts failed when rebooting Node with errors like this:

[secure004]  RESTART APHLICT  Restarting Aphlict servers...
[secure004] [2018-08-10 16:44:17] EXCEPTION: (CommandException) Command failed with error #255!
[secure004] COMMAND
[secure004] /core/lib/phabricator/bin/aphlict restart --config /core/conf/aphlict/secure.aphlict.json
[secure004] 
[secure004] STDOUT
[secure004] Reading configuration from: /core/conf/aphlict/secure.aphlict.json
[secure004] Aphlict Server (10402) exited normally.
[secure004] 
[secure004] 
[secure004] STDERR
[secure004] Stopping Aphlict Server (10402)...
[secure004] [2018-08-10 16:44:17] EXCEPTION: (CommandException) Command failed with error #1!
[secure004] COMMAND
[secure004] nodejs '--max-old-space-size=256' -- '/core/lib/phabricator/support/aphlict/server/aphlict_server.js' '--config=/core/conf/aphlict/secure.aphlict.json' '--test=true'
[secure004] 
[secure004] STDOUT
[secure004] [Fri Aug 10 2018 16:44:17 GMT+0000 (UTC)] 
[secure004] <<< UNCAUGHT EXCEPTION! >>>
[secure004] 
[secure004] Error: You need to install the Node.js "ws" module for websocket support. See "Notifications User Guide: Setup and Configuration" in the documentation for instructions. SyntaxError: Use of const in strict mode.
[secure004]     at Object.<anonymous> (/core/lib/phabricator/support/aphlict/server/aphlict_server.js:73:9)
[secure004]     at Module._compile (module.js:456:26)
[secure004]     at Object.Module._extensions..js (module.js:474:10)
[secure004]     at Module.load (module.js:356:32)
[secure004]     at Function.Module._load (module.js:312:12)
[secure004]     at Function.Module.runMain (module.js:497:10)
[secure004]     at startup (node.js:119:16)
[secure004]     at node.js:902:3
[secure004] 
[secure004] 
[secure004] STDERR
[secure004] 
[secure004] /core/lib/phabricator/sup... (1,359 more byte(s)) ... at [<phutil>/src/future/exec/ExecFuture.php:369]
[secure004] libcore(), phutil(head=master, ref.master=dd136d1c3712)
[secure004]   #0 ExecFuture::resolvex() called at [<phutil>/src/future/exec/execx.php:17]
[secure004]   #1 execx(string, string, string) called at [<libcore>/workflow/host/CoreHostUpgradeWorkflow.php:794]
[secure004]   #2 CoreHostUpgradeWorkflow::upgradeAphlictService() called at [<libcore>/workflow/host/CoreHostUpgradeWorkflow.php:45]
[secure004]   #3 CoreHostUpgradeWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441]
[secure004]   #4 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333]
[secure004]   #5 PhutilArgumentParser::parseWorkflows(array) called at [/core/scripts/host.php:19]
[secure004] [2018-08-10 09:44:17] EXCEPTION: (Exception) Command failed with error 255! at [<libcore>/workflow/remote/CoreRemoteWorkflow.php:59]
[secure004] libcore(), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=5a1a8323ee80)
[secure004]   #0 CoreRemoteWorkflow::passthruRemote(string) called at [<libcore>/workflow/remote/CoreRemoteDeployWorkflow.php:147]
[secure004]   #1 CoreRemoteDeployWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:441]
[secure004]   #2 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:333]
[secure004]   #3 PhutilArgumentParser::parseWorkflows(array) called at [/Users/epriestley/dev/core/scripts/remote.php:54]
[secure004] Command ("/Users/epriestley/dev/core/bin/remote 'deploy' 'secure004'") exited nonzero ("255")!

See T12171 for the previous chapter in this adventure.

Related Objects

Event Timeline

epriestley triaged this task as Low priority.Aug 10 2018, 4:50 PM
epriestley created this task.
Herald added a subscriber: eadler. · View Herald TranscriptAug 10 2018, 4:50 PM
epriestley added a comment.Aug 10 2018, 4:54 PM
[secure004] Error: You need to install the Node.js "ws" module for websocket support. See "Notifications User Guide: Setup and Configuration" in the documentation for instructions. SyntaxError: Use of const in strict mode.

I suspect nodejs upgraded and restored /usr/bin/nodejs, so we're now running nodejs (very old Node, v0.10.25) instead of node (manual bind to stable in T12171, v7.4.0 -- seven whole major versions ahead LUL).

I'm going to sudo mv /usr/bin/nodejs /usr/bin/nodejs.ubuntu, per T12171, to take nodejs out of path and see if that fixes it.

$ sudo mv /usr/bin/nodejs /usr/bin/nodejs.ubuntu
$ ./lib/phabricator/bin/aphlict start
Reading configuration from: phabricator/conf/aphlict/aphlict.default.json
Writing logs to: /var/log/aphlict.log
Aphlict Server started.

Yep. I'll mv the other three hosts and then redeploy.

epriestley closed this task as Resolved.Aug 10 2018, 5:26 PM

That appears to have fixed it, so this was just T12171-redux.

The current "fix" is awful but I think we're planning to move to Ubuntu 16 in T13076 which will probably moot this.

epriestley mentioned this in T12171: Node is now very exciting to install on Ubuntu 14?.Aug 10 2018, 5:26 PM
Phabricator · Built by Phacility