We've noticed that notifications haven't been working on our install the past several weeks. When attempting to do basic testing, the aphlict server quits unexpectedly:
cspeck@host ~> curl https://testing.company.com:22280 -vvvvv * Rebuilt URL to: https://testing.company.com:22280/ * Trying 10.x.x.x... * Connected to testing.company.com (10.x.x.x) port 22280 (#0) * Server aborted the SSL handshake * Closing connection 0 curl: (35) Server aborted the SSL handshake
[cspeck@testing phabricator]$ sudo -u nginx ./bin/aphlict debug --config ./conf/aphlict/aphlict.custom.json Reading configuration from: /usr/local/phacility/phabricator/conf/aphlict/aphlict.custom.json Starting Aphlict server in foreground... Launching server: $ node '--max-old-space-size=256' -- '/usr/local/phacility/phabricator/support/aphlict/server/aphlict_server.js' '--config=/usr/local/phacility/phabricator/conf/aphlict/aphlict.custom.json' [11/18/2016, 9:03:04 PM] Starting servers (service PID 10345). [11/18/2016, 9:03:04 PM] Logging to "/var/log/aphlict.log". [11/18/2016, 9:03:04 PM] Started client server (Port 22280, With SSL). [11/18/2016, 9:03:04 PM] Started admin server (Port 22281, No SSL). [11/18/2016, 9:03:04 PM] This server has fingerprint "um6kb7vpJTDQ88RM". >>> Server exited!
NodeJS 6.9.1 is the current LTS version of Node, which is what yum installs/upgrades to by default. Looking at the Changelog for Node 6.9.1, nothing explicitly mentions SSL but there are references to some hashing things. I suppose it's possible the issue might also be related to which version of OpenSSL is installed on the system as well, though I'm not as certain I know the best way to go about changing OpenSSL versions for compiling Node or ws.
I tried adding logs allover the place in the aphlict server and ws. In all the places I tried (I know maybe a little javascript), all I could conclude was that it's very likely the error happens very very early in the SSL handshake that it's deep inside ws or NodeJS's internal usage of OpenSSL.
Workaround
- I've tested swapping out the node binary for both v6.9.0 and v7.1.0, re-installing ws (just in case). Doing this seems to get things working again.
- Let nginx handle the websocket request and terminate the SSL (by following these steps). I tried this and it does work.
Workarounds that don't work: I tried installing/using old versions of ws as the aphlict project does not specify which version it relies on. The current version is v1.1.1, and I tried v0.8, v0.7, and v0.6 all with the same results.
Environment
| OS | CentOS 7 |
| Node | v6.9.1 |
| libphutil | b4f866bd75ec138397a16337bc5d326e586a6276 |
| arcanist | e17fe43ca3fe6dc6dd0b5ce056f56310ea1d3d51 |
| phabricator | 10c4dedd183092dbb0ffc5825cdd42120a0e1b16 |
[ { "type": "client", "host": "testing.company.com", "port": 22280, "protocol": "https" }, { "type": "admin", "host": "127.0.0.1", "port": 22281, "protocol": "http" } ]
{ "servers": [ { "type": "client", "port": 22280, "listen": "0.0.0.0", "ssl.key": "/etc/ssl/company_com.key", "ssl.cert": "/etc/ssl/company_com.pem", "ssl.chain": null }, { "type": "admin", "port": 22281, "listen": "127.0.0.1", "ssl.key": null, "ssl.cert": null, "ssl.chain": null } ], "logs": [ { "path": "/var/log/aphlict.log" } ], "pidfile": "/var/tmp/aphlict/pid/aphlict.pid" }