This report contains several issues that might be caused by a common problem so I refrained from opening multiple bug reports for it. I will split it up if requested.
Version Information
phabricator 3f9e8e675bd94a481974c65cbf4933a6a97e907c (Mon, Apr 11)
arcanist 737f5c0df976fe2b3178aac6ab7feb3d3621d99e (Sat, Apr 9)
phutil a46f7b7e5d8073a91e89f7facc9fea62baaca7e8 (Sun, Apr 10)
sprint df6e9dee03e4dd0da551199df33b6d04bf096d32 (Mon, Apr 11)
Setup (Steps to Reproduce?)
I am running this docker image of phabricator behind an apache reverse proxy to have a unified address with SSL/TLS for my phabricator and other services. As you can see above, I also added the sprint extension but I don't think that this matters.
notification.enabled is set to true.
Initially, I had problems with the settings of notification.client-uri and notification.server-uri that I solved as follows:
- notification.client-uri = https://phabricator.example.org/ws/ (proxy address that forwards to port 22280 on the phabricator docker container)
- notification.server-uri = http://localhost:22281/ (default setting)
- added preamble.php with $_SERVER['HTTPS']=true; to get wss:// links
Now I started aphlict in debug mode with parameter --client-port=22280 as it would not bind to that port due to notification.client-uri (see T9293#134595) and watched the debug output there when browsing/sending test notifications.
Firefox does not connect
On Firefox the notification area says "Setting Up Client" forever and nothing is visible in aphlict's output.
On this site the notifications work in the same Firefox.
Listeners accumulate (-> memory leak?)
On Chromium the notification area says "Connected" and aphlict says:
[Tue Apr 12 2016 12:10:40 GMT+0000 (UTC)] <Listener/1/> Connected from 172.18.0.1. [Tue Apr 12 2016 12:10:40 GMT+0000 (UTC)] <Listener/1/> Received message: {"command":"subscribe","data":["PHID-USER-daleu2naqv4ajmptqerr"]} [Tue Apr 12 2016 12:10:40 GMT+0000 (UTC)] <Listener/1/> Subscribed to: ["PHID-USER-daleu2naqv4ajmptqerr"]
where 172.18.0.1 is the IP of my apache proxy as seen by the docker container.
When I reload, I get
[Tue Apr 12 2016 12:11:24 GMT+0000 (UTC)] <Listener/1/> Disconnected. [Tue Apr 12 2016 12:11:27 GMT+0000 (UTC)] <Listener/2/> Connected from 172.18.0.1. [Tue Apr 12 2016 12:11:27 GMT+0000 (UTC)] <Listener/2/> Received message: {"command":"subscribe","data":["PHID-USER-daleu2naqv4ajmptqerr"]} [Tue Apr 12 2016 12:11:27 GMT+0000 (UTC)] <Listener/2/> Subscribed to: ["PHID-USER-daleu2naqv4ajmptqerr"]
so the first listener is disconnected and a new one is opened. This leads to an increasing listener number and to the warning:
(node) warning: possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit. Trace at WebSocketServer.addListener (events.js:239:17) at WebSocketServer.<anonymous> (/srv/phabricator/phabricator/support/aphlict/server/lib/AphlictClientServer.js:87:13) at emitOne (events.js:77:13) at WebSocketServer.emit (events.js:169:7) at /srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:79:14 at completeHybiUpgrade2 (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:262:5) at completeHybiUpgrade1 (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:287:13) at WebSocketServer.handleHybiUpgrade (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:315:3) at WebSocketServer.handleUpgrade (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:152:26) at Server.<anonymous> (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:77:12) (node) warning: possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit. Trace at WebSocketServer.addListener (events.js:239:17) at WebSocketServer.<anonymous> (/srv/phabricator/phabricator/support/aphlict/server/lib/AphlictClientServer.js:92:13) at emitOne (events.js:77:13) at WebSocketServer.emit (events.js:169:7) at /srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:79:14 at completeHybiUpgrade2 (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:262:5) at completeHybiUpgrade1 (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:287:13) at WebSocketServer.handleHybiUpgrade (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:315:3) at WebSocketServer.handleUpgrade (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:152:26) at Server.<anonymous> (/srv/phabricator/phabricator/support/aphlict/server/node_modules/ws/lib/WebSocketServer.js:77:12)
Subscribers are not counted
When visiting /notification/status/ I can see that notifications have been issued (messages.in > 0) but none have been delivered (messages.out == 0). Furthermore, even when there are Listeners in the aphlict.log that have been subscribed and have not yet disconnected, the clients.active and clients.total stay at zero.
Live notifications are not displayed
Now when I send a test notification I get this output from aphlict:
[Tue Apr 12 2016 12:10:45 GMT+0000 (UTC)] Received notification (/ws/): {"key":"6272641019226701868","type":"notification","subscribers":["PHID-USER-daleu2naqv4ajmptqerr"]}
I don't see any notification in the browser (just testing Chromium here as Firefox does not connect).
When I reload the page the notification area shows that I have new notifications.
Cause?
As the notification is working on this site with the same browser that does not even connect to the web socket on mine, I believe that there is an error in my setup. I'd like to find it and probably that should become part of the documentation as the setup was quite a hassle so far.