Page MenuHomePhabricator

Websocket notification not working anymore after upgrade from 2016 week 15 to 2016 week 18 (stable branch)
Closed, DuplicatePublic

Description

Background

In the timespan between week 15 and week18 Aphlict configuration has been changed to support clustering.

I've upgraded our install and reconfigured our installation as specified here: https://secure.phabricator.com/book/phabricator/article/notifications/

  • The "Cluster Notification Status" page now shows both the client and admin server up and running.
  • I can successfully connect to the websocket endpoint using a standard weboscket client and I can see the connection count going up in the "Cluster Notification Status" page.

Problem

Desktop notification are not working anymore within the Phabricator webapp.

I've brought up a test instance on Phacility for comparison and I can see our host is now missing a line similar to this that the test host serves in every page:

"aphlict-listen":[{"websocketURI":"wss:\/\/nlb001.phacility.com:443\/~test-ggh2o4cguhyk\/?instance=test-ggh2o4cguhyk"

Of course our wss:// URI would be different than this one but any page served by our install is totally missing the "aphlict-listen" field.

To confirm this I've used Chrome developer tools' network tab to check for any WS connection and I can also confirm that on the Phacility Test Host I can see the wss:// request with status 101.
That same line is not present anymore on hour host (websocket notifications were running fine before the upgrade).

Reproduction steps

Upgrade a single hosted, Debian based Phabricator instance with working websocket notifications from the old non-clustered configuration type (stable branch 2016 week 15) to the new Aphlict clustered notification configuration (stable branch 2016 week 18).

Event Timeline

Can you give precise reproduction steps, please? Specifically:

I've upgraded our install and reconfigured our installation as specified here: https://secure.phabricator.com/book/phabricator/article/notifications/

Knowing what steps you took are important for us to locally reproduce the issue. I should be able to completely follow each step and re-setup your configurations and observe the problem for ourselves.

  • I git pulled the 3 phabricator repos from the stable branch.
  • I changed how aphlict is started from:
/var/app/phab/phabricator/bin/aphlict start --client-host=localhost --client-port=22280

To:

/var/app/phab/phabricator/bin/aphlict start
  • I did not specify any custom aphlict json configuration file as phabricator's default one is okay for my configuration.
[
  {
    "type": "client",
    "host": "phab.duckma.com",
    "port": 443,
    "protocol": "https",
    "path": "/ws/"
  },
  {
    "type": "admin",
    "host": "127.0.0.1",
    "port": 22281,
    "protocol": "http"
  }
]

I attach a screenshot of the notification server status page:

Screen Shot 2016-05-04 at 15.53.31.png (370×2 px, 71 KB)

  • As final step I've removed the 2 old notification configuration variables because the "Setup Issues" page started complaining that there were obsolete configuration variables.
  • Nginx websocket proxy configuration has not been changed.

I suspect you may not be setting $_SERVER['HTTPS'] so Phabricator believes you are connecting over HTTP. It looks for a client using the http protocol, fails to find one (only https is available), and does not emit a websocket setup instruction.

I can make this more explicit in the notification panel, which should confirm if it's the problem or not.

This is an issue in general that we should raise as a setup issue, but it's difficult because we can only detect HTTPS on the client, and currently have no other client-side setup issues. I don't immediately see a way to build them that isn't fairly cumbersome. We can't trust X-Forwarded-Proto in the general case because it's a security risk to trust it if it came directly from a client instead of from a trusted loadbalancer.

@epriestley you nailed it.

I followed the instructions at https://secure.phabricator.com/book/phabricator/article/configuring_preamble/ and now everything is back to normal.

Feel free to close this task.

Thanks,
Marco

I suppose we can trust X-Forwarded-Proto for the purposes of raising a setup issue, although I'm not sure if that would help here, since I think the header usually won't be set if you use nginx to terminate SSL instead of, say, an ELB.

Although it's not a default setting,

proxy_set_header X-Forwarded-Proto $scheme;

is a common configuration option when using nginx to terminate SSL.

If configured this would ensure nginx sends the X-Forwarded-Proto header.

It could be added in the sample nginx.conf at: https://secure.phabricator.com/book/phabricator/article/configuration_guide/