Page MenuHomePhabricator

Intermittent issues with Aphlict / Notifications
Closed, InvalidPublic

Description

It's strange and non reproductible. Sometimes it works and then upon restart of aphlict server or phabricator daemon or nginx server it stops.
Aphlict logs are empty, just syaing that aphlict server started well or that testnotification publish a notification.

So the client can't join the servers sometimes and then sometimes it mysterialy works.

webReverse Proxy Nginx that encrypt (TLS upon EC self-signed cert)Web server Nginx (with documentation configuration)Nodejs + Ws

Shall I wait for T6889: Evaluate PHP alternatives to NodeJS for Aphlict ?

Event Timeline

tycho.tatitscheff raised the priority of this task from to High.
tycho.tatitscheff updated the task description. (Show Details)
tycho.tatitscheff added a project: Aphlict.
chad renamed this task from Notifications are really broken to Intermittent issues with Aphlict / Notifications.May 12 2015, 5:07 PM
chad raised the priority of this task from High to Needs Triage.
chad updated the task description. (Show Details)

I don't currently anticipate ever pursuing T6889.

We haven't experienced this issue or seen other reports of it, and don't currently have enough information to reproduce it, so this isn't actionable in its current state.

Ok. Is there some debug log I can activate (nginx or phabricator side ) ?

So some more information.
On the web server :

nginx phabricator server
server {                                                                                                                                                                                      
  listen 80;  
  server_name t.a.a.info;
  root        /srv/phabricator/phabricator/webroot;
  client_max_body_size 32m;
  charset utf-8;
  
  location / {
    index index.php;
    rewrite ^/(.*)$ /index.php?__path__=/$1 last;
  }

  location /index.php {
    fastcgi_pass   unix:/var/run/php5-fpm.sock;
    fastcgi_index   index.php;

    #required if PHP was built with --enable-force-cgi-redirect
    fastcgi_param  REDIRECT_STATUS    200;

    #variables to make the $_SERVER populate in PHP
    fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
    fastcgi_param  QUERY_STRING       $query_string;
    fastcgi_param  REQUEST_METHOD     $request_method;
    fastcgi_param  CONTENT_TYPE       $content_type;
    fastcgi_param  CONTENT_LENGTH     $content_length;

    fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;

    fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
    fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

    fastcgi_param  REMOTE_ADDR        $remote_addr;
  }
 
 location = /ws/ {
    proxy_pass http://websocket_pool;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 999999999;
  }
}

on the proxy (vlan 10.0.107.4/24 maped to our public ip) :

nginx phabricator server
server {                                                                                                                                                                                      
    listen 80;
    listen 443 ssl; 
    server_name t.a.a.info;
    client_max_body_size 32M; 

    ssl on;
    ssl_certificate_key /etc/nginx/ssl/a.key.pem;
    ssl_certificate /etc/nginx/ssl/a.crt;
  
    ssl_ciphers "AES128+EECDH:AES128+EDH";
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    
    ssl_prefer_server_ciphers on;
    ssl_dhparam /etc/nginx/ssl/dhparam.pem;
    
    if ($ssl_protocol = "") {
        rewrite ^ https://$host$request_uri? permanent;
    }     

    location / {
        proxy_pass http://10.0.107.4;
        proxy_set_header Host $host;                                                                   
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }    
}

As said the certificates are self-generated EC (prime256v1) certificates signed by a rsa4096 self-signed CA.

About the config :

configvalue
notification.enabledtrue
notification.client-urihttp://localhost/ws/
notification.server-urihttp://localhost:22281/
notification.ssl-keynull
notification.ssl-certnull

The aphlict server starts correctly and listen phabricator event without problem but cant connect to any client.
Some debugging atemps :

from webserver wcat port

wscat -c ws://127.0.0.1:22280/
... error: Error: connect ECONNREFUSED

from webserver wcat ws://url/ws/

wscat -c ws://t.a.a.info/ws/
... error: Error: unexpected server response (502)

from webserver wcat ws://url/ws

wscat -c ws://t.a.a.info/ws
... error: Error: unexpected server response (301)

So it seems to responds at /ws and not /ws .
Im still looking.

Up.

I think people over the net got alse problème behind nginx and proxy when using socket io and only websockets.
Once the connection is etablished, the problems disapeared.
So they configure socket.io to first make long term polling then degrade to websocket .
However I was on public computer when I browse this but I can try to found the link again ?

Do you think it could be an hint ?
How Wikimedia peoples are doing with their proxy configuration ? They seems to get issue too !

https://phabricator.wikimedia.org/T765#1273259
https://phabricator.wikimedia.org/T68989#726453

One solution may be to temporary replace ws dependency with something stronger that can do this degradation (like http://socket.io/).

  • Pro are that it can also allow old navigator to connect with other method.
  • Cons are tht it adds extra dependency (well it replace one with something bigger).

WDYT

@tycho.tatitscheff have you ever solved this? I'm experiencing the very same issue...

This doesn't describe a reproducible problem. See Providing Reproduction Steps.

We do not plan to degrade the protocol; clients must implement websockets.

T10697 may possibly help.