Page MenuHomePhabricator

Upload file stalles network connections
Closed, InvalidPublic

Description

Installation: OS Debian 8.6 Jessie
Phabricator version:
phabricator 62cf4e6b95abbebc8ae2b1591039967651188c6e (Sat, Jan 14)
arcanist ade25facfdf22aed1c1e20fed3e58e60c0be3c2b (Fri, Jan 6)
phutil 9d85dfab0f532d50c2343719e92d574a4827341b (Fri, Jan 13)
MySQL: mysql Ver 14.14 Distrib 5.5.53, for debian-linux-gnu (x86_64) using readline 6.3
Apache2:
Server version: Apache/2.4.10 (Debian)
Server built: Sep 15 2016 20:44:43
Server's Module Magic Number: 20120211:37
Server loaded: APR 1.5.1, APR-UTIL 1.5.4
Compiled using: APR 1.5.1, APR-UTIL 1.5.4

Storage is configured MySQL (storage.mysql-engine.max-size = 100000) and localdisk (storage.local-disk.path is set).

Phabricator is setup like vhost with next config:

<VirtualHost *:80>
        ServerName phabricator.domain.com
       ServerAlias phab.domain.org

        DocumentRoot /home/repo/phabricator/phabricator/webroot/
        RewriteEngine on
        RewriteRule ^/rsrc/(.*)     -                       [L,QSA]
        RewriteRule ^/favicon.ico   -                       [L,QSA]
        RewriteRule ^(.*)$          /index.php?__path__=$1  [B,L,QSA]
        ErrorLog /var/log/apache2/phabricator-error.log
       CustomLog /var/log/apache2/phabricator-access.log common

<Directory "/home/repo/phabricator/phabricator/webroot">
       Require all granted
</Directory>
</VirtualHost>

Debug of Phabricator initialized by:

darkconsole.enabled
Enable Phabricator's debugging console.
Customized Value: true

darkconsole.always-on
Activate DarkConsole on every page.
Customized Value: true

debug.time-limit
Limit page execution time to debug hangs.
Customized Value: 15

XHProf console plugin is not installed/enabled. If it's needed for diagnostics - I will.

The bug:

pasted_file (180×314 px, 20 KB)

After almost 2 year work with that installation of Phab we started to experience file upload problems. 5 users reported sporadically stalled upload files on 15-20-25% and freez. Network access to server is stalled. Must reset connection.
2 of users always have a problem with upload (Nor drag-and-drop way nor upload form). That appear in task edit and in File module. Testet with all available browsers - IE11, Chrome, Firefox.
Network recovers only by reset (enable/disable) of network adapter to brings back the network connection.
The size of files is from 500K to 2.5Mb. I thought is not a size critical - next time same file can be uploaded for few seconds.

Apache2 logs and Phabricator logs not report any errors which is strange.
All recommended settings in apache2 conf and PHP values are setup to the highest possible timeout or files/upload size.

DarkConsole don't show errors too (hereby can find screenshots)
http://prntscr.com/dw8mp5
http://prntscr.com/dw8mzc

If it's needed more info/actions/logs can provide.

Event Timeline

Additional info - Phab is updated to Master stream after appearance of problem.

Are you able to reproduce the bug on a test instance on Phacility? We need to be able to locally reproduce a bug to accept it.

Is it test facility Master branch synced. Because there all works but here is master updated not stable.
I'm afraid that can't revert to Stable due to database changes.

I can not reproduce this. Here's what I did:

  • I launched a new EC2 instance (a t2.micro) running Debian Jessie (ami-0343ae47).
  • I added a DNS record for the host: debian001.epriestley.com
  • I installed dependencies, following the install guide:
$ sudo apt-get update
$ sudo apt-get install git mysql-server apache2 php5 php5-mysql php5-curl php5-cli
  • I edited the default Apache site to serve Phabricator.
000-default.conf
<Directory "/var/devtools/phabricator/webroot">
  Require all granted
</Directory>

<VirtualHost *>
  ServerName debian001.epriestley.com
  DocumentRoot /var/devtools/phabricator/webroot
  RewriteEngine on
  RewriteRule ^(.*)$ /index.php?__path__=$1 [B,L,QSA]
</VirtualHost>
  • I cloned master:
/var$ sudo mkdir devtools
/var$ sudo chown admin:admin devtools
/var$ cd devtools/

$ git clone https://github.com/phacility/libphutil.git
$ git clone https://github.com/phacility/arcanist.git
$ git clone https://github.com/phacility/phabricator.git
  • I enabled mod_rewrite and restarted Apache:
$ sudo a2enmod rewrite
$ sudo service apache2 restart
phabricator/ $ ./bin/storage upgrade
  • I changed the configuration settings in the original report:
$ ./bin/config set storage.mysql-engine.max-size 100000
$ ./bin/config set storage.local-disk.path /var/storage
$ sudo mkdir /var/storage
$ sudo chmod 777 /var/storage # For simplicity, not a best practice.
  • I uploaded a bunch of files of varying sizes:

Screen Shot 2017-01-17 at 10.01.53 AM.png (1×1 px, 210 KB)

  • Everything worked perfectly:

Screen Shot 2017-01-17 at 10.01.58 AM.png (1×1 px, 207 KB)

I can't reproduce this, so we can't move forward. Next steps:

  • provide us complete, step-by-step instructions that allow us to reproduce this issue in a clean environment, starting from scratch; or
  • we will close this report in a few days if we don't receive a reproduction case.

Reports like this are a very poor use of our limited resources. They waste a lot of our time trying to reproduce issues which are almost inevitably configuration-related and unique to one install. This is why we require complete reproduction steps to accept reports.

If you'd like us to work with you to help diagnose and resolve this problem in your environment, we're happy to do that but charge $1,500/hr. See Consulting.

Wow.
With that charge you can retire at that moment ... ridiculous.
I'm pretty sure in lab clean env from scratch all will works flawlessly. The point is in working environment.
Should be some guide where to dig for a problem.
Anyway will try with new clean install and migration of real data.
And will inform my boss that my month salary some guys take it for an hour :)

What branch you checkout - master or stable ?
Never mind, I found what you've done, sorry.

This does not appear to be moving forward as a valid bug report which we can accept upstream.