Page MenuHomePhabricator

Downloading Attachment Extremely Slow
Closed, WontfixPublic

Description

It seems like downloading a file via /files/data/..., the download is incredibly slow. Not entirely too sure why, or where to look:

pasted_file (25×1 px, 6 KB)

pasted_file (209×417 px, 20 KB)

I know our connection to the phab instance we have is much faster, considering it takes only a few seconds to pull a git repo ten times the size of the image above.

Few notes: Files are stored on the server via the file backing store on their own volume. During these tests the volume was not busy/loaded.

❯❯❯ hdparm -Tt /dev/xvdb

/dev/xvdb:
 Timing cached reads:   18796 MB in  1.99 seconds = 9440.13 MB/sec
 Timing buffered disk reads: 1978 MB in  3.00 seconds = 659.04 MB/sec
❯❯❯ hdparm -Tt --direct /dev/xvdb

/dev/xvdb:
 Timing O_DIRECT cached reads:   394 MB in  2.00 seconds = 196.73 MB/sec
 Timing O_DIRECT disk reads: 2350 MB in  3.00 seconds = 782.81 MB/sec

We are running on Ubuntu 14.04, PHP 5.5.9-1, MariaDB 10.0.20 and nginx/1.4.6.

We are serving PHP via php-fpm, here's the nginx config:

server {
  server_name phabricator.int;
  root        /var/www/phab/phabricator/webroot;
  client_max_body_size 1g;

  location / {
    index index.php;
    rewrite ^/(.*)$ /index.php?__path__=/$1 last;
  }

  location = /favicon.ico {
    try_files $uri =204;
  }

  location /index.php {
    fastcgi_pass   php;
    fastcgi_index   index.php;
    fastcgi_buffers 16 64k;

    #required if PHP was built with --enable-force-cgi-redirect
    fastcgi_param  REDIRECT_STATUS    200;

    #variables to make the $_SERVER populate in PHP
    fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
    fastcgi_param  QUERY_STRING       $query_string;
    fastcgi_param  REQUEST_METHOD     $request_method;
    fastcgi_param  CONTENT_TYPE       $content_type;
    fastcgi_param  CONTENT_LENGTH     $content_length;
    fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;

    fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
    fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

    fastcgi_param  REMOTE_ADDR        $remote_addr;
  }
}

Event Timeline

jhgg raised the priority of this task from to Needs Triage.
jhgg updated the task description. (Show Details)
jhgg added a project: Files.
jhgg added a subscriber: jhgg.
jhgg updated the task description. (Show Details)

I've completely disabled the MySQL storage engine by setting storage.mysql-engine.max-size to 0 and then running the ./bin/files/migrate --engine local-disk --all to move the files from MySQL to the local disk. Thinking maybe MySQL blob data could have been the bottleneck... no improvements noticed.

epriestley triaged this task as Wishlist priority.Nov 24 2015, 4:59 PM

Just to set expectations:

  • This report has enough details that we can try to reproduce it, but reproducing it requires configuring a lot of stuff and could realistically take hours. I don't expect to try to reproduce it for a very long time (like, years) unless other users run into the same problem.
  • I suspect the problem almost certainly won't reproduce when I eventually go through your steps, because numerous similar setups don't have any issues.

If you want a more timely resolution to this, you could try narrowing down the problem on your own (e.g., do a fresh test install on EC2 and see if every install you touch has the same problem).

epriestley claimed this task.

We no longer offer support for this kind of problem (that technically has reproduction steps, but is sufficiently involved to reproduce that no one has time to follow them, e.g. build a new server from scratch with assorted specific software versions).

We routinely get 10MB/sec download rates for multi-gigabyte data exports from admin.phacility.com so I'm fairly confident this is a configuration problem in your environment, not a bug in Phabricator.

You can try these approaches:

  • Try reporting this via Discourse and see if another member of the community is able to follow your steps and reproduce the issue.
  • Simplify the reproduction case so that the issue is easier to reproduce, then report it via Discourse.
  • Pay us for support (see Support Pacts) and we'll hunt down weird environmental configuration stuff as long as you remain solvent.