Page MenuHomePhabricator
Create Task
Maniphest T10264

git clone using http hangs after update
Closed, ResolvedPublic
Actions

Description

After updating libphutil, arcanist, and phabricator, doing the storage upgrade, and then restarting all the daemons, I can no longer pip install from a git repo. I removed pip from the equation and found that git clone using http no longer works for me. It hangs indefinitely. I can clone using ssh, just not http. I've looked over all the config and it looks good. This has been working for many months, so unless the newer code requires a change to the config somewhere, I don't think this is a configuration issue. I tried both the stable and master branches with the same results. I'm on master now.

Here is the command I'm running:

% git clone https://srlib:<password>@phabricator.sproutretail.com/diffusion/SL/srlib.git@v1.2.15
Cloning into 'srlib.git@v1.2.15'...

And that is all I get before it hangs indefinitely.

Here is what I get if I set GIT_CURL_VERBOSE and GIT_TRACE:

trace: built-in: git 'clone' 'https://srlib:<password>@phabricator.sproutretail.com/diffusion/SL/srlib.git@v1.2.15'
trace: run_command: 'git-remote-https' 'origin' 'https://srlib:<password>@phabricator.sproutretail.com/diffusion/SL/srlib.git@v1.2.15'
Cloning into 'srlib.git@v1.2.15'...
* Couldn't find host phabricator.sproutretail.com in the .netrc file; using defaults
* Adding handle: conn: 0x970cd0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x970cd0) send_pipe: 1, recv_pipe: 0
* About to connect() to phabricator.sproutretail.com port 443 (#0)
*   Trying 107.170.39.60...
* Connected to phabricator.sproutretail.com (107.170.39.60) port 443 (#0)
* found 164 certificates in /etc/ssl/certs/ca-certificates.crt
* 	 server certificate verification OK
* 	 common name: bytevampire.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: C=US,postalCode=08807,ST=New Jersey,L=Bridgewater,STREET=Suite 2000,STREET=1200 Rt 22 E,O=Sprout Retail\, Inc.,OU=Multi-Domain SSL,CN=bytevampire.com
* 	 start date: Mon, 05 Oct 2015 00:00:00 GMT

* 	 expire date: Thu, 29 Dec 2016 23:59:59 GMT

* 	 issuer: C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Organization Validation Secure Server CA
* 	 compression: NULL
* 	 cipher: AES-128-CBC
* 	 MAC: SHA1
> GET /diffusion/SL/srlib.git@v1.2.15/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.8.3.2
Host: phabricator.sproutretail.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 401 You must log in to access repositories.
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Date: Tue, 02 Feb 2016 02:12:47 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/5.5.9-1ubuntu4.14
< WWW-Authenticate: Basic realm="Phabricator Repositories"
< 
* Ignoring the response-body
* Connection #0 to host phabricator.sproutretail.com left intact
* Issue another request to this URL: 'https://srlib:<password>@phabricator.sproutretail.com/diffusion/SL/srlib.git@v1.2.15/info/refs?service=git-upload-pack'
* Couldn't find host phabricator.sproutretail.com in the .netrc file; using defaults
* Found bundle for host phabricator.sproutretail.com: 0x911470
* Re-using existing connection! (#0) with host phabricator.sproutretail.com
* Connected to phabricator.sproutretail.com (107.170.39.60) port 443 (#0)
* Adding handle: conn: 0x970cd0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x970cd0) send_pipe: 1, recv_pipe: 0
* Server auth using Basic with user 'srlib'
> GET /diffusion/SL/srlib.git@v1.2.15/info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic c3JscWI6U3Xyb3V1ODk=
User-Agent: git/1.8.3.2
Host: phabricator.sproutretail.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Date: Tue, 02 Feb 2016 02:12:47 GMT
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/5.5.9-1ubuntu4.14
< X-Frame-Options: Deny
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< 
* Connection #0 to host phabricator.sproutretail.com left intact
trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--no-progress' 'https://srlib:<password>@phabricator.sproutretail.com/diffusion/SL/srlib.git@v1.2.15/'
trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--no-progress' 'https://srlib:<password>@phabricator.sproutretail.com/diffusion/SL/srlib.git@v1.2.15/'
trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--no-progress' 'https://srlib:<password>@phabricator.sproutretail.com/diffusion/SL/srlib.git@v1.2.15/'
* Couldn't find host phabricator.sproutretail.com in the .netrc file; using defaults
* Adding handle: conn: 0xc4ff20
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 1 (0xc4ff20) send_pipe: 1, recv_pipe: 0
* About to connect() to phabricator.sproutretail.com port 443 (#1)
*   Trying 107.170.39.60...
* Connected to phabricator.sproutretail.com (107.170.39.60) port 443 (#1)
* found 164 certificates in /etc/ssl/certs/ca-certificates.crt
* SSL re-using session ID
* 	 server certificate verification OK
* 	 common name: bytevampire.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: C=US,postalCode=08807,ST=New Jersey,L=Bridgewater,STREET=Suite 2000,STREET=1200 Rt 22 E,O=Sprout Retail\, Inc.,OU=Multi-Domain SSL,CN=bytevampire.com
* 	 start date: Mon, 05 Oct 2015 00:00:00 GMT

* 	 expire date: Thu, 29 Dec 2016 23:59:59 GMT

* 	 issuer: C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Organization Validation Secure Server CA
* 	 compression: NULL
* 	 cipher: AES-128-CBC
* 	 MAC: SHA1
* Server auth using Basic with user 'srlib'
> POST /diffusion/SL/srlib.git@v1.2.15/git-upload-pack HTTP/1.1
Authorization: Basic c3JscWI6U3Xyb3V1ODk=
User-Agent: git/1.8.3.2
Host: phabricator.sproutretail.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 1412

* upload completely sent off: 1412 out of 1412 bytes
< HTTP/1.1 200 OK
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Date: Tue, 02 Feb 2016 02:12:48 GMT
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/5.5.9-1ubuntu4.14
< X-Frame-Options: Deny
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< 
* Connection #1 to host phabricator.sproutretail.com left intact

So, it looks like authentication is working fine and at least a GET and POST to git-upload-pack both return 200, but it hangs after that. I'm not really sure what else to look for.

Thanks,
Tony.

Revisions and Commits

rPHU libphutil
D15483rPHUa4ca530644be Move request decoding/streaming to AphrontRequestStream
rP Phabricator
D15314rP42566379dc3c Fix HTTP body decompression in PHP 5.6
D15258rP5e3754828f7b Fix handling of gzip in VCS responses

Related Objects

Event Timeline

arandrea created this task.Feb 2 2016, 2:43 AM
avivey updated the task description. (Show Details)Feb 2 2016, 2:51 AM
avivey added a subscriber: avivey.Feb 2 2016, 2:56 AM
  • Do you see any errors in the server log?
  • Are you sure about adding @v1.2.15 to the clone uri? I've never seen this before.
arandrea added a comment.Feb 2 2016, 3:14 AM

Which server log are you asking about? I have looked around at various logs and can't find any errors, but maybe I'm not looking in the right place. I see the GET and POST commands, both returning 200, in the nginx logs and that is it. Nothing in syslog.

As far as the @v1.2.15 goes, that was just a cut-n-paste issue when I converted the pip install command to a git clone. It behaves the same, i.e. it hangs, even if I take the tag specifier off the command. But, you are right, normally I wouldn't specify a tag/branch on a git clone command.

joshuaspence added a subscriber: joshuaspence.Feb 2 2016, 12:24 PM
chad added a project: Diffusion.Feb 2 2016, 5:03 PM
epriestley mentioned this in T10265: VCS password using LDAP.Feb 3 2016, 2:02 PM
epriestley added a subscriber: epriestley.Feb 3 2016, 2:12 PM

I'm unable to reproduce this against this host or hosts in the Phacility cluster.

It is possible (but unlikely) that rP9d125b45 resolved this.

It is possible that rP8269fd6 caused this. You can try reverting that change and see if it resolves the issue.

If reverting rP8269fd6 resolves this, we're probably looking at something like this:

  • Perhaps some webserver setups (nginx? FCGI?) decode content bodies before passing them to PHP, while others (apache + mod_php) do not.
  • If so, we probably need to play some kind of guessing game to try to figure out if the content is compressed or not, although I believe this is impossible in the general case. We may be able to introduce configuration and then use a self-test to make sure it's set properly.
  • If the behavior of (string)file_get_contents('compress.zlib://php://input'); is sometimes to hang, it's garbage and we can't use it.
arandrea added a comment.Feb 3 2016, 4:49 PM

I updated to get rP9d125b45 and there is no change. Cloning a repo using http still hangs. I then reverted the change in rP8269fd6 and that did not fix the problem either.

epriestley edited projects, added Bug Report (Needs Information); removed Bug Report.Feb 3 2016, 4:52 PM

I'm out of ideas and can't reproduce this problem, then, so we can't move forward (see Contributing Bug Reports). Here's, e.g., this working fine for me against this server:

$ GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://secure.phabricator.com/diffusion/GITTEST/git-test.git git-testx
08:52:14.954957 git.c:348               trace: built-in: git 'clone' 'https://secure.phabricator.com/diffusion/GITTEST/git-test.git' 'git-testx'
Cloning into 'git-testx'...
08:52:14.962756 run-command.c:335       trace: run_command: 'git-remote-https' 'origin' 'https://secure.phabricator.com/diffusion/GITTEST/git-test.git'
* Couldn't find host secure.phabricator.com in the .netrc file; using defaults
*   Trying 52.8.228.58...
* Connected to secure.phabricator.com (52.8.228.58) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: secure.phabricator.com
* Server certificate: Starfield Secure Certificate Authority - G2
* Server certificate: Starfield Root Certificate Authority - G2
> GET /diffusion/GITTEST/git-test.git/info/refs?service=git-upload-pack HTTP/1.1
Host: secure.phabricator.com
User-Agent: git/2.5.4 (Apple Git-61)
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Wed, 03 Feb 2016 16:52:15 GMT
< Server: Apache
< X-Frame-Options: Deny
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Length: 437
< Content-Type: application/x-git-upload-pack-advertisement
< 
* Connection #0 to host secure.phabricator.com left intact
08:52:15.224866 run-command.c:335       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://secure.phabricator.com/diffusion/GITTEST/git-test.git/'
08:52:15.226881 exec_cmd.c:178          trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://secure.phabricator.com/diffusion/GITTEST/git-test.git/'
08:52:15.254751 git.c:348               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://secure.phabricator.com/diffusion/GITTEST/git-test.git/'
* Couldn't find host secure.phabricator.com in the .netrc file; using defaults
* Found bundle for host secure.phabricator.com: 0x7fac40c1f0d0
* Re-using existing connection! (#0) with host secure.phabricator.com
* Connected to secure.phabricator.com (52.8.228.58) port 443 (#0)
> POST /diffusion/GITTEST/git-test.git/git-upload-pack HTTP/1.1
Host: secure.phabricator.com
User-Agent: git/2.5.4 (Apple Git-61)
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 255

* upload completely sent off: 255 out of 255 bytes
< HTTP/1.1 200 OK
< Date: Wed, 03 Feb 2016 16:52:15 GMT
< Server: Apache
< X-Frame-Options: Deny
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Transfer-Encoding: chunked
< Content-Type: application/x-git-upload-pack-result
< 
remote: Counting objects: 215, done.
remote: Compressing objects: 100% (193/193), done.
08:52:15.403357 run-command.c:335       trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 20095 on Orbital.local' '--check-self-contained-and-connected' '--pack_header=2,215'
08:52:15.405404 exec_cmd.c:178          trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 20095 on Orbital.local' '--check-self-contained-and-connected' '--pack_header=2,215'
08:52:15.406877 git.c:348               trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 20095 on Orbital.local' '--check-self-contained-and-connected' '--pack_header=2,215'
* Connection #0 to host secure.phabricator.com left intact
remote: Total 215 (delta 135), reused 0 (delta 0)
Receiving objects: 100% (215/215), 345.92 KiB | 0 bytes/s, done.
Resolving deltas: 100% (135/135), done.
Checking connectivity... 08:52:15.765365 run-command.c:335       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all'
08:52:15.765779 exec_cmd.c:178          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all'
08:52:15.767385 git.c:348               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all'
done.
• kanarip added a subscriber: • kanarip.Feb 4 2016, 9:12 AM
• kanarip added a comment.Feb 4 2016, 9:26 AM

We are currently experiencing this problem on one of our repositories, but not the other 50 or so.

Reverting rP8269fd6e6c1f8f03f1645215c5748e794cc8486c fixed our issue (thanks for the hint . We are running Apache httpd-2.4.6-31.el7.x86_64 with php-5.4.16-23.el7_0.3.x86_64;

Phabricator at rP71bda66870d8ef832f4d048b11282f9ae0086f05
libphutil at rPHU9c472e7c9b64395424c6cd25734bf239cb3c113d

I've provided GIT_CURL_VERBOSE=1 GIT_TRACE=1 output for the unsuccessful attempt in P1937, and the successful attempt in P1936 in case that helps.

sicnervo added a subscriber: sicnervo.Feb 5 2016, 9:08 AM

I'm experiencing exactly same problem while accessing one of my repos via HTTP.

dtf added a subscriber: dtf.EditedFeb 5 2016, 12:30 PM

Same thing, just one repo out of many (biggest one, only 400 MB, though).
Reverting the rP71bda66870d8: (stable) Decode "Content-Encoding: gzip" content helps.

Our config:

  • Ubuntu 14.04.3
  • Apache 2.4.7
  • PHP 5.5.9-1ubuntu4.14
davidgiga1993 added a subscriber: davidgiga1993.Feb 9 2016, 3:32 PM

Got the same problem with only one of my repos (~ 50MB, largest one).
Reverting the commit mentioned by @dtf fixed the problem.

  • CentOS release 6.7
  • Apache 2.2.15
  • PHP 5.4.45
  • Git 1.7.1
alexkiousis added a subscriber: alexkiousis.Feb 10 2016, 10:05 AM

Reporting the same behaviour. Fetch hangs on our biggest repo.
Reverting the gzip commit fixes the problem.

vanmeeuwen added a subscriber: vanmeeuwen.Feb 11 2016, 7:43 PM
vanmeeuwen added a comment.Feb 11 2016, 7:45 PM

We run origin/stable from rP (times three), except for reverting rP71bda66870d8ef832f4d048b11282f9ae0086f05 to prevent this issue from occuring.

epriestley added a comment.Feb 12 2016, 1:07 AM

Can anyone who is able to reproduce this let me know if this fixes it?

diff --git a/src/applications/diffusion/controller/DiffusionServeController.php b/src/applications/diffusion/controller/DiffusionServeController.php
index 8d131ec..d1655d7 100644
--- a/src/applications/diffusion/controller/DiffusionServeController.php
+++ b/src/applications/diffusion/controller/DiffusionServeController.php
@@ -431,7 +431,6 @@ final class DiffusionServeController extends DiffusionController {
       'REQUEST_METHOD' => $_SERVER['REQUEST_METHOD'],
       'QUERY_STRING' => $query_string,
       'CONTENT_TYPE' => $request->getHTTPHeader('Content-Type'),
-      'HTTP_CONTENT_ENCODING' => $request->getHTTPHeader('Content-Encoding'),
       'REMOTE_ADDR' => $_SERVER['REMOTE_ADDR'],
       'GIT_PROJECT_ROOT' => $repository_root,
       'GIT_HTTP_EXPORT_ALL' => '1',

Specifically:

  • Un-revert the gzip commit if you previously reverted it (so your repository is clean and in sync with the upstream).
  • Apply that patch (delete the one line above).
  • Do whatever you do to reproduce the issue and see if it's still a problem.
epriestley added a comment.Feb 12 2016, 1:09 AM

Oh, and make sure you restart fully after changing the code -- I suspect that might have been the cause of the revert not fixing things earlier. Here's a guide if you aren't sure how:

https://secure.phabricator.com/book/phabricator/article/restarting/

epriestley edited projects, added Bug Report; removed Bug Report (Needs Information).Feb 12 2016, 2:24 AM
davidgiga1993 added a comment.Feb 12 2016, 5:39 AM

I just tested it and it seems like removing the line solves the problem.

dtf added a comment.Feb 12 2016, 7:56 AM

Confirm. After deleting of mentioned line on top of stable everything clones fine.

sicnervo added a comment.Feb 12 2016, 11:48 AM

I think that may be important:

root@srv:/opt/phabricator# GIT_CURL_VERBOSE=1 GIT_TRACE=1 git pull
trace: exec: 'git-pull'
trace: run_command: 'git-pull'
trace: built-in: git 'rev-parse' '--git-dir'
trace: built-in: git 'rev-parse' '--is-bare-repository'
trace: built-in: git 'rev-parse' '--show-toplevel'
trace: built-in: git 'ls-files' '-u'
trace: built-in: git 'symbolic-ref' '-q' 'HEAD'
trace: built-in: git 'config' 'branch.master.rebase'
trace: built-in: git 'config' 'pull.rebase'
trace: built-in: git 'rev-parse' '-q' '--verify' 'HEAD'
trace: built-in: git 'fetch' '--update-head-ok'
trace: run_command: 'git-remote-https' 'origin' 'https://secure.phabricator.com/diffusion/P/phabricator.git'
* Couldn't find host secure.phabricator.com in the .netrc file; using defaults
* Hostname was NOT found in DNS cache
*   Trying 52.8.228.58...
* Connected to secure.phabricator.com (52.8.228.58) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
*        server certificate verification SKIPPED
*        common name: secure.phabricator.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: OU=Domain Control Validated,CN=secure.phabricator.com
*        start date: Mon, 23 Mar 2015 16:47:38 GMT

*        expire date: Mon, 28 Mar 2016 18:47:22 GMT

*        issuer: C=US,ST=Arizona,L=Scottsdale,O=Starfield Technologies\, Inc.,OU=http://certs.starfieldtech.com/repository/,CN=Starfield Secure Certificate Authority - G2
*        compression: NULL
*        cipher: AES-128-CBC
*        MAC: SHA256
> GET /diffusion/P/phabricator.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.9.1
Host: secure.phabricator.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Fri, 12 Feb 2016 11:43:05 GMT
* Server Apache is not blacklisted
< Server: Apache
< X-Frame-Options: Deny
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Length: 373
< Content-Type: application/x-git-upload-pack-advertisement
<
* Connection #0 to host secure.phabricator.com left intact
trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://secure.phabricator.com/diffusion/P/phabricator.git/'
trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://secure.phabricator.com/diffusion/P/phabricator.git/'
trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://secure.phabricator.com/diffusion/P/phabricator.git/'
* Couldn't find host secure.phabricator.com in the .netrc file; using defaults
* Hostname was NOT found in DNS cache
*   Trying 52.8.228.58...
* Connected to secure.phabricator.com (52.8.228.58) port 443 (#1)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* SSL re-using session ID
*        server certificate verification SKIPPED
*        common name: secure.phabricator.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: OU=Domain Control Validated,CN=secure.phabricator.com
*        start date: Mon, 23 Mar 2015 16:47:38 GMT

*        expire date: Mon, 28 Mar 2016 18:47:22 GMT

*        issuer: C=US,ST=Arizona,L=Scottsdale,O=Starfield Technologies\, Inc.,OU=http://certs.starfieldtech.com/repository/,CN=Starfield Secure Certificate Authority - G2
*        compression: NULL
*        cipher: AES-128-CBC
*        MAC: SHA256
> POST /diffusion/P/phabricator.git/git-upload-pack HTTP/1.1
User-Agent: git/1.9.1
Host: secure.phabricator.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 997

* upload completely sent off: 997 out of 997 bytes
< HTTP/1.1 200 OK
< Date: Fri, 12 Feb 2016 11:43:05 GMT
* Server Apache is not blacklisted
< Server: Apache
< X-Frame-Options: Deny
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Length: 904
< Content-Type: application/x-git-upload-pack-result
<
* Connection #1 to host secure.phabricator.com left intact
* Couldn't find host secure.phabricator.com in the .netrc file; using defaults
* Found bundle for host secure.phabricator.com: 0xc184d0
* Re-using existing connection! (#1) with host secure.phabricator.com
* Connected to secure.phabricator.com (52.8.228.58) port 443 (#1)
> POST /diffusion/P/phabricator.git/git-upload-pack HTTP/1.1
User-Agent: git/1.9.1
Host: secure.phabricator.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 594

* upload completely sent off: 594 out of 594 bytes
< HTTP/1.1 200 OK
< Date: Fri, 12 Feb 2016 11:43:06 GMT
* Server Apache is not blacklisted
< Server: Apache
< X-Frame-Options: Deny
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Length: 0
< Content-Type: application/x-git-upload-pack-result
<
* Connection #1 to host secure.phabricator.com left intact
^C
root@srv:/opt/phabricator#

Everything work fine when I'm trying to clone repo to the other location.

epriestley added a comment.Feb 12 2016, 12:12 PM

Awesome, thanks for testing! I think this should be an easy fix, now.

epriestley added a revision: D15258: Fix handling of gzip in VCS responses.Feb 12 2016, 12:25 PM
epriestley closed this task as Resolved by committing rP5e3754828f7b: Fix handling of gzip in VCS responses.Feb 12 2016, 4:09 PM
epriestley added a commit: rP5e3754828f7b: Fix handling of gzip in VCS responses.
epriestley added a comment.Feb 12 2016, 4:09 PM

I believe this is now fixed at HEAD of master. It should promote to stable tomorrow morning.

Please let me know if anyone's still seeing issues after updating. Thanks for everyone's help in verifying/reproducing this!

dpatou added a subscriber: dpatou.EditedFeb 17 2016, 11:10 PM

I'm in the middle of a trip right now so can't test as much as I'd like but i'm still having the issue. On my desktop computer I can pull pretty fine the repos that were previously cloned but on this freshly reinstalled laptop I can't clone repositories, hanging forever after the git-upload-pack with connection left untouched, as the other comments stated here.

I can provide with greater details if needed in about a day.

For the record I just updated my instance of phabricator to https://secure.phabricator.com/rP8c3ca2a72971278b16eaad5a1f13cf34f0ca4068 (with proper server/daemons restarting) this morning so the fix of Feb 12 is supposedly in my instance and not solving.

psychomieze added a subscriber: psychomieze.Feb 18 2016, 3:01 PM
dpatou added a comment.Feb 18 2016, 6:09 PM

I still can't git clone, but if I create an empty folder, init a repo in it, put my phab-hosted repo as origin remote and pull/merge into master, it works just fine great (just as cloning would/should)

epriestley added a comment.Feb 18 2016, 6:20 PM

@dpatou Does reverting both 5e375482 and 8269fd6e fix the issue for you?

What webserver are you using (nginx, apache, ...)?

Broadly, which repository you are cloning does not directly matter as far as gzip / Content-Encoding is concerned. The reason this works for some repositories and request and does not work for other repositories and requests is that git only compresses the request body if it's over a certain size. So what actually matters is how big the request is, which is influenced by things like how many remote branches/tags the local repository knows about, which is presumably why larger/older repositories tend to have issues while smaller/cleaner ones not to.

Note that all the failing requests above have Content-Encoding: gzip in the request headers for the failing request. This presence of absence of this encoding is entirely a function of internal behavior in the git client.

dpatou added a comment.Feb 18 2016, 7:53 PM

@epriestley I will try reverting theses commits when I get a stable ssh into my phab instance and keep you posted.

Webserver is apache.

Cloning worked until I recently did a big update on my phab instance (I tend to pull updates every couple of monthes only)

Repositories don't matter indeed (no matter which repo the outcome is same), it's really cloning vs fetching that seems to matter. None of my repos are that big really one to three branches each. Same goes for tags, some repo have a dozen or so, while some others that fail too don't have any. Small and Big repos alike seem to have this problem for me.

I'll check reverting the commits. As for the gzip 'headers', I tried several versions of git on that rig (thought it was a rig issue at first). I'm not at home and can't remote into my home comp to check out how it behaves but my gut feeling is that it will be the same.

What puzzles me is that init + remote add + pull works whereas clone doesn't, even on single branch repos.

ppggff added a subscriber: ppggff.Feb 19 2016, 9:26 AM

I got the same problem with HEAD of master.
it seems relate to clone a git repo with gziped post data by http.

php version:

root@t450:/var/log/apache2# php --version
PHP 5.6.17-0+deb8u1 (cli) (built: Jan 13 2016 09:10:12)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies

apache's error log shows:

[Fri Feb 19 16:24:09.726119 2016] [:error] [pid 8281] [client 10.168.10.223:55964] PHP Warning:  file_get_contents(): cannot represent a stream of type Input as a File Descriptor in /home/gaozq/pha/phabricator/support/PhabricatorStartup.php on line 144
[Fri Feb 19 16:24:09.726151 2016] [:error] [pid 8281] [client 10.168.10.223:55964] PHP Warning:  file_get_contents(compress.zlib://php://input): failed to open stream: operation failed in /home/gaozq/pha/phabricator/support/PhabricatorStartup.php on line 144

after revert 5e375482 and 8269fd6e, problem fixed.

maybe a problem of php, some people seem to have same problem:

dpatou added a comment.EditedFeb 20 2016, 8:21 AM

@epriestley Reverting both 5e375482 and 8269fd6e make me able to clone normally again.

Can I do anything such as providing detailled server conf (what's needed?) to allow this being fixed in upstream ?

Edit: I guess this is pretty obvious but my instance only exposes http for repos (as opposed to ssh)

epriestley reopened this task as Open.Feb 20 2016, 12:32 PM

@dpatou Do you have errors in your error log similar to the ones @ppggff describes?

dpatou added a comment.Feb 20 2016, 3:29 PM

@epriestley yes, pretty much the same ones :

[Sat Feb 20 09:04:56.337247 2016] [:error] [pid 5801] [client 86.219.113.93:56185] PHP Warning:  file_get_contents(): cannot represent a stream of type Input as a File Descriptor in /home/w4w/domains/REDACTED/phabricator/support/PhabricatorStartup.php on line 144
[Sat Feb 20 09:04:56.350142 2016] [:error] [pid 5801] [client 86.219.113.93:56185] PHP Warning:  file_get_contents(compress.zlib://php://input): failed to open stream: operation failed in /home/w4w/domains/REDACTED/phabricator/support/PhabricatorStartup.php on line 144

Ubuntu 14.04
Apache : 2.4.16
Php : 5.6.12-1
Php being through mod_php

epriestley added a comment.Feb 20 2016, 3:34 PM

Alright, thanks. Both you and @ppggff have PHP 5.6, I'll see if I can reproduce with 5.6.

My initial guess is that it might be related to this change in 5.6:

php://input may now be reopened and read as many times as required.

dpatou added a comment.EditedFeb 20 2016, 4:23 PM

This comment seem to indicate 5.6 indeed broke file_get_contents("compress.zlib://php://input"). I didn't delve much into the proposed workaround yet but it might give a hint of what's going on.

epriestley added a comment.Feb 20 2016, 4:52 PM

Yeah, I'd like to avoid that "simple workaround" of 200 lines of spaghetti if I can, since I'd guess this task is going to go through 50 more rounds if I have to implement that.

avivey removed a subscriber: avivey.Feb 20 2016, 6:24 PM
epriestley added a comment.EditedFeb 20 2016, 7:24 PM
Edit: don't bother with this.

If you have a chance, can anyone with the "compress.zlib" issue on PHP 5.6 let me know if this patch fixes it (when applied against a clean stable or master):

diff --git a/support/PhabricatorStartup.php b/support/PhabricatorStartup.php
index bc7daf4..67a9742 100644
--- a/support/PhabricatorStartup.php
+++ b/support/PhabricatorStartup.php
@@ -141,7 +141,14 @@ final class PhabricatorStartup {
       $source_stream = 'php://input';
     }
 
-    self::$rawInput = (string)file_get_contents($source_stream);
+    $input_stream = fopen($source_stream, 'rb');
+    $input_data = '';
+    while (!feof($input_stream)) {
+      $input_data .= (string)fread($input_stream, 16 * 1024);
+    }
+    fclose($input_stream);
+
+    self::$rawInput = $input_data;
   }

It doesn't seem to break other versions, but I haven't repro'd the 5.6 issue locally yet.

epriestley added a comment.Feb 20 2016, 7:44 PM

Never mind, that patch doesn't work.

epriestley added a revision: D15314: Fix HTTP body decompression in PHP 5.6.Feb 20 2016, 7:55 PM
epriestley added a commit: rP42566379dc3c: Fix HTTP body decompression in PHP 5.6.Feb 20 2016, 10:55 PM
epriestley added a comment.Feb 20 2016, 10:58 PM

I think this is now resolved for PHP 5.6 at HEAD of master.

dpatou added a comment.Feb 21 2016, 9:15 PM

I can confirm it now works properly for me (5.6) at HEAD.

<3 for fixin so fast.

epriestley closed this task as Resolved.Feb 21 2016, 9:16 PM
epriestley claimed this task.

Thanks for verifying the fix!

I'm hopeful that this one is really, seriously resolved now, but let us know if anyone is still seeing issues.

ppggff added a comment.Feb 22 2016, 3:29 AM

Still hang...
dumping decompressed result shows input was truncated.

change the window value to 31 or 47 fix it.
according to zlib's manual:

windowBits can also be greater than 15 for optional gzip decoding. Add 32 to windowBits to enable zlib and gzip decoding with automatic header detection, or add 16 to decode only the gzip format (the zlib format will return a Z_DATA_ERROR).

epriestley added a comment.Feb 22 2016, 3:30 AM

Does changing window to 31 actually fix it for you?

ppggff added a comment.Feb 22 2016, 3:32 AM

yes

In T10264#160688, @epriestley wrote:

Does changing window to 31 actually fix it for you?

ppggff added a comment.Feb 22 2016, 3:39 AM

test:

php test.php 31 && tail test.txt
php test.php 30 && tail test.txt
ppggff added a comment.Feb 23 2016, 4:59 PM

@epriestley, test files in last comment indicate 31 can decompress gzip properly.
please check it.

ps: where does the 30 come from?

epriestley mentioned this in T7789: Support Git Large File Storage.Mar 15 2016, 12:29 PM
epriestley added a revision: D15483: Move request decoding/streaming to AphrontRequestStream.Mar 16 2016, 5:51 PM
epriestley added a commit: rPHUa4ca530644be: Move request decoding/streaming to AphrontRequestStream.Mar 17 2016, 3:07 PM
epriestley added a comment.Mar 17 2016, 3:34 PM

D15483 changes the constant to 47 (15 window size + 32 for gzip + zlib), which should accept and decode the broadest possible range of inputs. If this somehow breaks something, 31 (15 window size + 16 for gzip only) is probably the next thing to try. See that change for discussion.

In particular, the test.php script above works for me when executed as test.php 47.

Phabricator · Built by Phacility