Page MenuHomePhabricator
Create Task
Maniphest T10228

Switching imported repository to hosted, then cloning over HTTP in Phacility production fails
Closed, ResolvedPublic
Actions

Description

A user reports an issue with:

  • importing a repository from GitHub on a Phacility instance;
  • switching it from imported to hosted;
  • cloning over HTTP.

I can reproduce this in production, but not in development:

git clone https://test-ji2ubpx2swcl.phacility.com/diffusion/POEMS/poems.git poemhttp1
Cloning into 'poemhttp1'...
error: RPC failed; result=22, HTTP code = 500
fatal: The remote end hung up unexpectedly

Neither the web or repo nodes currently log anything relevant.

Our logging around clone issues is generally rather deficient so I'm going to start by improving that.

Revisions and Commits

rP Phabricator
D15145rP71bda66870d8 (stable) Decode "Content-Encoding: gzip" content
D15144rP482da7f5dfb3 (stable) Apply phutil_utf8ize() to stderr output from VCS commands prior to…
D15145rP8269fd6e6c1f Decode "Content-Encoding: gzip" content
D15144rPe5947e08d389 Apply phutil_utf8ize() to stderr output from VCS commands prior to logging
D15127rP30473549acfb Add a basic pull event log for debugging repository cloning

Event Timeline

epriestley created this task.Jan 27 2016, 5:36 PM
Herald added a subscriber: chad. · View Herald TranscriptJan 27 2016, 5:36 PM
joshuaspence added a subscriber: joshuaspence.Jan 28 2016, 9:49 AM
epriestley added a revision: D15127: Add a basic pull event log for debugging repository cloning.Jan 28 2016, 2:40 PM
epriestley added a commit: rP30473549acfb: Add a basic pull event log for debugging repository cloning.Jan 28 2016, 4:18 PM
epriestley added a comment.Jan 30 2016, 11:49 PM

Logging for this is now in production, but the initial data is totally useless:

mysql> select * from repository_pullevent;
+----+--------------------------------+--------------------------------+------------+--------------------------------+---------------+----------------+------------+------------+------------+
| id | phid                           | repositoryPHID                 | epoch      | pullerPHID                     | remoteAddress | remoteProtocol | resultType | resultCode | properties |
+----+--------------------------------+--------------------------------+------------+--------------------------------+---------------+----------------+------------+------------+------------+
|  1 | PHID-PULE-to3l26kkvy6grc3eym6d | PHID-REPO-xupsohlap33awie3agus | 1454197482 | PHID-USER-votbrdt4fwqfwyn3q7c2 |    XXXXXXXXXX | http           | wild       |        200 | null       |
+----+--------------------------------+--------------------------------+------------+--------------------------------+---------------+----------------+------------+------------+------------+
1 row in set (0.00 sec)
epriestley added a comment.Jan 31 2016, 12:01 AM

Slightly more useful:

{"response.message":"Error 1: fatal: protocol error: bad line length character: \u001f\ufffd\b\n"}
epriestley added a revision: D15144: Apply phutil_utf8ize() to stderr output from VCS commands prior to logging.Jan 31 2016, 12:04 AM
epriestley added a comment.Jan 31 2016, 12:42 AM

Alright, I made some progress on this.

In production, the failing request is being sent gzipped with Content-Encoding: gzip. I'm not sure why git chooses to gzip in production and not in development. I can see no obvious difference between the server responses. The trigger may actually be the request exceeding an arbitrary 1KB threshold:

remote-curl.c
...
	} else if (use_gzip && 1024 < rpc->len) {
...

When we receive a gzipped response, we aren't decoding it correctly.

epriestley added a revision: D15145: Decode "Content-Encoding: gzip" content.Jan 31 2016, 12:46 AM
epriestley added a commit: rPe5947e08d389: Apply phutil_utf8ize() to stderr output from VCS commands prior to logging.Jan 31 2016, 12:46 AM
epriestley closed this task as Resolved by committing rP8269fd6e6c1f: Decode "Content-Encoding: gzip" content.Jan 31 2016, 12:48 AM
epriestley added a commit: rP8269fd6e6c1f: Decode "Content-Encoding: gzip" content.
epriestley added a commit: rP482da7f5dfb3: (stable) Apply phutil_utf8ize() to stderr output from VCS commands prior to….
epriestley added a commit: rP71bda66870d8: (stable) Decode "Content-Encoding: gzip" content.
epriestley added a comment.Jan 31 2016, 12:52 AM

I merged these to stable and upgraded web001 to fix the immediate issue. The utf8 fix technically needs to go to the repo hosts, but we don't need it there immediately since it only aided in identifying and debugging the gzip issue.

Phabricator · Built by Phacility