Just writing this up for completeness: a user in the Phacility cluster reported issues performing clones over HTTP. I was able to reproduce this for proxied requests locally, and get useful diagnostic information with GIT_CURL_VERBOSE:

epriestley@orbital ~/dev/scratch $ rm -rf localtesttest && GIT_CURL_VERBOSE=1 git clone http://local.phacility.com/diffusion/TESTY/git-test.git localtesttest
Cloning into 'localtesttest'...

... <snip> ...

< X-Phabricator-Proxy: true
< Content-Type: application/x-git-upload-pack-result
< 
* Malformed encoding found in chunked-encoding
* Closing connection 0
error: RPC failed; result=56, HTTP code = 200
fatal: The remote end hung up unexpectedly

The hint here is * Malformed encoding found in chunked-encoding, which seemed to be because we would read and then re-emit the Transfer-Encoding: chunked header from the service host, but cURL would implicitly un-chunk the request body, so we'd end up emitting a Transfer-Encoding: chunked header and a non-chunked body. The fix is rP23367265e1, and I cherry-picked it to stable in rP70079f47db3d, then hotfixed it to web nodes in the cluster (see: https://twitter.com/phacilitystatus/status/685258767705882625).

Stripping the header if we found it before re-emitting the response appears to have resolved things (I get clean HTTP clones now locally and in the cluster), and I'm reasonably satisfied with this explanation from a technical point of view.

I'm a little unsure about what triggered the issue, since it definitely worked at some point but it isn't obvious what changed. However, it is likely rarely used (HTTP is off in the cluster by default, SSH is on and works properly without additional steps) and it might be something mundane like the size of the repository.