Rare hosted Git SSH error "fatal: The remote end hung up unexpectedly"
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	epriestley
	Dec 5 2013, 4:34 PM

Description

I've seen this once or twice, and at least one user is reporting running into it rarely:

http://pastebin.com/uJYV7iDY

This is somewhat similar to an issue I was able to reproduce quasi-reliably earlier on, in D7558.

D7558 suggests that Git is sensitive to the order in which we close pipes. My guess is that we need to close something which we aren't currently closing explicitly, and 95% of the time that's fine because it tears down before the connection tears down, but 5% of the time we lose a race and end up with this error.

Digging around the Git source code a bit, there's an envvar you can define to get packet-level information:

GIT_TRACE_PACKET=1 git pull

If this is a race, I'd expect git to receive all of the data whether it hits the fatal or not. That would tend to support the theory that this is an out-of-order-teardown issue.

Revisions and Commits

rPHU libphutil
	D7775	rPHUa5b32ff541df Allow PhutilChannel to select only write sockets, and expose read buffer status
rP Phabricator
	D7776	rPf7464400a510 Limit memory usage of `ssh-exec` during large pull operations

Event Timeline

epriestley created this task.Dec 5 2013, 4:34 PM

epriestley claimed this task.

epriestley raised the priority of this task from to Normal.

epriestley updated the task description. (Show Details)

epriestley added a project: Diffusion.

epriestley added a subscriber: epriestley.

@zeeg / @dctrwatson, let me know if you've run into this. I think I've seen it myself, but just at a slightly-higher-than-line noise level.

Haven't had this happen yet... and doing dozens (lol) of pulls on a couple repos a day

Web scale!

It's possible this was fixed by the fwrite() fix, in that I haven't seen it since pushing that. On the other hand, I haven't pushed that much stuff since then either. If anyone sees this after rPHU69490c5, let me know. I'll assume this is fixed if I don't see it for the next week or so.

Gosh darn it. I who'd actually gotten around to making a GIT_TRACE_PACKET = 1 push and managed to reproduce it. I've just now updated to the latest revision and will see if I get it again, though.
Here is the log, for posterity:

FAILED

$ git push
packet:         push< a7287f3d1c6f62ced99c4684451172e9ae4b0941 refs/heads/master\0 report-status delete-refs side-band-64k quiet ofs-delta
packet:         push< 0000
packet:         push> a7287f3d1c6f62ced99c4684451172e9ae4b0941 eb83e4ee7abb0e02aff32e4140eb12755a26096f refs/heads/master\0 report-status side-band-64k
packet:         push> 0000
Counting objects: 15, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 780 bytes | 0 bytes/s, done.
Total 8 (delta 4), reused 0 (delta 0)
packet:         push< \1000eunpack ok0019ok refs/heads/master0000
packet:         push< unpack ok
packet:         push< ok refs/heads/master
packet:         push< 0000
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer
To ssh://vcs@phabricator/diffusion/S/
   a7287f3..eb83e4e  master -> master
error: failed to push some refs to 'ssh://vcs@phabricator/diffusion/S/'

SUCCESSFUL

$ git push
packet:         push< eb83e4ee7abb0e02aff32e4140eb12755a26096f refs/heads/master\0 report-status delete-refs side-band-64k quiet ofs-delta
packet:         push< 0000
packet:         push> eb83e4ee7abb0e02aff32e4140eb12755a26096f fe4e5d47d7652980ffeaba118c4abeccceb94186 refs/heads/master\0 report-status side-band-64k
packet:         push> 0000
Counting objects: 15, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 792 bytes | 0 bytes/s, done.
Total 8 (delta 4), reused 0 (delta 0)
packet:         push< \1000eunpack ok0019ok refs/heads/master0000
packet:         push< unpack ok
packet:         push< ok refs/heads/master
packet:         push< 0000
packet:         push< 0000
To ssh://vcs@phabricator/diffusion/S/
   eb83e4e..fe4e5d4  master -> master

I'll let you know if I run into it again!

No, issue still remains and is rather prevalent. Around 25-50% in my environment.

Ah, interesting. Thanks!

I think I can probably come up with a repro case here, it will just take some fiddling.

epriestley edited this Maniphest Task.Dec 16 2013, 7:52 PM

epriestley edited this Maniphest Task.Dec 16 2013, 7:57 PM

D7775 / D7776 might fix this once they land.

epriestley edited this Maniphest Task.Dec 16 2013, 8:36 PM

epriestley edited this Maniphest Task.

During diligent testing (the empirically safe amount of 7 commits) in my environment I have not managed to successfully reproduce this since the latest commits. I leave it up to you to decide whether this constitutes enough evidence to close it or not!

Awesome! I'll leave this open for another day or two and close it the next time I remember it, if no one runs into new issues between now and then. Thanks for your help!

Haven't seen this anymore myself so I'm going to assume we fixed it. Thanks again! Let us know if anything else comes up.

Herald added a subscriber: eadler. · View Herald TranscriptSep 15 2017, 5:26 PM