Page MenuHomePhabricator

Cloning repository keeps entire repository in memory as it's being streamed over SSH
Closed, ResolvedPublic

Description

We are currently in the process of cloning a 400MB repository over SSH hosted in Phabricator. Currently there are about 15 people cloning the repository and we are noticing when running the top command that there are PHP processes with the entire repository held in memory:

11193 git       20   0  891m 483m  360 R 33.3  6.3   1:16.99 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user dulanjanl
11009 git       20   0  932m 648m  128 R 26.5  8.4   1:33.07 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user terence.tham
11862 git       20   0  938m 471m  424 R 25.6  6.1   0:17.16 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user chrism
11941 git       20   0 1278m 892m  416 R 24.7 11.6   0:09.62 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user hannahw
11516 git       20   0  932m 452m  368 R 24.4  5.9   0:38.10 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user leviq
11432 git       20   0  476m 313m  368 R 23.8  4.1   0:50.77 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user marcusw
10632 git       20   0 1245m 1.0g   48 R 22.0 13.9   2:00.44 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user lucasr
10155 git       20   0 1235m 790m   52 R 20.5 10.3   2:35.66 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user anitah
10759 git       20   0  883m 624m  128 R 19.6  8.1   1:52.08 php /srv/phabricator/phabricator/bin/ssh-exec --phabricator-ssh-user scottb
  788 git       20   0  168m  23m   36 S  4.5  0.3   4:15.74 php-fpm: pool phabricator
  789 git       20   0  254m  36m   36 S  3.6  0.5   4:15.26 php-fpm: pool phabricator
 1906 git       20   0  254m  36m   28 S  3.0  0.5   4:11.79 php-fpm: pool phabricator

Event Timeline

waynea assigned this task to epriestley.
waynea raised the priority of this task from to Needs Triage.
waynea updated the task description. (Show Details)
waynea updated the task description. (Show Details)
waynea added a subscriber: waynea.
epriestley triaged this task as Normal priority.Dec 16 2013, 3:09 PM

If you aren't at HEAD, there are some recent changes which should improve some aspects of this. Particularly, rPHU69490c53 works around a bug in PHP which could cause these processes to remain open even after the client disconnects.

However, the fundamental issue is that we don't throttle reads from git-upload-pack, so we can end up with the entire output held in memory. I'll add throttling so we maintain only a small buffer in the process itself.

epriestley edited this Maniphest Task.
epriestley edited this Maniphest Task.

Heads up on this, you should update at your convenience and kill any long-running / suspicious looking ssh-exec hanging around on your server.

(Oh, @hach-que too, since we talked about this briefly.)

Thanks for reporting this. I think it's fixed at HEAD, let us know if you still see it or run into anything else.