Page MenuHomePhabricator

arc land hangs if git pre-push hook creates child daemon processes
Closed, InvalidPublic

Description

I spent a lot of time debugging this but have yet to determine exactly what the underlying issue is..

Basically, we have a fairly complex git pre-push hook, that attempts to ensure that all our unit tests have run prior to continuing with the push. One of the pre-requisites for getting through the unit tests successfully is a clean build, which involves running a cleanup script we have which kill off various local daemons and restarts them.

What I've found is that this all works when using 'git push origin' directly, but when attempting 'arc land', everything hangs at the end of the arc command. Arc appears to be waiting for 'ssh <our fabricator instance> git-receive-pack' to terminate, but all operations have finished.

Meanwhile, if I kill the ssh git-receive-pack instance explicitly, arc wakes up and finishes.

This is all on OS X Yosemite 10.10.4. arc version reports:

arcanist 18a01dcf1fbd65e97a281e260cea422f85b99227 (8 Aug 2015)
libphutil f0d26839031c93c8bff5f208e1bedf0c3be40baa (31 Jul 2015)

If I run dtruss -p on the ssh process it seems to have been terminated already, and I believe the problem is perhaps that arc is confused about which child PIDs it should be waiting for, but I'm not sure. Moreover, it only happens when these new long-lived daemonized processes are created as part of the git pre-push hook.

Event Timeline

ramesh_dharan raised the priority of this task from to Needs Triage.
ramesh_dharan updated the task description. (Show Details)
ramesh_dharan added a project: Arcanist.
ramesh_dharan added a subscriber: ramesh_dharan.

Can you simplify your hooks into a minimal test case which reproduces this issue? I don't have access to them, and can't really guess where to begin. I tried this:

.git/hooks/pre-commit
#!/bin/sh

(sleep 60 &) &

This completes normally and leaves a backgrounded sleep process running after arc land exits.

If I put it in pre-push instead, I can get a more hang-like behavior, but the hang also reproduces with normal git push:

.git/hooks/pre-push
#!/bin/sh

(sleep 10 &) &

Produces:

$ git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 357 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To ssh://local@localvault.phacility.com/diffusion/TESTY/git-test.git
...

Then, a 10 second hang, then:

...
 7eb056e..26e4d1b  master -> master

Worth adding:

We've forked arcanist. The commit hash Ramesh linked to is on our fork; it's based off of rARC7d15b85.

@ramesh_dharan, when reproing, can you please use arc land-direct (which corresponds to upstream's arc land) rather than our arc land (which is heavily modified)?

So we have several variables at play: the two-month old arcanist, our custom fork of arcanist, and the hooks. I wouldn't be surprised if a combination of these variables is causing the problem; we should try to isolate the cause.

ramesh_dharan claimed this task.

Thanks @jhurwitz; actually, I run typically run my own installation of stock/upstream arc and libphutil that I have set up directly from GitHub, but I had downgraded to the 'arc' provisioned on our machines to see if that affected anything.

@epriestley you're right that it does seem to happen with stock git pre-push as well; not sure why I wasn't seeing it before but I definitely see it now with git 2.5.0 on my machine.

I'm comfortable with closing this out since it seems to be a git issue, not an arc issue.