Phabricator hosted git repositories appear to error with 'failed to push some refs' without any error detail..
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Fordy
	Jul 31 2016, 7:47 AM

Description

Latest phabricator hosted git repositories appear to error with 'failed to push some refs' without any error detail, even though the push works?

To reproduce (on a new install):

Setup sshd wrapper to phabricator using provided script on port 2222, following diffusion hosting guide.
Create the new git repository in phabricator and activate - leave settings to default.
Clone this repository via phabricator URI

The clone works ok - however if I attempt to push, this errors - even though it is actually a success (if I commit any changes, these are pushed ok). Below example is when everything is in sync:

-bash-4.3$ git push origin master:master
error: failed to push some refs to 'ssh://vcs-user@phabricator:2222/diffusion/13'

Verbose version:

-bash-4.3$ git push -v origin master:master
Pushing to ssh://vcs-user@phabricator:2222/diffusion/13
To ssh://vcs-user@phabricator:2222/diffusion/13
 = [up to date]      master -> master
updating local tracking ref 'refs/remotes/origin/master'
error: failed to push some refs to 'ssh://vcs-user@phabricator:2222/diffusion/13'

I've tried this with a new git repository, latest version of phabricator. I can git clone the repository ok, but when I attempt to git push origin master, i get the error.

If I create a new file in the repo, the push works fine (not a fast-forward error) but then I still get the error:

-bash-4.3$ echo data-in-test-file > testfile3
-bash-4.3$ git add testfile3
-bash-4.3$ git commit -m "test file comment" testfile3
[master 9ac6b75] test file comment
 1 file changed, 1 insertion(+)
 create mode 100644 testfile3
-bash-4.3$ git push origin master:master
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 294 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://vcs-user@phabricator:2222/diffusion/13
 0f6f149..9ac6b75  master -> master
error: failed to push some refs to 'ssh://vcs-user@phabricator:2222/diffusion/13'

-bash-4.3$ git push origin master:master
error: failed to push some refs to 'ssh://vcs-user@phabricator:2222/diffusion/13'

I've scoured the internet and there are many answers to this particular error, but all assume someone else committed first and this is a fast-forward error. This is not true here, and even if it was - my pushes aren't being rejected by remote.

What gives, what is stuck and causing this error? The process appears to be working fine other than this error. (I can see repository updates in phabricator).

If I check this phabricator-created repository out in git locally (not via vcs-user and the phabricator wrapper) push works fine with no error.

I can't fully say its a 'git specific issue' because if I remove phabricator from the equasion it all works fine..

I've rm'd all hooks (both from checked out copy, and the copy hosted by phabricator) to exclude these as potentail causes.
I've checked persmissions and vcs-user can checkout and push the the repository locally without error. (avoiding phabricator wrapper).

Am I being dim or is something more wrong afoot?

Event Timeline

Fordy created this task.Jul 31 2016, 7:47 AM

I can't reproduce this here, locally, or in the Phacility cluster. Can you? You should be able to push to rGITTEST on this install, or launch a test instance from admin.phacility.com and create a repository there.

I have never seen this myself, nor seen any reports of it. Problems like this are usually specific to something unusual in the reporting user's environment.

We can't move forward until we have reproduction steps. See Providing Reproduction Steps.

As an alternative to providing reproduction steps, you can give us root access to a mutable test environment where this issue reproduces. This may be faster or easier than identifying the abnormality in your environment yourself.

If you can't give us steps to reproduce the problem and also can't give us access to an environment where the problem reproduces, we're happy to go back and forth asking questions and trying to get to the bottom of things at consulting rates (see Consulting).

Thanks for your help / advice Evan!

I thought I'd provided reproduction steps in the above, obviously not in the correct format :o)

I've managed to resolve this issue for now after much painstaking diagnostics - it definitely wasn't a git, or phabricator issue, but a weird OS issue with sudo, which I've still not quite got to the bottom of.

For those that are intrigued:

I freshly installed phabricator on a new machine elsewhere which didn't have the problem. Then managed to replicate the problem there by breaking random things. Specifically phabricator/bin/ssh-exec - I changed the exit line to be exit(3) and this caused the problem... So it was that likely exiting with an error code that was causing me trouble!

I couldn't find any phabricator error / diagnostic log where it was logging to - so I modified and put my own diags in temporarily.

I stored the error code and found it was 140 - then found this error with sudo:

-bash-4.3$ sudo -E -n -u phd /usr/bin/git-receive-pack
Bad System Call
-bash-4.3$ echo $?
140

Bad system call? Eugh! No idea why sudo does this on this system yet, however I've worked around by avoiding using sudo for now until I can resolve - by having vcs-user use unix perms to access the repositories.

phabricator/src/infrastructure/daemon/PhabricatorDaemon.php

Within sudoCommandAsDaemonUser()
+ $processUser = posix_getpwuid(posix_geteuid())['name'];
+
+ // sudo user is same as the current user, just run the command.
+ if ( $user == $processUser )
+ return $command;
+

Ta da! It now works and git doesn't look ill. Ta for your help. Best avenue is probably fixing sudo on this box!

Sounds like this is resolved? Or should we attempt to catch this case?

If you can identify why sudo was broken and what you did to fix it, we can probably add a setup warning to test for this problem. However, since we haven't seen other users with broken sudo and we couldn't help users resolve the issue with available information even if we did detect it, I don't think pursuing a check in the upstream is worthwhile.

If you figure this out, feel free to file a new task and we can look at what sort of tests/warnings are appropriate. Until then, I don't plan to pursue any more action in the upstream since it seems like this is a bizarre environmental haunted system sort of issue.

Phabricator hosted git repositories appear to error with 'failed to push some refs' without any error detail..Closed, ResolvedPublicActions

Description

Event Timeline

Phabricator hosted git repositories appear to error with 'failed to push some refs' without any error detail..
Closed, ResolvedPublic
Actions