In Mercurial 3.4/3.5 a new wire-protocol bundle2 was introduced. In 3.4 it was turned on by default only for servers, in 3.5 it was turned on for clients as well.

It's possible that something may need updated in Phabricator to support the new wire protocol, but this could be an issue with just Mercurial and not directly related to Phabricator.

What version of Mercurial are you using for client vs. server?

Im using 3.5.1 on both the server and the client. It only happens if I push to the mercurial repos via Phabricator, directly works fine. So maybe Phabricator is not passing through some information needed by Mercurial?

Ah that's good information to know. My guess is that Phabricator needs an updated version of the wire protocol to account for bundle2.

For reference, the information I could find about the new bundle wire format:
https://mercurial.selenic.com/wiki/BundleFormat2
Which also essentially says to read the code (this should be at the 3.5 stable release):
https://selenic.com/hg/file/21aa1c313b05/mercurial/bundle2.py

I setup a test environment with mercurial 3.5.1 on both client and server and did not see this issue. Provide some further detail about your setup:

1. Are you using SSH or HTTP(S) for push/pull?
2. Are you using any extensions such as largefiles?
3. Does the commit you're pushing contain binary file changes, or perhaps something else aside from simple text file modifications?
4. Could the server have multiple versions of mercurial, and might not be using 3.5.1?
5. Could you post your local machine repository of .hg/hgrc, possibly even your account's ~/.hgrc or ~/Mercurial.ini?

In T9450#137728, @cspeckmim wrote:

I setup a test environment with mercurial 3.5.1 on both client and server and did not see this issue. Provide some further detail about your setup:

That's weird, I'll investigate it further.

1. Are you using SSH or HTTP(S) for push/pull?

HTTPS

2. Are you using any extensions such as largefiles?

No, I've removed all extensions from my .hgrc. So it's a plain install.

3. Does the commit you're pushing contain binary file changes, or perhaps something else aside from simple text file modifications?

It happens with simple text modifications, I didn't test with binaries.

4. Could the server have multiple versions of mercurial, and might not be using 3.5.1?

I'm certain there is just a single install of mercurial on the server, I've double checked the 'hg' path configured in Phabricator too.

5. Could you post your local machine repository of .hg/hgrc, possibly even your account's ~/.hgrc or ~/Mercurial.ini?

I've emptied my .hgrc files but the same problem occurs.

Also I've tried to downgrade hg back to 3.4.2 and the problem doesn't occur anymore. I haven't seen any trace of bundle2 in the debug log using that version. So it's pretty clear the new bundle2 wire-protocol causes this issue.

I can reproduce this issue. It does not occur with SSH but does with HTTP.

According to Mercurial they released bundle2 with 3.4 but it was only enabled by default for "servers", whereas in 3.5 they enabled it by default for clients as well -- so with a 3.4 client it should fall back to old-style bundle format and not cause issues.

I've tried a few things but I haven't successfully interpreted Mercurial's documentation. I think ultimately the issue is here:
https://secure.phabricator.com/diffusion/P/browse/master/src/applications/diffusion/controller/DiffusionServeController.php;8ded0927aa7f1912d8c3b744b2f5d743b052c98f$536

I think we have to package up the $stderr into a proper bundle2 bundle. According to the documentation in the bundle2.py code I believe the return should be something like:

I've played a bit and have been able to get different errors to occur client-side~, however I think this approach would just be a tide-over to prevent the exception from appearing. I suspect there might be significantly more work involved in adding full support for bundle2.

Thanks, I'll also try to play with it the coming week. What's the best way to debug PHP? Is there a way to get a CLI debugger at that point of the code or should I just try to print vars to stdout or something? (not much PHP experience).

I don't have much PHP experience either. I installed Phabricator on a VM, then for making these changes I just nano the file, restart the services, and test it out.

I was hoping to get a vagrant of it working as I've read that it automatically shares the host's base folder with the guest system, which would mean faster iterations and being able to modify the files locally in my own editor.

For getting some output I've thrown exceptions with the data I want to see and check in error log, or occasionally I find a $console variable in scope that I can spew at.

For HTTP stuff which doesn't return a response (like this), the best debugging tool is sometimes using phlog(), then tail -f on the apache/php-fpm error log in another window.

Thanks!

I've installed Phabricator on a freshly installed server and switched to Mercurial over SSH. Now I'm getting the same problem, but it's kinda random somehow. Sometimes I don't get the error, but most of the times I do:

abort: not a Mercurial bundle
remote: abort: stream ended unexpectedly (got 0 bytes, expected 4)

In addition to this I now also get this error sometimes:

[255] » hg push --debug                                                                                                                      
no terminfo entry for sitm
no terminfo entry for dim
pushing to ssh://phd-vcs@somedomain.com:8022/diffusion/TEST/test/
running ssh  phd-vcs@somedomain.com -p 8022 'hg -R diffusion/TEST/test/ serve --stdio'
...
remote:   File "/usr/local/lib/python2.7/dist-packages/mercurial/sshserver.py", line 94, in serve_forever
remote:     while self.serve_one():
remote:   File "/usr/local/lib/python2.7/dist-packages/mercurial/sshserver.py", line 112, in serve_one
remote:     rsp = wireproto.dispatch(self.repo, self, cmd)
remote:   File "/usr/local/lib/python2.7/dist-packages/mercurial/wireproto.py", line 515, in dispatch
remote:     args = proto.getargs(spec)
remote:   File "/usr/local/lib/python2.7/dist-packages/mercurial/sshserver.py", line 32, in getargs
remote:     arg, l = argline.split()
remote: ValueError: need more than 1 value to unpack

It might be related.
I'm still investigating.

This is my latest debugging result:

sending unbundle command
remote: [2015-09-28 14:45:32] PHLOG: null at [/opt/phabricator/src/applications/diffusion/ssh/DiffusionMercurialWireClientSSHProtocolChannel.php:124]
remote: [2015-09-28 14:45:32] PHLOG: 'heads 10' at [/opt/phabricator/src/applications/diffusion/ssh/DiffusionMercurialWireClientSSHProtocolChannel.php:124]
remote: [2015-09-28 14:45:32] PHLOG: 10 at [/set/phabricator/src/applications/diffusion/ssh/DiffusionMercurialWireClientSSHProtocolChannel.php:131]
remote: [2015-09-28 14:45:32] PHLOG: '666f726365' at [/opt/phabricator/src/applications/diffusion/ssh/DiffusionMercurialWireClientSSHProtocolChannel.php:149]
bundle2-output-bundle: "HG20", 4 parts total
bundle2-output-part: "replycaps" 155 bytes payload
bundle2-output-part: "check:heads" streamed payload
bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/mercurial/dispatch.py", line 163, in _runcatch
    return _dispatch(req)
  File "/usr/local/lib/python2.7/site-packages/mercurial/dispatch.py", line 895, in _dispatch
    cmdpats, cmdoptions)
  File "/usr/local/lib/python2.7/site-packages/mercurial/dispatch.py", line 656, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/local/lib/python2.7/site-packages/mercurial/dispatch.py", line 1013, in _runcommand
    return checkargs()
  File "/usr/local/lib/python2.7/site-packages/mercurial/dispatch.py", line 982, in checkargs
    return cmdfunc()
  File "/usr/local/lib/python2.7/site-packages/mercurial/dispatch.py", line 892, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/local/lib/python2.7/site-packages/mercurial/util.py", line 792, in check
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/mercurial/commands.py", line 5286, in push
    bookmarks=opts.get('bookmark', ()))
  File "/usr/local/lib/python2.7/site-packages/mercurial/exchange.py", line 240, in push
    _pushbundle2(pushop)
  File "/usr/local/lib/python2.7/site-packages/mercurial/exchange.py", line 650, in _pushbundle2
    reply = pushop.remote.unbundle(stream, ['force'], 'push')
  File "/usr/local/lib/python2.7/site-packages/mercurial/wireproto.py", line 411, in unbundle
    ret = bundle2.getunbundler(self.ui, stream)
  File "/usr/local/lib/python2.7/site-packages/mercurial/bundle2.py", line 606, in getunbundler
    raise util.Abort(_('not a Mercurial bundle'))
Abort: not a Mercurial bundle
abort: not a Mercurial bundle
remote: abort: stream ended unexpectedly (got 0 bytes, expected 4)
remote: abort: unexpected error 'pushkey'.

I've tried a few things, but I'm unable to fix it unless I take a deep dive into the inner workings of both Phabricator and Mercurial. Maybe this is finally a reason to switch Git anyway...

I guess it's best to add a "Mercurial 3.5>= unsupported" statement somewhere in the docs.

You said it doesn't happen every time with SSH? I had setup SSH on my 3.5.1 <-> 3.5.1 setup and it seemed to push without issues, but I think I only tried once or twice.

Right, sometimes it works perfectly. I just don't get it, it seems kinda random. Maybe I should try once more on another new VM. Please note it only happens when there were commits to push. Pushing without changes always works.

I have a proposed change to resolve the issue in the short-term. The idea is to remove "bundle2" from the list of capabilities which the server responds with. This forces the client/server to negotiate using a legacy wire format. I think this solution is an elegant intermediary solution as it doesn't require determining version of client/server mercurial, nor having to package up bundles in a format that is not yet fully known/documented (aside from source code...).

Going forward however, I suspect there may need to be significantly more effort put into packaging up bundles being sent back and forth. I initially looked at being able to package up a bundle2 to send down, however I was under the impression that it would only be "half an implementation" needed for bundle2. I found it rather difficult even doing this, as there is no clear documentation about what headers are required (parameters appear totally optional and apparently none exist in core Mercurial as yet) -- but it looks like there's at least 1 header required. Discovering this may involve working directly with the mercurial developer community which I'm not yet ready to invest in. I think I was also confusing the command server format and the bundle2 format.

One thing to note, even though the changesets are pushed up while the client sees an exception, the client fails to update the internal phases of commits pushed up (leaving them in 'draft' phase instead of 'public'). This has high potential to mess with users' typical workflows in Mercurial, which I think is a consideration for fixing something in the short-term.

I haven't done any testing with hg pul so do not yet know what if anything fails in that situation.

Thanks, I guess that's the best fix in the short-term. I've also tried implementing the bundle2 protocol, but no luck. Without any documentation it's no fun.

I can get this working over HTTP by just removing the special casing around the "unbundle" command. However, doing this prevents hg from emitting errors from commit hooks.

I can't immediately figure out how we're suppose to send stderr over the wire, or even if Mercurial is capable of this. For example:

except util.Abort:
    # The old code we moved used sys.stderr directly.
    # We did not change it to minimise code change.
    # This need to be moved to something proper.
    # Feel free to do it.
    sys.stderr.write("abort: %s\n" % exc)
    return pushres(0)

We do get some hint that it's a commit hook issue, at least, so maybe this is an improvement:

$ echo spaghetti >> README ; hg commit -m tmp ; hg push
pushing to http://admin:***@local.phacility.com/diffusion/HGTEST/mercurial/
searching for changes
abort: pretxnchangegroup.phabricator hook exited with status 1

I'm not having much success getting SSH to work at all.

The Mercurial wire protocol suffers badly from being a cobbled-together pile of sub-protocols with a grammar which is not application-independent: you have to know about every command in order to parse the protocol. bundle2 appears to add a whole new sub-grammar to the wire protocol. I think working with the Mercurial wire protocol is pretty easily the least pleasant thing about Phabricator. I am dismayed that they have made it even more complex.

This is literally part of the new sub-protocol design:

Name MUST start with a letter. If this first letter is lower case, the
parameter is advisory and can be safely ignored. However when the first
letter is capital, the parameter is mandatory and the bundling process MUST
stop if he is not able to proceed it.

As far as I can tell, this was in no way forced by legacy badness.

I suspect we may need to parse the bundle2 parts, although I'm not really sure yet.

As far as I can tell, this was in no way forced by legacy badness.

I think they do this with the command server protocol as well -- channels with lower-case are optional but upper-case are mandatory. So at most it may just be a pattern that they enjoy for whatever reason >:|
https://www.mercurial-scm.org/wiki/CommandServer#Channels

I suspect we may need to parse the bundle2 parts, although I'm not really sure yet.

Is it worthwhile to pursue the changes I've been looking at, which decline the bundle2 functionality altogether? I was going to poke at updating the SSH portion. My thoughts were that it might be a quick win which shouldn't cause much (or any?) problems since clients should fallback to doing push/pull over bundle1. The only case where it might cause issue is with sites that are expecting bundle2 functionality with their 3.4+ install and finding that it doesn't.

I believe one of the purposes of the upgrade to bundle2 was to cover being "more-transactional" -- the only example I've seen of this is spamming Ctrl+C while pushing to try and get past server-side pretxn hooks (shh!). Supposedly bundle2 fixes this.

The same problem occurred on my site with the latest Phabricator and the latest Mercurial 3.7.2 installed on both sides.

Interestingly, it can be cured setting a special development debug flag for in the Mercurial configuration file:
in /etc/mercurial/hgrc or in vcs-user's ${HOME}/.hgrc adding the following snippet

[devel]
bundle2.debug = true

caused a hg push to a hosted Phabricator repository to work without a problem. But do not ask me, why.

In T9450#163090, @WTreutterer wrote:

The same problem occurred on my site with the latest Phabricator and the latest Mercurial 3.7.2 installed on both sides.

Interestingly, it can be cured setting a special development debug flag for in the Mercurial configuration file:
in /etc/mercurial/hgrc or in vcs-user's ${HOME}/.hgrc adding the following snippet

[devel]
bundle2.debug = true

caused a hg push to a hosted Phabricator repository to work without a problem. But do not ask me, why.

"Me too" - although I find that it is sporadic without and does some times work.

Thanks for the work around :)

Actually turns out that the work around isn't reliable - I still get that error occasionally (I'm using SSH)