All evidence points to this and T9548 being the same task, right? I'm just going to merge them.

I believe there's more to it than just the bundle2 setting that causes it to not work. It still breaks for me even with devel.legacy.exchange=bundle1:

In T10382#191728, @durin42 wrote:

For now, I'd suggest that if phabricator breaks bundle2, filter the response to the capabilities command and omit bundle2=.* from the response.

To be clear my suggestion was only for using 3.4 from client machines. Server machines can still be kept up to date.

Note that my suggestion in https://secure.phabricator.com/T10382#191728 is still what I think /should/ happen: if Phabricator insists on trying to parse the Mercurial protocol, it should absolutely not include capabilities it doesn't understand when queried by the client.

Running vanilla Mercurial 3.4 should not be recommended as all vanilla versions older than 3.7.3 are vulnerable to a trio of CVEs, the most likely exploitable being CVE-2016-3630. Instead, Mercurial <4.0 can be configured to disable bundle2 via the experimental.bundle2-exp=false config option and Mercurial >=4.0 can do the same via devel.legacy.exchange=bundle1. Both of these options are undocumented. I discourage their use. But as a workaround for Phabricator, they are preferred to exposing clients to known security bugs.

Yeah, this is in stable, just didn't make the cut for individual mention in the changelog. I think it promoted in Week 52 and I just lumped it into "Differential got infrastructure changes".

Ah ok

I don't think we mention every commit in the changelog?

I may have missed it but it doesn't look like this was mentioned in any changelog in December/January - or is this not yet in stable?

@jeremy.norris - You should be able to push if you use a mercurial client 3.4 or below. What I did is download the hg-3.4 sources and do make local, then moved the renamed binary into my path (/usr/local/bin/ I think), and I use hg-3.4 push -r master when pushing.

So reading through all of this, am I understanding correctly that Phabricator and Mercurial > 3.4 aren't compatible with each other? I've spent the past couple days attempting to push an existing repository with a few thousand changesets into a Phabricator hosted instance to no avail (using mercurial 4.0.1 for both the client and server).

@gabe - That's the same issue, I believe. You should be able to push using Mercurial 3.4. I've been meaning to do some testing to try and fix this over the winter break but a day is no longer what it used to be.

Not sure if this is the same issue, this is on a new phacility hosted phabricator instance, when attempting to push to a new mercurial repo:

This field is only populated when you update a revision, and always contains the first line of the update message, which is also posted as a comment.

As a user of both Phabricator and Mercurial, opening up HTTP isn't an option we want to consider and would instead stick to having the repository hosted elsewhere.

Today, we already decode the SSH wire protocol and figure out what's going on inside it up to the bundle2 stuff -- here's the, uh, "protocol spec":

It should be possible to determine whether an operation is read or write by the command name: a command either performs writes or it doesn't. The only write commands you should see are unbundle and pushkey. While it might be possible for batch to contain pushkey operations, I don't think that's done today (but I'd have to audit the code to be sure). One thing you should be able to rely on is the HTTP method: GET requests should never perform any significant writes (although some cache files may be written to as a result of resolving data, but that shouldn't be important). It should be perfectly valid to treat GET and POST differently. Although if you route requests to different endpoints within the same TCP/HTTP stream, inconsistent state of a read-only mirror from a master could lead to race conditions (e.g. push fails because a mirror is out of sync). If you route by HTTP request method, you should not need to know anything about the Mercurial wire protocol.

@indygreg - Thanks for reaching out and discussing a way forward. I am not on the phabricator team but I do have an interest in its integration capabilities with mercurial. At best I might be able to summarize my understanding of some of phabricator's needs based on the history of this task, but I do not feel I could properly represent them on mercurial's mailing-list or wiki. For hosting repositories I believe this task would require needing to know if the mercurial operation is read or write -- the two primary reasons being read/write access control and cluster picking.

While I'm hesitant to throw stones here and don't really have any context on bundle2 or the wire protocol in general and there may be plenty of reasons that it works like it does, my general impression is that the Mercurial wire protocol is significantly under-designed, and is now on at least the 3rd and possibly 4th iteration of hacks to try to address the lack of actual protocol design.

In the read/write case, it's too late once we hit a hook -- we need to be able to decide which cluster node to send the traffic to, and we need to know if it's a read or write request before we make that decision, since there may be a nearby read-only node and a distant read/write node. If we can examine the stream and make a determination that the request is read-only, we can freely pick a nearby read-only node.

It's not entirely phabricator specific, as others probably have similar use cases. I'm literally the only hg dev that knows this is an issue (someone flagged me on twitter a month ago, and it's only a strange inability to sleep tonight that got me deep enough in my stack taht I got to this), but if you go to our ML with a decent problem statement ("we need a hook to let us identify read vs write transactions and do our own ACL checking" sounds like a start?) you'll get more help than just me when I'm unable to sleep. :)

Aha

I'm not a phabricator developer so I'm not sure if supporting mercurial hooks is something they're willing to take upstream. I can join the hg mailing list, but wouldn't this conversation be specific to phabricator and beneficial to be carried out here?

Today, we technically do not need to decode the protocol, but we will need to be able to distinguish between read and write requests to implement cluster read-only nodes (T10883) in the future and need to be able to rewrite the protocol to keep parity with fork-like/virtual-ref features that we plan to support in Git (T10691, T8092).

Ah, okay. We can probably do a lot better with some sort of pre-transaction hook then (that was my gut feeling to begin with). Once you've got an idea what the use case is, maybe we can chat on the hg mailing list?

I am totally on-board with finding something more elegant/maintainable, and would be able to help with some testing and possibly development. I'll try to look over the code in the near future to recall what it was doing, but I believe it's trying to determine whether the mercurial operations are read/write in order to determine whether the actor has necessary permissions to do so. I *believe* that's what it's doing, but it's been close to a year since I was looking through this.

We've got a volunteer (slowly) working on documenting formats, but right now bundle2 is not documented extensively outside of the code and the wiki page already mentioned in this thread.

@durin42 - that would probably be better than parsing the bundle stream but, is that documented anywhere? I tried looking fairly extensively for details on the bundle2 protocol and never came across that. Also, the bundle stream itself isn't what's being parsed/modified, it's the capabilities/outer protocol which I found at least some documentation, which is what the parsing/filtering is based on.

I'm not sure why phabricator is trying to parse the bundle stream (I don't know anything about phabricator), but you should be able to invoke hg with bundle2 disabled by setting experimental.bundle2-advertise to false in an hgrc (or using --config if you're shelling out to hg). Given that it's an experimental flag, it's not something I can promise will exist forever, but it might be a start.

So, yes, the bundle2 thing is definitely orthogonal to this. For now, I'd suggest that if phabricator breaks bundle2, filter the response to the capabilities command and omit bundle2=.* from the response.

See T9548#139908 for some discussion.

Could someone create a Mercurial bug (https://bz.mercurial-scm.org) with explanation about how Phabricator handles the bundle2 ssh stream? They might be able to give some hints.
I would like to do it, but I'm not sure how Phabricator handles the stream exactly.

For reference, I use this small script to allow myself to easily use arc land with bookmarks:

@cspeckmim Sadly I don't have any Mercurial host that is not Phabricator.

@fcoelho oops sorry about the mixup. I've previously had success by downgrading the client to 3.4 or less.

@cspeckmim I think you probably confused me with @urzds, I didn't post any logs here yet.

@fcoelho - Would you be able to test your log change with pushing to a mercurial repository not hosted by Phabricator, to compare with the log from your previous comment?

I've been looking at this the last couple of days as it's a pain point for me -- I'd like to host my repositories on Diffusion, and I would like to stay with Mercurial rather than git..

Tried running 2.6.2 on the server side, and that indeed does fix the bundle2 issue, but creates more issues when trying to push from a 3.7.3 hg client.. :-(

If I host externally, I basically lose the access control and herald rules, right?

We host most of our Mercurial repositories externally - this is primarily the infrastructure we had prior to adopting Phabricator. We have a few repositories hosted by Diffusion, however our server runs 2.6.2 which does not utilize the bundle2 protocol.

Ah, you're right, thanks! I misread that page -- that does make Mercurial tags a lot easier to deal with.

In T1130#118470, @epriestley wrote:

They're a function of an individual commit instead of being a function of the repository as a whole, so we'd have to do a lot more work to support them (it's not meaningful to, say, "list all tags in the repository", as it is with Git), ...

The same problem occurred on my site with the latest Phabricator and the latest Mercurial 3.7.2 installed on both sides.

Reading the code around /usr/lib/python2.7/dist-packages/mercurial/sshserver.py:32:

def getargs(self, args):
    data = {}
    keys = args.split()
    for n in xrange(len(keys)):
        argline = self.fin.readline()[:-1]
        arg, l = argline.split()
        if arg not in keys:
            raise util.Abort("unexpected parameter %r" % arg)
        if arg == '*':
            star = {}
            for k in xrange(int(l)):
                argline = self.fin.readline()[:-1]
                arg, l = argline.split()
                val = self.fin.read(int(l))
                star[arg] = val
            data['*'] = star
        else:
            val = self.fin.read(int(l))
            data[arg] = val
    return [data[k] for k in keys]

This implies a question: Could it be Phabricator which modifies the stream between the client and the server in an illegal way? Or is this some bug in Mercurial itself?

Reading the code around /usr/lib/python2.7/dist-packages/mercurial/sshserver.py:32:

def getargs(self, args):
    data = {}
    keys = args.split()
    for n in xrange(len(keys)):
        argline = self.fin.readline()[:-1]
        arg, l = argline.split()
        if arg not in keys:
            raise util.Abort("unexpected parameter %r" % arg)
        if arg == '*':
            star = {}
            for k in xrange(int(l)):
                argline = self.fin.readline()[:-1]
                arg, l = argline.split()
                val = self.fin.read(int(l))
                star[arg] = val
            data['*'] = star
        else:
            val = self.fin.read(int(l))
            data[arg] = val
    return [data[k] for k in keys]

Our projects make heavy use of mercurial tags too, for version tagging and it would be great to see them in Phabricator to easily compare files between versions (they're way more easy to remember than changeset hashes).

The issue is back, but this time it affects an existing repository, which I was using successfully for a long time. Please reopen.

@cspeckmim I just fixed that, unfortunately, it doesn't help. Here is a complete debug output when trying to push:

@andyboeh - I think there might be an issue with some of the fixes I was looking at. When removing the bundle2 capability from reply, the message preceding the capabilities output is a length of the following line, usually something like

242
capabilities: a, b, c, d, bundle2=http%%%%%...

By just removing bundle2 it causes the preceding number to be off, which is what the client uses for parsing the stream. I haven't gotten back to looking at this but I think that's the issue you're seeing.

I am on Mercurial 3.6 on both, the server and the client, and just tried what @cspeckmim suggested: Removing bundle2 from the capabilities in reply to the hello command. Now my client gets one step further, but still fails to push most of the time with remote: abort: stream ended unexpectedly (got 0 bytes, expected 4)

@cspeckmim - you're right. But for now I'm not going to create Maniphest Task because I'm a novice in Phabricator. Actually I don't think, but maybe I did wrong something. Before writing this question I've seen T9450. When I become a more experienced user, I'll use Maniphest tasks. Thanks for your advice and sharing your experiences.

Yeah, the Q&A section isn't monitored for these kinds of things, it's more for support.
However, since this is hard to reproduce (See Contributing Bug Reports), it would be very hard to get any resources to solve.

@efkan - you should probably make a Maniphest Task for errors you run into rather than a Ponder Question - errors will likely be seen and fixed sooner than questions. For example I am not an administrator and I don't believe @avivey is either (but a very active user/contributor).

This issue had been created for to create awareness to administrators and users.

It's late...

The mercurial server is sending down capabilities which includes bundle2 at the beginning of the remote SSH during push (which tells the client it can use new bundle2). It's not sent as part of a command which the client invokes which is what T9450 is handling. Instead when the mercurial command server initializes it immediately sends over a list of capabilities.

I think I may see what the issue is. I believe the 'hello' command returns its list of capabilities, which is not filtered by T9450. I'm trying to test out some fixes but having difficulty with nano~

I cloned Phabricator on 2016-01-16 by using that. Because of I use Windows OS, I cannot run arc upgrade.

I use ubuntu 14.04.2 as my server and its hg version is 3.3.2.
I had upgraded when I couldn't push to the server.

I just tried pushing a commit and this worked fine. I tried this with 3.6.3 on the client and tried pushing commits with these different versions on the server:

• 2.6.2
• 3.4.2
• 3.5.1
• 3.6.3

In every case the push when through without issues. I suspect @efkan may have an outdated install/configuration.

What's the hg version on the server?

Re: LargeFiles Patch

@michaeljs1990. You seem to have another issue.