(This isn't related to the "Search" application, even though it's about searching for things.)

This is likely just some of the secure hosts having different configuration. Ideally, we'd get the rCORE deployment stuff aligned to deploy them with it.

We are having the same issue with one of our Mercurial repositories. This particular repository has 206 branches: 14 open, 192 closed. When a new commit is pushed to this repository, the repository update task is spawning an hg log process for every branch, causing high CPU. This process is repeated every 15 seconds for several hours.

Having recently installed and configured Phabricator for our work environment, we need to host hg repos with largefiles in it, so we can't wait for this ticket to be completed. Feel free to ask if you need some testing, as we have no development capabilities to offer for this project.

We probably would not use an option to turn the feature off (versus just supporting it) so don't worry about adding it on our behalf. At the point where I was set up to convincingly test that turning the feature off worked -- and we had navigated bundle2 in some way -- I'd probably be most of the way toward proper support anyway.

Heads up: Mercurial 4.1 (to be released ~February 1) introduces media type negotiation in the HTTP protocol. https://www.mercurial-scm.org/repo/hg/rev/753b9d43ca81 captures the wire protocol changes and the commit message explains why things are done the way they are.

Ah ok I was just thinking that if anything happened to be depending on the state of the working copy then removing the -u argument could affect those.

The other side of the Conduit call still issues an hg (or git, or whatever) command in most cases, but sending everything through Conduit means a different host (one which actually does have a working copy) can give us the answer if necessary.

They shouldn't interact -- T10753 is mostly about clustering. When Diffusion and the daemons need to ask questions about a Mercurial repository, they're supposed to issue a Conduit call so Phabricator can get an answer by asking another host if it needs to. In a few cases, they currently just issue an hg command, which only works if the repository is present (and up to date) on local disk.

This might depend on T10753?

乁༼☯‿☯✿༽ㄏ

I'm just going to merge this into T9548 since I think bundle2 support will fix it?

All evidence points to this and T9548 being the same task, right? I'm just going to merge them.

I believe there's more to it than just the bundle2 setting that causes it to not work. It still breaks for me even with devel.legacy.exchange=bundle1:

In T10382#191728, @durin42 wrote:

For now, I'd suggest that if phabricator breaks bundle2, filter the response to the capabilities command and omit bundle2=.* from the response.

To be clear my suggestion was only for using 3.4 from client machines. Server machines can still be kept up to date.

Note that my suggestion in https://secure.phabricator.com/T10382#191728 is still what I think /should/ happen: if Phabricator insists on trying to parse the Mercurial protocol, it should absolutely not include capabilities it doesn't understand when queried by the client.

Running vanilla Mercurial 3.4 should not be recommended as all vanilla versions older than 3.7.3 are vulnerable to a trio of CVEs, the most likely exploitable being CVE-2016-3630. Instead, Mercurial <4.0 can be configured to disable bundle2 via the experimental.bundle2-exp=false config option and Mercurial >=4.0 can do the same via devel.legacy.exchange=bundle1. Both of these options are undocumented. I discourage their use. But as a workaround for Phabricator, they are preferred to exposing clients to known security bugs.

Yeah, this is in stable, just didn't make the cut for individual mention in the changelog. I think it promoted in Week 52 and I just lumped it into "Differential got infrastructure changes".

Ah ok

I don't think we mention every commit in the changelog?

I may have missed it but it doesn't look like this was mentioned in any changelog in December/January - or is this not yet in stable?

@jeremy.norris - You should be able to push if you use a mercurial client 3.4 or below. What I did is download the hg-3.4 sources and do make local, then moved the renamed binary into my path (/usr/local/bin/ I think), and I use hg-3.4 push -r master when pushing.

So reading through all of this, am I understanding correctly that Phabricator and Mercurial > 3.4 aren't compatible with each other? I've spent the past couple days attempting to push an existing repository with a few thousand changesets into a Phabricator hosted instance to no avail (using mercurial 4.0.1 for both the client and server).

@gabe - That's the same issue, I believe. You should be able to push using Mercurial 3.4. I've been meaning to do some testing to try and fix this over the winter break but a day is no longer what it used to be.

Not sure if this is the same issue, this is on a new phacility hosted phabricator instance, when attempting to push to a new mercurial repo:

This field is only populated when you update a revision, and always contains the first line of the update message, which is also posted as a comment.

As a user of both Phabricator and Mercurial, opening up HTTP isn't an option we want to consider and would instead stick to having the repository hosted elsewhere.

Today, we already decode the SSH wire protocol and figure out what's going on inside it up to the bundle2 stuff -- here's the, uh, "protocol spec":

It should be possible to determine whether an operation is read or write by the command name: a command either performs writes or it doesn't. The only write commands you should see are unbundle and pushkey. While it might be possible for batch to contain pushkey operations, I don't think that's done today (but I'd have to audit the code to be sure). One thing you should be able to rely on is the HTTP method: GET requests should never perform any significant writes (although some cache files may be written to as a result of resolving data, but that shouldn't be important). It should be perfectly valid to treat GET and POST differently. Although if you route requests to different endpoints within the same TCP/HTTP stream, inconsistent state of a read-only mirror from a master could lead to race conditions (e.g. push fails because a mirror is out of sync). If you route by HTTP request method, you should not need to know anything about the Mercurial wire protocol.

@indygreg - Thanks for reaching out and discussing a way forward. I am not on the phabricator team but I do have an interest in its integration capabilities with mercurial. At best I might be able to summarize my understanding of some of phabricator's needs based on the history of this task, but I do not feel I could properly represent them on mercurial's mailing-list or wiki. For hosting repositories I believe this task would require needing to know if the mercurial operation is read or write -- the two primary reasons being read/write access control and cluster picking.

While I'm hesitant to throw stones here and don't really have any context on bundle2 or the wire protocol in general and there may be plenty of reasons that it works like it does, my general impression is that the Mercurial wire protocol is significantly under-designed, and is now on at least the 3rd and possibly 4th iteration of hacks to try to address the lack of actual protocol design.

In the read/write case, it's too late once we hit a hook -- we need to be able to decide which cluster node to send the traffic to, and we need to know if it's a read or write request before we make that decision, since there may be a nearby read-only node and a distant read/write node. If we can examine the stream and make a determination that the request is read-only, we can freely pick a nearby read-only node.

It's not entirely phabricator specific, as others probably have similar use cases. I'm literally the only hg dev that knows this is an issue (someone flagged me on twitter a month ago, and it's only a strange inability to sleep tonight that got me deep enough in my stack taht I got to this), but if you go to our ML with a decent problem statement ("we need a hook to let us identify read vs write transactions and do our own ACL checking" sounds like a start?) you'll get more help than just me when I'm unable to sleep. :)

Aha

I'm not a phabricator developer so I'm not sure if supporting mercurial hooks is something they're willing to take upstream. I can join the hg mailing list, but wouldn't this conversation be specific to phabricator and beneficial to be carried out here?

Today, we technically do not need to decode the protocol, but we will need to be able to distinguish between read and write requests to implement cluster read-only nodes (T10883) in the future and need to be able to rewrite the protocol to keep parity with fork-like/virtual-ref features that we plan to support in Git (T10691, T8092).

Ah, okay. We can probably do a lot better with some sort of pre-transaction hook then (that was my gut feeling to begin with). Once you've got an idea what the use case is, maybe we can chat on the hg mailing list?

I am totally on-board with finding something more elegant/maintainable, and would be able to help with some testing and possibly development. I'll try to look over the code in the near future to recall what it was doing, but I believe it's trying to determine whether the mercurial operations are read/write in order to determine whether the actor has necessary permissions to do so. I *believe* that's what it's doing, but it's been close to a year since I was looking through this.

We've got a volunteer (slowly) working on documenting formats, but right now bundle2 is not documented extensively outside of the code and the wiki page already mentioned in this thread.

@durin42 - that would probably be better than parsing the bundle stream but, is that documented anywhere? I tried looking fairly extensively for details on the bundle2 protocol and never came across that. Also, the bundle stream itself isn't what's being parsed/modified, it's the capabilities/outer protocol which I found at least some documentation, which is what the parsing/filtering is based on.

I'm not sure why phabricator is trying to parse the bundle stream (I don't know anything about phabricator), but you should be able to invoke hg with bundle2 disabled by setting experimental.bundle2-advertise to false in an hgrc (or using --config if you're shelling out to hg). Given that it's an experimental flag, it's not something I can promise will exist forever, but it might be a start.

So, yes, the bundle2 thing is definitely orthogonal to this. For now, I'd suggest that if phabricator breaks bundle2, filter the response to the capabilities command and omit bundle2=.* from the response.

See T9548#139908 for some discussion.

Could someone create a Mercurial bug (https://bz.mercurial-scm.org) with explanation about how Phabricator handles the bundle2 ssh stream? They might be able to give some hints.
I would like to do it, but I'm not sure how Phabricator handles the stream exactly.

For reference, I use this small script to allow myself to easily use arc land with bookmarks:

@cspeckmim Sadly I don't have any Mercurial host that is not Phabricator.

@fcoelho oops sorry about the mixup. I've previously had success by downgrading the client to 3.4 or less.

@cspeckmim I think you probably confused me with @urzds, I didn't post any logs here yet.

@fcoelho - Would you be able to test your log change with pushing to a mercurial repository not hosted by Phabricator, to compare with the log from your previous comment?

I've been looking at this the last couple of days as it's a pain point for me -- I'd like to host my repositories on Diffusion, and I would like to stay with Mercurial rather than git..

Tried running 2.6.2 on the server side, and that indeed does fix the bundle2 issue, but creates more issues when trying to push from a 3.7.3 hg client.. :-(

If I host externally, I basically lose the access control and herald rules, right?

We host most of our Mercurial repositories externally - this is primarily the infrastructure we had prior to adopting Phabricator. We have a few repositories hosted by Diffusion, however our server runs 2.6.2 which does not utilize the bundle2 protocol.

Ah, you're right, thanks! I misread that page -- that does make Mercurial tags a lot easier to deal with.

In T1130#118470, @epriestley wrote:

They're a function of an individual commit instead of being a function of the repository as a whole, so we'd have to do a lot more work to support them (it's not meaningful to, say, "list all tags in the repository", as it is with Git), ...

The same problem occurred on my site with the latest Phabricator and the latest Mercurial 3.7.2 installed on both sides.

Reading the code around /usr/lib/python2.7/dist-packages/mercurial/sshserver.py:32:

def getargs(self, args):
    data = {}
    keys = args.split()
    for n in xrange(len(keys)):
        argline = self.fin.readline()[:-1]
        arg, l = argline.split()
        if arg not in keys:
            raise util.Abort("unexpected parameter %r" % arg)
        if arg == '*':
            star = {}
            for k in xrange(int(l)):
                argline = self.fin.readline()[:-1]
                arg, l = argline.split()
                val = self.fin.read(int(l))
                star[arg] = val
            data['*'] = star
        else:
            val = self.fin.read(int(l))
            data[arg] = val
    return [data[k] for k in keys]

This implies a question: Could it be Phabricator which modifies the stream between the client and the server in an illegal way? Or is this some bug in Mercurial itself?

Reading the code around /usr/lib/python2.7/dist-packages/mercurial/sshserver.py:32:

def getargs(self, args):
    data = {}
    keys = args.split()
    for n in xrange(len(keys)):
        argline = self.fin.readline()[:-1]
        arg, l = argline.split()
        if arg not in keys:
            raise util.Abort("unexpected parameter %r" % arg)
        if arg == '*':
            star = {}
            for k in xrange(int(l)):
                argline = self.fin.readline()[:-1]
                arg, l = argline.split()
                val = self.fin.read(int(l))
                star[arg] = val
            data['*'] = star
        else:
            val = self.fin.read(int(l))
            data[arg] = val
    return [data[k] for k in keys]

Our projects make heavy use of mercurial tags too, for version tagging and it would be great to see them in Phabricator to easily compare files between versions (they're way more easy to remember than changeset hashes).