Page MenuHomePhabricator

Support for Mercurial extensions in hosted repositories
Open, WishlistPublic

Description

  • We don't currently support extensions (like "Largefiles" or "LockExtension") for hosted repositories.
  • You can try enabling them them in the global .hgrc anyway, which may or may not work (or may work, but create security/policy problems).
  • Proper support requires us to evaluate the capabilities each extension provides on an extension-by-extension basis and make sure that relevant rules and policies aren't being violated. We can't do this easily in the general case.
  • This is a very low priority because there isn't much interest in any specific extension.

Event Timeline

Thanks Evan, the LockExtension has worked as desired for us using the fix suggested above. It would be great to have inbuilt support in the future for these.

The ability to do this with a self hosted solution is one of the things which differentiates Phabricator from the rest.

Particularly, for us the locking capability is why we use Mercurial over Git as some of our tools produce unmergable files (even if they are text).

This is a very low priority because there isn't much interest in any specific extension.

Our project is a game with lots of graphics. As we plan to move from SVN to Mercurial, it is a limitation that Phab does not support Largefiles (which is a must in our situation). Though I'll try a patch from T5189 as temporal solution.

We also utilize the largefiles extension as our repositories tend to have have numerous images, jars, other resources.

I tried configuring largefiles on the hosted repository's .hg/hgrc file, but this config file appears to be constantly overwritten by a phabricator daemon.

I tried configuring largefiles on the system's global config (according to: http://www.selenic.com/mercurial/hgrc.5.html), but this did not appear to have an effect.


I setup a hosted mercurial repository and tried to push an existing repo (which has largefiles turned on) to it, but failed with:

machine:repo cspeck$ hg pus -r master
pushing to ssh://hg@phab.mycompany.com/diffusion/MIM/repo/
searching for changes
abort: ssh://hg@phab.mycompany.com/diffusion/MIM/repo/ does not appear to be a largefile store

Actually, changing the largefile extension from being defined in:
/etc/mercurial/hgrc.d/hgrc
instead moving to:
/etc/mercurial/hgrc.d/largefiles.rc

Causes mercurial to properly pick up on the largefiles extension, however pushing to the repository still fails:

machine:repo cspeck$ hg pus -r master
pushing to ssh://hg@phab.mycompany.com/diffusion/MIM/repo/
searching for changes
abort: largefile da39a3ee5e6b4b0d3255bfef95601890afd80709 missing from store (needs to be uploaded)

I made sure that the global-largefiles-store was copied over from the existing repo, so all that content should exist. I'm not sure what exactly the problem is. I'm guessing I would need to convert the repo over to non-largefiles, push/import into phabricator, then try turning on largefiles.

I'm still investigating whether this can be resolved and if so whether it's worth pursuing.

I wanted largefiles support on my instance so I updated the patch file for the current version of Phabricator:

To Enable Largefiles:

  1. edit the server's hgrc file located at /etc/mercurial/hgrc to enable the largefiles extension
  2. apply patch to phabricator repo root: patch -p1 < /path/to/largefiles-ext.patch
  3. use with converted or new repos with largefiles support

@argylelabcoat - I'm not too familiar with phabricator code-base but I think the patch you suggested only allows the ssh retrieval/storage of largefiles. I would imagine arcanist and the differential (possibly diffusion) applications would need updated as well to account for largefiles.

Since largefiles actually stores the files in a largefiles-store configured on disk and replaces the repo file with a hash, I think differential would need to know about the file deception. Ex: it's not possible to apply a patch of a diff which modifies a largefile. I've not actually tried your patch however, and maybe differential/arcanist can work with your changes.

When cloning/importing a repository which is largefiles enabled, in order for history to work appropriately the largefiles cache would need copied over. I rsync'd the directory in our system to account for this.

T9393 discusses the "Blackbox" extension. To summarize:

  • This extension does not serve any clear purpose in Phabricator working copies.
  • This extension will not work without weakening the security model (by allowing the webserver to write to repository working copies).

Consequently, we are unlikely to support this extension because it seems pointless and dangerous given our current understanding of behavior and use cases.

The use case I was trying to accommodate was source control for projects with large binary files -- often multimedia files of some sort that get updated and integrated into a complex build process. Presently, SVN handles this better than HG or Git unless extensions such as hg largefiles or git-lfs are utilized

@argylelabcoat - The last comment made by @epriestley is describing the use case of @bcran's need for the Blackbox extension available for Mercurial. See T9393 for their discussions. This task is a generic task for providing some support for Mercurial extensions in general. He was just relaying a summary of the conversation.

I just realized this is specifically targeted towards hosted repositories. Would it be a separate effort for support extensions for externally-hosted repositories? For example, in order for the Phabricator server to keep its local clone of the repository it must have largefiles enabled. This can currently be worked around by turning on the extension for all repositories by modifying /etc/mercurial/..., however there will be no support for arc diff/patch etc.

Re: LargeFiles Patch

If I wanted to improve the patch to better meet Phabricator expectations, I assume I'd need:

  • HTTP and SSH paths to work
  • Arcanist support
  • Differential support
  • ?Files support?

I'm certainly interested in improving my patch and getting better support for hg-largefiles within Phabricator. The patch has worked for me with limited success so far. Some of my Hg w/ largefiles repos have broken down and became read-only and I haven't yet managed to debug this on my Phabricator server.

Having recently installed and configured Phabricator for our work environment, we need to host hg repos with largefiles in it, so we can't wait for this ticket to be completed. Feel free to ask if you need some testing, as we have no development capabilities to offer for this project.

I'd propose for simplicity that the focus switch from largefiles to lfs support for Mercurial repositories. lfs has been added to Phabricator for Git already T7789 and it seems that there'd be less work involved to piggyback on that.

Mercurial announced plans to switch to LFS a few years ago https://www.mercurial-scm.org/wiki/LfsPlan, and ships the lfs extension by default now.

We are currently using Phabricator and will likely to continue to do so for future projects. After we finish our first project I will take a stab at porting lfs support to Mercurial repos so that we can at least share a patch for the community.