Distribution mechanism for arc extensions
Open, WishlistPublic

"Dislike" token, awarded by svemir."Mountain of Wealth" token, awarded by jparise."Like" token, awarded by featherless."Mountain of Wealth" token, awarded by joshuaspence.
Assigned To
Authored By
champo, May 14 2014


The idea is to have a way to distribute libphutil libraries for arcanist in a simpler way than emailing a tar-ball to all users.

Discussion at https://secure.phabricator.com/chatlog/channel/6/?at=137189

The two big use-cases are:

  • Get the company's extensions/configuration to all users in an easy way (Without adding them to each repo)
  • Distribute 3rd party extensions, just like apt-get/npm/etc.

A more formal list of requirements (Mostly gathered from epriestley's comments around):

  • It should support installing arcanist extensions, Phabricator applications, and libphutil libraries.
    • i.e., it should handle configuration for phabricator and arcanist
    • Maybe it should even support installing third-party stuff like linters.
    • Maybe it should even support installing third-party dependencies like Node?
  • Packages should be signed by the author, and you should only need to trust the author to trust the package.
    • totally compromising a Phabricator install should be insufficient to compromise users of that install by tainting packages. If you (@avivey) sign a package, I (@epriestley) should be unable to taint it, even if you distribute it through secure.phabricator.com.
  • Packages should be able to define dependencies, and it should handle installing them.
  • for arcanist, packages may be specified via either by the project (.arcconfig) or by global configuration (.arcrc)
  • It should handle running different versions of the same package in different projects.
  • Have a way to require/alert users it's time to upgrade a package
  • Should not require the phab-marketplace to know about my extension (Because it's internal to my company and has all my secrets).
  • Support Linux, Mac OS X, and Windows.
  • "List all the things I have loaded/installed"
  • Should work in an environment where arc is mounted in a read-only location.

Important challenges:

  • Organization: Dumping directories next to things won't last very long and will run into issues with everything else here, as well as making it hard for us to do things like "list all the stuff that is installed". We would quickly need to have better rules about where stuff goes.
  • Versioning: How do we know something needs to be updated? How do we organize, store, and include multiple versions of a package?
  • Dependencies: How do we manage dependencies? How do we deal with cases like "diamond dependencies", where A depends on B and C, and B and C depend on different versions of D?
  • Security: How do we make sure that compromised user accounts don't lead to remote code execution on all users' machines? Code signing is probably the solution here, but it's complicated.

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes

From T8117, I think we can just call this "Packages", and rename the Owners objects (perhaps to "Claims") if there's confusion.

joshma added a subscriber: joshma.Feb 3 2016, 1:45 AM
nickz added a subscriber: nickz.Feb 18 2016, 7:02 PM
featherless added a subscriber: featherless.
epriestley moved this task from Backlog to vMajor on the Arcanist board.Jul 21 2016, 12:11 PM
isfs added a subscriber: isfs.Jul 24 2016, 7:49 AM

Changes connected to T8116 implement the initial server-side version of this. It's still very skeletal, but we probably need to make some client changes to move forward. In particular, the next object to implement is probably Signature, but signature algorithms should live in the client since the client will need to be able to verify signatures.

Roughly, arc will get new/expanded workflows:

arc upgrade: Today, this means "upgrade Arcanist". In the future, it will potentially mean several things:

  • Upgrade Arcanist, the client.
  • Upgrade extensions installed in Arcanist (very rare?).
  • Upgrade the current working directory (impossible/never?).
  • Upgrade extensions installed in libraries in the current working directory.
  • Upgrade global system software (future?)

I expect these to all live in the upgrade command. The default behavior will either become "upgrade everything" or "prompt, asking the user what to upgrade".

arc install: New command. This now gets several meanings:

  • Install a new extension into Arcanist.
  • Install a new extension into the Arcanist configuration for the current project.
  • Install a new extension or application into a library in the current working directory.
  • Install software on the system globally.
  • Download configured extensions for the current project or library.

There's some ambiguity here too, but arc install with no arguments probably means "synchronize everything so it is up to date", while arc install <package> probably means either "guess" or "prompt".

We can narrow down what arc install <package> means by giving packages types, like "Arcanist Extension", "Phabricator Application", "Library", "System Package", etc. It can then select a narrower range of reasonable install behaviors.

arc sign: New command. Sign a publisher, package, or version. This is used when publishing or attesting to the correctness of packages.

arc version: Today, this means "show Arcanist version". In the future, it will likely mean "show versions of all installed stuff" instead.

We also probably need these capabilities, but can figure them out in the future:

  • Search packages? arc search jslint?
  • Add a new package source (URI of an installed version of Packages)? Just arc set-config for now?
  • Remove a package -- some --remove flag on arc install? Separate arc uninstall?

Making major additions and changes to arc workflows dovetails heavily with T10329 and adjacent tasks. I expect to pursue that first, provide a more solid foundation for arc workflow to build upon, and then implement the new workflows.

Specifically, the next pieces I expect to build are:

  • Package types ("Arcanist Extension", "Phabricator Application", "Library", "System Software" (future)).
  • PackageSignature on the server, and arc sign on the client (initially, only for Publishers and Packages, probably).
  • Additional properties on Versions so they can actually point at a Git repository to clone (this will be modular in the future, but only support "clone a Git repo" for now).
  • Client-side support for cloning repos (arc install, arc version) and loading extensions.
  • Some sense of an upgrade channel / pathway and arc upgrade. Currently, "Versions" are not related to one another, so there's no way to specify how to upgrade version X. We can limp along without this initially since "look it up, then arc install" is fine for administrators while this is a prototype and I don't expect anyone to publish and sign 200 versions of an extension in the first week.


  • To move forward, Packages needs a mixture of client and server changes.
  • Arcanist workflows are getting modernized before the client changes (T10329).
  • After that, Packages can move forward on both the client and server.
fabe added a subscriber: fabe.Aug 2 2016, 1:46 PM

It might be beneficial to generally support gpg signed commits / tags in phabricators git repos and then use the same mechanism for the arcanist packages.
Since github recently started pushing this feature a bit (https://github.com/blog/2144-gpg-signature-verification) quite a few library maintainers started signing their release tags.
And i just recently spoke to a lib maintainer about extending composer to verify all packages (on install or upgrade) against a list of authors trusted by the user (or an enterprise wide list).

I've not experimented with this stuff yet and setting up gpg is still a pain in the ass. But the integration into git afterwards is quite straight-forward.

You could optionally use the users gpg keyring in addition to an arcanist specific one.

I generally expect all signing to be external, at least for the foreseeable future.

One of the major concerns I have with Composer is that it conflates the software developer and the software packager, often assuming they are one in the same (and generally having no mechanism to identify or verify the packager).

For example, I think having strong trust mechanism for the developer is of limited use (and potentially quite misleading) if the packager can release an "update" from v2.7.3 to v2.7.4 which actually reverts to v2.7.2, re-opening a widely disclosed and easily exploited security hole. If the developer removes the v2.7.2 tag from their repository as dangerous, the publisher can copy the repository elsewhere, restore the tag (which will have a valid signature!), and then point the package at the new repository. Generally, there's no way that I'm aware of to "unpublish" a GPG signature, but over time many signed versions of software become trivially unsafe as vulnerabilities are discovered and disclosed.

I want to primarily focus on trusting the publisher, and making it clear to the user that this is who they are trusting, and that they are trusting the publisher more or less completely. It's possibly even desirable not to show any developer-signature information to the user, as this implies that the barrier of trust the publisher must meet is lower: seeing that the package is signed by qmysteryman but the code is signed by Facebook "seems" trustworthy, but is not actually much different from only seeing that the package is signed by qmysteryman. I think a sophisticated attacker with complete control of a package is not made substantially less dangerous by only being able to undo security fixes vs being able to deploy arbitrary code.

Showing this kind of information to the publisher at the time they sign a version (to make it easier for publishers to perform due diligence before signing a package) could be useful, but it's probably some ways away.

This specific attack may not be entirely possible in practice (I'm not familiar with the Composer workflows), but I think it's broadly difficult to establish a clear capability gap between an attacker who can deploy any code at all and an attacker who can "only" deploy any code which a particular developer ever signed. At any given time, most code which a particular developer has ever signed is probably unsafe to run. I think the cryptographic assertion that the developer considered it safe to run at one time is not a very strong one: I would have made this assertion about all versions of Phabricator as I released them in the past, but would no longer make this assertion about those versions because users have discovered and reported security issues since then.

fabe added a comment.Aug 2 2016, 4:17 PM

The whole "identify the packager problem" with composer is indeed a big one and i'm not suggesting you use composer in any way for the arcanist/phabricator packages! Generally i agree that i want to primarily trust the packager and not the developer. This is the way this is handled with os packages (rpm/deb) and companies are used to the workflow: Getting new packages from a somewhat trusted/signed source (e.g. redhat/canonical), testing them yourself, signing them with your own key as well and then distributing to the internal repositories and "end users" only trust your own internal key.

I don't know of any package manager that currently supports "revoking" a package signature, but it would be a cool feature.
GPG generally allows for "unpublishing" a key or your signature of a key by using revocation certs. I think you would have to use a different key (maybe subkey) for every release and publish your trust to them with your "publisher" key. When a new version is released you would publish a revocation cert for the old key stating that you no longer trust it and signature validation should fail. But as GPG is a big box of black magic i'd need to test this properly. I might be wrong and validation will succeed and the key can only no longer be used for future signatures which wouldn't help much.

Going the other way and using a X.509 CA with a new key/cert for every release using CRLs / OCSP for revocation should work just as well.

urzds added a subscriber: urzds.Aug 17 2016, 8:35 AM
jcox added a subscriber: jcox.Sep 28 2016, 11:03 AM
mtsgrd added a subscriber: mtsgrd.Feb 22 2017, 1:49 PM
fcoelho added a subscriber: fcoelho.Aug 7 2017, 9:31 PM
cmmata added a subscriber: cmmata.Aug 10 2017, 8:18 AM