This is probably related to (and maybe the same as) T5055.

yes, this is probably a scope-limited sub-set of that task. Feel free to close as dup if you don't want this now.

I think this is reasonable to keep open separately since it's almost certainly the first use case, but I think T5055 is probably most of the groundwork ("build a general package management system to support Phabricator applications and Arcanist extensions").

I mostly just want to make sure we approach this in a way that considers the enormous set of burdensome implications associated with building a package management system, since there are a bunch of cheap/easy things we could do today that would improve things in the short term but wouldn't work in the long run.

Also we need a cool name.

The use case I'm thinking of is mostly "Make sure my company's custom lint is installed/configured, when I have lots of repos and I don't control the user's setup".

I can also assert that this is painful. Some teams find some linter they like online and submodule it, which is vaguely terrifying.

@epriestley: I want to pursue just "allow plugins to integrate into arc version and arc upgrade" soon - do you think that would fit into the long-term plans?

I don't want to build a hacky version of this blindly without having a future plan for solving the real package management problems we'll eventually encounter (off the top of my head: versioning, dependencies, trust, signing, channels/distribution, support for Phabricator applications and general-purpose libphutil libraries rather than just arc extensions).

If we can come up with a design which has a reasonable long-term plans there I'm fine with moving forward with it even if the short-term versions of those plans are stubs/nonexistent/completely fake, but I want to try to avoid building something and then having to completely redo it to get, e.g., a reasonable trust model. In particular, I consider the trust model of Composer to be crazy (see T6012).

If you want to move this forward, I'd like to see this:

• List all of the problems this thing will solve in 2018.
• Figure out how it might solve them.
• For each thing the future version of it does, what does it do now, and how do we get from here to there in a relatively clean way?

The stuff I can come up with offhand is:

• It should support installing arcanist extensions, Phabricator applications, and libphutil libraries.
• Maybe it should even support installing third-party stuff like linters.
• Maybe it should even support installing third-party dependencies like Node?
• Packages should be signed by the author, and you should only need to trust the author to trust the package. Particularly, totally compromising a Phabricator install should be insufficient to compromise users of that install by tainting packages. If you (@avivey) sign a package, I (@epriestley) should be unable to taint it, even if you distribute it through secure.phabricator.com.
• Packages should be able to define dependencies, and it should handle installing them.
• It should handle running different versions of the same package in different projects.

Some of this stuff is probably easy to add later, but some of it would impact decisions now.

Part of the tension here is that any package-related command potentially has two targets: arc itself, or the current working directory (Phabricator, some third-party app you're developing, etc).

For example, these are both reasonable things to want/expect: you might be targeting the command at arc:

phabricator/ $ arc version # Which version of arc extensions?
arcanist abc123 (12 Jul 2016)

...or at the current working directory:

phabricator/ $ arc version # Which version of Phabricator applications?
phabricator 123abc (12 Jul 2016)
phadventure 123def (12 Jul 2016)

We can partially resolve this by having two commands with different implicit targets. For example:

• arc version: Get arc version.
• arc packages: Get installed package versions.

However, these will do almost the same thing in many cases, and may have confusing names (say arc install vs arc extend -- it could be hard to remember which one you needed to use).

There are also potential future cases where this is especially confusing -- for example, if we add a way to just install system packages (like arc install pygments, which installs it as a global package), then you either have to mix the commands or they have to do the same thing in that case.

For most users, targeting arc is probably far more common than targeting the current working directory. However, I don't want to bias these workflows completely against working with Phabricator, since I'll probably be doing that a lot and I envision a thriving ecosystem with two or three good third-party applications by, say, 2025.

We can possibly muddle through by guessing things. For example:

• arc version: Print all versions. Have a --self flag to just print the arc version, perhaps.
• arc install: Install arcanist extension to arc, normal packages to CWD, global stuff globally. This is slightly ambiguous when installing normal packages, but normal users probably never do this. We can add a --self flag to force normal packages to install to arcanist. Also maybe there's a "Phabricator application" package type which further resolves this ambiguity by reducing the number of "normal packages".
• arc upgrade: Upgrade everything? This feels sort of bad/dangerous, maybe? But maybe fine? We could prompt? With --self flag to just upgrade arc.

That has its own set of problems, but those behaviors feel mostly reasonable/solvable to me.

arc upgrade: Upgrade everything? This feels sort of bad/dangerous, maybe? But maybe fine? We could prompt?

I may be misunderstanding what this is all about, but in case it's helpful:

gcloud (the google cloud master binary) does this -- you say gcloud components update and it has output like this:

% gcloud components update


Your current Cloud SDK version is: 97.0.0
You will be upgraded to version: 118.0.0

+--------------------------------------------------------+
|           These components will be updated.            |
+---------------------------------+------------+---------+
|               Name              |  Version   |   Size  |
+---------------------------------+------------+---------+
| BigQuery Command Line Tool      |     2.0.24 | < 1 MiB |
| Cloud SDK Core Libraries        | 2016.07.18 | 4.0 MiB |
| Cloud Storage Command Line Tool |       4.19 | 2.6 MiB |
+---------------------------------+------------+---------+
+----------------------------------------------------------------------------+
|                    These components will be installed.                     |
+-----------------------------------------------------+------------+---------+
|                         Name                        |  Version   |   Size  |
+-----------------------------------------------------+------------+---------+
| BigQuery Command Line Tool (Platform Specific)      |     2.0.24 | < 1 MiB |
| Cloud SDK Core Libraries (Platform Specific)        | 2016.03.28 | < 1 MiB |
| Cloud Storage Command Line Tool (Platform Specific) |       4.18 | < 1 MiB |
+-----------------------------------------------------+------------+---------+

A lot has changed since your last upgrade.  For the latest full release notes,
please visit:
  https://dl.google.com/dl/cloudsdk/channels/rapid/RELEASE_NOTES

Do you want to continue (Y/n)?

I think you can say gcloud components update <foo> to just update foo.

Isn't this more or less the same issue as arc set-config needs to update either local or global configs?

I'm just going to consolidate this into T5055, I think they're pretty much the same task at this point.

I think the different meanings of package-related commands in arc are more different than it looks like gcloud update is, or than arc set-config is, but that the ambiguity is probably surmountable with some attention to it, and a future with one arc upgrade is probably better than a future with separate arc upgrade and arc update commands which mean different things.

We can also afford to be very prompt-heavy on these workflows, since upgrading is rare. I think it would even be fine if the behavior of arc upgrade was to do nothing and tell you to run one of arc upgrade --self, arc upgrade --project, arc upgrade --everything (or ask what you mean with a prompt). Other package systems are generally interactive-by-default and being prompt-heavy will probably not generate as much of a pain point as typing an argument in arc diff <something> does.