Page MenuHomePhabricator

Upgrading: Early 2020 Changes to Arcanist
Open, LowPublic

Description

Significant changes to Arcanist (and Phabricator in general) have promoted to stable in 2020 Week 16. See the 2020 Week 16 (Late April) Changelog for more information.

This task describes changes to Arcanist. See the changelog for general upgrade guidance.

Most of these changes are minor and/or don't require any direct action on the part of users, but there are enough changes that you're likely to hit a few of them. It is not important to upgrade immediately: older versions of Arcanist will work against newer versions of Phabricator, and vice versa.

(The last stable version of arcanist/ prior to these changes is available on the legacy-2019 branch, as a possible workaround if some aspect of these changes is significantly disruptive to your workflow.)


The minimum PHP version required to run Arcanist is now PHP 5.5, which was released seven years ago in 2013. Requiring this version of PHP allows Arcanist to use a few new language features, including: traits; yield; and finally. See T11968 for some details.

libphutil is no longer required. All the code in "libphutil/" which Arcanist requires has been merged into "arcanist/". "libphutil/" is now an empty repository with a README file. See T13395 for discussion. You can safely delete the "libphutil/" directory if it is empty and everything is working for you.

These flags to common arc workflows have been removed:

arc diff--plan-changes Removed, see T13010. Try --draft.
--preview Removed, see T13010. Try --draft.
--no-diff Removed, see T13010. Legacy flag.
--cache Removed, see T13010. Legacy flag.
arc lint--only-changed Removed. Try "warning" severity.
--only-new Removed. Almost never worked correctly.
--cache Removed. Legacy flag.

The way shell completion is configured has changed. If you use this feature, see below for update instructions.


These workflows have been removed.

In some cases, these workflows were added a long time ago (circa 2012-2013) by a contributor, do something trivial, and have never been improved by later patches. This is likely to indicate that no one (or almost no one) uses them.

In the case of Phrequent workflows, Phrequent was contributed, has remained a prototype, and has no interest from current customers. It's not likely to move forward any time soon and isn't worth the effort to maintain the CLI integration, given the unclear argument for the value it's providing.

arc backoutTrivial wrapper around backout/revert.
arc closeCloses a task.
arc flagFlags objects, or lists flagged objects.
arc startStarts time tracking in Phrequent.
arc stopStops time tracking in Phrequent.
arc timeLists time tracking in Phrequent.
arc revertAlias of "backout".

These minor behaviors have also changed:

Reduced Compatibility with Ancient Phabricator

Some workflows which previously used last-generation APIs or fell back to last-generation APIs now require modern APIs. These APIs include:

  • paste.edit, introduced in D14393 in Nov 2015.
  • user.search, introduced in D15959 in May 2016.
  • differential.revision.search, introduced in D16089 in June 2016.
  • diffusion.commit.search, introduced in D17194 in Jan 2017.
  • file.search, introduced in D17612 in April 2017.

If you have not upgraded Phabricator in several years, this is a great time to upgrade.

CLI/Scripting Behaviors

  • Configuration flags must now be specified before workflows. For example, arc --config x=y diff is valid, but arc diff --config x=y is not valid.
  • When stdout is not a TTY (usually, this means you're running arc in a script/non-interactive environment), command line flags must now be explicitly terminated with -- (the "end of flags" argument), even if there are no flags, or no arguments, or the command you are executing can not take flags or arguments or is otherwise completely unambiguous. See T13491 for more discussion.
  • arc no longer exits with error code 77 to indicate invalid flags.

Workflow: Alias

  • arc alias will no longer remove redundant flags. Previously, you could define arc draft -> arc diff --draft, then run arc draft --draft (which would naively resolve to arc diff --draft --draft) and have your duplicate flag cleaned up. This isn't safe in the general case and deviates from the explicit approach we generally prefer elsewhere.
  • Shell aliases (like arc alias ls -- '!ls') are now stored as individual arguments, so existing aliases may need to be redefined by running arc alias ... again. In particular, arc alias x -- '!y z' now defines an alias x which runs the shell command 'y z' (i.e., a binary name with a space in it), not the binary y with argument z.
  • Shell aliases now escape their arguments, and thus no longer support inline shell code. (You can put shell code in a shell script on disk and have arc alias execute that, or perhaps define an !sh -c ... alias).
  • You can now define aliases that resolve to other aliases, creating alias chains. Exceptionally useful! Thousands of users have demanded this feature!
  • alias is now a global workflow for any toolset.
  • Note that only "modern" workflows can be resolved by "arc alias".

Workflow: Download

  • The --show flag (which printed the file to stdout) has been removed. Use --as - to achieve the same effect.
  • arc download can now download files which are too large to fit into memory, although see T13352 if you are downloading files larger than 20GB.

Workflow: Help

  • The arc help --full flag no longer exists. arc help is more verbose than it was before.
  • help is now a global workflow for any toolset.
  • Note that only "modern" workflows appear in "arc help".

Workflow: Liberate

  • arc liberate now liberates all libraries under the current working directory if given no arguments.
  • The flags --library-name, --force-update, --remap, and --verify no longer exist.
  • The flag --upgrade no longer exists. v1 libraries can no longer be upgraded to v2. If you have a library which is 5+ years old, remove all the __init__.php files and then just do a clean liberate on it.
  • The flag --all has been renamed to --clean (like make clean), for consistency with other commands and to disambiguate the new default behavior of rebuilding all libraries in subdirectories.

Workflow: Paste

  • The --json flag has been removed. Modern programs can interact with Paste more flexibly by calling the API directly, or by calling the API via arc call-conduit.

Workflow: Shell Complete

  • arc shell-complete now installs itself (when you run arc shell-complete).
  • You should edit anywhere you stuck the old source path/to/arcanist/.../bash-completion.sh line (like ~/.profile) and remove it. Then, run arc shell-complete to install the updated hook. Finally, start a new shell to pick up the changes.
  • The behavior of completion has changed somewhat, but probably not in any meaningful ways for most users.
  • Support for ZSH has been removed. This was contributed and I wasn't able to figure out how to make it work on master, so I suspect it may never have worked for more than one person in 2012. I'm open to restoring support if you're a ZSH wizard and can convince me that your implementation is correct and supportable, and it works out of the box for me by magic. See https://github.com/phacility/arcanist/pull/11 for the heady days of an upstream with low contribution barriers.
  • arc shell-complete is now a global workflow for any toolset.
  • Note that only "modern" workflows can be completed by arc shell-complete.

Workflow: Version

  • version is now a global workflow for any toolset. (But it isn't useful for other toolsets yet.)

.arcrc

  • The internal format of this file has changed, and it will now be written with keys at top level (instead of inside a "config" key). The old format will still be read. This only impacts you if you're manually reading .arcrc files, which you probably should not be.

Global Flags

  • Removed --conduit-version. We do capability testing, now.
  • Removed --conduit-timeout. Not clear this ever really had any callers.
  • Removed --skip-arcconfig. Unclear what this was for. FB only option because someone created an invalid file at /home/.arcconfig?

Configuration

  • Removed the arc.lint.cache configuration setting.
  • Removed the arcanist_configuration configuration setting. This is a legacy/FB-only option which doesn't fit into the roadmap.
  • The http.basicauth.user and http.basicauth.pass configuration options are no longer supported. These served a very niche use case. If you're a customer and need this, we can provide some similar capability via PhutilHTTPEngineExtension; file a support request.
  • The https.blindly-trust-domains configuration option is no longer supported. It's 2020, legitimate SSL is completely free now, and you should configure it properly. If you "need" this, you can implement it via PhutilHTTPEngineExtension.

Some internal APIs have also changed. This list is not exhaustive, but some of the most important changes are:

  • Future->resolve() no longer accepts a timeout. If you need to return execution to the caller after a certain amount of time, wrap the future in a FutureIterator and use setUpdateInterval().

And:

  • The preg_quote() lint rule has been removed. This had false positives on safe/correct use of preg_quote() with ( and ) as the expression delimiters, which is the modern coding style in this codebase.
  • The "use __CLASS__ instead of hard-coding a class name" lint rule no longer raises a warning if you use the class name as part of a longer string. So the string "Filesystem" (class name hard-coded as a string literal) still raises a warning if it appears in the class Filesystem, but the string "Filesystem path X does not exist." (class name used as an English word in a longer string) no longer raises this warning.

Event Timeline

epriestley triaged this task as Low priority.Feb 13 2020, 1:49 PM
epriestley created this task.
epriestley updated the task description. (Show Details)Feb 13 2020, 2:01 PM
epriestley updated the task description. (Show Details)Feb 13 2020, 10:09 PM
epriestley updated the task description. (Show Details)Feb 13 2020, 10:16 PM
epriestley updated the task description. (Show Details)Feb 13 2020, 10:57 PM
epriestley updated the task description. (Show Details)Feb 14 2020, 12:21 PM
cmmata added a subscriber: cmmata.Feb 14 2020, 4:22 PM
epriestley updated the task description. (Show Details)Feb 14 2020, 4:33 PM
epriestley updated the task description. (Show Details)Feb 14 2020, 4:41 PM
epriestley updated the task description. (Show Details)Feb 14 2020, 4:46 PM
epriestley updated the task description. (Show Details)Feb 14 2020, 4:50 PM
epriestley updated the task description. (Show Details)Feb 14 2020, 5:07 PM
epriestley updated the task description. (Show Details)Feb 15 2020, 5:58 PM
epriestley updated the task description. (Show Details)Feb 15 2020, 6:04 PM
epriestley updated the task description. (Show Details)Feb 17 2020, 5:22 PM
epriestley updated the task description. (Show Details)
epriestley updated the task description. (Show Details)Feb 26 2020, 4:38 PM
epriestley updated the task description. (Show Details)Feb 26 2020, 4:52 PM
epriestley updated the task description. (Show Details)
jnv added a subscriber: jnv.Feb 28 2020, 3:13 PM

The new method of installing the shell completion seems to error out for me with EXCEPTION: (Error) Call to undefined method ArcanistShellCompleteWorkflow::getPrompt() on a fresh install:

seth@ip-10-1-0-41:~$ mkdir arc_tmp_install
seth@ip-10-1-0-41:~$ cd arc_tmp_install/
seth@ip-10-1-0-41:~/arc_tmp_install$ git clone https://github.com/phacility/libphutil.git
Cloning into 'libphutil'...
remote: Enumerating objects: 25, done.
remote: Counting objects: 100% (25/25), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 16127 (delta 6), reused 11 (delta 3), pack-reused 16102
Receiving objects: 100% (16127/16127), 8.63 MiB | 30.92 MiB/s, done.
Resolving deltas: 100% (9634/9634), done.
seth@ip-10-1-0-41:~/arc_tmp_install$ git clone https://github.com/phacility/arcanist.git
Cloning into 'arcanist'...
remote: Enumerating objects: 132, done.
remote: Counting objects: 100% (132/132), done.
remote: Compressing objects: 100% (90/90), done.
remote: Total 21482 (delta 70), reused 79 (delta 41), pack-reused 21350
Receiving objects: 100% (21482/21482), 9.94 MiB | 30.21 MiB/s, done.
Resolving deltas: 100% (12693/12693), done.
seth@ip-10-1-0-41:~/arc_tmp_install$ arc

Command 'arc' not found, but can be installed with:

sudo apt install arc     
sudo apt install arcanist

seth@ip-10-1-0-41:~/arc_tmp_install$ export PATH=~/arc_tmp_install/arcanist/bin/:$PATH
seth@ip-10-1-0-41:~/arc_tmp_install$ arc
 USAGE EXCEPTION  Choose a workflow!
seth@ip-10-1-0-41:~/arc_tmp_install$ arc shell-complete
 DETECT  Detecting current shell...
 SHELL  The "SHELL" environment variable has value "/bin/bash", so the target shell was detected as "bash".
 INSTALL  Installing shell completion support for "bash".
To install shell completion support for "bash", this line will be added to your existing "~/.profile" file:

     source /home/seth/arc_tmp_install/arcanist/support/shell/hooks/bash-completion.sh # arcanist-shell-complete
[2020-04-05 15:33:46] EXCEPTION: (Error) Call to undefined method ArcanistShellCompleteWorkflow::getPrompt() at [<arcanist>/src/toolset/workflow/ArcanistShellCompleteWorkflow.php:299]
arcanist(head=master, ref.master=32005f26a4e7)
  #0 ArcanistShellCompleteWorkflow::runInstall() called at [<arcanist>/src/toolset/workflow/ArcanistShellCompleteWorkflow.php:144]
  #1 ArcanistShellCompleteWorkflow::runWorkflow(PhutilArgumentParser) called at [<arcanist>/src/workflow/ArcanistWorkflow.php:171]
  #2 ArcanistWorkflow::executeWorkflow(PhutilArgumentParser) called at [<arcanist>/src/toolset/ArcanistPhutilWorkflow.php:21]
  #3 ArcanistPhutilWorkflow::execute(PhutilArgumentParser) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:485]
  #4 PhutilArgumentParser::parseWorkflowsFull(array) called at [<arcanist>/src/runtime/ArcanistRuntime.php:151]
  #5 ArcanistRuntime::executeCore(array) called at [<arcanist>/src/runtime/ArcanistRuntime.php:30]
  #6 ArcanistRuntime::execute(array) called at [<arcanist>/support/init/init-arcanist.php:6]
  #7 require_once(string) called at [<arcanist>/bin/arc:10]
seth@ip-10-1-0-41:~/arc_tmp_install$

Additionally, https://secure.phabricator.com/book/phabricator/article/arcanist/ still references the old method of installing: "source /path/to/arcanist/resources/shell/bash-completion"

epriestley updated the task description. (Show Details)Apr 11 2020, 5:17 PM
epriestley updated the task description. (Show Details)Apr 12 2020, 9:40 PM
epriestley updated the task description. (Show Details)Apr 12 2020, 10:02 PM
epriestley updated the task description. (Show Details)Apr 12 2020, 11:09 PM
epriestley updated the task description. (Show Details)Apr 12 2020, 11:12 PM
epriestley updated the task description. (Show Details)Apr 13 2020, 1:06 PM
epriestley updated the task description. (Show Details)Apr 13 2020, 1:09 PM
epriestley updated the task description. (Show Details)Apr 13 2020, 8:42 PM
epriestley updated the task description. (Show Details)Apr 13 2020, 9:09 PM
hskiba added a subscriber: hskiba.Apr 25 2020, 5:01 AM
epriestley updated the task description. (Show Details)Apr 27 2020, 7:41 PM

We were using the --skip-arcconfig option like this in our CI environment (jenkins).

echo  '{
              "buildTargetPHID": "PHID-HMBT-....",
              "artifactKey": "jenkins.uri",
              "artifactType": "uri",
              "artifactData": {
                  "uri": "<url>",
                  "name": "Jenkins",
                  "ui.external": true
              }
          }'  | <arcanist>/bin/arc --skip-arcconfig call-conduit--conduit-uri <uri> harbormaster.createartifact

We did this because we have some libraries included in the .arcconfig file that need to be installed with Composer. Because we want to notify Harbormaster asap about the build starting, the --skip-arcconfig flag allowed us to do that before running $ composer install. Now we're getting the error:

 LOAD ERROR  Failed to load library at location "vendor/appsinet/arc-phpstan". This library is specified by "Configuration (Project Config File (/path/to/project/.arcconfig))". Check that the setting is correct and the library is located in the right place.


    Continue without loading library? [y/N] [2020-05-01 07:41:48] EXCEPTION: (PhutilConsoleStdinNotInteractiveException) The program is attempting to read user input, but stdin is being piped from some other source (not a TTY). at [<arcanist>/src/console/format.php:196]
arcanist(head=stable, ref.master=ade25facfdf2, ref.stable=c1d12ff7784b)
  #0 phutil_console_require_tty() called at [<arcanist>/src/console/format.php:47]
  #1 phutil_console_prompt(string) called at [<arcanist>/src/console/format.php:15]
  #2 phutil_console_confirm(string) called at [<arcanist>/src/runtime/ArcanistRuntime.php:507]
  #3 ArcanistRuntime::loadLibrary(ArcanistConfigurationEngine, string, string) called at [<arcanist>/src/runtime/ArcanistRuntime.php:372]
  #4 ArcanistRuntime::loadLibraries(ArcanistConfigurationEngine, ArcanistConfigurationSourceList, PhutilArgumentParser) called at [<arcanist>/src/runtime/ArcanistRuntime.php:102]
  #5 ArcanistRuntime::executeCore(array) called at [<arcanist>/src/runtime/ArcanistRuntime.php:37]
  #6 ArcanistRuntime::execute(array) called at [<arcanist>/support/init/init-arcanist.php:6]
  #7 require_once(string) called at [<arcanist>/bin/arc:10]

I've been trying to get it to run non-interactively to skip the question "Continue without loading library? [y/N]" by changing our arc call to <arcanist>/bin/arc --conduit-uri <uri> call-conduit -- harbormaster.createartifact, but without effect. For now, I'll have to move the call to after the composer install command. But would be better to get the build url reported before that. Any suggestions on how I can configure the command to work without the libraries from composer?

(This kind of question is best asked on Discourse.)

Try one of these approaches?

Option one:

  • Add a .arcconfig for the builds to your project somewhere. This file can be empty (just {}) or specify a minimal set of options (like just the Phabricator URI).
  • Run arc --config-file path/to/arcconfig-for-builds call-conduit ....
  • This should skip all other working copy / project configuration.

Option two:

  • Run in a CWD which is not a repository with an ".arcconfig" (like cd /tmp && arc call-conduit ...).

Thanks for the quick response! I'll give your suggestions a try, and else will continue on Discourse. Was not aware of its existence.