Page MenuHomePhabricator

arc commands (diff, version) fails on Windows - terrible fix inside
Closed, ResolvedPublic

Description

My problem

I upgraded Arcanist recently and it broke my work flow considerably.

I have seen a few tickets about similar/related error but I figured I would create a new issue since my findings are most likely not perfect due to the fact that I am neither a PHP coder nor very versed with the platform differences between Windows and everything else.

escapeshellargs

Anyway, the most evident issue is the wrongful transformation made by commands like this

execxLocal('log -1 --format=%s', '%H %ct');

On Windows the command that will be run is

git log -1 --format=" H  ct"

And the output when running an arc command is something along these lines

[2015-07-10 09:31:06] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git log -1 --format=" H  ct"

STDOUT
(empty)

STDERR
fatal: invalid --pretty format:  H  ct

As noted already (by @epriestley) this is because of how escapeshellargs works, and that it replaces % (amongst others) with just a blank space.

Funny side not. When I started digging around for the cause of this problem I read "escapeshellargs" as "escape hell args" - which I think is appropriate for this situation actually :)

Fix

My fix for this was to change the line to the following

arcanist/src/workflow/ArcanistVersionWorkflow.php

54      list($stdout) = $repository->execxLocal('log -1 --format="%C"', '%H %ct');

This will output the correct format on Windows

git log -1 --format="%H %ct"

This should be cross platform safe since it should, to my understanding, output the same on OSX and Linux. However, I will leave it up to you to make that call and test it.

default root

The second issue I ran into was represented by the dreaded and very non-descriptive error message below. I have seen this error on many issues reported, but the cause of it seems to be very different on each of these issues (i.e. none of the issues I found did help me in finding this solution for this)

Usage Exception: `arc diff` is only supported under git, hg, svn.

Or with some more tracing, something along these lines

libphutil loaded from 'C:\Users\Martin\Arcanist\libphutil\src'.
arcanist loaded from 'C:\Users\Martin\Arcanist\arcanist\src'.
Config: Reading user configuration file "C:\Users\Martin\AppData\Roaming/.arcrc"...
Config: Did not find system configuration at "C:\ProgramData\Phabricator/Arcanist/config".
Working Copy: Reading .arcconfig from "C:\Users\Martin\Repositories\code/.arcconfig".
Working Copy: Path "C:\Users\Martin\Repositories\code" is part of `git` workingcopy "C:\Users\Martin\Repositories\code".
Working Copy: Project root is at "C:\Users\Martin\Repositories\code".
Config: Did not find local configuration at "C:\Users\Martin\Repositories\code\.git\arc/config".
>>> [0] <exec> $ where "git"
<<< [0] <exec> 68,539 us
Working Copy: No candidate locations for .arcconfig from this working directory.

Working Copy: Path "C:\Users\Martin\Arcanist\arcanist" is not in any working copy.

[2015-07-10 08:42:15] EXCEPTION: (Exception) The current working directory is not part of a working copy for a supported version control system (Git, Subversion or Mercurial). at [<arcanist>\src\repository\api\ArcanistRepositoryAPI.php:82]
arcanist(head=master, ref.master=999eb9376568), phutil(head=master, ref.master=aa6cd8f7e5e5)
  #0 ArcanistRepositoryAPI::newAPIFromConfigurationManager(ArcanistConfigurationManager) called at [<arcanist>\src\workflow\ArcanistVersionWorkflow.php:47]
  #1 ArcanistVersionWorkflow::run() called at [<arcanist>\scripts\arcanist.php:382]

Shameful fix

The default value for root is represented by \ in the code; but that will not work on Windows.

What I did was a super ugly hack and just changed it to C:\ but a better solution would be to run the command CD \ to get to the correct root directory (i.e. if you are on another drive than C:).

libphutil/src/filesystem/FileSystem.php

742  public static function walkToRoot($path, $root = 'C:/') {

What did this fix for me

At the time of writing I have only tested the arc diff and arc land command.

I will most likely also run arc feature soon and see if I run into any problems and can fix this in a similar fashion. But at least this is a start for those of us who think Windows is a valid platform to develop on and who love the Phabricator workflow.

Please try to make a fix for this. I know the problem of running Arcanist on Windows has been in the code for some weeks now, if not more, and it would be really appreciated if it was fixed so that those of us who is on Windows actually can be productive. I know we are few compared to the rest of the community but even a dirty fix for this would be OK (phutil_is_windows()).

Ref T8298, T8291, T6966

Event Timeline

quinyx_martin raised the priority of this task from to Needs Triage.
quinyx_martin updated the task description. (Show Details)

I don't understand where these issues are coming from; we have several developers that use the latest version of Arcanist and libphutil on Windows without issue.

I think what needs to be established:

  • How did you install PHP?
  • What version of PHP are you using?
  • Does Arcanist work fine under msysGit Bash?
  • Does Arcanist work fine under Windows Command Prompt?
  • Does Arcanist work fine under Windows PowerShell?
  • What value do you have for system-wide PATH?
  • Is both arc.bat and php.exe in your system-wide PATH?

For reference, this is the Powershell script we use to install Arcanist on developer machines (replace the URL https://code.redpointsoftware.com.au/ with your own instance). On a fresh install of Windows (or an install of Windows that does not have PHP or Arcanist installed), run this under an Administrator Powershell prompt and restart the computer.

Does Arcanist work then?

NOTE: This is a third-party script and is not supported by upstream (I am not an upstream developer). Run this script at your own risk.
$ErrorActionPreference = 'Stop'

trap
{
    Write-Host "An error occurred"
    Write-Host $_
    Write-Host $_.Exception.StackTrace
    cd $PSScriptRoot
    exit 1
}

if (!(Test-Path C:\Tools)) {
    mkdir C:\Tools
}
if (!(Test-Path C:\Tools\PHP)) {
    mkdir C:\Tools\PHP
}

pushd C:\Tools
if (!(Test-Path C:\Tools\libphutil)) {
    git clone https://github.com/phacility/libphutil.git
}
if (!(Test-Path C:\Tools\arcanist)) {
    git clone https://github.com/phacility/arcanist.git
}
popd

if (!(Test-Path C:\Tools\PHP.zip)) {
    $releases = wget -Uri http://windows.php.net/downloads/releases/
    if (!($releases.Content -match "php-5.6.[0-9]+\-Win32-VC11-x86.zip")) {
        Write-Error "Unable to find right version of PHP to download (ask June to fix this!)"
        exit 1
    }
    $php = $Matches[0]
    wget -Uri http://windows.php.net/downloads/releases/$php -OutFile C:\Tools\PHP.zip
}

if (!(Test-Path C:\Tools\PHP\php.exe)) {
    $shell = new-object -com shell.application
    $zip = $shell.NameSpace("C:\Tools\PHP.zip")
    foreach($item in $zip.items())
    {
        $shell.Namespace("C:\Tools\PHP").copyhere($item)
    }
}

try {
    $env = [Environment]::GetEnvironmentVariable("Path", [System.EnvironmentVariableTarget]::Machine)
    if (!$env.Contains("C:\Tools\PHP")) {
        $env = ($env + ";C:\Tools\PHP;C:\Tools\arcanist\bin")
        [Environment]::SetEnvironmentVariable("Path", $env, [System.EnvironmentVariableTarget]::Machine)
        Write-Warning "Re-open this console to use Arcanist"
    }
} catch {
    Write-Warning "You must run this script as Administrator to add Arcanist to your PATH."
}

if (!(Test-Path "C:\Tools\PHP\php.ini")) {
    Copy-Item C:\Tools\PHP\php.ini-development C:\Tools\PHP\php.ini
}

$content = Get-Content -Raw "C:\Tools\PHP\php.ini"
$content = $content.Replace(";extension=php_curl.dll", "extension=ext\php_curl.dll")
Set-Content -Path "C:\Tools\PHP\php.ini" -Value $content

$env:Path += ";C:\Tools\PHP;C:\Tools\Arcanist\bin"
arc set-config editor "\`"C:\Program Files (x86)\GitExtensions\GitExtensions.exe\`" fileeditor"
arc set-config default https://code.redpointsoftware.com.au/
$output = (echo '{}' | arc call-conduit user.whoami)
if ($output -eq $null) {
    arc install-certificate
}

cd $PSScriptRoot

I don't understand where these issues are coming from; we have several developers that use the latest version of Arcanist and libphutil on Windows without issue.
I think what needs to be established:

  • How did you install PHP?
  • What version of PHP are you using?
  • Does Arcanist work fine under msysGit Bash?
  • Does Arcanist work fine under Windows Command Prompt?
  • Does Arcanist work fine under Windows PowerShell?
  • What value do you have for system-wide PATH?
  • Is both arc.bat and php.exe in your system-wide PATH?
  • I have installed PHP as anyone would; enabling curl and making sure that it is in my PATH.
  • Latest, and a few versions back (I have had the same issue on two machines, one home and one at work)
  • I do not run msysGit Bash
  • I do not run PowerShell
  • I have added PHP, Arcanist, GIT etc. to the path, I can execute the commands just fine - only that it does not work past that.
  • Yes

To add a little more about my environment. I run Windows 8, and I try to make everything work with Command Prompt (without Administrator prompt). The only real issue with this approach is that there is a limit to the PATH environment string so what I had to do was to shorten all my entries by converting everything to DOS paths.

It worked just fine on my work machine prior to upgrading to the latest version, so I do not think it has anything to do with the rest of my setup. I had the exact same problem on my home machine when I tried to set it up a few weeks ago.

Awesome script, will try to use that next time I setup a new environment, but I have done all those steps - even though I did it manually.

I am only trying to share my findings, I would love to point the finger at myself and have it be an issue that only affected me. But since I did solve it this way, I have seen others with the same issue and the fact that I could reproduce it on another machines I still do believe it is a valid issue to look into.
But, if it turns out to be some configuration step I have missed I would love to pinpoint it and make sure that it ends up in the documentation.

We used the above script to install Arcanist this Friday (2 days ago). If you are running a version near that date or before it, then it's definitely an environment issue as the script works fine.

The only real issue with this approach is that there is a limit to the PATH environment string so what I had to do was to shorten all my entries by converting everything to DOS paths.

This is somewhat concerning; I have no idea what effect this will have.

I do not run msysGit Bash

You'll need Git installed for even the most basic of Arcanist commands to work (how did you clone Arcanist / libphutil without having Git installed?)

We used the above script to install Arcanist this Friday (2 days ago). If you are running a version near that date or before it, then it's definitely an environment issue as the script works fine.

The only real issue with this approach is that there is a limit to the PATH environment string so what I had to do was to shorten all my entries by converting everything to DOS paths.

This is somewhat concerning; I have no idea what effect this will have.

I do not run msysGit Bash

You'll need Git installed for even the most basic of Arcanist commands to work (how did you clone Arcanist / libphutil without having Git installed?)

I have msysGit, just not using its Bash. Like I said, I have been running this fine for well over a year.

Can you try running Arcanist commands under msysGit?

Can you try running Arcanist commands under msysGit?

Not at work so I cannot do that now. I assume also that you mean the Bash environment - and even if it would work there I prefer a solution where I can run it from Command Prompt. Call me a neat freak but I just like not having to rely on too many programs when developing.

The problem is that escaping commands on Windows is thoroughly broken; there is no one solution that will work universally for everyone. Fixing this issue by changing the escaping so that it works under Command Prompt may well break it under msysGit, and it's important to understand in what situations the error occurs so that we don't cause regressions for anyone else.

There's potentially a fix in D10555, which basically gives up on any hope of securely escaping arguments on Windows and just does very basic quote escaping.

Of course, I understand that the solution needs to be done properly in order to not break for others. I like the implementation in the diff you mentioned - it is not pretty but at least we contain the "hacks" in one place for the different environments. And who knows - maybe Windows 10 will give us a proper bash to use so that these problems will be long gone.

Thank you for looking into to this and trying to think of a good solution.

rbalik added a subscriber: rbalik.Jul 7 2016, 4:10 AM
rbalik added a comment.Jul 7 2016, 4:20 AM

I just ran into the issue with this error on Windows:
Usage Exception: arc diff` is only supported under git, hg, svn.`

I got around it by running arc diff in the root of the repo. Looks like it doesn't work in subdirs due to the walkToRoot issue.

epriestley moved this task from Backlog to Windows on the Arcanist board.Jul 21 2016, 11:50 AM
epriestley closed this task as Resolved.Oct 2 2018, 12:49 PM
epriestley claimed this task.

I believe this is essentially just T13209.