Page MenuHomePhabricator

Detect if pygmentize exists but fails to execute
Closed, ResolvedPublic

Description

Dear PhabricatorS,
I'm sorry this is probably a newby question, but I can't find a solution or, better, I found some docs on the internet, applied all suggestions but they did not work for me.
In our organization we are evaluating Phabricator and, being the best tool we have tested so far, I'm willing to have it perfectly functional in all the required parts in order to have it approved for everyday use by CTO.

Problem: pygmentize is not working on my setup.

Setup:
Server: VMWare VM Ubuntu Server 14.04
DB: MySQL
Phabricator: installed in /var/opt/phabricator
Webserver. Apache2 installed using apt-get of ubuntu server and modifying the config as required by Phabricator docs

diego@ServerLinux1:/var/opt/phabricator/phabricator$ ps axu
[snip!]
mysql     1047  0.0  8.1 889516 167984 ?       Ssl  Sep29   1:18 /usr/sbin/mysqld
root      1150  0.0  0.2  91772  4636 ?        S    Sep29   0:44 /usr/sbin/vmtoolsd
memcache  1258  0.0  0.0 325396  1184 ?        Sl   Sep29   0:01 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
root      1358  0.0  0.0  25344  1676 ?        Ss   Sep29   0:00 /usr/lib/postfix/master
postfix   1367  0.0  0.0  27460  1552 ?        S    Sep29   0:00 qmgr -l -t unix -u
root      1536  0.0  0.8 416980 18088 ?        Ss   Sep29   0:01 /usr/sbin/apache2 -k start
www-data  1540  0.0  3.3 447232 69112 ?        S    Sep29   0:22 /usr/sbin/apache2 -k start
www-data  1541  0.0  2.9 440224 60732 ?        S    Sep29   0:06 /usr/sbin/apache2 -k start
www-data  1543  0.0  3.7 454808 76200 ?        S    Sep29   0:30 /usr/sbin/apache2 -k start
root      1602  0.0  0.0  14540   960 tty1     Ss+  Sep29   0:00 /sbin/getty -8 38400 tty1
root      1631  0.0  0.0      0     0 ?        S    Sep29   0:00 [kauditd]
www-data  1775  0.0  2.4 431512 50672 ?        S    Sep29   0:33 /usr/sbin/apache2 -k start
www-data  1776  0.0  2.7 435016 56236 ?        S    Sep29   0:24 /usr/sbin/apache2 -k start
www-data  1777  0.0  2.8 439920 58748 ?        S    Sep29   0:23 /usr/sbin/apache2 -k start
www-data  1779  0.0  2.5 429736 52196 ?        S    Sep29   0:12 /usr/sbin/apache2 -k start
www-data  1780  0.0  3.0 440132 62188 ?        S    Sep29   0:07 /usr/sbin/apache2 -k start
root      5352  0.0  1.0 276860 21432 ?        S    Sep29   0:25 php ./phd-daemon PhabricatorRepositoryPullLocalDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
root      5353  0.0  1.3 278592 28036 ?        Ss   Sep29   1:02 php ./exec_daemon.php PhabricatorRepositoryPullLocalDaemon --load-phutil-library=/var/opt/phabricator/arcanist/src --load-phutil-library=/var/opt/phabricator/phabricator/src --log=/var/tmp/phd/log/daemons.l
root      5361  0.0  1.0 276860 21420 ?        S    Sep29   0:16 php ./phd-daemon PhabricatorGarbageCollectorDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
root      5362  0.0  1.4 279544 29212 ?        Ss   Sep29   0:00 php ./exec_daemon.php PhabricatorGarbageCollectorDaemon --load-phutil-library=/var/opt/phabricator/arcanist/src --load-phutil-library=/var/opt/phabricator/phabricator/src --log=/var/tmp/phd/log/daemons.log
root      5380  0.0  1.0 276860 21424 ?        S    Sep29   0:16 php ./phd-daemon PhabricatorTaskmasterDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
root      5381  0.0  2.8 310320 57964 ?        Ss   Sep29   0:10 php ./exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/var/opt/phabricator/arcanist/src --load-phutil-library=/var/opt/phabricator/phabricator/src --log=/var/tmp/phd/log/daemons.log --
root      5392  0.0  1.0 276860 21428 ?        S    Sep29   0:15 php ./phd-daemon PhabricatorTaskmasterDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
root      5393  0.0  2.7 307864 55908 ?        Ss   Sep29   0:12 php ./exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/var/opt/phabricator/arcanist/src --load-phutil-library=/var/opt/phabricator/phabricator/src --log=/var/tmp/phd/log/daemons.log --
root      5397  0.0  1.0 276860 21428 ?        S    Sep29   0:15 php ./phd-daemon PhabricatorTaskmasterDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
root      5398  0.0  2.7 308776 56980 ?        Ss   Sep29   0:11 php ./exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/var/opt/phabricator/arcanist/src --load-phutil-library=/var/opt/phabricator/phabricator/src --log=/var/tmp/phd/log/daemons.log --
root      5402  0.0  1.0 276860 21432 ?        S    Sep29   0:15 php ./phd-daemon PhabricatorTaskmasterDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
root      5403  0.0  2.9 312444 60496 ?        Ss   Sep29   0:10 php ./exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/var/opt/phabricator/arcanist/src --load-phutil-library=/var/opt/phabricator/phabricator/src --log=/var/tmp/phd/log/daemons.log --
www-data  6077  0.0  2.2 426980 45576 ?        S    Sep29   0:04 /usr/sbin/apache2 -k start
root      6694  0.0  0.0      0     0 ?        S    01:37   0:01 [kworker/u2:3]
root     12347  0.0  0.2 105628  4336 ?        Ss   10:56   0:00 sshd: diego [priv]
diego    12437  0.0  0.1 105628  2100 ?        S    10:56   0:00 sshd: diego@pts/0
diego    12438  0.0  0.2  22096  4692 pts/0    Ss   10:56   0:00 -bash
www-data 15073  0.0  1.4 420860 29804 ?        S    12:32   0:00 /usr/sbin/apache2 -k start
postfix  16471  0.0  0.0  27408  1520 ?        S    13:20   0:00 pickup -l -t unix -u -c
diego    18643  0.0  0.0  17168  1288 pts/0    R+   14:37   0:00 ps axu

Pygments: last version 1.6 with a modified lexer to correctly "pygmentize" my assembly files (verified it is functioning properly). Pygmentize installed in /usr/local/bin:

diego@ServerLinux1:/var/opt/phabricator/phabricator$ ls /usr/local/bin/pygmentize -l
-rwxr-xr-x 1 root root 304 Sep 26 17:12 /usr/local/bin/pygmentize
diego@ServerLinux1:/var/opt/phabricator/phabricator$

Added the '/usr/local/bin' path to phabricator configuration, as required using the comment:

sudo ./bin/config set environment.append-paths '["/usr/local/bin"]'

And the result is this:

diego@ServerLinux1:/var/opt/phabricator/phabricator$ ./bin/config get environment.append-paths
{
  "config" : [
    {
      "key"    : "environment.append-paths",
      "source" : "local",
      "value"  : [
        "\/usr\/local\/bin"
      ]
    }
  ]
}

Verified the path /usr/local/bin is present in $PATH variable of www-data user (running Apache):

diego@ServerLinux1:/var/opt/phabricator/phabricator$ sudo -u www-data printenv PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
diego@ServerLinux1:/var/opt/phabricator/phabricator$

Verified one task running apache2 has correct environment:

diego@ServerLinux1:/var/opt/phabricator/phabricator$ sudo cat /proc/1775/environ
APACHE_RUN_DIR=/var/run/apache2APACHE_PID_FILE=/var/run/apache2/apache2.pidPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/binAPACHE_LOCK_DIR=/var/lock/apache2LANG=CAPACHE_RUN_USER=www-dataAPACHE_RUN_GROUP=www-dataAPACHE_LOG_DIR=/var/log/apache2PWD=/diego@ServerLinux1:/var/opt/phabricator/phabricator$

Verified the user www-data can run pygmentize:

diego@ServerLinux1:/var/opt/phabricator/phabricator$ sudo -u www-data pygmentize -V
Pygments version 1.6, (c) 2006-2013 by Georg Brandl.
diego@ServerLinux1:/var/opt/phabricator/phabricator$

Enabled Pygments from the configuration, restarted phd and ...

The Phabricator is still complaining that pygmentize cannot be found in current path...

phabricator_error_pygmentize.png (415×765 px, 20 KB)

WHERE AM I WRONG? :-(

Thank you very much, kudos for your great software!
Diego

Event Timeline

dserafin raised the priority of this task from to Needs Triage.
dserafin updated the task description. (Show Details)
dserafin added a project: Phabricator.
dserafin added a subscriber: dserafin.

Offhand, I'm not sure. We actually run pygmentize as pygmentize -h to check it, does that exit with a zero exit code? It's possible that the -h flag has changed across versions, or -h is somehow broken on your install.

The only other thing I can think of is some kind of SELinux/AppArmor thing preventing the webserver process from running the binary?

Thank you,
I must confess that I'm using Linux since Linus' first version, but I have little sysadmin experience.

Anyhow,
here it is the app-armor status:

diego@ServerLinux1:/var/opt/phabricator/phabricator$ sudo aa-status
apparmor module is loaded.
5 profiles are loaded.
5 profiles are in enforce mode.
   /sbin/dhclient
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/mysqld
   /usr/sbin/tcpdump
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode.
   /usr/sbin/mysqld (1047)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
diego@ServerLinux1:/var/opt/phabricator/phabricator$

and this is what happens if I try to launch pygmentize -h:

diego@ServerLinux1:/var/opt/phabricator/phabricator$ sudo -u www-data pygmentize -h
Usage: /usr/local/bin/pygmentize [-l <lexer> | -g] [-F <filter>[:<options>]] [-f <formatter>]
          [-O <options>] [-P <option=value>] [-o <outfile>] [<infile>]

       /usr/local/bin/pygmentize -S <style> -f <formatter> [-a <arg>] [-O <options>] [-P <option=value>]
       /usr/local/bin/pygmentize -L [<which> ...]
       /usr/local/bin/pygmentize -N <filename>
       /usr/local/bin/pygmentize -H <type> <name>
       /usr/local/bin/pygmentize -h | -V

Highlight the input file and write the result to <outfile>.

If no input file is given, use stdin, if -o is not given, use stdout.

<lexer> is a lexer name (query all lexer names with -L). If -l is not
given, the lexer is guessed from the extension of the input file name
(this obviously doesn't work if the input is stdin).  If -g is passed,
attempt to guess the lexer from the file contents, or pass through as
plain text if this fails (this can work for stdin).

Likewise, <formatter> is a formatter name, and will be guessed from
the extension of the output file name. If no output file is given,
the terminal formatter will be used by default.

With the -O option, you can give the lexer and formatter a comma-
separated list of options, e.g. ``-O bg=light,python=cool``.

The -P option adds lexer and formatter options like the -O option, but
you can only give one option per -P. That way, the option value may
contain commas and equals signs, which it can't with -O, e.g.
``-P "heading=Pygments, the Python highlighter".

With the -F option, you can add filters to the token stream, you can
give options in the same way as for -O after a colon (note: there must
not be spaces around the colon).

The -O, -P and -F options can be given multiple times.

With the -S option, print out style definitions for style <style>
for formatter <formatter>. The argument given by -a is formatter
dependent.

The -L option lists lexers, formatters, styles or filters -- set
`which` to the thing you want to list (e.g. "styles"), or omit it to
list everything.

The -N option guesses and prints out a lexer name based solely on
the given filename. It does not take input or highlight anything.
If no specific lexer can be determined "text" is returned.

The -H option prints detailed help for the object <name> of type <type>,
where <type> is one of "lexer", "formatter" or "filter".

The -h option prints this help.
The -V option prints the package version.

diego@ServerLinux1:/var/opt/phabricator/phabricator$ echo $?
0
diego@ServerLinux1:/var/opt/phabricator/phabricator$

The return code should be 0 ....

Still in the dark...
Thank you for help.
Diego

I have no clue how AppArmor works, sorry -- I just know some users have run into this kind of problem with it.

You can try putting this in phabricator/support/debug.php:

debug.php
<?php

var_dump(exec_manual('pygmentize -h'));

Then visit http://your.install.com/debug/. That will show you the output from the point of view of the webserver -- the three values are exit code, stdout, and stderr. That might be helpful (for example, it will show you if pygmentize is failing because of some Python environmental issue).

If that's not helpful, you can change the command to /usr/local/bin/pygmentize -h or which pygmentize to check for path issues.

(The debug.php file does not normally exist, you'll have to create it.)

Here's an example of the output on my machine:

array(3) {
  [0]=>
  int(0)
  [1]=>
  string(2412) "Usage: /usr/local/bin/pygmentize [-l <lexer> | -g] [-F <filter>[:<options>]] [-f <formatter>]
          [-O <options>] [-P <option=value>] [-o <outfile>] [<infile>]

       /usr/local/bin/pygmentize -S <style> -f <formatter> [-a <arg>] [-O <options>] [-P <option=value>]
       /usr/local/bin/pygmentize -L [<which> ...]
       /usr/local/bin/pygmentize -N <filename>
       /usr/local/bin/pygmentize -H <type> <name>
       /usr/local/bin/pygmentize -h | -V

Highlight the input file and write the result to <outfile>.

If no input file is given, use stdin, if -o is not given, use stdout.

<lexer> is a lexer name (query all lexer names with -L). If -l is not
given, the lexer is guessed from the extension of the input file name
(this obviously doesn't work if the input is stdin).  If -g is passed,
attempt to guess the lexer from the file contents, or pass through as
plain text if this fails (this can work for stdin).

Likewise, <formatter> is a formatter name, and will be guessed from
the extension of the output file name. If no output file is given,
the terminal formatter will be used by default.

With the -O option, you can give the lexer and formatter a comma-
separated list of options, e.g. ``-O bg=light,python=cool``.

The -P option adds lexer and formatter options like the -O option, but
you can only give one option per -P. That way, the option value may
contain commas and equals signs, which it can't with -O, e.g.
``-P "heading=Pygments, the Python highlighter".

With the -F option, you can add filters to the token stream, you can
give options in the same way as for -O after a colon (note: there must
not be spaces around the colon).

The -O, -P and -F options can be given multiple times.

With the -S option, print out style definitions for style <style>
for formatter <formatter>. The argument given by -a is formatter
dependent.

The -L option lists lexers, formatters, styles or filters -- set
`which` to the thing you want to list (e.g. "styles"), or omit it to
list everything.

The -N option guesses and prints out a lexer name based solely on
the given filename. It does not take input or highlight anything.
If no specific lexer can be determined "text" is returned.

The -H option prints detailed help for the object <name> of type <type>,
where <type> is one of "lexer", "formatter" or "filter".

The -h option prints this help.
The -V option prints the package version.

"
  [2]=>
  string(0) ""
}

Great!
With the debug code I (in fact indirectly YOU) solved the issue. This was my output of the ../debug page:

array(3) {
  [0]=>
  int(1)
  [1]=>
  string(0) ""
  [2]=>
  string(2649) "Traceback (most recent call last):
  File "/usr/local/bin/pygmentize", line 9, in <module>
    load_entry_point('Pygments==1.6', 'console_scripts', 'pygmentize')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/local/lib/python2.7/dist-packages/Pygments-1.6-py2.7.egg/pygments/cmdline.py", line 19, in <module>
    from pygments.formatters import get_all_formatters, get_formatter_by_name, \
  File "/usr/local/lib/python2.7/dist-packages/Pygments-1.6-py2.7.egg/pygments/formatters/__init__.py", line 14, in <module>
    from pygments.formatters._mapping import FORMATTERS
  File "/usr/local/lib/python2.7/dist-packages/Pygments-1.6-py2.7.egg/pygments/formatters/_mapping.py", line 19, in <module>
    from pygments.formatters.img import BmpImageFormatter
  File "/usr/local/lib/python2.7/dist-packages/Pygments-1.6-py2.7.egg/pygments/formatters/img.py", line 20, in <module>
    from PIL import Image, ImageDraw, ImageFont
  File "build/bdist.linux-x86_64/egg/PIL/Image.py", line 61, in <module>
  File "build/bdist.linux-x86_64/egg/PIL/_imaging.py", line 7, in <module>
  File "build/bdist.linux-x86_64/egg/PIL/_imaging.py", line 4, in __bootstrap__
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 936, in resource_filename
    self, resource_name
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1567, in get_resource_filename
    self._extract_resource(manager, self._eager_to_zip(name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1595, in _extract_resource
    self.egg_name, self._parts(zip_path)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1002, in get_cache_path
    self.extraction_error()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 982, in extraction_error
    raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg
cache:

  [Errno 13] Permission denied: '/var/www/.python-eggs'

The Python egg cache directory is currently set to:

  /var/www/.python-eggs

Perhaps your account does not have write access to this directory?  You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.

"
}

The .python-eggs directory was not present in /var/www. I created the dir, changed ownership to www-data and voilà, bug solved!

Pygmentize is now working!

Thank you for your fast, precise and effective help!
Diego

Awesome. I think we can improve this setup check so that it reports a more useful error if pygmentize exists and just fails when we try to run it.

That would be very useful. You can also add a warning on the Pygments installation instructions to carefully check that temporary directory and, in any case, double check it's functional using the debug function page you showed me: that was really clear and shot the bug.

chad renamed this task from Pygmentize not working on Ubuntu server installation to Detect if pygmentize exists but fails to execute.Sep 30 2014, 5:58 PM
chad triaged this task as Low priority.
chad added a project: Setup.

It could help a lot to not only say the return code is invalid, but also display its value, and stdout and stderr.