PHP7, particularly before 7.1.2, segfaults a whole lot while running CLI scripts
Closed, ResolvedPublic

Description

NOTE: I'm still looking into this, and it's not clear to me if this is some kind of environmental bug with my PHP builds, a problem with Phabricator, or even something else.

When using arc with PHP 7.1, and I attempt to run arc liberate on the Phabricator repository (e.g. because I wrote a patch), the process will mysteriously crash due to a segmentation fault in the underlying xhpast binary (I assume), causing the overall liberation process to fail spectacularly. This happens when xhpast is being used to enumerate and process the symbols/class dependencies of the PHP code, so that e.g. arc lint can tell you "hey, I can't resolve that symbol", etc.

I strongly believe this is a PHP 7.1-related flaw, though I have not investigated completely. The reason I think this is because I was able to reproduce it on two completely separate Linux systems (Fedora 25 and Ubuntu) with radically different toolchains/environments, yet I was able to concretely reproduce the failure on both of them. I have never seen this on any PHP 5.x related build, even on recent Fedora/Ubuntu machines (I only recently started using PHP 7.1 on my brand-new laptop).

Note this is made more complex by the fact the segmentation fault occurs seemingly randomly; it does not always fault on the same file, but it always eventually faults, with a probability of 1, on some random file. Even if I run the xhpast subcommand that crashed on a specific Phabricator file like 1,000 times repeatedly in a loop, I never get a failure twice. But I can always cause it to crash, no matter what, if I just run it and wait. Sometimes arc liberate fails 10% in, other times 20%, etc.

I'm reporting this from my desktop so I'll attach some traces and more information when I move over to my laptop later.

chad added a subscriber: chad.Feb 3 2017, 3:40 AM

like this:

chad@phac-dev:/var/www/html/dev/phabricator$ arc liberate
[2017-02-03 03:39:35] EXCEPTION: (CommandException) Command failed with error #139!
COMMAND
php '/var/www/html/dev/core/lib/libphutil/src/moduleutils/../../scripts/phutil_symbols.php' --ugly -- '/var/www/html/dev/core/lib/phabricator/src/applications/settings/panel/PhabricatorEmailPreferencesSettingsPanel.php'

STDOUT
(empty)

STDERR
Segmentation fault (core dumped)
 at [<phutil>/src/future/exec/ExecFuture.php:369]
phutil(head=master, ref.master=9d85dfab0f53)
  #0 ExecFuture::resolvex(NULL) called at [<phutil>/src/future/exec/ExecFuture.php:391]
  #1 ExecFuture::resolveJSON() called at [<phutil>/src/moduleutils/PhutilLibraryMapBuilder.php:483]
  #2 PhutilLibraryMapBuilder::analyzeLibrary() called at [<phutil>/src/moduleutils/PhutilLibraryMapBuilder.php:77]
  #3 PhutilLibraryMapBuilder::buildMap() called at [<phutil>/src/moduleutils/PhutilLibraryMapBuilder.php:109]
  #4 PhutilLibraryMapBuilder::buildAndWriteMap() called at [<phutil>/scripts/phutil_rebuild_map.php:75]
thoughtpolice added a comment.EditedFeb 3 2017, 3:40 AM

xhprof isn't needed to reproduce this; it's xhpast that's involved. Maybe. (xhpast is just a simple bison/flex parser/lexer combo so it isn't dependent on PHP internals in any way, and it's what drives the module parsing, etc)

Yup, that's the one @chad. I haven't tracked it down any further with strace or anything -- but if you try running that phutil_symbols.php command a billion times, note it doesn't fail every time, or possibly even at all (try like, for x in $(seq 1 1000); do php ... >/dev/null; echo $?; done | grep 1 or something). But if you run arc liberate again, it *will* fail every time, just on a different file, at a different point.

chad added a comment.Feb 3 2017, 3:45 AM

well i've ruined my dev environment now...

a devastating self-own. we've all owned ourselves.

Just curious -- did you just notice this, or did you upgrade just to test this bug? If it was the first, that's an interesting data point! (that arc liberate is less common than I expected)

If the second -- unless you compiled yourself, I think in most environments, you *can* downgrade pretty easily. I used prepackaged repositories on Ubuntu/Fedora. Uhhh, well, maybe not on Fedora 25 anymore, the php5.6 repo may not be active now... On Ubuntu though there's an actively maintained PHP 5.6 repo, by one of the PHP developers themselves, no less. So maybe my laptop is screwed, but you might still have a chance until I figure out some more information..

chad added a comment.Feb 3 2017, 3:52 AM

I updated to test this bug. I'll have to roll back if I need to add a new class.

chad added a comment.Feb 3 2017, 3:52 AM
chad@phac-dev:/var/www/html/dev/phabricator$ php -v
PHP 7.1.1-1+deb.sury.org~xenial+1 (cli) (built: Jan 20 2017 09:20:20) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.1.1-1+deb.sury.org~xenial+1, Copyright (c) 1999-2017, by Zend Technologies

Yeah, the sury repository is the one maintained by one of the PHP devs. I think if you just do something like apt remove php71* or apt remove php7.1* you can just reinstall all the php5.6 packages and you'll be dandy, at least as far as arc is concerned. They're all in the same repo, just separated by version suffix.

Also the Fedora repo I'm using has a new PHP 7.1.2 RC and it looks like it has a bunch of bugfixes, too, so I'll try that.

Awwwww yissssssss, looks like PHP 7.1.2 RC1 fixes it! Using the Fedora 25 "PHP 7.1 Remi Testing" repository, 7.1.2 just completely finished a run of arc liberate on Phabricator without problem.

Unfortunately, it's very unclear to me which commit in this giant mergefest is actually responsible for the solution. We may never know.

iiam

PHP7.1 segfaults about a dozen times a day for me, every day, with little apparent rhyme or reason to the segfaults.

I can only assume I am being punished for not rushing it into production as soon as possible. Perhaps we can deploy PHP 7.2 to production directly from master without waiting for an actual release.

PHP7.1 segfaults about a dozen times a day for me, every day, with little apparent rhyme or reason to the segfaults.

I can only assume I am being punished for not rushing it into production as soon as possible. Perhaps we can deploy PHP 7.2 to production directly from master without waiting for an actual release.

With all the improved performance characteristics, PHP sometimes goes too fast and drives into a wall accidentally I guess. Maybe if you scream "HOLY COW I'M TOTALLY GOING SO FAST OH F***" every time it faults you'll be given a sign or a vision of when the lashings will stop.

epriestley renamed this task from `arc liberate` fails on Phabricator mysteriously with PHP 7.1 to `arc liberate` fails on Phabricator miraculously with PHP 7.1.Feb 3 2017, 1:15 PM

Per above, it sounds like upgrading fixed what you were seeing?

I'm seeing segfaults all over the place, including in arc land -- which does not run lint / arc liberate -- so I think the total scope of "PHP 7.1 segfaults a whole lot" probably isn't restricted to arc liberate. (Curiously, I haven't seen segfaults in arc liberate locally). This is probably worth keeping open anyway since I imagine other users will hit it, and maybe we'll eventually end up chasing these down ourselves. I also haven't upgraded to PHP 7.1.2 myself so maybe that just fixed everything.

If not, the next step is probably to run arc in gdb, try to get a stack trace and then go from there. I gave up on even building PHP7 on OSX though so my current plan is "wait for the segfaults to get bored and go away on their own", since I could easily imagine spending a very long time trying to hunt them down. As far as I know, there isn't too much we can do from the arc side to make this process easier (there isn't a magic dump_a_nice_php_stack_when_you_segfault=true flag or a way to build one, as far as I know, and the PHP stack trace probably wouldn't be terribly illuminating anyway).

Yeah, 7.1.2 RC1 seems to fix the problems. There are like 2 dozen random "uuhh fix off by one error, revert thing that overflowed something, you bastard, fix this too" commits in that commit log, even some affecting stuff like libgd (which you'd think would have been pretty hardened), so it's really impossible to tell what fixed what without it becoming tedious.

As a side note, Phabricator on the server seems to be doing OK with 7.1.1... My daemon logs do indicate one segfault when running repository update, but it's relatively sporadic and the daemons restart it and it moves on (this is when importing a copy of the Linux Kernel). Otherwise I haven't noticed any other issues. Maybe it would be worth testing some of these point releases and reasonably put in the user guide "PHP before 7.x.y has ghosts that will haunt you", or something. Or not, since users might just come to this ticket.

epriestley renamed this task from `arc liberate` fails on Phabricator miraculously with PHP 7.1 to PHP7, particularly before 7.1.2, segfaults a whole lot while running CLI scripts.
epriestley added a subscriber: der.internethering.

Just as a side note: PHP 7.1.2 was released like a week ago, and I have not seen any more spurious segfaults on my instances when doing imports or anything like that on the server. The arc liberate bug remains dead and I have not seen any more faults in arc either. Builds should now be available in most PHP 7.1 repositories (in multiple distros/systems), so I suggest an upgrade for anyone who's affected.

@epriestley I'm not sure if we should close this. Looking at T12101, this ticket is essentially the only "Random crashes will happen in PHP 7 due to instability" that's open right now, I think. Perhaps this ticket should be kept as a general "Dumping ground for all PHP 7 segfaults", and alternative tickets should be merged into this one -- until the general consensus seems to be the edges are all gone? If so I'll update the original description a bit.

Yeah, I'm basically just keeping this open as a dumping ground for the moment and purely selfishly because my segfaults aren't resolved yet -- brew didn't have PHP 7.1.2 last I checked, although it seems like it now does. I've upgraded and I'll see if that improves things.

epriestley closed this task as Resolved.Apr 7 2017, 12:55 PM
epriestley claimed this task.

The segfaults appear to have subsided for me so I'm just going to close this as "upgrade to PHP 7.1.2".