T9805?

like this:

chad@phac-dev:/var/www/html/dev/phabricator$ arc liberate
[2017-02-03 03:39:35] EXCEPTION: (CommandException) Command failed with error #139!
COMMAND
php '/var/www/html/dev/core/lib/libphutil/src/moduleutils/../../scripts/phutil_symbols.php' --ugly -- '/var/www/html/dev/core/lib/phabricator/src/applications/settings/panel/PhabricatorEmailPreferencesSettingsPanel.php'

STDOUT
(empty)

STDERR
Segmentation fault (core dumped)
 at [<phutil>/src/future/exec/ExecFuture.php:369]
phutil(head=master, ref.master=9d85dfab0f53)
  #0 ExecFuture::resolvex(NULL) called at [<phutil>/src/future/exec/ExecFuture.php:391]
  #1 ExecFuture::resolveJSON() called at [<phutil>/src/moduleutils/PhutilLibraryMapBuilder.php:483]
  #2 PhutilLibraryMapBuilder::analyzeLibrary() called at [<phutil>/src/moduleutils/PhutilLibraryMapBuilder.php:77]
  #3 PhutilLibraryMapBuilder::buildMap() called at [<phutil>/src/moduleutils/PhutilLibraryMapBuilder.php:109]
  #4 PhutilLibraryMapBuilder::buildAndWriteMap() called at [<phutil>/scripts/phutil_rebuild_map.php:75]

xhprof isn't needed to reproduce this; it's xhpast that's involved. Maybe. (xhpast is just a simple bison/flex parser/lexer combo so it isn't dependent on PHP internals in any way, and it's what drives the module parsing, etc)

Yup, that's the one @chad. I haven't tracked it down any further with strace or anything -- but if you try running that phutil_symbols.php command a billion times, note it doesn't fail every time, or possibly even at all (try like, for x in $(seq 1 1000); do php ... >/dev/null; echo $?; done | grep 1 or something). But if you run arc liberate again, it *will* fail every time, just on a different file, at a different point.

well i've ruined my dev environment now...

a devastating self-own. we've all owned ourselves.

Just curious -- did you just notice this, or did you upgrade just to test this bug? If it was the first, that's an interesting data point! (that arc liberate is less common than I expected)

If the second -- unless you compiled yourself, I think in most environments, you *can* downgrade pretty easily. I used prepackaged repositories on Ubuntu/Fedora. Uhhh, well, maybe not on Fedora 25 anymore, the php5.6 repo may not be active now... On Ubuntu though there's an actively maintained PHP 5.6 repo, by one of the PHP developers themselves, no less. So maybe my laptop is screwed, but you might still have a chance until I figure out some more information..

I updated to test this bug. I'll have to roll back if I need to add a new class.

Yeah, the sury repository is the one maintained by one of the PHP devs. I think if you just do something like apt remove php71* or apt remove php7.1* you can just reinstall all the php5.6 packages and you'll be dandy, at least as far as arc is concerned. They're all in the same repo, just separated by version suffix.

Also the Fedora repo I'm using has a new PHP 7.1.2 RC and it looks like it has a bunch of bugfixes, too, so I'll try that.

Awwwww yissssssss, looks like PHP 7.1.2 RC1 fixes it! Using the Fedora 25 "PHP 7.1 Remi Testing" repository, 7.1.2 just completely finished a run of arc liberate on Phabricator without problem.

Unfortunately, it's very unclear to me which commit in this giant mergefest is actually responsible for the solution. We may never know.

PHP7.1 segfaults about a dozen times a day for me, every day, with little apparent rhyme or reason to the segfaults.

I can only assume I am being punished for not rushing it into production as soon as possible. Perhaps we can deploy PHP 7.2 to production directly from master without waiting for an actual release.

In T12196#210013, @epriestley wrote:

PHP7.1 segfaults about a dozen times a day for me, every day, with little apparent rhyme or reason to the segfaults.

I can only assume I am being punished for not rushing it into production as soon as possible. Perhaps we can deploy PHP 7.2 to production directly from master without waiting for an actual release.

With all the improved performance characteristics, PHP sometimes goes too fast and drives into a wall accidentally I guess. Maybe if you scream "HOLY COW I'M TOTALLY GOING SO FAST OH F***" every time it faults you'll be given a sign or a vision of when the lashings will stop.

Per above, it sounds like upgrading fixed what you were seeing?

I'm seeing segfaults all over the place, including in arc land -- which does not run lint / arc liberate -- so I think the total scope of "PHP 7.1 segfaults a whole lot" probably isn't restricted to arc liberate. (Curiously, I haven't seen segfaults in arc liberate locally). This is probably worth keeping open anyway since I imagine other users will hit it, and maybe we'll eventually end up chasing these down ourselves. I also haven't upgraded to PHP 7.1.2 myself so maybe that just fixed everything.

If not, the next step is probably to run arc in gdb, try to get a stack trace and then go from there. I gave up on even building PHP7 on OSX though so my current plan is "wait for the segfaults to get bored and go away on their own", since I could easily imagine spending a very long time trying to hunt them down. As far as I know, there isn't too much we can do from the arc side to make this process easier (there isn't a magic dump_a_nice_php_stack_when_you_segfault=true flag or a way to build one, as far as I know, and the PHP stack trace probably wouldn't be terribly illuminating anyway).

Yeah, 7.1.2 RC1 seems to fix the problems. There are like 2 dozen random "uuhh fix off by one error, revert thing that overflowed something, you bastard, fix this too" commits in that commit log, even some affecting stuff like libgd (which you'd think would have been pretty hardened), so it's really impossible to tell what fixed what without it becoming tedious.

As a side note, Phabricator on the server seems to be doing OK with 7.1.1... My daemon logs do indicate one segfault when running repository update, but it's relatively sporadic and the daemons restart it and it moves on (this is when importing a copy of the Linux Kernel). Otherwise I haven't noticed any other issues. Maybe it would be worth testing some of these point releases and reasonably put in the user guide "PHP before 7.x.y has ghosts that will haunt you", or something. Or not, since users might just come to this ticket.

Just as a side note: PHP 7.1.2 was released like a week ago, and I have not seen any more spurious segfaults on my instances when doing imports or anything like that on the server. The arc liberate bug remains dead and I have not seen any more faults in arc either. Builds should now be available in most PHP 7.1 repositories (in multiple distros/systems), so I suggest an upgrade for anyone who's affected.

@epriestley I'm not sure if we should close this. Looking at T12101, this ticket is essentially the only "Random crashes will happen in PHP 7 due to instability" that's open right now, I think. Perhaps this ticket should be kept as a general "Dumping ground for all PHP 7 segfaults", and alternative tickets should be merged into this one -- until the general consensus seems to be the edges are all gone? If so I'll update the original description a bit.

Yeah, I'm basically just keeping this open as a dumping ground for the moment and purely selfishly because my segfaults aren't resolved yet -- brew didn't have PHP 7.1.2 last I checked, although it seems like it now does. I've upgraded and I'll see if that improves things.