This seems to have quieted down, now.

This change broke the search dialog on tags typeaheads...

Both tables have this key:

It's true that it has been 12 years, but I'm sure fwrite() will start returning an error code when it encounters a permanent, fatal EPIPE error condition soon.

That upstream bug report is really something else.

Ok gotcha.

The "mess" part is the broken PHP. In C, where you can tell what happened, the behavior is all reasonable.

I wouldn't exactly say it's a huge mess. It's just an interrupted system call, which is fairly well documented. See for example http://poincare.matf.bg.ac.rs/~ivana/courses/ps/sistemi_knjige/pomocno/apue/APUE/0201433079/ch10lev1sec5.html. It doesn't exactly help that PHP gives you a broken abstraction over the underlying C implementation.

Here's a script which can hit these behaviors without cloning 9000 working copies in a loop:

PHP 7.3.1 is exiting abruptly for me (no error message, return code 0) on preg_match('(\\G\s+)sS', '');. This occurs in the daemons. I can't reproduce this in a separate test script. I have no idea what's going on.

Let's compile Gentoo nine times (Gentoo flavor is macOS Mojave 10.14.2):

D20049 introduces a piece of code which could be simplified if the minimum version was 5.4.0 or newer, so PHP_QUERY_RFC3986 is available for http_build_query().

In T13224#243817, @epriestley wrote:

@nochum, if you want to try replacing 'pygmentize ... with 'exec pygmentize ... near line 25 of PhutilPygmentsSyntaxHighlighter.php and that fixes it, I'll upstream it. exec will stop dash from leaving an sh -c process running.

Worked perfectly!

(Also: the time limit should be 15 seconds, not 15 minutes! If you're seeing the sh processes die after 15 minutes, something else is likely afoot.)

Yes, it's 15 seconds, I misspoke.

@nochum, if you want to try replacing 'pygmentize ... with 'exec pygmentize ... near line 25 of PhutilPygmentsSyntaxHighlighter.php and that fixes it, I'll upstream it. exec will stop dash from leaving an sh -c process running.

D212: Fix daemon issues caused by Ubuntu's surprising intermediary shell?

It seems the commit didn't fully fix the issue. When pygmentize runs for phabricator, there are processed that are forked from the initial process that are still never killed and they cause the cpu to spike. For example, when running into this issue on our install, we see something like:
#ps aux | grep pygm
user 13190 0.0 0.0 4456 680 ? S 16:46 0:00 sh -c pygmentize -O encoding=utf-8 -O stripnl=False -f html -l 'bash'
user 13191 0.0 0.0 4456 784 ? S 16:46 0:00 sh -c pygmentize -O encoding=utf-8 -O stripnl=False -f html -l 'bash'
user 13192 74.7 0.0 23996 9336 ? R 16:46 0:05 /usr/bin/python /usr/local/bin/pygmentize -O encoding=utf-8 -O stripnl=False -f html -l bash
user 13193 76.7 0.0 23636 9028 ? R 16:46 0:06 /usr/bin/python /usr/local/bin/pygmentize -O encoding=utf-8 -O stripnl=False -f html -l bash

Thanks! I get the same behavior locally, I filed this upstream: https://bugs.php.net/bug.php?id=77496

I can't get MYSQLI_OPT_LOCAL_INFILE to work on secure, either. I tried on secure001 and secure004 (where the database is not local). As far as I can tell, this option doesn't do anything, anywhere, ever?

We're probably done here, but ideally the next steps are:

Maybe another point in favor of this claim is that the option does not work is the behavior of this:

I think that maybe mysql_nonapi.c just overrides the conn->options() call? Near line 269 of PHP 7.2.3:

I can't get MYSQLI_OPT_LOCAL_INFILE to work on secure, either. I tried on secure001 and secure004 (where the database is not local). As far as I can tell, this option doesn't do anything, anywhere, ever? I'm going to look at the source and see if I can figure out what's going on, but I'll back it out of D19998 if I can't find some evidence that it's useful.

I'm unable to get the MySQLi option MYSQLI_OPT_LOCAL_INFILE to actually work. Here's the script I'm using:

It looks like we don't need to do anything about mysql on the CLI since this option is, thankfully, not enabled by default:

I'm more about not living that Homebrew life.

This doesn't help if you're committed to that Apache life, but FWIW I'm using nginx 1.15.8 via Homebrew on Mojave without any issues.

In moving forward here, we're generally moving from manually-configured HMAC keys to automatic ones. This is generally good: it's simpler (less configuration); and I believe almost no one configured the old ones, so installs now actually get unique HMAC keys; and the new keys have more entropy, too.

...it is virtually inconceivable to me that this is something we would ever bring upstream in any form.

Yeah, that's T10769.

In T13219#242865, @joshuaspence wrote:

I am seeing a similar issue on our install:

[2018-12-13 12:20:12] EXCEPTION: (PhabricatorClusterImproperWriteException) Unable to establish a write-mode connection (to application database "phabricator_repository") because Phabricator is in read-only mode. Whatever you are trying to do does not function correctly in read-only mode. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:119] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
  #0 PhabricatorLiskDAO::raiseImproperWrite(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:60]
  #1 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructre/storage/lisk/LiskDAO.php:1011]
  #2 LiskDAO::establishConnection(string) called at [<phabricator>/src/applications/repository/stora... (619 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:380]
[13-Dec-2018 12:20:12 Etc/UTC] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
[13-Dec-2018 12:20:12 Etc/UTC]   #0 <#3> ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:446]
[13-Dec-2018 12:20:12 Etc/UTC]   #1 phlog(PhutilProxyException) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:453]
[13-Dec-2018 12:20:12 Etc/UTC]   #2 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:222]
[13-Dec-2018 12:20:12 Etc/UTC]   #3 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[13-Dec-2018 12:20:12 Etc/UTC]   #4 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]

I am seeing a similar issue on our install:

There are probably some stragglers that have yet to turn up, but we appear to have survived this largely unscathed.

Ah, thanks! Yeah, that looks like the same thing. I think the bug title isn't quite correct (runtime is dramatically worse than O(input), but not infinite) and the analysis is slightly misleading (escaping the $ doesn't matter, I think) but the patch looks like it will get the job done (although, offhand, I'm unsure about the ($|(?=")|(?=\$) groups).

FYI, I think this upstream issue with pygments addresses this issue (was open since 2016, but only recently had a commit made to fix it): https://bitbucket.org/birkenfeld/pygments-main/issues/1257/infinite-loops-in-specific-files

The next step here is to find (almost) every instance of %T in the codebase and convert it to %R. In most cases, the conversion is trivial:

I'm going to start landing this stuff now. master will start complaining about unsafe queries all over the place (although much less frequently than it was when I first added the warning). Depending on how much complaining still exists on Friday I might make the warning developer-only, but I'm currently hopeful that I can clean up most of it before the next release promotes.