This should now be fixed in HEAD of master. Thanks for the report!

Great! It looks like we're down from about 18K to 3K on repo001 too, so it appears that this pretty much works as expected.

After merging with upstream, we were able to increase our taskmasters up to 32, and still not come anywhere close to the number of connections that were previously used. So far, so good on our end.

I deployed this stuff, and I'm actually seeing more used ports now on secure001 than I did in D16389. However, I think most of these are coming from web requests and the larger count is because we have more traffic during US daytime than we did earlier in the morning. It didn't occur to me earlier that I'd be accidentally counting connections from both sources.

I have the similar problem.. Sometimes web UI shows setup issue warning "Phabricator Daemons Are Not Running". Then I restart daemons. They work fine for some period (may be a week or two). And then again "Phabricator Daemons Are Not Running" is showed...

(Presuming that fixed things until we see evidence otherwise.)

Alright, that's promising at least. Thanks for testing!

In T11291#185175, @epriestley wrote:

It's possible that we're in the wrong by falling back to set character set utf8;.

For anyone who can reproduce this, does this fail?

mysql> set character set utf8mb4;

I think it would be better not to fallback at all and raise a setup warning. We'll likely have other issues in the future for not using the native driver.

(I made about six typos in the original version of that comment, so reload if you're following the steps from email.)

It's possible that we're in the wrong by falling back to set character set utf8;.

impressive

@eadler, I was just able to resolve this on our installation by installing the php56w-mysqlnd package instead of php56w-mysql, which is what we had installed before.

I'm thinking the problem with our DB can be fixed by this:
https://mathiasbynens.be/notes/mysql-utf8mb4

In T11291#185026, @eadler wrote:

We're seeing an identical issue on our install. What version of MySQL are you using? Maybe we can work together and figure out how to get Emoji to work!

We're seeing an identical issue on our install. What version of MySQL are you using? Maybe we can work together and figure out how to get Emoji to work!

it's definitely the 🎪 that's causing the issue, not the redacted bit. I think it's an issue with our DB, but ./storage status thinks it's fine.

Whatever string this is:

Here are the results of that command:

(The harbormaster_object table is just a test table used by unit tests.)

You could also try running these commands against MySQL:

You can likely reproduce the issue in a repeatable way with this command:

Unfortunately, the repo is confidential source, we might be on our own to debug this one. Perhaps there are issues with our database configuration @edibiase might have more info.

Seems like that commit imported fine.

./storage adjust returned no errors. Pushing to the GITTEST repo gave me the following error:

I can't reproduce this. Bug reports MUST have reproduction steps.

It was merged into master today.

Looks like this should be available in 7.1 btw: https://wiki.php.net/rfc/async_signals#proposed_voting_choices.

Of course! Thanks for poking around!

(And thanks for providing access to an environment where the issue reproduces!)

@Mnkras, I think I've undone everything I did and terminated all sessions on the host. You can revoke my key at your convenience.

I was writing my reply while you were writing the above. Just figured I'd mention it.

Yes, that's synchronous signal handling.

note that in the above we're explicitly handling signals. The signal_dispatch call must be placed inside of some activity loop.

function onsignal($signal) {

echo "Got SIGINT.\n";
  exit;

}

Yes, this reproduces on a local build of PHP 7.0.8. Here's what I did:

I'm building PHP 7.0.8 locally. In the meantime, I found these possibly similar reports:

Here's a smaller, self-contained test case:

I produced this test script to try to see about version issues via 3v4l.org, but you can't write files there. Requiring C as part of a different file is important, so I don't see a way to use that.

I put the global signal handler in run(), unresponsive.

I no-op'd the constructor and put the logic in run(), still unresponsive.

I disabled the entire actual daemon behavior, still unresponsive.

Removing the SIGINT handler entirely made it responsive.

I changed the SIGINT handler to exit immediately, still unresponsive to SIGINT:

This trivial script works correctly:

The PhutilNiceDaemon daemon also does not respond to ^C.

The debug daemon does not appear to respond to an external SIGTERM (via kill <pid> either).

With --trace, the daemon isn't doing any work, and also isn't hung (just normal queries to look at the queue).

I ran bin/phd stop and saw similar behavior:

@Mnkras gave me access to the host.

I can't reproduce that either:

Im really not sure, when I run restart this is what my daemon page looks like:

I can't reproduce this. Here's what I did:

eventually the dead ones appear to fall off the edge, but in the meantime UI doesn't give me a warning that my daemons are not actually running.

Ok, I just installed a new instance of the server in another VM.
Steps to reproduce, as far as I can tell, are:

Bug reports MUST include reproduction steps that allow us to reproduce the issue locally. Without this information, we can not move forward.

There's a very small possibility that D16110 fixed a race condition which was causing this, but that seems like quite a stretch.

You can email me directly at epriestley@phacility.com, or create a private thread in Conpherence on this install via the icon in the header.

I worked on this problem for a couple hours last night, tailing log files
and checking various security features. This is our development server, the
lead Dev and I have complete control over it. I also had this problem on
another machine which is setup very similarly (CloudLinux, but locked done
a ton as its our hosting server), but on that machine I did not put the
Phabricator installation in a cPanel account. Had the same problem there,
so I don't think it's the security / CageFS getting in the way. Something
is timing the script out or killing it, and it's not the Kernel.

D15977 improves some behaviors slightly and adds more extensive logging for signal handling. I don't think I caught anything with real meat to it.

D15975 should fix the (Exception) No such daemon "39084:6ra7uz"! error. I'm not sure if it will do anything for stability or not, but it should reduce noise. I didn't uncover anything interesting in resolving it (it wasn't really even a race, just a fairly straightforward set of state management bugs).

Sure thing. I left the office shortly after posting on Ponder (hence the
answer delay).

This is almost certainly not a resource problem, but I have no idea what it is.