Voila.

You can also safely cherry-pick that to test if it resolves things, it doesn't interact with anything else.

That is: rP65c1c758ed261f65c8164d200ce01373ae5b822c

If you upgrade past 65c1c758ed261f65c8164d200ce01373ae5b822c (both stable and master have this change), does it reproduce?

(And again, to emphasize, I'm happy to drop this if y'all aren't interested. I have my solution and am happy ignoring this, but just thought it might be something you guys would want to check out.)

Head as of my Dec. 17 post in this thread. Can roll forward, but was hoping to wait till the edit engine migration settled a bit.

What version of Phabricator are you currently running?

@epriestley any suggestions?

I did some further digging and made a little progress

This should do it:

Yep, here is the default value on all the installs of PHP I've done from source:

Am glad to get ahead of it. What's the exact incantation we need to put into our php.ini?

(After T12071 we will almost certainly require "E", so you could get ahead of that now.)

Actually, it looks like PHP defaults to "EGPCS" but ships with config files that use "GPCS" (??? why would you do this ???), so this is probably not something that got changed or configured intentionally or accidentally.

Perfect, thanks! That's what I expected, so D17148 should resolve this. Alternatively, setting that value to "EGPCS" (without upgrading, just restarting the daemons) should also resolve this.

While I'm chasing this down, can you show me the output of this command?

This is an unintended side effect of D17100 but resolution isn't straightforward.

I think the desired behavior is to silently drop the notifications, yes. Specifically, they should be dropped in PhabricatorMailReplyHandler->filterRecipientUsers().

Reference in PHP 7.1 migration guide: http://php.net/manual/en/migration71.new-features.php#migration71.new-features.asynchronous-signal-handling

I've been playing with it for maybe half an hour, and couldn't reproduce:

• Made revision public
• stopped the daemons
• mention a user
• updated the revision to a new diff
• made the revision not-visible to that user (visible only to the acting user)
• start daemons and let them finish the queue

Thanks for the advice. That yields the trace below. My understanding is that it was a diff with policy set to subscribers which had an author plus a single subscriber. That subscriber was added by herald upon the diff's creation, so there's certainly grounds for something going on there. I had trouble divining the exact issue, but tinkering with the permissions for the object ended up letting the emails go through.

We used to have a tool to show a task's data, but it must have gotten lost in some refactoring.

• In Daemons, find the task ID in the leftmost column of "Leased Tasks".
• Run bin/worker execute --id <id> --trace to execute the task in the foreground.

That will give you a more detailed error information which you can paste here.

Just updated to head and am seeing the same issues.

When this is in stable and we upgrade I will have a reminder to keep an eye on logs and come back here

Although it's still a little early to be sure, we appear to have fixed the policy issue with unit test failures (D17029). Not sure if that actually impacted this or not.

(Feel free to follow up with reproduction steps once you've sorted everything out)

You have to be up to date and you must provide reproduction information in order to file a bug report. Please use Ponder otherwise. Thanks!

Hmmm I'm not seeing those lines in that file anywhere. On the current head of master, I'm pretty sure the viewer is always set to PhabricatorUser::getOmnipotentUser() in TransactionPublishWorker. Although maybe that isn't making it all the way to the policy aware query in every case.

@jcox I just noticed this line from you:

@jcox, I dove into this a bit, but like you, I didn't come up with much. We have not used bin/remove destroy for any of our repos and there doesn't appear to be any meaningful relationship between the failing tasks that I have found yet. The script that @epriestley kindly provided was able to load any diff I threw at it. The script was helpful in determining the right queries to make to investigate further.

See Consulting if you'd like me to help you personally. Thanks!

Just checked this morning and many tasks have returned and started to fail again. I'm sure there's a way to reproduce this, but our setup is a bit complex and I'm not sure where to start. @epriestley Could you tell me how to determine what these tasks are even doing? Is there a way to see what action triggered them and why they will fail indefinitely?

@jessjohnson I still haven't had a chance to dig further into this, but I'll report back when I do (or if you figure anything out, please let me know). I don't think killing off the tasks permanently is really an option since this seems to happen for specific degenerate instances of otherwise-needed tasks. Perhaps not allowing these tasks to retry would be an okay bandaid, but I'd still like to figure out the underlying problem if we can.

Also, like @jcox, the diffs always load in that advanced space-age script like there's no problem. Perhaps these exceptions are aware of our attempts to end them. @epriestley Do you have an idea of something else we could try to see why so many exceptions are occurring?

The upstream still doesn't have a reproduction case; we won't act until we have one. See Providing Reproduction Steps.

We're also seeing about 100k of these failures in our logs over the past few weeks. I clear out all the tasks and they quickly return. Have you found the root cause?

Ah yeah it could be related to that (policy errors/unit test failures), although I wouldn't really describe this as "extremely rare" (except of course when I'm trying to make it happen). Some stats from grepping through logs: since last week it looks like this error has happened 73k times involving 193 distinct diffs. It's like the daemons are laughing at my inability to repro it.

I think there is an extremely rare policy error somewhere in the policy code that we've occasionally seen in the form of sporadic unit test failures. I haven't been able to reproduce them in the same environment and they're false negatives (e.g., prevent access when it would be OK to allow access) so I haven't spent a ton of effort hunting them down, but it's possible that they have the same cause.

Just spent a bit of time trying to repro this in a script but could not. The diffs always load successfully from the script. I even tried waiting until right after one of the errors popped up in the logs and then queried for that diff via the script right away.

Also possible is stuff like "the diff has a custom policy which you destroyed", etc.

If an object exists but is failing to load, it's usually because a required related object does not exist (in most cases, "required" means "required in order to evaluate policy checks").

I believe I've seen these exceptions for the past year or so on our setup as well, though I hadn't actually confirmed the diffs do exist. I did confirm that nobody ever manually deleted diffs though.

Thanks!

Yeah, ON DUPLICATE KEY UPDATE is sensitive to order. Setup:

I think this is:

I clicked "Update Now" and the update frequency for our most active repository has dropped back down to 15 seconds.

This went out a while ago and I confirmed the fix in production.

I think "Waiting" doesn't actually count as "Running", because it's "Waiting to run again" after an exception made the daemon die.

I have installed Phabricator from official documentation and get the same problem. I didn't create a bug report as I don't know the production steps. The problem occurred when I have upgraded Phabricator in July.

We don't provide support for other installation guides. Please contact that maintainer or install Phabricator from our official documentation,

It's a persistent problem. I've already tried to restart the deamons, the OS itself, but the problem persists. Phabricator was installed in a CentOS 7.2 system following the steps in this link: https://www.vultr.com/docs/how-to-install-and-configure-phabricator-on-centos-7

Part of this is likely a change to the semantics of setTimeout(). Current uses are:

FWIW, PHP 7.1.0 RC1 was released today, and it includes the async signal change. The following amended script from T11270#184087 seems to work:

Hi, we're still strugling with this issue, anybody had to deal with this already ? Is there a plan on optimizing commit parsing and daemons scalability ?

The particular issue is that that the change removed a file containing the bytes ... 0xE5 0x76 .... The 0xE5 is probably an å in Latin 1.

I extracted the response body prior to encoding as a serialized()'d object so I can narrow down what's causing the encoding to fail and it looks reasonable, looking like it's almost certainly a UTF8 issue now.

Not too surprisingly, the response is empty. AphrontResponse->encodeJSONForHTTPResponse() uses json_encode() and does not detect UTF8-related encoding failures, so the chance that this is a UTF8 issue is pretty high. We should also probably fix that so fail earlier and more explicitly (and don't emit HTTP 200).

The next step I want to take is figuring out what the actual problematic response body looks like, but that's slightly tricky right now because there's no way to make Conduit calls from the CLI.

I believe this is resolved with modern clustered repositories. secure has been spread across two hosts for some time now without encountering any of this.