Thanks for the quick response!

Update past D16623 (Sep 28). See also T11705.

This is a screenshot of the status page. This can usually be fixed by manually rescheduling an update by either clicking "Update now" or calling "./bin/repository update" in command line.

@jessjohnson I still haven't had a chance to dig further into this, but I'll report back when I do (or if you figure anything out, please let me know). I don't think killing off the tasks permanently is really an option since this seems to happen for specific degenerate instances of otherwise-needed tasks. Perhaps not allowing these tasks to retry would be an okay bandaid, but I'd still like to figure out the underlying problem if we can.

Also, like @jcox, the diffs always load in that advanced space-age script like there's no problem. Perhaps these exceptions are aware of our attempts to end them. @epriestley Do you have an idea of something else we could try to see why so many exceptions are occurring?

The upstream still doesn't have a reproduction case; we won't act until we have one. See Providing Reproduction Steps.

We're also seeing about 100k of these failures in our logs over the past few weeks. I clear out all the tasks and they quickly return. Have you found the root cause?

Hrrm -- what is max_allowed_packet set to in MySQL?

This helped! Thanks for another fast fix.

I have a patch to Arc that will use a non-printable control character (INFORMATION SEPARATOR THREE) instead of | for the separator.

I believe this should now be fixed in HEAD of master. It should promote to stable within 24 hours.

T11744 has a vaguely-related special case where "split + count" was unreasonably slow for very large inputs.

D16683 makes us bail early if the combined length of the old and new lines is larger than 1280 bytes, which is a crude test for approximately two reasonable-length lines of emoji source code.

It looks like this is largely just PHP being PHP. To handle UTF8 combining characters correctly, we're producing a list of strings like this internally as the atoms of comparison:

This should now be fixed in HEAD of master.

I'll look at this tomorrow but can probably figure it out from this. Initial guess is that we're doing this (which is O(big constant * string length)):

Thanks also for the tip about SIGHUP to get a backtrace! Had not known that, and it looks extremely useful.

Sweet -- thanks for the fast fix! I'll try applying it locally to clear out the affected tasks we have, and to make sure any new ones don't get me paged tonight. :-)

The root issue was that we would compute a ProseEditDistanceMatrix (roughly, a Levenshtein distance matrix) for large inputs, but this is O(N*M) in the size of the inputs and both of these inputs had ~2,200 "atoms" at the top prose diff level, so the matrix had ~5M elements.

Oh, one diagnostic tool that I didn't mention is that you can send SIGHUP to a running bin/* command to get a stacktrace of what it's doing dumped to /tmp/phabricator_backtrace_<pid> (usually). This one looks like it is a smoking gun:

Thank you! Not every day someone invites me to cause a crash on their production server. :-D

Awesome, thanks! I'll dig into it and update you once I get somewhere.

Just made P2011, and I think that's a repro -- the "Show Details" link hangs.

Yeah, just take a shot at making a paste here (secure.phabricator.com) with the same degenerate behavior. If you can get us crashing I suspect I can get a fix in about 5 minutes through the power of adding var_dump() all over the place on the production server. If not I have a few ideas but there's a lot of code that could be causing the problem and that trace isn't an obvious smoking gun to me.

Nope -- pstree shows no subprocesses when the worker is stuck.

One other quick diagnostic is: when the worker is stuck, is there a pygmentize subprocess running?

Meaning here on secure.phabricator.com? Sure -- no guarantee it'll work, but I have some guesses I can throw at it.

Can you attempt to create a paste on this install which reproduces the issue? I have a few ideas about what's up, but that's almost certainly the fastest path to a fix if it works.

@HarryOtto, your message is off topic in this thread and is so tenuously related to anything here that it looks like it might be SEO linkbuilding spam, so I've removed it.

Ah yeah it could be related to that (policy errors/unit test failures), although I wouldn't really describe this as "extremely rare" (except of course when I'm trying to make it happen). Some stats from grepping through logs: since last week it looks like this error has happened 73k times involving 193 distinct diffs. It's like the daemons are laughing at my inability to repro it.

I think there is an extremely rare policy error somewhere in the policy code that we've occasionally seen in the form of sporadic unit test failures. I haven't been able to reproduce them in the same environment and they're false negatives (e.g., prevent access when it would be OK to allow access) so I haven't spent a ton of effort hunting them down, but it's possible that they have the same cause.

Just spent a bit of time trying to repro this in a script but could not. The diffs always load successfully from the script. I even tried waiting until right after one of the errors popped up in the logs and then queried for that diff via the script right away.

See Restarting Phabricator, this is required any time you update Phabricator.

This is a PHP error, what PHP are you running, XML is likely missing from it.

You should read Contributing Bug Reports because your report is missing some informations like arcanist version. And I think that upstream is not supporting linters anymore.

behold the dangers of laptop only web development

we're testing changes to Conpherence right now.

I probably caused this now that I think about it

Sounds like everything resolved as expected though!

@born2webdesign Unfortunately nothing about this ticket says "bug report", as in I see nothing about an issue with Phabricator that can be resolved by the upstream. Installation in unique environments can be tricky, but we expect you to resolve these issues on your own or use Ponder here to ask for help from the community. What you're asking falls outside of what we classify as "free support".

@chad Yes, it ist ;)
@asherkin Thanks, yes that is the doc I was linked to, and indeed, putting the following in the preamble solved it:

$_SERVER['HTTPS'] = true;

Sorry, I said it was late ;)
However, I don't consider this a special installation (and other apps on the server don't show such behaviour), so I find it kind of "interesting" that I have to resort to such measures?
Thanks

You got linked to https://secure.phabricator.com/book/phabricator/article/configuring_preamble/, right?

Is this a bug report?

Looks like we use the function to mean "codepoints" in some places in phabricator/.

With no claim that this is the world's most elegant fix, I think this may be a fix:

In the short term, I'll accept a patch to do this if you want to test it?

T9640 and T2383 are somewhat related.

When you upload a file that is stored as chunks, the chunks are also files. For example, file F123 may have chunks F124, F125, F126, etc. These files are not normally visible in the web UI but you can see that the next file you upload will skip some numbers (e.g., be F127).