Ok thanks -- here are the leased tasks which I'll cancel:

Alright. In the short term, you can bin/worker cancel --id <id> any tasks with a bunch of failures to stop that specific task from retrying. Presumably a handful of problem tasks are causing the majority/entirety of the issue.

In T9603#141211, @epriestley wrote:

Have you used bin/remove destroy to delete any diffs or revisions?

Have you used bin/remove destroy to delete any diffs or revisions?

The Mercurial upstream fixed this but we need to retain the (1 or 2) or (3 or 4) code for backward compatibility anyway, so I don't plan to remove it until we run into a real problem it causes other than "it is icky".

Okay. This report doesn't appear to describe a reproducible problem. See:

may be i was to lazy to wait enough. it was a repo with 40.000+ changeset. and one more thing i add the repo with V callsign then delete it and add again with the same callsign. but may be it just dump the exception in the log many thousand time and i've to wait more.

Okay, here's what I did to try to reproduce this:

yes i'n certain!
it was running all night and at the morning it's still running. when i look into the daemon's queue i see the reason is failed for each job so will repeat later in an exponential delay time.

Are you certain these tasks aren't being removed automatically? The expectation is that they will naturally fail over time when they are unable to load the corresponding commits.

Phurl is a prototype application. See:

I can't reproduce this:

I've experienced this too. When credentials are wrong, it should at least give an option to correct them.

We haven't seen reports about this from other installs or experienced it ourselves, and we've run an instanced production cluster with a mixture of workloads for more than 6 months now. All the daemon stuff has also changed a lot over the course of the last two years since this was originally reported.

From discussion on IRC, I think this was a UI issue (these parsers do not block the queue, it just isn't clear that "Leased" includes tasks being actively worked on and tasks waiting on lease expiry), not a queue issue? We can improve this UI but I'm probably not going to touch it until I can collect some more stuff there (cf T7307).

Oh, I didn't look at the stack trace carefully when dealing with T9458. I think the fix for that (D14142) almost certainly resolved this, too.

Thanks for looking into this for me. I ended up totally re-installing everything manually (or not really manually but using Homebrew) and that seemed to work much better. Something about MAMP must have been causing all kinds of problems.

I can't reproduce this on any of the systems I have access to (although none are replicas of your system) and, in general, there is nothing Phabricator should be able to do in PHP to cause a process to segfault (except maybe willfully sending ourselves signals). With very rare exception, segfaults are caused by PHP itself or an extension, not by us. I also haven't seen other reports of anything similar.

Actually, I can't immediately reproduce this. Here's what I did to set the test up:

My previous DELETE queries just reduced amount of tasks created, but I'm still getting few of those tasks.

We can and should reduce the amount of memory required here -- I believe it's roughly O(N^2) because we load every user for every user, in order to respect visibility policies. We can reduce it to O(N) by dumping the handles after building the mail.

Thank you for full explanation.

Here's my best effort to figure out what's happening here:

@epriestley any thoughts? This issue is blocking entire queue.

Execution log: P1855

Er, execution log, I suppose.

What happens earlier in the stack trace?

Better yet: Instead of mentioning the PID, link to /daemon/log/.../, it already contains the necessary information (except the UID). There actually is no need for the lengthy list of "expected/actual/started" usernames, which I suggested, if you provide a list of affected daemons with links to their log.

This was false alarm. key_chronological index was not added on feed_storynotification table on production db (but was added on local). I'm sorry for wasting your time :-)

This appears to have been fixed: I've updated to a newer revision and the problem no longer occurs.

What if we do this?

I get some odd results locally, in MySQL too, although I have a very small table on my laptop (~1100 rows):

I doubt it is Percona issue. I've just tried perform same queries on my laptop with MySQL 5.6 and small data set. Still getting the same results as above.

The fact that it's not using the key at all explains why the GC was slow for you and why removing the ORDER BY improved performance, at least, but it seems like it should be able to use the key to execute this query efficiently, and appears to do so in MySQL (we don't have as much data as you do, but I don't recall the GC having any issues when we pushed this change here, and other large installs haven't reported issues but I'd expect them to if the GC was not able to use the key).

I'm tentatively inclined to think that this is a bug in Percona, since:

Getting the same as above still.

Yuck. Can you come up with any query where it uses the key? What if we do something really dumb like this to try to trick it?

Same

mysql> EXPLAIN SELECT * FROM feed_storynotification WHERE chronologicalKey < 6175228766710272490 ORDER BY chronologicalKey DESC LIMIT 100;
+----+-------------+------------------------+-------+---------------+------------+---------+------+---------+------------------------------------------+
| id | select_type | table                  | type  | possible_keys | key        | key_len | ref  | rows    | Extra                                    |
+----+-------------+------------------------+-------+---------------+------------+---------+------+---------+------------------------------------------+
|  1 | SIMPLE      | feed_storynotification | index | NULL          | userPHID_2 | 133     | NULL | 2941950 | Using where; Using index; Using filesort |
+----+-------------+------------------------+-------+---------------+------------+---------+------+---------+------------------------------------------+
1 row in set (0.00 sec)

Out of curiosity, what's the query plan for this (DESC instead of ASC)?

I'm getting different results:

Hmm, that's surprising. At least on our dataset, I get a worse-looking query plan without the ORDER BY:

Yes, I've solved this way. chronologicalKey already existed on the table but we had over 13 million records there. And performing DELETE query took quite long. Apparently it locked inserts to feed_storynotification table what caused deadlocks.

Yes, this looks consistent with the GC and publisher deadlocking.

Alternate style:

A daemon is running as user //observed_username//, while the Phabricator config specifies phd.user to be //expected_username//.

The 3rd paragraph is outright confusing:

phd Daemons will try to use `root` to start as the configured user. Make sure that the user who starts `phd` has the correct sudo permissions to start `root` daemons as `phd`

My previous Phabricator version was early 2015-May. I see, a bit later FeedStoryNotificationGarbageCollector was introduced (D13233).

Looking T9080: Service-based repositories in Phacility cluster fail too abruptly if cloning fails this isn't fixed for me. Until the UI allows you to configure the repository, I don't think this should be closed.

This appears to be resolved. It's been in master for about a week without issues and deployed to the cluster for about 48 hours.

I'm happy to take a crack at this, but I'm not sure what the desired behavior is... Silently drop notifications to users that don't have the correct permissions? Filter the list of users to notify before triggering notifications?

FWIW The particular commit that was causing trouble eventually (!?!) was processed. Since I could reproduce multiple times times with 4x the RAM this feels spooky. Maybe the memory consumption is non-deterministic? Or somehow very different when running in the daemon instead of the cli?

T8612 is related.

On a side note, does anyone know of a way to clear (PhabricatorApplicationTransactionPublishWorker) tasks from the daemon queue? These will fail forever on my system since this user does not have access. Multiple have already failed over 6000 times to send the email.

@epriestley Did a little more digging in phabricator and it seems to be trying to email an employee who has a herald rule that emails him on any commit anywhere (he doesn't have access to all repositories though).

well, I can still see the same error with new tasks.

OK, as @shrinidhirao suggested, the fix only works for new tasks. I canceled all failing tasks and wait to see how new tasks work.

I have the same error with the latest code even after I applied the fix from @shrinidhirao.