Right, try stopping the pull daemons on the other tiers and leave only one running on the repository machine and see if that resolves your issue.

Sure, I get that... But the import is failing even on the instance running without the --no-discovery flag.

It means you don't need to run the taskmaster daemons on the same machines as the repositories because it can just make a Conduit call instead. You no longer need to run separate daemon tiers or use the --no-discovery flag.

I don't think that would fix the deadlock issue, which seems to be the root cause?

You probably need to apply D11874, which fixes this issue.

Seeing this on the repository host:

Here are my diggings in the logs to determine how many diffs are problematic (there are 27)

cspeckrun@specktop ~/D/daemon_logs> grep -o -E "PHID-DIFF-[a-z0-9]{20}" daemons.log | sort | uniq
PHID-DIFF-2tkk6i73v4o32wqfqcsb
PHID-DIFF-4tnlystz7it5w44j7imd
PHID-DIFF-5i5uoael4eeuhdj5hsgw
PHID-DIFF-75mxqtmqo6f7xot6ciar
PHID-DIFF-7kwzsmaymdbdbxxo7lru
PHID-DIFF-c5qhg3kcivjarfwd2wek
PHID-DIFF-cptimb5cxdariuh4xufh
PHID-DIFF-eqo6o5le2phkdlflqyxu
PHID-DIFF-f5fbtt6ap6odesigjbmj
PHID-DIFF-fyfkn25bo6gtkutuepax
PHID-DIFF-fzyhpnm7eb6gkwlo42pf
PHID-DIFF-ghrwxjwtf7x7ngxyctfn
PHID-DIFF-hv7oxcttgnlksvixhya3
PHID-DIFF-ial33nwcdim4it3ne2pd
PHID-DIFF-irahpyhwfl4mkrvwpcpf
PHID-DIFF-j3ucrnpm6ytqhnb6yv4y
PHID-DIFF-lu3vilalrz6bl4e3aexj
PHID-DIFF-lvwde63gotffedkrmqlm
PHID-DIFF-nl2q7sybe77ydc3rnh2d
PHID-DIFF-o4ugvxgy3djdptvi5dn7
PHID-DIFF-pzrxnjtrm4hmbik4d7af
PHID-DIFF-rn3eaygvie4ylbxqxmdd
PHID-DIFF-rxuv3g5g3pibqq7o4gxj
PHID-DIFF-szgjp2bc6znknxaddesg
PHID-DIFF-vohgcadppoygmn3judfy
PHID-DIFF-yyziqbxidx575ey5b3w2
PHID-DIFF-zplnxsntawlxmhu7sag4
cspeckrun@specktop ~/D/daemon_logs> ls -l
total 294424
-rw-r--r--+ 1 cspeckrun  staff  120935264 Dec 17 23:12 daemons.log
cspeckrun@specktop ~/D/daemon_logs> grep -o -E "PHID-DIFF-[a-z0-9]{20}" daemons.log | wc -l
   45645

Noting here, this started happening again within the past day or two. I'm going to cancel the leased tasks again, but would like to find some way to determine what's going on. I suspect there might be some diff's that are no longer around (not sure how) which are causing this. Will grep through the daemon logs at some point.

Assuming this is in good shape now.

Phabricator newbie here. This error message doesn't really make any sense without Phabricator experience.
Suggesting a way to fix it would be much better.

I'll see if I can come up with a reproduction case.

It would be helpful to understand how to reproduce this. Specifically, I think what's happening is something like this:

Ah! That could definitely explain it.

Hi, a co-worker of @vintrax here.
It seems this happened because I messed up. I wrote a bot that accidentally left hundreds of comments in a single task in the span of 20 minutes. The large amount of comments caused both the a lot of search index jobs to be slow (DELETEs/UPDATEs on a big MYSQL record) and big in amount (an index job per comment). To top it off these jobs couldn't be run in parallel efficiently because of search_documentfield table locks (because it's MyISAM). This caused the queue to still be clogged more than 24 hours after the 'bot mishap' happened, which is why I initially did not think this had caused the issues

Wauw, Evan, thank you so much for your debugging effort, for sharing your thoughts and your elaborate explanation. It's (seriously) highly appreciated!

I can't reproduce this. Here's what I did:

You are about 4 months out of date.

Do you have any steps we can use to reproduce the bug? See Contributing Bug Reports.

Some more information:

For what it's worth, we have a very similar setup (40ish repositories in Phabricator, all hosted in Gitlab). We have not observed any performance issues such as those described in this ticket.

Since we don't have access to whatever GitLab is measuring, we can't understand this issue in greater detail. Particularly, we can not reproduce the underlying heavy load in a way we can examine. We have not seen similar reports from other users, except @jcarrillo7, who has not filed any details. We have many installs with hundreds of repositories and my expectation is that the operation is cheap, so I suspect this may be an issue unique to GitLab.

Sounds like you could easily fork and change the polling frequency; The calculation is at https://secure.phabricator.com/diffusion/P/browse/master/src/applications/repository/storage/PhabricatorRepository.php$1527, and you can drop it to an hour just by changing that line.

I am not sure of the underlying implementation of GitLab, but our phabricator instance has been flagged as the #1 user of GitLab resources. Our GitLab instance hosts thousands of repositories, so being the single largest user of GitLab is concerning since we only have ~50 repositories.

huge amount of traffic

At that scale, is there any reason to not use Phabricator's repository hosting instead of GitLab? You can mirror repositories from Phabricator to GitLab if you still need some GitLab functionality (see the Mirrors option under Edit Repository).

Oops! Thats embarrassing. These numbers were gathered from multiple places/multiple people, and mass confusion ensued. The 2Hz number came from the total requests to GitLab instead of requests per repository. Ill update the title/description accordingly.

I'm not sure I understand your math -- here's what I get:

@epriestley I know you mentioned that a diffusion.looksoon would trigger this, but we have no known users of this command.

Only due to the fact that it failed 250 times and got removed by Phabricator.... I suspect it will happen again the next time that file changes.

It sounds like this no longer reproduces for you.

sadly I can't run this

bin/worker execute --id <id>

since the task has been archived

I have metamta.differential.inline-patches set to 100
metamta.email-body-limit is set to the Global Default 524288

I can't immediately reproduce this. Here's what I did:

In T9803#144799, @epriestley wrote:

The default value of metamta.email-body-limit is 512KB. Does your server reject 512KB messages, or have you raised this limit, or do you expect 512KB messages to send cleanly?

The default value of metamta.email-body-limit is 512KB. Does your server reject 512KB messages, or have you raised this limit, or do you expect 512KB messages to send cleanly?

I just noticed the Diff UI also attempts to show this file.

The particular file in question was a .yuv file which is used for video data format file.

I ran this command out of curiosity:

As a side work around, is there anyway to "complete" this task or kick it off the queue? It really is not that important if one email fails to be sent.

In T7053#144095, @epriestley wrote:

This also seems to work properly on this host, although I hit the "daemons are not running" setup warning once, presumably by racing their restart. If this is an issue we can tweak how that warning works, I think.

In T7053#144090, @epriestley wrote:

• When editing the small amount of config which can't auto-restart (phd.user, phd.start-taskmasters). I think phd.user already has an existing warning anyway.

Yeah, I would like to handle this properly but I'm not sure if it is worthwhile doing so?

This also seems to work properly on this host, although I hit the "daemons are not running" setup warning once, presumably by racing their restart. If this is an issue we can tweak how that warning works, I think.

One possible minor followup to maybe keep on the radar is showing some extra warnings in a few cases:

We now have execute as well.

Based on that screenshot in the log, looks like you're running lots of other extensions?

Sorry, my problem is, that I get this:

@tycho.tatitscheff is correct, it's very hard to provide you with help on any of your questions because they lack our expected information for reports. We really want the community to be mindful we're a very small development team and going on "fishing expeditions" to provide free support is taxing on us (and likely to have us not help you at all). Please read up on https://secure.phabricator.com/book/phabcontrib/article/bug_reports/.

Can you gives more info/context. It is hard help without.

Ok, after what you said I stopped phd and went to sudo su, and tried again. After storage adjust and phd start seems like everything working. Thank you :).

I have changed that to en_US.UTF-8 and then destroy and upgrade the storage but still

Also, it's possible that ./bin/storage is running in one locale, and the webserver is running in another; That would cause the most harm.

Well... yeah, that's I is against regulations.

and php -r "print strtolower('ABCDEFGHIJKLMNOPQRSTUVWXYZ');"

Can you run locale and show me the output?

Ok sorry, this:

Navigate there in the web interface
http://your.install/config/all/

My phabricator folder is at ~/Phabricator/phabricator/bin .... etc. but I couldn't find any config/all or config/module/versions folders, I was looking for them :/

and Phabricator versions?

What versions of mysql, php are you using?

Ah that was it. I increased that and also had to increase the innodb_log_file_size.

What is your max_allowed_packet set to in MySQL?