We need specific reproduction steps to move forward, and "a repository with about 2800 commits" isn't sufficiently specific.

Please provide the URI of a repository which reproduces the problem which we can clone in a test environment.

Unless you can make this task private to a group of people who would be prepared to sign a NDA in French, that's not going to happen.

What would work though, is to create a repo where I could import an existing, open source project, and see how it fares. I'm thinking maybe TortoiseHg. Would that work?

An existing, open source project is perfectly fine, provided you first verify that it reproduces the problem.

(A synthetic repository which you create by running hg commit in a loop, or instructions which allow us to build such a synthetic repository, are also fine.)

Noted. I'll try and reproduce the problem with a different repository.

The repository at https://bitbucket.org/LeCoyote/brokentest is enough to see unusual CPU usage. Of course, it is nowhere near as bad as the other one, but the CPU is still coasting a noticeable 2 or 3% above average, which seems disproportionate for such a small repo. Also, when calling the update manually:

$ time ./bin/repository update 3
Updated repository "cieltest".

real    0m0.934s
user    0m2.500s
sys     0m0.268s

(edited after creating dummy branches, total user time jumped from .8s to 2.5s)

Here's what I did to try to reproduce this:

• I reused the host I recently built in T12112 (debian001.epriestley.com), see that task for setup steps.
• I installed hg:

$ sudo apt-get install mercurial

• I created a new, empty Mercurial repository, and a working copy directory for the repository.
• I configured phabricator.base-uri and diffusion.allow-http-auth.
• I set a VCS password in Settings.
• I ran bin/repository update 1 to verify that the config was in good shape.
• I configured phd.user to admin.
• I allowed www-data to sudo to hg in /etc/sudoers.

www-data ALL=(admin) SETENV: NOPASSWD: /usr/bin/hg

• I cloned the provided test repository:

$ hg clone https://bitbucket.org/LeCoyote/brokentest

• I pushed the repository to the server:

$ hg push http://debian001.epriestley.com/diffusion/1/test-repository/
pushing to http://debian001.epriestley.com/diffusion/1/test-repository/
http authorization required for http://debian001.epriestley.com/diffusion/1/test-repository/
realm: Phabricator Repositories
user: admin
password: 
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 12 changesets with 12 changes to 12 files (+9 heads)

• I started the daemons on the server:

$ ./bin/phd start

• I waited for the repository to fully import:

• I monitored load on the server for a few minutes and saw load below 1 with the CPU 99% idle:

$ while [ true ]; do date; uptime; mpstat -P 0 | tail -n2; echo; echo; sleep 15; done;
Thu Jan 19 22:00:43 UTC 2017
 22:00:43 up 2 days,  4:14,  1 user,  load average: 1.17, 0.69, 0.36
10:00:43 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
10:00:43 PM    0    0.06    0.00    0.02    0.02    0.00    0.00    0.01    0.00    0.00   99.89


Thu Jan 19 22:00:58 UTC 2017
 22:00:58 up 2 days,  4:14,  1 user,  load average: 0.91, 0.66, 0.35
10:00:58 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
10:00:58 PM    0    0.06    0.00    0.02    0.02    0.00    0.00    0.01    0.00    0.00   99.89


Thu Jan 19 22:01:13 UTC 2017
 22:01:13 up 2 days,  4:14,  1 user,  load average: 0.71, 0.62, 0.35
10:01:13 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
10:01:13 PM    0    0.06    0.00    0.02    0.02    0.00    0.00    0.01    0.00    0.00   99.89


Thu Jan 19 22:01:28 UTC 2017
 22:01:28 up 2 days,  4:15,  1 user,  load average: 0.55, 0.59, 0.34
10:01:28 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
10:01:28 PM    0    0.06    0.00    0.02    0.02    0.00    0.00    0.01    0.00    0.00   99.89

...

Thu Jan 19 22:02:28 UTC 2017
 22:02:28 up 2 days,  4:16,  1 user,  load average: 0.20, 0.48, 0.32
10:02:28 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
10:02:28 PM    0    0.06    0.00    0.02    0.02    0.00    0.00    0.01    0.00    0.00   99.89

I am unable to reproduce the issue you described by following the steps provided. To move forward:

• provide steps which allow us to reproduce this issue in a test environment (perhaps by correcting mistakes I made in following the reproduction process, or providing additional information); or
• we will close this report after a few days if we are still unable to reproduce it.

Note that the test host above (debian001.epriestley.com) is a t2.micro, so the CPU is an underclocked potato. I would expect the host to be enormously sensitive to CPU utilization issues.

I was afraid this would happen. However, short of giving you access to our server, there is no way I can exactly describe our setup: there are fare too many variables. Perhaps we could narrow it down a bit? What could be a factor here? The obvious non-standard features of our setup are :

• linux-vserver
• 32bit userland
• LVM on soft raid
• daemon user = web user

That said, I can't understand why the same user can run hg log with negligible CPU impact, and run the same hg log through phabricator with significant performance impact. Clearly phabricator does something in addition to calling mercurial that is causing the problem, and I don't know what it could be. That's where your input would help :-)

Also @epriestley what's the output for time ./bin/repository update 1 on your repo? Our system is a Xeon W3520 so the CPU itself shouldn't be a problem. We're going to try and install phabricator elsewhere as a last resort, but it won't help in telling what's wrong with this setup.

I'd be happy to work with you one-on-one to help troubleshoot issues in your environment, but we can't offer that kind of support for free. This process almost always takes up a large amount of our time and very rarely uncovers any real bugs in Phabricator or helps anyone except the single user experiencing a configuration problem. See Support Resources for more discussion of the kinds of free and paid support we offer. If you'd like to move forward with one-on-one configuration and environment support, see Consulting.

We are certainly not going to engage into paid consulting without knowing how long it might take to solve the problem, given your hourly rate :)
I would have appreciated at least a tiny hint on how to solve this ourselves instead, so I guess I'm going to dive into the code and implement a workaround somewhere. For example, tinkering with the $smart_wait calculation might be a start.

Could you at least give me the time output for the broken/test repository update on your test system, so that I have a baseline for comparison later? Thanks!

I'd be happy to, see Consulting to move forward.

What'd be the fee for that 1 second command and a 5 second copy/paste?

Obviously, we can't just hand out estimates for free. Converting units and multiplying numbers together takes away valuable time and resources we could be using to develop Phabricator.

And if I gave you a precise estimate you could be sneaky and just divide it by $1,500/hr to get the answer without paying for it!

Erm… I was just asking for time repository update on the repository you have already cloned, hence the 15 seconds. If I understand this correctly, you seem to indicate that it would still be too much. I'm having a hard time understanding, but if you're adamant about this, then there is nothing left to say :)