Applying the bundle with arc patch --arcbundle also takes a long time:

arc patch --arcbundle bundle.tar.gz --trace
libphutil loaded from '/path/to/libphutil/src'.
arcanist loaded from '/path/to/arcanist/src'.
Config: Reading user configuration file "/home/josh/.arcrc"...
Config: Did not find system configuration at "/etc/arcconfig".
Working Copy: Reading .arcconfig from "/path/to/.arcconfig".
Working Copy: Path "/path/to" is part of `git` working copy "/path/to".
Working Copy: Project root is at "/path/to".
Config: Did not find local configuration at "/path/to/.git/arc/config".
Loading phutil library from '/path/to/support/flarc/src'...
>>> [0] <exec> $ tar tfO '/path/to/bundle.tar.gz'
<<< [0] <exec> 3,721,369 us
>>> [1] <exec> $ tar xfO '/path/to/bundle.tar.gz' meta.json
<<< [1] <exec> 3,588,870 us
>>> [2] <exec> $ tar xfO '/path/to/bundle.tar.gz' changes.json
<<< [2] <exec> 4,040,191 us
>>> [3] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'0'
<<< [3] <exec> 3,771,322 us
>>> [4] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'1'
<<< [4] <exec> 3,674,722 us
>>> [5] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'2'
<<< [5] <exec> 3,646,983 us
>>> [6] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'3'
<<< [6] <exec> 3,651,432 us
>>> [7] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'4'
<<< [7] <exec> 3,651,113 us
>>> [8] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'5'
<<< [8] <exec> 3,787,710 us
>>> [9] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'6'
<<< [9] <exec> 3,669,368 us
>>> [10] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'7'
<<< [10] <exec> 3,775,023 us
>>> [11] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'8'
<<< [11] <exec> 3,710,657 us
>>> [12] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'9'
<<< [12] <exec> 3,775,201 us
>>> [13] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'10'
<<< [13] <exec> 3,772,889 us
>>> [14] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'11'
<<< [14] <exec> 3,653,123 us
>>> [15] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'12'
<<< [15] <exec> 3,652,439 us
>>> [16] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'13'
<<< [16] <exec> 3,675,907 us
>>> [17] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'14'
<<< [17] <exec> 3,673,572 us
>>> [18] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'15'
<<< [18] <exec> 3,794,988 us
>>> [19] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'16'
<<< [19] <exec> 3,700,176 us
>>> [20] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'17'
<<< [20] <exec> 3,566,099 us
>>> [21] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'18'
<<< [21] <exec> 3,555,604 us
>>> [22] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'19'
<<< [22] <exec> 3,587,953 us
>>> [23] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'20'
<<< [23] <exec> 3,701,705 us
>>> [24] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'21'
<<< [24] <exec> 3,662,087 us
>>> [25] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'22'
<<< [25] <exec> 3,522,353 us
>>> [26] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'23'
<<< [26] <exec> 3,642,239 us
>>> [27] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'24'
<<< [27] <exec> 3,558,719 us
>>> [28] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'25'
<<< [28] <exec> 3,678,668 us
>>> [29] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'26'
<<< [29] <exec> 3,652,127 us
>>> [30] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'27'
<<< [30] <exec> 3,727,473 us
>>> [31] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'28'
<<< [31] <exec> 3,654,216 us
>>> [32] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'29'
<<< [32] <exec> 3,744,392 us
>>> [33] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'30'
<<< [33] <exec> 3,657,835 us
>>> [34] <exec> $ tar xfO '/path/to/bundle.tar.gz' hunks/'31'

I can't immediately reproduce this locally. Here's what I did:

• I moved src/ to notsrc/ in this project (Phabricator).
• This produces a ~30MB diff affecting 8,100 files.
• I ran time arc diff --only.
• It produced a diff in 4m 30s.

Here's the transcript:

$ time arc diff --only


    This diff has a very large number of changes (8,164). Differential works
    best for changes which will receive detailed human review, and not as
    well for large automated changes or bulk checkins. See
    https://secure.phabricator.com/book/phabricator/article/differential_large_changes/
    for information about reviewing big checkins. Continue anyway? [y/N] y

Uploaded binary data for "CHA.git.tgz".
Uploaded binary data for "CHA.git.tgz".
Uploaded binary data for "CHB.hg.tgz".
Uploaded binary data for "CHB.hg.tgz".
Uploaded binary data for "CHC.svn.tgz".
Uploaded binary data for "CHC.svn.tgz".
Uploaded binary data for "CHD.svn.tgz".
Uploaded binary data for "CHD.svn.tgz".
Uploaded binary data for "CHE.svn.tgz".
Uploaded binary data for "CHE.svn.tgz".
Uploaded binary data for "GT.git.tgz".
Uploaded binary data for "GT.git.tgz".
Uploaded binary data for "HT.hg.tgz".
Uploaded binary data for "HT.hg.tgz".
Uploaded binary data for "ST.svn.tgz".
Uploaded binary data for "ST.svn.tgz".
Upload complete.
 PUSH STAGING  Pushing changes to staging area...
Counting objects: 12, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (12/12), 326.39 KiB | 0 bytes/s, done.
Total 12 (delta 4), reused 0 (delta 0)
To ssh://dweller@secure.phabricator.com/diffusion/STAGING/staging.git
 * [new tag]         9282243fb4467f363b09cbf2485209cd587331ba -> phabricator/diff/36182
 STAGING PUSHED  Pushed a copy of the changes to tag "phabricator/diff/36182" in the staging area.
Created a new Differential diff:
        Diff URI: https://secure.phabricator.com/differential/diff/36182/

Included changes:
  M       .arcconfig
  A       notsrc/.phutil_module_cache
  V       notsrc/__phutil_library_init__.php
             from: src/__phutil_library_init__.php
  V       src/__phutil_library_init__.php
             to: notsrc/__phutil_library_init__.php

... <snip thousands of lines> ...

  V       src/view/widget/hovercard/PhabricatorHovercardView.php
             to: notsrc/view/widget/hovercard/PhabricatorHovercardView.php

real	4m14.569s
user	0m58.105s
sys	0m51.252s

Here's a revision I created from it:

D14976: Move "src/" to "notsrc/"

Although that takes quite a while to load, it renders in about 30s for me and the page is somewhat usable (e.g., I can find particular files and review them).

However, this is a textbook case of Differential not being appropriate, per https://secure.phabricator.com/book/phabricator/article/differential_large_changes/ -- no human will review these changes.

@eadler, per discussion offline, maybe you have a more narrow use case here? I believe the behavior of more realistic (say, ~a couple hundred files) changes is already quite reasonable, but if you're running into issues I'd like to tailor improvements to the specific use case you're encountering.

For use cases that will not receive human scrutiny, I'd like to understand why they're being sent to Differential anyway (and the prompt saying "do not send this change to Differential" is being bypassed).

For example, if the root cause here is just that users are super lazy (or, in the future, use cases emerge where commit access is controlled through "Land Revision" in the web UI) maybe we could provide an arc diff --no-human-review mode which pushes the changes to staging but does not attempt to display them, and tells users how to fetch and review them locally instead. Then you could arc diff your "move src/ to notsrc/", "check in a 5GB logfile", "commit the entire MySQL 5.3 source", etc., diffs and go through the motions of review without us actually needing to do any work, which is all wasted anyway since no one is going to look at the individual changes to the 8,000 files you moved.

I disagree that moving a large number of files is something that shouldn't be reviewed. In fact, in my original case it is really only one path that is being moved (the top level directory). As a side effect, all subdirectories and subfiles are also moved.

Let me clarify: do you think a human should open a web page which has 8,000 rows on it showing that 8,000 files moved in order to review "move src/ to notsrc/", or that it's sufficient for them to review the metadata ("I'm going to move src/ to notsrc/") without reviewing the actual details of the change, as described in the "Large Changes" document?

Broadly, I think there are three classes of use case here:

• Things that should not be going to Differential because they will not receive detailed human review. I'd like to figure out why users are sending these changes to Differential and help them find more suitable alternative workflows. Examples:
  • Checking in a 5GB logfile.
  • Cutting a branch in SVN with a million files, or renaming src/ to notsrc/.
  • Committing the MySQL 5.1 source to third-party/mysql-5.1/.
  • Updating the copyright header on 100,000 files from 2015 to 2016.
• Codebase-spanning manual changes which need manual review but exceed the scope of a single revision and are not atomic. I'd like to build an "audit a codebase" sort of workflow for these (discussed in T4348), and expect revisions to be small and incremental ("convert callsites in Maniphest", "convert callsites in Diffusion"). This includes projects on the multiple-man-weeks of work scale, where many many thousands of files will receive human changes. Examples:
  • Adding pht() or similar around all text strings in a codebase (this took dozens of engineers many months at Facebook).
  • Replacing some old pattern with some new pattern in 2000 places.
• Codebase-spanning manual changes which need manual review and are large, but small enough to be done atomically. I believe behavior is already relatively reasonable here (we aren't super elegant about 800-file diffs, but I believe we have generally reasonable performance and behaviors), but am open to improving it in cases where it currently isn't working as well as it could be.
  • Replacing an old API call with a new API call at several hundred callsites, but few enough that one engineer can make all the changes and reasonably test the results in a day or two.

In T8612#152644, @epriestley wrote:

Let me clarify: do you think a human should open a web page which has 8,000 rows on it showing that 8,000 files moved in order to review "move src/ to notsrc/", or that it's sufficient for them to review the metadata ("I'm going to move src/ to notsrc/") without reviewing the actual details of the change, as described in the "Large Changes" document?

We have had a number of cases in which we do use Differential for reviewing large changes and in these cases we review the metadata + a random sample of the files as a sanity check. I think though there are two separate issues:

• A relatively simple change (git mv src/ notsrc/) causes a complex diff. It would be better if the diff showed "Move src/ to notsrc/ rather than showing the change on every individual file.
• It is not possible to upload this change to Phabricator in the first place.

What's your ideal behavior when a user changes Copyright 2015 to Copyright 2016 in 1 million files, adds a 5GB access.log file, adds the MySQL 5.1 source, all in one diff, and then runs arc diff?

In T8612#152666, @epriestley wrote:

What's your ideal behavior when a user changes Copyright 2015 to Copyright 2016 in 1 million files, adds a 5GB access.log file, adds the MySQL 5.1 source, all in one diff, and then runs arc diff?

I have a philosophy that every commit should be reviewed pre-push. In the case of large diffs, I still think that they should go through the formal review process, although I don't necessarily expect the reviewer to spend a significant amount of time reviewing a "Change copyright year" diff, but in these cases I would expect the diff summary and test plan to be scrutinized rather than the actual contents of the diff. For example, if I was a reviewer on a diff which "Changed the copyright year" then I would expect the author to explain how they made the bulk change. For example, did they just ran sed -e 's/2015/2016/' or did they parse PHP comments and perform a regex replacement on comments only?

In the case of a large file (or even perhaps a binary) I would probably try to produce the same result myself and then compare. For example, I might arc patch the diff locally, rebuild the binary and then see if I end up with the same result.

Sure, I agree with all that. But what do you want arc diff to actually do, technically speaking, in that situation?

I think the behavior of arc diff is reasonable at the moment, in that it warns when a change is too large to be reviewed by a human. I think that it should still be possible to submit the changes for review in any case. Differential should probably show similar warnings on these diffs as well, like "This diff is too large to be reviewed, click here to review it anyway". Maybe Differential should even behave similar to Diffusion: "This commit is enormous, and affects more than 1000 files. Changes are not shown."

I think at the time I filed this ticket, I ended up just submitting a garbage diff (moving only a single file or something) and asked the reviewer to review the metadata. Once the diff had been accepted, I then ran git mv old/ new/ locally and then ran arc amend --revision D22186.

Interestingly, if I attempt to view that diff in the UI just now then it fails with a 504 Gateway Time-out. I can, however, load the corresponding commit in Diffusion, it just hides all of the actual changes with: "This commit is enormous, and affects more than 1000 files. Changes are not shown."

Differential already does that. See D14976, above:

Hmm, okay... I wonder why I can't load the diff after it has been landed then

Differential does not have any mode where it actually hides files, because this behavior does not make any sense to me:

• User touches 100M files.
• User runs arc diff.
• arc diff spends an hour uploading the changes.
• Differential hides 99.999% of the changes.

Didn't we just waste an hour of the user's time? No human will ever page through 100M files even if the web UI lets you. If there are even 8,000 files, as with D14976, never mind 1M, surely no user will ever inspect them individually?

I think the real question is why is it taking arc diff hours to upload? Something seems grossly inefficient.

The specific case of an auto-generated update from a huge commit we could probably improve, I don't think we benefit from auto-updating the revision to show the 100M-commit change and could reasonably just say "this is huge, go look at it in Diffusion instead".

Above, it took me ~4m to upload 8,000 files, which is about 33 files per second. That could probably be faster, but seems reasonable to me?

I'm not sure why it took hundreds of times longer for you.

For what its worth, I was able to reproduce this:

As an alternative, I tried uploading the git diff output manually at /differential/diff/create/, but this timed out after 30 seconds.

On this install by uploading a trivial directory rename to the Linux repository.

I have also been running time arc diff --conduit-uri=https://secure.phabricator.com --only and it has been executing for 16 minutes now.

The git diff on each file costs about 10ms for me, maybe it's much slower in your (presumably larger) repository?

>>> [113] <exec> $ git diff --no-ext-diff --no-textconv --no-color --src-prefix=a/ --dst-prefix=b/ -U32767 '2c293bdca88d048ee413f9496ce34d75f3c45d7e' -- 'notsrc/aphront/response/AphrontReloadResponse.php'
<<< [113] <exec> 9,921 us
>>> [114] <exec> $ git diff --no-ext-diff --no-textconv --no-color --src-prefix=a/ --dst-prefix=b/ -U32767 '2c293bdca88d048ee413f9496ce34d75f3c45d7e' -- 'src/aphront/response/AphrontReloadResponse.php'
<<< [114] <exec> 11,451 us

Hard to imagine it's hundreds of times slower, though...

arc diff took at least an hour for my test on the Linux repository. Sometime after this, my laptop froze and I had to restart.

If you run with --trace, what does the first 1,000ish lines of output look like?

Oops.