The git diff on each file costs about 10ms for me, maybe it's much slower in your (presumably larger) repository?

For what its worth, I was able to reproduce this:

Above, it took me ~4m to upload 8,000 files, which is about 33 files per second. That could probably be faster, but seems reasonable to me?

The specific case of an auto-generated update from a huge commit we could probably improve, I don't think we benefit from auto-updating the revision to show the 100M-commit change and could reasonably just say "this is huge, go look at it in Diffusion instead".

I think the real question is why is it taking arc diff hours to upload? Something seems grossly inefficient.

Differential does not have any mode where it actually hides files, because this behavior does not make any sense to me:

Hmm, okay... I wonder why I can't load the diff after it has been landed then

Differential already does that. See D14976, above:

I think at the time I filed this ticket, I ended up just submitting a garbage diff (moving only a single file or something) and asked the reviewer to review the metadata. Once the diff had been accepted, I then ran git mv old/ new/ locally and then ran arc amend --revision D22186.

I think the behavior of arc diff is reasonable at the moment, in that it warns when a change is too large to be reviewed by a human. I think that it should still be possible to submit the changes for review in any case. Differential should probably show similar warnings on these diffs as well, like "This diff is too large to be reviewed, click here to review it anyway". Maybe Differential should even behave similar to Diffusion: "This commit is enormous, and affects more than 1000 files. Changes are not shown."

Sure, I agree with all that. But what do you want arc diff to actually do, technically speaking, in that situation?

In T8612#152666, @epriestley wrote:

What's your ideal behavior when a user changes Copyright 2015 to Copyright 2016 in 1 million files, adds a 5GB access.log file, adds the MySQL 5.1 source, all in one diff, and then runs arc diff?

What's your ideal behavior when a user changes Copyright 2015 to Copyright 2016 in 1 million files, adds a 5GB access.log file, adds the MySQL 5.1 source, all in one diff, and then runs arc diff?

In T8612#152644, @epriestley wrote:

Let me clarify: do you think a human should open a web page which has 8,000 rows on it showing that 8,000 files moved in order to review "move src/ to notsrc/", or that it's sufficient for them to review the metadata ("I'm going to move src/ to notsrc/") without reviewing the actual details of the change, as described in the "Large Changes" document?

Broadly, I think there are three classes of use case here:

Let me clarify: do you think a human should open a web page which has 8,000 rows on it showing that 8,000 files moved in order to review "move src/ to notsrc/", or that it's sufficient for them to review the metadata ("I'm going to move src/ to notsrc/") without reviewing the actual details of the change, as described in the "Large Changes" document?

I disagree that moving a large number of files is something that shouldn't be reviewed. In fact, in my original case it is really only one path that is being moved (the top level directory). As a side effect, all subdirectories and subfiles are also moved.

For example, if the root cause here is just that users are super lazy (or, in the future, use cases emerge where commit access is controlled through "Land Revision" in the web UI) maybe we could provide an arc diff --no-human-review mode which pushes the changes to staging but does not attempt to display them, and tells users how to fetch and review them locally instead. Then you could arc diff your "move src/ to notsrc/", "check in a 5GB logfile", "commit the entire MySQL 5.3 source", etc., diffs and go through the motions of review without us actually needing to do any work, which is all wasted anyway since no one is going to look at the individual changes to the 8,000 files you moved.

I can't immediately reproduce this locally. Here's what I did: