Please update to HEAD before filing a bug report. We can't take reports against old versions of Phabricator. You can use Phacility if you'd like to test an up to date installation.

Hm; we weren't running HEAD when we reported T11748. We have an upgrade planned to latest-stable in a week or so -- will that be sufficient to re-test with? Or is there a repository that we can apply a 4,800-file diff to on Phacility?

We weren't running quite HEAD then, but we were running a version that was our patches on top of something about a week old -- see the end of the description of T11743, which was the first report in that chain. Our version today is a couple of months old, so I see why that'd feel different.

After our next upgrade to latest-stable, we'll be back to a similar level of HEAD-ness to what we were running then.

Note that in T11743 I ended up reproducing on Phacility anyway at Evan's request so that he could debug it live.

You can always create a test instance in Phacility, and upload the diff there - those are free and temporary - see Reproduction/Resources
There's also rGITTEST, although maybe it's not so good for a super huge diff...

It will likely be easiest to replicate again after upgrading to Week 52 stable this Friday, since I can't use precise same diff, and don't know the precise set of replication criteria yet.

I strongly suspect this report has enough information to let us replicate the issue locally, so you don't need to take any further action yet. I'll follow up if I have difficulty replicating it and we can figure out what differs between my synthetic test case and your real environment.

(Because of churn in T11114 and other upstream changes I'm not exactly sure what's happening with Week 52 anyway.)

Alright, I'm actually having trouble reproducing this. Here's what I tried:

• I reduced max_allowed_packet to 1MB.
• I restarted mysqld.
• I verified the change was in effect:

mysql> select @@max_allowed_packet;
+----------------------+
| @@max_allowed_packet |
+----------------------+
|              1048576 |
+----------------------+
1 row in set (0.00 sec)

• I checked out a new branch:

$ git checkout -b pathtest

• I created 10,000 moderately long files:

$ for i in `seq 1 10000`; do echo $i > moderately_long_directory_name/long_long_path_name_$i.txt; done;

• I committed the change and created a revision from it:

$ git commit -a -m paths
$ arc diff HEAD^ --conduit-uri=http://local.phacility.com/

• I received a prompt advising me not to do this, but entered "y" to continue anyway:

This diff has a very large number of changes (10,000). Differential works
best for changes which will receive detailed human review, and not as
well for large automated changes or bulk checkins. See
https://secure.phabricator.com/book/phabricator/article/differential_large_changes/
for information about reviewing big checkins. Continue anyway? [y/N] y

• I ran bin/phd debug to generate the mail in the foreground.

Mail generation worked correctly. We generate these paths in the mail:

AFFECTED FILES
  moderately_long_directory_name/long_long_path_name_1.txt
  moderately_long_directory_name/long_long_path_name_10.txt
  moderately_long_directory_name/long_long_path_name_100.txt
  moderately_long_directory_name/long_long_path_name_1000.txt
<...snip...>
  moderately_long_directory_name/long_long_path_name_122.txt
  moderately_long_directory_name/long_long_path_name_1220.txt
  moderately_long_directory_name/long_long_path_name_1221.txt
  (9750 more files...)

Note that the "snip" is me snipping text, but the "9750 more files..." is Phabricator automatically snipping the file list and showing only 250 files.

I had inline diffs configured, but they were correctly dropped from this mail.

The mail message Phabricator generates as a result of this is only about 38KB, combining both text and HTML mail and fit easily in the 1MB packet limit, so I'd expect half as many files should fit handily into the ~64MB packet limit described, unless their paths are incredibly long -- on the order of 256KB each (which isn't actually possible until D17062, which is ahead of you).

Perhaps the most direct diagnostic is to add something like this near line 1178 of PhabricatorApplicationTransactionEditor.php:

var_dump($mail);
die("Aborting.");

Then run bin/worker execute --id 35362582.

That should dump >65MB of output to stdout, and the content of that output should tell us which field is growing unreasonably large. It is quite possible that there is some relevant configuration difference and I'm just not hitting it right, although I think all the relevant config is still set to values similar to the cases in T11748.

Alternatively, you can attempt to reproduce it on a Phacility instance and I'm happy to fish the root cause out of the environment there.

bin/worker execute --id 35362582 did not work because the task was archived. Instead I ran ./bin/phd stop; ./bin/worker retry --id 35362582; ./bin/phd debug PhabricatorTaskmasterDaemon. The key seems to be the production of the .patch file itself, not the number of files. Somehow these users managed to cram a >100M patchfile into Differential:

Such a large diff is clearly not a reasonable thing to begin with, but ideally one that wouldn't wedge the task. The most coherent thing might be to not attach patchfiles over a certain size?

Aha -- the diff as posted for review was mostly-reasonably sized. The diff as committed was the huge one:

mysql> select differential_diff.id as diff_id, creationMethod, SUM(addLines), SUM(delLines)
    ->   from differential_changeset
    ->   join differential_diff on differential_changeset.diffID = differential_diff.id
    ->  where differential_diff.revisionID = 253041
    ->  group by differential_diff.id;
+---------+----------------+---------------+---------------+
| diff_id | creationMethod | SUM(addLines) | SUM(delLines) |
+---------+----------------+---------------+---------------+
|  837437 | arc            |         11845 |          8567 |
|  837558 | commit         |       1752986 |        396108 |
+---------+----------------+---------------+---------------+

Thanks, I'll take a more in-depth look at this.

This might be sort of a generalization of T11767.

What is metamta.diffusion.byte-limit set to?

Oh, never mind, that's the Differential patch responding to the commit, so metamta.diffusion.byte-limit isn't relevant.

Also, is metamta.email-body-limit set to something crazy? I think that only means main-body (not body + attachments) right now, but it should probably mean body + attachments + headers.

metamta.email-body-limit is unset.

I'm going to presume this is resolved by D18598 -- I updated T11767 with more plans and context for smoothing over the behavior, but I think it will no longer be disastrously bad. I'll follow up in PHI36 PHI55.

Do you mean "follow up in PHI55"?

Oh, yes, sorry, looked at the wrong tab.