Page MenuHomePhabricator

Render only the first 1,000 changed paths by default when viewing a commit in Diffusion
Closed, ResolvedPublic

Description

E.g. http://llvm-reviews.chandlerc.com/rL170845.

That commit came up as one of my automated audits and from clicking on the link into the audit and having the "comment" box appear (so that I can just resign, since the commit is not really meant to be reviewed, it's mostly a large import), takes at least a minute.

(some caching effect may be speeding it up now, even though it still takes a while).

It seems like progress first goes into the sidebar indicating which files are modified, which blocks the main part of the page from loading, then the main part of the page is loading a bunch of entries for each file (which just say "load"). Finally the comment form appears. And the page is even getting the "This commit is very large." treatment.

Part of the problem is that 12MB+ of HTML is being sent down the wire. Also just the rendering is taking forever (the tab is using 280MB of memory by itself in Chrome).

I think a possible fix could be to introduce another fractal level of "This commit is very large." that not only collapses the diffs, but also presents the changed files in a way that is that requires less than a linear amount of content (w.r.t. # changed files) to be sent for the initial page load.

Event Timeline

One problem is that llvm-reviews.candlerc.com appears not to be able to gzip responses, so the response is uncompressed and the wire size of the response is roughly ~10MB. On secure.phabricator.com, a similar page (with 8MB of uncompressed data) gets delivered zipped and has a wire size of just 580KB (at least, this is what I'm seeing from my machine using Charles to examine the requests and responses). Enabling gzip should provide an immediate and potentially dramatic improvement in performance for every page, but particularly for large pages like this.

(If whoever administrates the server has no idea how to do this, I can probably point you in the right direction. Possibly we can automatically detect it by making requests to ourselves -- I'll file a task.)

The example page takes 14 seconds for me to load on my machine, and a similar commit on secure.phabricator.com takes about 8 seconds (rP6cc196a2e557ffe1e6eeeef063358553ccdd3481). These aren't great, but they seem relatively manageable provided such commits are one-in-a-thousand (this is roughly the rate of such commits in the Phabricator repository, at least). Some of the discrepancy is probably gzip-related. Are you on slower/older hardware? I have a 2010 Macbook Pro -- how slow is https://secure.phabricator.com/rP6cc196a2e557ffe1e6eeeef063358553ccdd3481 for you?

Generally, we probably should break how we degrade large commits into more stages. We currently distinguish between normal commits and commits which are too large to load diffs for (at 100 changes), but do not distinguish between those commits and commits which are too large to load all the paths for (maybe this should be at something like 1000 changes). In existing installs, this has been a fairly rare case and performance has been good enough given the rare incidence. There are also a lot of subtle interactions which make implementing this sort of thing more difficult than it might seem like it should be (for example, users can link to specific changesets with anchors, but if we stop showing changesets past the first 1,000 by default then an anchor beyond those first 1,000 won't go anywhere. We can drop this case on the floor but that really just pushes it off until someone eventually files a bug about it), so it's hard to prioritize this edge case stuff against things that affect the other 99.9% of commits.

The sidebar thing is probably going to start defaulting to "off" soon, which should also help here.

how slow is https://secure.phabricator.com/rP6cc196a2e557ffe1e6eeeef063358553ccdd3481 for you?

I'm getting performance in the same ballpark (~10 seconds) as you are reporting. My internet connection where I am right now is particularly slow (150KB/s down). Slow internet + lack of gzip was, I suspect, the cause of the load times measured in minutes.

such commits are one-in-a-thousand

My intuition is that this is about the right ballpark in general. However, in recent (last month) of my development on LLVM, my watches have picked up 10+ such commits. I suspect that part of this is due to the fact that in the last month we have been preparing for a release, which has caused a lot of "full repo" branches to be created. Additionally, as in the commit I linked, the final resting place for some large blocks of generated files (such as all of the generated doxygen documentation for the release) happen inside of SVN (not sure why), so those also trigger these massive commits.

Part of the annoyance is that there is no way to resign-from-audit/accept/etc without loading the full page---for these commits, I usually know ahead of time that they are huge (and that there is nothing to audit anyway). This is also independently useful (e.g., I have already seen the commit email) so I will file it as a separate task. Adding this feature seems like a better time investment overall.

epriestley renamed this task from Unresponsive page for large commits. to Render only the first 1,000 changed paths by default when viewing a commit in Diffusion.Apr 9 2013, 5:30 PM
epriestley triaged this task as Low priority.
chad changed the visibility from "All Users" to "Public (No Login Required)".Jul 3 2015, 5:20 AM
epriestley claimed this task.

This was fixed a couple of years ago in D4730, and works fine on the modern LLVM install: http://reviews.llvm.org/rL170845