One possible improvement remains here: in streaming mode, HTTPSFuture will not "Accept-Encoding: gzip" and can not inflate responses. Today, this only applies to arc download.

There's also one minor related concern in PHI1638 that I want to take care of here, since it's adjacent.

In fairness, "SetInputFilter DEFLATE" is documented as uniquely opinionated, if you catch this box and read between the lines a little bit:

A grand aventure in PHI1679 led to identifying the Apache "SetInputFilter DEFLATE" directive as a problem with implementing compression.

"FutureGraph" is dead. Long live "HardpointEngine".

Generators can't "return" until PHP 7:

PHP has a setting called post_max_size. If a POST request is larger than this size, PHP does not populate $_POST or $_FILES.

In T11968#251717, @epriestley wrote:

XHPAST currently can't build an AST for $result = yield ..., even though this is a valid construct. This is probably a straightforward fix.

XHPAST currently can't build an AST for $result = yield ..., even though this is a valid construct. This is probably a straightforward fix.

I have some code which runs and looks plausible (i.e., not covered in piles of callback garbage), at least:

Here's an actual example of loadHardpoints($objects, $hardpoint):

In the specific case of the Hardpoints, we currently often have code which loads objects but doesn't do anything with them. For example, most Query classes use didFilterResults() to fill things-that-sure-look-like-hardpoints, but few do anything with the results.

The core idea in D5104 + D5105 is that $future->resolve() and id(new FutureIterator(array($future)))->next() (like, roughly) execute meaningfully different code paths.

Somewhere in experimental or wilds, I introduced ArcanistConduitEngine. This has some weird fake future stuff going on, so this is probably now ripe.

Probably require PHP 5.4 regardless.

A problem in moving forward here is that we ultimately do very little complex data access in Phabricator, and what complex data access we do perform can often be faked. We likely have more use cases in Arcanist and provisioning code: API calls are slower, workflows are more parallel/interactive, and we can't just fake it all with AJAX.

See also PHI1628, which reports that a 4MB blob of test details is slow to render.

There's no specific performance/scale issue here, so waiting until one arises or this pops up on a profile.

See T13110 for some followups.

For what it's worth, after spot-checking a few tasks on my install, all seem to query the same policy 10ish times in succession near the end of the load.

Some of this got fixed, some ended up in T13081.

With that stuff, we're down from 473 to 227 on T9950. That's still awful, but not quite as ridiculous.

This was largely tackled above.

I'm having some difficulty merging (5) and (6), or either of them into (2). My local test also has a lot of nonsense going on that a real install shouldn't. I'm going to land what I've got so far and see how much progress we've made here.

Queries are down to 139 for me locally.

Queries are now down from 151 to 145 for me locally.

Next problem:

For my local test task (T369), the initial page query load was 199 queries. After D19222 we're down slightly to 151 queries.

Some of this is:

This doesn't seem to have caused any issues.

PHI109 captured a slightly more illuminating error. D18692 has some discussion.

Yeah I just didn't want to add the (...) view to DiffusionHistory if it's going to slow those pages down. I'd just leave the feature out.

I think using a Pool (like HandlePool) is not too much more work (at least, if I'm implementing it) than AJAX it and will be useful in more cases. I believe we need the Pool implementation regardless, sooner or later.

@epriestley maybe this should just AJAX?

Calling this resolved since the page is now tolerably bad (~5s) which more or less pulls this out of the realm of critical scaling issues.

@jmeador, I believe this is now resolved at HEAD. Let us know if you're still seeing issues after upgrading?

After implementing PHP7 support for XHProf and updating Lipsum, I used bin/lipsum to generate 4,000 packages owning 150,000 paths. I think I have an approximately-reasonable reproduction case locally now, which generates this profile (6.5s for a change affecting about 100 files):