Pagination isn't quite as simple as I'd remembered. Specifically, we do not expose internal pagination cursors to viewers today because doing so permits an information discovery attack that goes like this:

I think this behavior is very good

This is also a potential problem in the general case, although the general case of this is very rare.

Broadly, most applications can execute Spaces queries cheaply by querying ... AND spacePHID IN (<list of spaces the viewer can see>) when Spaces are configured, which is efficient until some install decides to create 30,000 Spaces for some reason.

This helped! Thanks for another fast fix.

I believe this should now be fixed in HEAD of master. It should promote to stable within 24 hours.

I've banged on this a reasonable amount locally without issues and the originating instance reports that this seems to have calmed things down in production with these patches, so it seems like this pretty much just worked.

For people who suffer from this: Note that D16483 might have actually made things a little worse, by searching the DB for old messages (Not loading them though); I tested with 1M messages on an SSD drive, and I didn't see any changes, but YMMV.

I deployed this stuff here. I neglected to get a production timing beforehand so I can't really claim it's actually faster, but it seems fast-ish now. Let me know if things look better and/or I ruined everything when you get around to deploying it.

I don't immediately have an attack on the JIRA thing (it should be less expensive in normal cases), and don't want to do the Auth stuff without more planning, and nothing else leaps out at me as having a similar sort of value for the cost.

There's still quite a bit of room here. Locally, I see a 220ms call for differential.revision.search. Here are savings which look fairly easy to capture:

That sort of leaves you in trouble with custom edge-based fields like the one above, but a reasonable short term approach is the one you've already taken:

Ah, there are really a couple of issues here.

That looks broadly correct to me, I just want to try to remove the assumption that the first object's first field's storage is universally the right storage in fixing this upstream. It's always true today and for the foreseeable future, just obviously not a Fundamental Axiom of the System. Let me review D16346 and I'll spend 10 minutes on this, I'm pretty sure it's straightforward, not a weird snowbally mess, and that the upstreamable patch is only cosmetically different from that one.

This is brittle but it works if your objects all use the native storage.

I think the issue is that there's no callable thing for bulk-loading custom fields right now, which is why SearchEngineExtension doesn't have a hook for it. Other SearchEngine extensions don't need it since none currently load any data.

(Only unique queries are saved, so this should be a small amount of data in most cases, but the Phacility setup tends to mean we issue a lot of unique queries.)

Using an extended policy sort of fixes this, except that extended policies currently only strengthen policies, so I also have to weaken the default View policy. This potentially makes the UI misleading, since it suggests that "all users" can view an instance, which isn't true.

One possible approach is to cache policy checks (when we've determined that user X can see object Y, cache that for the remainder of the request), but I don't want to do this idly since it has far-reaching implications, and there are cases where this cache could potentially produce the wrong result (for example, when we predict the effects of making a policy change during an edit, we evaluate objects as though they had a different policy than they really have).

D16045 reduced this to one query. We can cache it more aggressively and amortize the cost to 0 queries, but I'll leave that for T11096. This is no longer wildly out of line in terms of cost/value.

D16001 likely provides an appropriate general-purpose mechanism for the user cache pool.

The following queries are full table scans:

In the extreme case where you have, say, a million tests, I'd expect there is probably little value in reporting each test into Harbormaster as a "pass", and your harness might want to summarize all passes into "999,998 additional tests passed" and submit two failures and one aggregate-pass.

T9704 also discusses slowness on inserting the tests. That may be unnecessarily slow right now, but I don't expect clients to submit unlimited numbers of tests in one API call. Instead, submit tests in chunks (say, of 1K tests per page or whatever) by calling harbormaster.sendmessage repeatedly with a work status.

We don't currently have a sortable column on the unit message table, since pass, fail, etc., aren't naturally sortable by any key MySQL can construct.

See also T2225 for a concrete issue caused by showing enormous numbers of paths (unbearably slow page loads because of the raw wire filesize).

@paladox: Your last question was already answered before in T8612#163215

Oh ok, is there a way to set it to unlimited so that a user can choose to do it per file but yes it is unlikely a user will review a million changes but it is always good to check.

You can manually adjust the $hard_limit in your local installation here: https://secure.phabricator.com/diffusion/P/browse/master/src/applications/diffusion/controller/DiffusionCommitController.php;ac729278328ed9679229d11ba1eefdad784b59e2$148

I don't intend to change that limit. I believe there is very little value in reviewing 1000+ file changes from the web UI, and that the 1K file limit is a reasonable one.

Yes that's what I would like to see please, please allow it to show that over large commits please.

That commit is changing more than 1,000 files.

@epriestley oh because on https://phabricator.wikimedia.org/rAPAW76c6444cfb64e872b8b15688a8999d96af84a406 it dosent do that yet it is only chaning a 100 files.

In T8612#163197, @paladox wrote:

What about per file diff viewing it would hide them by default but add an option to each file that says show diff.