Page MenuHomePhabricator

Implement a composite query which unifies cursor paging across subqueries
Closed, WontfixPublic

Description

For T5750, we need a CompositeQuery which can walk a single cursor across a collection of subqueries. The goal is to end up with an object which works something like this and produces the expected results:

$results = id(new CompositeQuery())
  ->addQuery(new PeopleQuery())
  ->addQuery(new ProjectQuery())
  ->executeWithCursorPager($pager);

This is complicated to build and requires at least some infrastructure cleanup. Concretely, it's currently not possible to implement getPagingValue() for a query like this, because the result (which is really something like "cursor material", not a "paging value") depends on which direction the cursor is iterating.

On the other hand, implementing this without being able to implement getPagingValue() seems like a dead end that will saddle us with weird limitations. In particular, I think it's a requirement to allow a CompositeQuery to be embedded inside another CompositeQuery rather than arbitrarily restricting them to be 1-ply deep.

More broadly, most paging/order/cursor code in Query subclasses is not robust, and many of the methods related to paging are confusing, too general, not general enough, generally don't make sense, etc. It's unclear how much work this needs.

Details

Commits
D12380 / rP156b156e77b6: Give Conduit params/return/errors protected visibility
D12383 / rP8efdc4aabf09: Replace getPagingValue() with cursor methods
D12381 / rP09ad69238ee7: Drive conduit result ordering through Query order specifications
D12379 / rP6e4f508bebdb: Provide "builtin" high-level result orders
D12372 / rPbdd1edea7a40: Modernize ManiphestTask paging and ordering
D12371 / rP411456084448: Modernize more paging/order queries
D12378 / rP2794c69db550: Remove getPagingColumn() / getReversePaging()
D12363 / rP9c7c13ffc88a: Modernize Phrequent and Commit query ordering/paging
D12361 / rP9b5198f46330: Remove ORDER_PATH_MODIFIED from Differential
D12362 / rP51dabc500714: Modernize Differential paging/ordering
D12360 / rPe0fa0fbdee1b: Modernize Phriction ordering/paging
D12358 / rPa4a198342e2c: Modernize ReleephProjectQuery ordering/paging
D12356 / rP4fba6e7730aa: Remove trivial implementations of getPagingColumn()
D12359 / rP8bd1ab9d136f: Modernize Feed and Phlux ordering/paging
D12357 / rPd496f4d28c7c: Modernize ProjectQuery paging/ordering
D12353 / rP604d1409f16f: Make buildPagingClauseFromMultipleColumns() safer
D12355 / rPa40c40fade6a: Drive query ordering and paging more cohesively
D12354 / rPe6174ed45cbd: Fix an issue where pastes could be reordered as a side effect of cache fills
D12352 / rPa43473c4b65c: Begin formalizing query orders
D12351 / rP9dc114d1159a: Make formatOrderClause() safer
D12374 / rPHU99e9235ec6a4: Actually support "%Lf" in qsprintf()
D12346 / rP15b41f563934: Remove Herald rule edit log

Related Objects

StatusAssignedTask
Resolvedchad
ResolvedNone
Resolvedepriestley
DuplicateNone
ResolvedNone
Wontfixepriestley
Resolvedepriestley
Openepriestley
Resolvedepriestley
ResolvedNone
ResolvedNone
DuplicateNone
Wontfixepriestley
DuplicateNone
Resolvedepriestley
OpenNone
Wontfixepriestley
Resolvedepriestley
Resolvedchad
Resolvedepriestley
Wontfixepriestley
OpenNone
OpenNone
Resolvedepriestley
Resolvedepriestley
Resolvedepriestley
Wontfixepriestley

Event Timeline

epriestley claimed this task.
epriestley raised the priority of this task from to Normal.
epriestley updated the task description. (Show Details)
epriestley added a subscriber: epriestley.
epriestley closed this task as Wontfix.Apr 15 2015, 3:20 PM

I think I'm going to give up and use offset paging.

We can implement the original class (CompositeQuery) today with relatively few issues (although not "no issues"), but that isn't sufficient in the general case, because I also need some kind of ProxyQuery which turns each user into @user + projects(@user), and #project into #project + not(#project) + any(#project) + member(#project), and can page across them with a cursor.

That sounds pretty reasonable, but I haven't had success generating an implementation I'm comfortable with: you end up with cursors starting in the middle of pages of virtual results and a whole lot of other really horrible edge cases. The amount of time I'm spending on this is just not justified by the value of the feature. Realistically, no one is going to page through 20,000 users or projects by clicking "next" 200 times.