Page MenuHomePhabricator

Paginate transactions on objects once there are a huge number of them
Closed, ResolvedPublic

Description

If objects (like tasks) accumulate a huge number of transactions (say, many thousands), we'll load and display all the transactions. Although this is is very rare under normal circumstances, some types of long-lived objects (configuration? repositories?) will probably tend to accumulate a huge number of transactions over time, and automated scripts can produce a large number of transactions quickly.

Instead, we should show the most recent 500, and then let the user click somewhere to load more in chunks of 500 or something.

Event Timeline

epriestley raised the priority of this task from to Low.
epriestley updated the task description. (Show Details)
epriestley added a project: Transactions.
epriestley added subscribers: epriestley, frgtn.
epriestley raised the priority of this task from Low to Normal.

T6088 is an example of this disrupting things in the wild, rather than just being a performance issue. It can also help with T5834.

  • We have a large number of callsites where code executes a TransactionQuery and then builds a TransactionView. Pull this somewhere common (maybe PhabricatorControler or somewhere in ApplicationTransactions) with some signature like buildTransactionTimeline($viewer, $object, new WhateverQuery(), new WhateverView()).
  • This will generally clean things up, and incidentally make T5834 easier to implement by letting us add needTokens(true) in one place.
  • Instead of loading all transactions, use a pager to load the 100 most recent (for whatever value of 100 feels good).
  • Pass the pager to the transaction view and have it render a "this thread is very long. Show older changes" kind of thing.
  • Some new endpoint in Transactions ajaxes things in 100 at a time.
  • The "changes from before the last time you commented" thing interacts with this. It might be nice to combine them:
    • After loading the first page, apply the "hide transactions before your last comment" filter.
    • Only render what isn't filtered out.
    • Start the ajaxing pager at the first hidden transaction.
    • A single new endpoint serves both use cases.
    • If you've never commented, we kick in at 100 transactions.

There's probably some optimization opportunity in the rendering pipeline too, but the cost on T603 isn't wildly unreasonable (~5ms per transaction) so we probably can't easily make it like 10x faster.

I was holding off marking this as resolved to 1) wait a bit to fix the bugs as they came in and 2) clean up some reverse paging code.

I think the incoming bugs have stopped for a few weeks now - yay...!
I have been trying to clean up the reverse paging stuff added in D10887, but have just decided its actually pretty good to default as is, post looking at all sorts of callsites of course.