Page MenuHomePhabricator

Task detail pages are issuing an egregious number of queries?
Closed, ResolvedPublic

Description

When I load T9950 on this install, DarkConsole reports 473 queries. We should require fewer than 473 queries to render this page.

The DarkConsole "Analyze Query Plans" button does not work with Quicksand. This button should probably always execute a real navigation. This likely isn't trivial to fix so I just noted it on T13081.

The DarkConsole "Services" tab has some weird looking rendering on the tables, particularly an odd header border. This element looks janky, even for the award-winning epriestley design in DarkConsole.

There is no way to get stack traces for service calls in DarkConsole. It would be nice to provide these when "Analyze Query Plans" is activated.

There is no way to download/export a DarkConsole trace to share with someone. This would occasionally be useful for support workflows.

The "Events" tab could probably be removed.

See PHI448. This isn't exactly related, but it looks like D19213 probably introduced an infinite loop in policy evaluation.

Event Timeline

epriestley triaged this task as Normal priority.Mar 14 2018, 5:17 PM
epriestley created this task.
epriestley updated the task description. (Show Details)Mar 14 2018, 5:31 PM

Some of this is:

  • Project handles issue a needImages() query.
  • This doesn't have a trivial cache and must always execute at least one additional query.
  • The fallback query behavior for builtins from D18174 is common and unbatched.
epriestley updated the task description. (Show Details)Mar 14 2018, 6:05 PM

For my local test task (T369), the initial page query load was 199 queries. After D19222 we're down slightly to 151 queries.

Next problem:

  • When we load builtin files, we load all the objects they're attached to so that we can do policy checks.
  • This is utterly pointless since all users can always see builtin files. They're default files we ship with and freely available on disk regardless of how carefully we apply policy checks to them.

Queries are now down from 151 to 145 for me locally.

Next issue:

  • When we load builtin files, we load transform sources so we can do policy checks.
  • This is very similar to the attachment thing.
  • This is pointless since builtins are: never transforms; and always visible anyway.

Queries are down to 139 for me locally.

Next issue, which may be tough: we're loading projects in 6 separate places, without apparently taking advantage of handle pools:

  • PhabricatorApplicationTransactionQuery uses older handle code (because some transactions may still use it) and forces handles to load inline with iterator_to_array(). However, this should prime the handle pool?
  • PhabricatorProjectsCurtainExtension -> PhabricatorBoardLayoutEngine is loading project profile images as a side effect of loading objects to do policy checks. This is unnecessary, since we don't need profile images to do policy checks (or most other things).
  • ManiphestEditEngine -> PhabricatorBoardLayoutEngine is doing something similar to test if the viewer can reassign the task while building curtain actions. This is unused and can be trivially removed.
  • And then it does this again (!). This is all moot.
  • Then we do it again to figure out the values for "Change Project Tags" in the comment dropdown.
  • Then we do it again.

So (3) and (4) can be trivially removed. I suspect (5) and (6) can be combined. We may be able to combine (2) and (5+6), although this may be a larger change. It may be possible to reduce the cost of this combined load. Handle pools appear to be working correctly; only the initial load is an actual handle load.

I'm having some difficulty merging (5) and (6), or either of them into (2). My local test also has a lot of nonsense going on that a real install shouldn't. I'm going to land what I've got so far and see how much progress we've made here.

With that stuff, we're down from 473 to 227 on T9950. That's still awful, but not quite as ridiculous.

epriestley closed this task as Resolved.Mar 15 2018, 3:37 AM

Some of this got fixed, some ended up in T13081.

These would be nice but they're reasonably covered elsewhere:

There is no way to download/export a DarkConsole trace to share with someone. This would occasionally be useful for support workflows.
The "Events" tab could probably be removed.

And performance can always improve.

For what it's worth, after spot-checking a few tasks on my install, all seem to query the same policy 10ish times in succession near the end of the load.

Copy-paste from dark console. Above and below are omitted for brevity, but no intermediate lines.

query	+479 ms	285 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+480 ms	128 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+481 ms	121 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+482 ms	126 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+483 ms	121 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+484 ms	124 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+485 ms	126 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+486 ms	122 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+487 ms	126 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+487 ms	121 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 	Disabled
query	+489 ms	125 us	SELECT * FROM `policy` WHERE phid IN ('PHID-PLCY-n4665lvgpguiwju4opw2') 
event	+493 ms	22 us	-	
event	+494 ms	23 us	-	
kvcache-get	+495 ms	25 us	-`