Page MenuHomePhabricator

Reduce the impact of `bin/storage dump`
ClosedPublic

Authored by epriestley on Apr 26 2017, 6:57 PM.
Tags
None
Referenced Files
F13159731: D17795.diff
Mon, May 6, 2:38 AM
Unknown Object (File)
Sun, Apr 28, 2:29 AM
Unknown Object (File)
Tue, Apr 9, 9:49 PM
Unknown Object (File)
Apr 3 2024, 2:36 PM
Unknown Object (File)
Apr 1 2024, 11:56 PM
Unknown Object (File)
Mar 29 2024, 8:30 AM
Unknown Object (File)
Jan 19 2024, 4:52 PM
Unknown Object (File)
Jan 15 2024, 5:22 PM
Subscribers
None

Details

Summary

Ref T12646.

  • Use "wb1" instead of "wb" to use level 1 gzip compression (faster, less compressy). Locally, this went about 2x faster and the output only grew 4% larger.
  • LinesOfALargeExecFuture does a lot of unnecessary string operations, and can boil down to a busy wait. The process is pretty saturated by I/O so this isn't the end of the world, but just use raw ExecFuture with FutureIterator so that we wait in select().
  • Also, nice the process to +19 so we try to give other things CPU.
Test Plan
  • Ran bin/storage dump --compress --output ....
  • Saw CPU time for my local database drop from ~240s to ~90s, with a 4% larger output. Most of this was adding the 1, but the ExecFuture thing helped a little, too.
  • I'm not sure what a great way to test nice in a local environment is and it's system dependent anyway, but nothing got worse / blew up.
  • Used gzcat | head and gzcat | tail on the result to sanity-check that everything was preserved.

Diff Detail

Repository
rP Phabricator
Branch
dump1
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 16719
Build 22305: Run Core Tests
Build 22304: arc lint + arc unit

Event Timeline

This revision is now accepted and ready to land.Apr 26 2017, 7:08 PM
This revision was automatically updated to reflect the committed changes.