Page MenuHomePhabricator

Reduce the impact of `bin/storage dump`
ClosedPublic

Authored by epriestley on Apr 26 2017, 6:57 PM.
Tags
None
Referenced Files
F15413563: D17795.id.diff
Wed, Mar 19, 7:41 PM
Unknown Object (File)
Feb 9 2025, 3:55 AM
Unknown Object (File)
Feb 9 2025, 3:55 AM
Unknown Object (File)
Feb 9 2025, 3:55 AM
Unknown Object (File)
Feb 3 2025, 8:49 AM
Unknown Object (File)
Jan 30 2025, 3:08 AM
Unknown Object (File)
Jan 28 2025, 6:14 PM
Unknown Object (File)
Jan 28 2025, 7:08 AM
Subscribers
None

Details

Summary

Ref T12646.

  • Use "wb1" instead of "wb" to use level 1 gzip compression (faster, less compressy). Locally, this went about 2x faster and the output only grew 4% larger.
  • LinesOfALargeExecFuture does a lot of unnecessary string operations, and can boil down to a busy wait. The process is pretty saturated by I/O so this isn't the end of the world, but just use raw ExecFuture with FutureIterator so that we wait in select().
  • Also, nice the process to +19 so we try to give other things CPU.
Test Plan
  • Ran bin/storage dump --compress --output ....
  • Saw CPU time for my local database drop from ~240s to ~90s, with a 4% larger output. Most of this was adding the 1, but the ExecFuture thing helped a little, too.
  • I'm not sure what a great way to test nice in a local environment is and it's system dependent anyway, but nothing got worse / blew up.
  • Used gzcat | head and gzcat | tail on the result to sanity-check that everything was preserved.

Diff Detail

Repository
rP Phabricator
Branch
dump1
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 16719
Build 22305: Run Core Tests
Build 22304: arc lint + arc unit

Event Timeline

This revision is now accepted and ready to land.Apr 26 2017, 7:08 PM
This revision was automatically updated to reflect the committed changes.