Page MenuHomePhabricator

Reduce the impact of `bin/storage dump`
ClosedPublic

Authored by epriestley on Apr 26 2017, 6:57 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Nov 9, 1:33 PM
Unknown Object (File)
Sat, Nov 9, 1:33 PM
Unknown Object (File)
Sat, Nov 9, 1:33 PM
Unknown Object (File)
Tue, Nov 5, 11:49 PM
Unknown Object (File)
Sat, Oct 26, 7:45 PM
Unknown Object (File)
Thu, Oct 24, 3:55 PM
Unknown Object (File)
Oct 20 2024, 3:33 AM
Unknown Object (File)
Oct 19 2024, 12:05 PM
Subscribers
None

Details

Summary

Ref T12646.

  • Use "wb1" instead of "wb" to use level 1 gzip compression (faster, less compressy). Locally, this went about 2x faster and the output only grew 4% larger.
  • LinesOfALargeExecFuture does a lot of unnecessary string operations, and can boil down to a busy wait. The process is pretty saturated by I/O so this isn't the end of the world, but just use raw ExecFuture with FutureIterator so that we wait in select().
  • Also, nice the process to +19 so we try to give other things CPU.
Test Plan
  • Ran bin/storage dump --compress --output ....
  • Saw CPU time for my local database drop from ~240s to ~90s, with a 4% larger output. Most of this was adding the 1, but the ExecFuture thing helped a little, too.
  • I'm not sure what a great way to test nice in a local environment is and it's system dependent anyway, but nothing got worse / blew up.
  • Used gzcat | head and gzcat | tail on the result to sanity-check that everything was preserved.

Diff Detail

Repository
rP Phabricator
Branch
dump1
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 16719
Build 22305: Run Core Tests
Build 22304: arc lint + arc unit

Event Timeline

This revision is now accepted and ready to land.Apr 26 2017, 7:08 PM
This revision was automatically updated to reflect the committed changes.