Page MenuHomePhabricator

Reduce the impact of `bin/storage dump`
ClosedPublic

Authored by epriestley on Apr 26 2017, 6:57 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Nov 9, 1:33 PM
Unknown Object (File)
Sat, Nov 9, 1:33 PM
Unknown Object (File)
Sat, Nov 9, 1:33 PM
Unknown Object (File)
Tue, Nov 5, 11:49 PM
Unknown Object (File)
Sat, Oct 26, 7:45 PM
Unknown Object (File)
Thu, Oct 24, 3:55 PM
Unknown Object (File)
Oct 20 2024, 3:33 AM
Unknown Object (File)
Oct 19 2024, 12:05 PM
Subscribers
None

Details

Summary

Ref T12646.

  • Use "wb1" instead of "wb" to use level 1 gzip compression (faster, less compressy). Locally, this went about 2x faster and the output only grew 4% larger.
  • LinesOfALargeExecFuture does a lot of unnecessary string operations, and can boil down to a busy wait. The process is pretty saturated by I/O so this isn't the end of the world, but just use raw ExecFuture with FutureIterator so that we wait in select().
  • Also, nice the process to +19 so we try to give other things CPU.
Test Plan
  • Ran bin/storage dump --compress --output ....
  • Saw CPU time for my local database drop from ~240s to ~90s, with a 4% larger output. Most of this was adding the 1, but the ExecFuture thing helped a little, too.
  • I'm not sure what a great way to test nice in a local environment is and it's system dependent anyway, but nothing got worse / blew up.
  • Used gzcat | head and gzcat | tail on the result to sanity-check that everything was preserved.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

This revision is now accepted and ready to land.Apr 26 2017, 7:08 PM
This revision was automatically updated to reflect the committed changes.