Page MenuHomePhabricator

Add a `bin/bulk export` CLI tool to make debugging and profiling large exports easier
ClosedPublic

Authored by epriestley on Jan 30 2018, 1:54 PM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, Dec 26, 10:13 AM
Unknown Object (File)
Sat, Dec 21, 11:10 PM
Unknown Object (File)
Sat, Dec 21, 11:10 PM
Unknown Object (File)
Sat, Dec 21, 11:10 PM
Unknown Object (File)
Sat, Dec 21, 11:10 PM
Unknown Object (File)
Tue, Dec 10, 2:36 PM
Unknown Object (File)
Mon, Dec 9, 12:24 AM
Unknown Object (File)
Sun, Dec 8, 3:52 AM
Subscribers
None

Details

Summary

Ref T13049. When stuff executes asynchronously on the bulk workflow it can be hard to inspect directly, and/or a pain to test because you have to go through a bunch of steps to run it again.

Make future work here easier by making export triggerable from the CLI. This makes it easy to repeat, inspect with --trace, profile with --xprofile, etc.

Test Plan
  • Ran several invalid commands, got sensible error messages.
  • Ran some valid commands, got exported data.
  • Used --xprofile to look at the profile for a 300MB dump of 100K tasks which took about 40 seconds to export. Nothing jumped out as sketchy to me -- CustomField wrangling is a little slow but most of the time looked like it was being spent legitimately.

Diff Detail

Repository
rP Phabricator
Branch
export1
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 19275
Build 26043: Run Core Tests
Build 26042: arc lint + arc unit