Page MenuHomePhabricator

Add a `bin/bulk export` CLI tool to make debugging and profiling large exports easier
ClosedPublic

Authored by epriestley on Jan 30 2018, 1:54 PM.
Tags
None
Referenced Files
F15444059: D18965.id.diff
Thu, Mar 27, 7:34 AM
F15442802: D18965.diff
Thu, Mar 27, 12:34 AM
F15411070: D18965.diff
Wed, Mar 19, 8:48 AM
F15410205: D18965.diff
Wed, Mar 19, 6:54 AM
F15408584: D18965.diff
Wed, Mar 19, 12:01 AM
F15404166: D18965.id45487.diff
Tue, Mar 18, 6:33 AM
F15393912: D18965.id45492.diff
Sat, Mar 15, 11:15 PM
F15393911: D18965.id45487.diff
Sat, Mar 15, 11:15 PM
Subscribers
None

Details

Summary

Ref T13049. When stuff executes asynchronously on the bulk workflow it can be hard to inspect directly, and/or a pain to test because you have to go through a bunch of steps to run it again.

Make future work here easier by making export triggerable from the CLI. This makes it easy to repeat, inspect with --trace, profile with --xprofile, etc.

Test Plan
  • Ran several invalid commands, got sensible error messages.
  • Ran some valid commands, got exported data.
  • Used --xprofile to look at the profile for a 300MB dump of 100K tasks which took about 40 seconds to export. Nothing jumped out as sketchy to me -- CustomField wrangling is a little slow but most of the time looked like it was being spent legitimately.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable