Page MenuHomePhabricator

Support export of data in files larger than 8MB
ClosedPublic

Authored by epriestley on Jan 29 2018, 3:04 AM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Apr 19, 2:45 AM
Unknown Object (File)
Wed, Apr 17, 5:33 AM
Unknown Object (File)
Thu, Apr 11, 10:06 AM
Unknown Object (File)
Mon, Apr 8, 2:28 PM
Unknown Object (File)
Fri, Mar 29, 3:20 PM
Unknown Object (File)
Mar 23 2024, 10:31 AM
Unknown Object (File)
Mar 5 2024, 1:36 AM
Unknown Object (File)
Feb 16 2024, 12:07 PM
Subscribers
None

Details

Summary

Depends on D18952. Ref T13049. For files larger than 8MB, we need to engage the chunk storage engine. PhabricatorFile::newFromFileData() always writes a single chunk, and can't handle files larger than the mandatory chunk threshold (8MB).

Use IteratorUploadSource, which can, and "stream" the data into it. This should raise the limit from 8MB to 2GB (maximum size of a string in PHP).

If we need to go above 2GB we could stream CSV and text pretty easily, and JSON without too much trouble, but Excel might be trickier. Hopefully no one is trying to export 2GB+ datafiles, though.

Test Plan
  • Changed the JSON exporter to just export 8MB of the letter "q": return str_repeat('q', 1024 * 1024 * 9);.
  • Before change: fatal, "no storage engine can store this file".
  • After change: export works cleanly.

Diff Detail

Repository
rP Phabricator
Branch
export3
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 19236
Build 25988: Run Core Tests
Build 25987: arc lint + arc unit