Allow diffusion.filecontentquery to load data for arbitrarily large files
ClosedPublic
Actions

Authored by epriestley on Jan 21 2016, 5:42 PM.

Details

Reviewers

chad

Maniphest Tasks

T10186: Landing a diff with files larger than 8M leads to storage engine error

Commits

Restricted Diffusion Commit
rPb51a859636ac: Allow diffusion.filecontentquery to load data for arbitrarily large files

Summary

Fixes T10186. After D14970, diffusion.filecontentquery puts the content in a file and returns the file PHID.

However, it does this in a way that doesn't go through the chunking engine, so it will fail for files larger than the chunk threshold (generally, 8MB).

Instead, stream the file from the underlying command directly into chunked storage.

Test Plan

Made a commit including a really big file: https://github.com/epriestley/poems/commit/4dcd4c492b31d82f0fec6f447cf3719b9cc079e0
Used diffusion.filecontentquery to load file content.
Parsed/imported commit locally.
Used diffusion.filecontentquery to load content for smaller files (README, etc).

Diff Detail

Repository

rP Phabricator

Branch

upload1

Lint

Lint Passed

Unit

Tests Passed

Build Status

Buildable 10229
Build 12435: Run Core Tests
Build 12434: arc lint + arc unit

Event Timeline

epriestley updated this revision to Diff 36393.Jan 21 2016, 5:42 PM

epriestley retitled this revision from to Allow diffusion.filecontentquery to load data for arbitrarily large files.

epriestley updated this object.

epriestley edited the test plan for this revision. (Show Details)

epriestley added a reviewer: chad.

epriestley added a task: T10186: Landing a diff with files larger than 8M leads to storage engine error.

I also tried to make the table on Applications → Files → Configure a little more clear and less technical, since it currently looks like only 8MB files are supported even when a chunk engine is configured. It now just shows "Limit: Unlimited" for the chunk engine: