Page MenuHomePhabricator

Fix some issues with Diffusion file data limits
ClosedPublic

Authored by epriestley on Jan 19 2018, 10:17 PM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, Apr 25, 1:42 AM
Unknown Object (File)
Sun, Apr 21, 3:23 AM
Unknown Object (File)
Sun, Apr 21, 2:03 AM
Unknown Object (File)
Fri, Apr 19, 1:29 AM
Unknown Object (File)
Thu, Apr 11, 9:31 AM
Unknown Object (File)
Tue, Apr 9, 2:03 AM
Unknown Object (File)
Tue, Apr 2, 3:26 AM
Unknown Object (File)
Fri, Mar 29, 2:22 AM
Subscribers
None

Details

Summary

See https://discourse.phabricator-community.org/t/files-created-from-repository-contents-slightly-over-one-chunk-in-size-are-truncated-to-exactly-one-chunk-in-size/988/1. Three issues here:

  • When we finish reading git cat-file ... or whatever, we can end up with more than one chunk worth of bytes left in the internal buffer if the read is fast. Use while instead of if to make sure we write the whole buffer.
  • Limiting output with setStdoutSizeLimit() isn't really a reliable way to limit the size if we're also reading from the buffer. It's also pretty indirect and confusing. Instead, just let the FileUploadSource explicitly implement a byte limit in a straightforward way.
  • We weren't setting the time limit correctly on the main path.

Overall, this could cause >4MB files to "write" as 4MB files, with the rest of the file left in the UploadSource buffer. Since these files were technically under the limit, they could return as valid. This was intermittent.

Test Plan
  • Pushed a ~4.2MB file.
  • Reloaded Diffusion a bunch, sometimes saw the while/if buffer race and produce a 4MB file with a prompt to download it. (Other times, the buffer worked right and the page just says "this file is too big, sorry").
  • Applied patches.
  • Reloaded Diffusion a bunch, no longer saw bad behavior or truncated files.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable