Page MenuHomePhabricator

Provide a streaming HTTP response parser
ClosedPublic

Authored by epriestley on Feb 7 2018, 12:14 AM.
Tags
None
Referenced Files
Unknown Object (File)
Wed, Feb 26, 4:55 PM
Unknown Object (File)
Fri, Feb 21, 3:32 PM
Unknown Object (File)
Sat, Feb 15, 11:08 PM
Unknown Object (File)
Thu, Feb 13, 4:16 AM
Unknown Object (File)
Jan 30 2025, 10:19 PM
Unknown Object (File)
Jan 30 2025, 7:49 AM
Unknown Object (File)
Jan 28 2025, 3:14 AM
Unknown Object (File)
Jan 28 2025, 3:14 AM
Subscribers
None

Details

Summary

Ref T12907. The major blocker to doing a shard rebalance/migration in the cluster is that we can't download files which are larger than 2GB via native stuff, since the entire response is held in memory.

We could just make the thing shell out to wget, but this limitation affects other things like arc download so it would be nice to fix it properly.

I'd like to replace the "hold the whole thing in memory" parser with a streaming parser, and then let the streaming parser stream the response bodies to disk.

To this end, provide a streaming parser with some tests. These aren't exhaustive, but should at least cover the basics fairly well.

This doesn't actually do anything yet.

Test Plan

Ran tests.

Diff Detail

Repository
rPHU libphutil
Lint
Lint Not Applicable
Unit
Tests Not Applicable