Ref T12907. The major blocker to doing a shard rebalance/migration in the cluster is that we can't download files which are larger than 2GB via native stuff, since the entire response is held in memory.
We could just make the thing shell out to wget, but this limitation affects other things like arc download so it would be nice to fix it properly.
I'd like to replace the "hold the whole thing in memory" parser with a streaming parser, and then let the streaming parser stream the response bodies to disk.
To this end, provide a streaming parser with some tests. These aren't exhaustive, but should at least cover the basics fairly well.
This doesn't actually do anything yet.