Previously, see T12907.
Downloading a >2GB file via HTTPSFuture currently fails. Normally this isn't much of an issue since regular users rarely interact with arc in a way that prompts it to download 2GB files, but this operation is required by export and migration operations in the Phacility cluster (see T13393 and PHI1329 for recent examples). We could switch the transfer channel to something else, but this limitation is silly and we should overcome it even if a different channel is ultimately more suitable for these particular operations down the line.
This is largely a combination of two related issues:
- We hold the entire response in memory.
- We hold the entire response in memory in a string, and PHP has a 2GB string limit.
The fix is conceptually straightforward: stream responses, include options for buffering/streaming them directly to disk, and probably swap the internal storage from a string to a rope in the case that we aren't disk-buffering.
This is somewhat tricky because the available primitives for doing this are limited and it amounts to requiring we have a streaming HTTP parser. We do now, after D19011, but it's not yet clear if this parser is actually functional or robust.