I have no idea why this works through the web UI...

I'm also seeing this issue, even just using curl -d api.token=... https://phabricator.url/api/file.download

This might be because the file in question is ~150MB, so it's probably trying to download the whole thing and base64 it (?!) before spitting it back.

In T8348#143186, @nornagon wrote:

I'm also seeing this issue, even just using curl -d api.token=... https://phabricator.url/api/file.download

I wouldn't expect curl to ever support this. The way arc download will support chunked files is by downloading chunks and then pieces that back together client side. This wouldn't be possible with pure curl.

@epriestley, I was thinking of implementing this as follows:

1. Have file chunks inherit policies from the parent file, so that individual chunks can be downloaded with file.download.
2. Return file PHIDs for chunks from file.querychunks so that we can iterate over all chunks.
3. Write an ArcanistFileDownloader class similar to ArcanistFileUploader.

I was hoping to copy whatever "Download File" does, but its not obvious to me how this functionality works.

A possibly easier approach is to let the user just get an HTTP URI for the download (i.e. a normal, non-Conduit URI) and then call it in a standard way. Maybe by adding a needDownloadURIs parameter to file.info (or waiting for T7715 and adding it to file.query) -- the links are slightly expensive to generate because we have to do one-time token writes so it's probably better not to generate them by default.

This is much simpler, although it won't support parallel downloads or resumable downloads by default. However, we should accept and respect the "Range" HTTP header and be able to process it efficiently, so if we did want to build those features it would probably be less complex overall to just use "Range" headers on the client.

I think that would leave us with only two potential scale issues:

• We'll still buffer the whole file in memory on the client. For very large files, we need to adjust HTTPSFuture to be able to stream response bodies directly to disk (this isn't too difficult).
• We may still need a set_time_limit(-1) call somewhere in the pathway of PhabricatorFileDataController to actually prevent the 30s timeout.

Could the server do the chunk stitching and stream the response to the
client, rather than fetching it all up front?

Could the server do the chunk stitching and stream the response to the client, rather than fetching it all up front?

This is what it does for normal HTTP downloads.

Can you point me at the code where the chunk stitching happens?

PhabricatorFileChunkIterator does the stitching. Basically the flow is like this:

• The controller loads the PhabricatorFile and does policy, etc., checks.
• It asks the StorageEngine for an iterator, then configures it with start/end if there's a "Range" header.
• It wraps the iterator in a FileResponse and returns that.

The FileResponse iterates the iterator and dumps the results out as a response stream.

The iterator loads chunks one at a time and hands them off to the response.

After D17614, arc download uses file.search to retrieve a URI it can GET, then does a normal HTTP GET to that URI, retrieving the file content in the HTTP response body.