Unable to upload file: failed to read 4583864320 bytes after offset 0
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	joshuaspence
	Apr 10 2017, 10:21 PM

Description

I'm trying to upload a 4.3GB tar containing log files to Files and am hitting an exception:

> arc upload --trace logs.tar
 ARGV  '/usr/local/src/arcanist/bin/../scripts/arcanist.php' 'upload' '--trace' 'logs.tar'
 LOAD  Loaded "phutil" from "/usr/local/src/libphutil/src".
 LOAD  Loaded "arcanist" from "/usr/local/src/arcanist/src".
Config: Reading user configuration file "/home/josh/.arcrc"...
Config: Did not find system configuration at "/etc/arcconfig".
Working Copy: Reading .arcconfig from "/home/josh/workspace/REDACTED/.arcconfig".
Working Copy: Path "/home/josh/workspace/REDACTED" is part of `git` working copy "/home/josh/workspace/REDACTED".
Working Copy: Project root is at "/home/josh/workspace/REDACTED".
Config: Did not find local configuration at "/home/josh/workspace/REDACTED/.git/arc/config".
Loading phutil library from '/home/josh/workspace/REDACTED/support/flarc/src'...
>>> [0] <conduit> user.whoami() <bytes = 117>
>>> [1] <http> https://phabricator.REDACTED/api/user.whoami
<<< [1] <http> 958,490 us
<<< [0] <conduit> 959,524 us
>>> [2] <conduit> file.allocate() <bytes = 282>
>>> [3] <http> https://phabricator.REDACTED/api/file.allocate
<<< [3] <http> 330,927 us
<<< [2] <conduit> 331,305 us

[2017-04-10 22:18:00] EXCEPTION: (Exception) Unable to upload file data: Unable to upload file: failed to read 4583864320 bytes after offset 0 from file at path "logs.tar". at [<arcanist>/src/workflow/ArcanistUploadWorkflow.php:87]
arcanist(head=master, ref.master=a59cfca5f190), flarc(), phutil(head=master, ref.master=fb9e0642c4ea)
  #0 ArcanistUploadWorkflow::run() called at [<arcanist>/scripts/arcanist.php:394]

The log files are sensitive, so I can't upload them to this install in an attempt to reproduce. Let me see if I can come up with a dummy file to reproduce the error.

Phabricator	d1421bc3a1ee464b30d826c13d3ccb763394a8df
Arcanist	a59cfca5f190c44403dfc7449c678a2aa1626bb4
phutil	c581e769f10c6d2b427900897edba74e01a572bd

Revisions and Commits

rP Phabricator
	D17651	rP944f7da48670 (stable) Correct two parameter strictness issues with file uploads
	D17651	rPa7a068f84cd0 Correct two parameter strictness issues with file uploads

Related Objects

Mentioned In: T12515: Upgrading: File Integrity Hashing and SHA1
T12532: `syntax.filemap` does not validate input
Mentioned Here: D17651: Correct two parameter strictness issues with file uploads
D17616: Make the Files "TTL" API more structured
rPHUc581e769f10c: Allow daemons to read their own idle state
rARCa59cfca5f190: Upgrade "arc upload" to use SHA256
rPd1421bc3a1ee: Add "bin/storage optimize" to run OPTIMIZE TABLE on everything

Event Timeline

joshuaspence created this task.Apr 10 2017, 10:21 PM

Herald added a subscriber: eadler. · View Herald TranscriptApr 10 2017, 10:21 PM

Okay, I managed to reproduce it with non-sensitive data:

> dd if=/dev/zero of=test.img bs=1024 count=0 count=$[4*1024*1024]
4194304+0 records in
4194304+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 43.7755 s, 98.1 MB/s

> arc upload test.img
Exception
Unable to upload file data: Unable to upload file: failed to read 4294967296 bytes after offset 0 from file at path "test.img".
(Run with `--trace` for a full exception trace.)

> arc --conduit-uri https://secure.phabricator.com upload test.img
Exception
Unable to upload file data: Unable to upload file: failed to read 4294967296 bytes after offset 0 from file at path "test.img".
(Run with `--trace` for a full exception trace.)

joshuaspence mentioned this in T12532: `syntax.filemap` does not validate input.Apr 10 2017, 10:33 PM

From the server-side error logs:

[11-Apr-2017 08:34:27 Australia/Sydney] [2017-04-11 08:34:27] EXCEPTION: (PhutilTypeExtraParametersException) Got unexpected parameters: chunkedHash at [<phutil>/src/parser/PhutilTypeSpec.php:150]
[11-Apr-2017 08:34:27 Australia/Sydney] arcanist(head=master, ref.master=a59cfca5f190), phabricator(head=master, ref.master=d1421bc3a1ee), phlab(head=master, ref.master=84b62e6917ec), phutil(head=master, ref.master=c581e769f10c)
[11-Apr-2017 08:34:27 Australia/Sydney]   #0 <#2> PhutilTypeSpec::checkMap(array, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:1312]
[11-Apr-2017 08:34:27 Australia/Sydney]   #1 <#2> PhabricatorFile::readPropertiesFromParameters(array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:274]
[11-Apr-2017 08:34:27 Australia/Sydney]   #2 <#2> PhabricatorFile::newChunkedFile(PhabricatorChunkedFileStorageEngine, integer, array) called at [<phabricator>/src/applications/files/engine/PhabricatorChunkedFileStorageEngine.php:114]
[11-Apr-2017 08:34:27 Australia/Sydney]   #3 <#2> PhabricatorChunkedFileStorageEngine::allocateChunks(integer, array) called at [<phabricator>/src/applications/files/conduit/FileAllocateConduitAPIMethod.php:116]
[11-Apr-2017 08:34:27 Australia/Sydney]   #4 <#2> FileAllocateConduitAPIMethod::execute(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/method/ConduitAPIMethod.php:123]
[11-Apr-2017 08:34:27 Australia/Sydney]   #5 <#2> ConduitAPIMethod::executeMethod(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:131]
[11-Apr-2017 08:34:27 Australia/Sydney]   #6 <#2> ConduitCall::executeMethod() called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:81]
[11-Apr-2017 08:34:27 Australia/Sydney]   #7 <#2> ConduitCall::execute() called at [<phabricator>/src/applications/conduit/controller/PhabricatorConduitAPIController.php:83]
[11-Apr-2017 08:34:27 Australia/Sydney]   #8 phlog(PhutilTypeExtraParametersException) called at [<phabricator>/src/applications/conduit/controller/PhabricatorConduitAPIController.php:103]
[11-Apr-2017 08:34:27 Australia/Sydney]   #9 PhabricatorConduitAPIController::handleRequest(AphrontRequest) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:269]
[11-Apr-2017 08:34:27 Australia/Sydney]   #10 AphrontApplicationConfiguration::processRequest(AphrontRequest, PhutilDeferredLog, AphrontPHPHTTPSink, MultimeterControl) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:181]
[11-Apr-2017 08:34:27 Australia/Sydney]   #11 AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phabricator>/webroot/index.php:17]

Does this reproduce with a 5MB file?

Probably just fallout from D17616 but easier to debug if I don't need to upload 4GB.

(Actually maybe you need a 9MB file.)

I originally tried a 1GB file and wasn't able to reproduce...

Hmm, I got a different error with a 9MB file. But this looks like a problem on our end:

> arc upload test.img
Exception
Unable to upload file data: ERR-CONDUIT-CORE: No configured storage engine can store this file. See "Configuring File Storage" in the documentation for information on configuring storage engines.
(Run with `--trace` for a full exception trace.)

Alright, let me take a look. That exception is pretty specific and it may be obvious from the code.

I think the first error is causing (maybe?) the chunk engine to fail, so we're falling back to the "upload the entire file in one shot" engine, which ain't great on 4GB files.

epriestley added a revision: D17651: Correct two parameter strictness issues with file uploads.Apr 10 2017, 10:49 PM

Yeah, pretty sure what's happening is:

We try to do a chunked upload.
The trivial chunkedHash failure kills that.
We fall back to deciding that reading 4GB of data into a big string, base64 + JSON encoding it, and then POSTing it is a great idea.
With the 9MB file I think we make it fairly far before the backend is like "hey, you should be chunking this, what's going on"?
With the 4.3GB file we die much sooner. If we made it slightly further, we'd hit PHP's 2GB string limit anyway.

Not sure if this is useful:

> arc --conduit-uri https://secure.phabricator.com upload 1GB.img
Exception
Unable to upload file data: [HTTP/500] Internal Server Error
As received by the server, this request had a nonzero content length but no POST data.

Normally, this indicates that it exceeds the 'post_max_size' setting in the PHP configuration on the server. Increase the 'post_max_size' setting or reduce the size of the request.

Request size according to 'Content-Length' was '1431655947', 'post_max_size' is set to '64M'.

Yeah, that looks like it's consistent with the hypothesis above. I expect:

Files larger than 2GB: failure in fread().
Files smaller than ~2GB but larger than ~64MB: failure to POST.
Files smaller than ~64MB but larger than ~4MB: "no storage engine can store this".
Files smaller than ~4MB: work.

D17651 should make all of them work.