Page MenuHomePhabricator

Landing a diff with files larger than 8M leads to storage engine error
Closed, ResolvedPublic

Assigned To
None
Authored By
OCram
Jan 20 2016, 4:38 PM
Referenced Files
F1065706: pasted_file
Jan 21 2016, 7:16 AM
F1065704: pasted_file
Jan 21 2016, 7:16 AM
F1065702: pasted_file
Jan 21 2016, 7:16 AM
F1065253: pasted_file
Jan 20 2016, 4:38 PM

Description

Situation

I tried to land a diff containing binary files larger than 8M. Therefore I already checked the config descibed in: Configuring File Storage. The landing itself worked fine but the commit is still in progress.

System config

  • My apache config looks like:
NameVirtualHost *:443
<VirtualHost XX.XXX.XXX.XX:443>
        SSLEngine on
        SSLCertificateFile /etc/pki/tls/certs/wildcard.crt
        SSLCertificateKeyFile /etc/pki/tls/private/wildcard.pem

        # Change this to the domain which points to your host.
        ServerName phab.myside.net
        ServerAlias phab2.myside.net

        # Change this to the path where you put 'phabricator' when you checked it
        # out from GitHub when following the Installation Guide.
        #
        # Make sure you include "/webroot" at the end!
        DocumentRoot /opt/phabricator/webroot

        <IfModule mod_rewrite.c>
                RewriteEngine on
                RewriteRule ^/rsrc/(.*)     -                       [L,QSA]
                RewriteRule ^/favicon.ico   -                       [L,QSA]
                RewriteRule ^(.*)$          /index.php?__path__=$1  [B,L,QSA]
        </IfModule>

        <Directory "/opt/phabricator/webroot">
                Require all granted
        </Directory>

        ## mod_php5 directives
        <IfModule mod_php5.c>
                php_admin_flag  engine                  On
                php_value       upload_max_filesize     1G
                php_value       post_max_size           1G
                php_value       output_buffering        16384
                php_admin_value date.timezone           'Europe/Berlin'
                php_value       include_path            '.:/opt/PHPExcel/Classes/'
                php_value       memory_limit            256M

        </IfModule>

        <LocationMatch "\.(?i:os)$">
                ForceType application/octet-stream
                Header set Content-Disposition attachment
        </LocationMatch>
</VirtualHost>
  • php.ini
php.ini
[PHP]
engine = On
short_open_tag = Off
asp_tags = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 32M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 2M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
[CLI Server]
cli_server.color = On
[Date]
date.timezone = 'Europe/Berlin'
[filter]
[iconv]
[intl]
[sqlite]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
sendmail_path = /usr/sbin/sendmail -t -i
mail.add_x_header = On
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQL]
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 60
mysql.trace_mode = Off
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[Sybase-CT]
sybct.allow_persistent = On
sybct.max_persistent = -1
sybct.max_links = -1
sybct.min_server_severity = 10
sybct.min_client_severity = 10
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.bug_compat_42 = Off
session.bug_compat_warn = Off
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[MSSQL]
mssql.allow_persistent = On
mssql.max_persistent = -1
mssql.max_links = -1
mssql.min_error_severity = 10
mssql.min_message_severity = 10
mssql.compatability_mode = Off
mssql.secure_connection = Off
[Assertion]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[mcrypt]
[dba]
  • The storage Engine config looks like:

pasted_file (255×1 px, 32 KB)

Error Details

The phd daemon log shows:

[20-Jan-2016 17:24:35 Europe/Berlin] [2016-01-20 17:24:35] EXCEPTION: (PhutilProxyException) Error while executing Task ID 67700. {>} (Exception) No configured storage engine can store this file. See "Configuring File Storage" in the documentation for information on configuring storage engines. at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:306]
[20-Jan-2016 17:24:35 Europe/Berlin] arcanist(head=master, ref.master=aeb374b33348), phabricator(head=master, ref.master=e84693f589d3), phutil(head=master, ref.master=0fa7efbf09d7)
[20-Jan-2016 17:24:35 Europe/Berlin]   #0 <#2> PhabricatorFile::buildFromFileData(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:391]
[20-Jan-2016 17:24:35 Europe/Berlin]   #1 <#2> PhabricatorFile::newFromFileData(string, array) called at [<phabricator>/src/applications/files/storage/PhabricatorFile.php:218]
[20-Jan-2016 17:24:35 Europe/Berlin]   #2 <#2> PhabricatorFile::buildFromFileDataOrHash(string, array) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionFileContentQueryConduitAPIMethod.php:73]
[20-Jan-2016 17:24:35 Europe/Berlin]   #3 <#2> DiffusionFileContentQueryConduitAPIMethod::newFile(DiffusionGitRequest, string) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionFileContentQueryConduitAPIMethod.php:49]
[20-Jan-2016 17:24:35 Europe/Berlin]   #4 <#2> DiffusionFileContentQueryConduitAPIMethod::getResult(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionQueryConduitAPIMethod.php:148]
[20-Jan-2016 17:24:35 Europe/Berlin]   #5 <#2> DiffusionQueryConduitAPIMethod::execute(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/method/ConduitAPIMethod.php:118]
[20-Jan-2016 17:24:35 Europe/Berlin]   #6 <#2> ConduitAPIMethod::executeMethod(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:135]
[20-Jan-2016 17:24:35 Europe/Berlin]   #7 <#2> ConduitCall::executeMethod() called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:85]
[20-Jan-2016 17:24:35 Europe/Berlin]   #8 <#2> ConduitCall::execute() called at [<phabricator>/src/applications/diffusion/query/DiffusionQuery.php:81]
[20-Jan-2016 17:24:35 Europe/Berlin]   #9 <#2> DiffusionQuery::callConduitWithDiffusionRequest(PhabricatorUser, DiffusionGitRequest, string, array) called at [<phabricator>/src/applications/differential/engine/DifferentialDiffExtractionEngine.php:158]
[20-Jan-2016 17:24:35 Europe/Berlin]   #10 <#2> DifferentialDiffExtractionEngine::isDiffChangedBeforeCommit(PhabricatorRepositoryCommit, DifferentialDiff, DifferentialDiff) called at [<phabricator>/src/applications/differential/engine/DifferentialDiffExtractionEngine.php:226]
[20-Jan-2016 17:24:35 Europe/Berlin]   #11 <#2> DifferentialDiffExtractionEngine::updateRevisionWithCommit(DifferentialRevision, PhabricatorRepositoryCommit, array, PhabricatorContentSource) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:233]
[20-Jan-2016 17:24:35 Europe/Berlin]   #12 <#2> PhabricatorRepositoryCommitMessageParserWorker::updateCommitData(DiffusionCommitRef) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryGitCommitMessageParserWorker.php:11]
[20-Jan-2016 17:24:35 Europe/Berlin]   #13 <#2> PhabricatorRepositoryGitCommitMessageParserWorker::parseCommitWithRef(PhabricatorRepository, PhabricatorRepositoryCommit, DiffusionCommitRef) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:40]
[20-Jan-2016 17:24:35 Europe/Berlin]   #14 <#2> PhabricatorRepositoryCommitMessageParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:40]
[20-Jan-2016 17:24:35 Europe/Berlin]   #15 <#2> PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
[20-Jan-2016 17:24:35 Europe/Berlin]   #16 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
[20-Jan-2016 17:24:35 Europe/Berlin]   #17 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[20-Jan-2016 17:24:35 Europe/Berlin]   #18 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:183]
[20-Jan-2016 17:24:35 Europe/Berlin]   #19 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:125]

final question

  • So is it possible to increase the limit for commited files?

Event Timeline

upload_max_filesize: Maximum file size PHP will accept in a raw file upload. This is not normally used when uploading files via drag-and-drop, but affects some other kinds of file uploads. If you exceed this, Phabricator will give you a useful error. This often defaults to 2M. Set this to at least 32MB.

It'd be good to know if this resolves your issue, since our error message is debatable on "useful".

This is an issue in the implementation, not a configuration issue.

I increased upload_max_filesize in the vhost to 1G. Afterwards the arc land itself worked but the commit done by phd still shows the descibed log entry.

Here are some more details about the error:

  • Diffusion / Commit page:

pasted_file (592×786 px, 64 KB)

  • Daemon stats:

pasted_file (820×1 px, 111 KB)

pasted_file (428×686 px, 48 KB)

So I really think increased all apache and php vars which could limit a upload size. In general I was wondering where the limit of 8M for max file size on the local disk storage comes from.

Sorry, I wasn't very clear -- I know exactly why this is happening. It isn't related to configuration or arcanist (it's an implementation issue related to D14970), so I don't need any more information.

(And there are no configuration settings you can change which will possibly impact or resolve it -- it can only be resolved by an upstream fix to the problematic implementation.)

I believe this is fixed at head. Let us know if you're still seeing issues.

Thanks for the report!

Thank you very much! You solved all my problems :D

itsbeautiful