Page MenuHomePhabricator

Eternally failing mail send task
Closed, InvalidPublic

Description

Phabricator daemons are eternally failing to send a mail message on my server because the message is too big.

[2015-11-16 15:19:35] EXCEPTION: (PhutilProxyException) Error while executing Task ID 847959. {>} (CommandException) Command failed with error #75!
COMMAND
/usr/sbin/sendmail -oi -t

STDOUT
(empty)

STDERR
postdrop: warning: uid=501: File too large
sendmail: fatal: osystem(501): message file too big
 at [<phutil>/src/future/exec/ExecFuture.php:416]
arcanist(head=master, ref.master=66ab1c955d27), phabricator(head=master, ref.master=26a235ab8a18), phutil(head=master, ref.master=e9ed72483a14)
  #0 <#2> ExecFuture::resolvex() called at [<phabricator>/externals/phpmailer/class.phpmailer-lite.php:540]
  #1 <#2> PHPMailerLite::SendmailSend(string, string) called at [<phabricator>/externals/phpmailer/class.phpmailer-lite.php:499]
  #2 <#2> PHPMailerLite::Send() called at [<phabricator>/src/applications/metamta/adapter/PhabricatorMailImplementationPHPMailerLiteAdapter.php:104]
  #3 <#2> PhabricatorMailImplementationPHPMailerLiteAdapter::send() called at [<phabricator>/src/applications/metamta/storage/PhabricatorMetaMTAMail.php:733]
  #4 <#2> PhabricatorMetaMTAMail::sendNow() called at [<phabricator>/src/applications/metamta/PhabricatorMetaMTAWorker.php:26]
  #5 <#2> PhabricatorMetaMTAWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
  #6 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
  #7 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
  #8 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:183]
  #9 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:125]

Related Objects

Event Timeline

jcarrillo7 updated the task description. (Show Details)
jcarrillo7 added projects: Mail, Daemons.
jcarrillo7 added a subscriber: jcarrillo7.

As a side work around, is there anyway to "complete" this task or kick it off the queue? It really is not that important if one email fails to be sent.

I ran this command out of curiosity:

./bin/mail show-outbound --id 156266

and it appears phabricator has attempted to shove the diff of a binary file into the email which is why its too large to send.

The particular file in question was a .yuv file which is used for video data format file.

I just noticed the Diff UI also attempts to show this file.

The default value of metamta.email-body-limit is 512KB. Does your server reject 512KB messages, or have you raised this limit, or do you expect 512KB messages to send cleanly?

The default value of metamta.email-body-limit is 512KB. Does your server reject 512KB messages, or have you raised this limit, or do you expect 512KB messages to send cleanly?

The mail server does not reject at 512KB. The limit on phabricator is still set at 512KB. The email it tried to send was 41.7 MB.

I can't immediately reproduce this. Here's what I did:

  • I set metamta.differential.inline-patches to 99999999.
  • I set metamta.email-body-limit to 1024 (i.e., 1KB).
  • I restarted the daemons for the changes to take effect.
  • I created a revision out of some arbitrary changes.

We don't actually apply truncation until we hand the message to the mailer, so it's difficult to observe the truncation. I did this, in lieu of configuring actual mail delivery locally:

I patched in a dump of the body handed to the mailer:

$ git diff HEAD
diff --git a/src/applications/metamta/storage/PhabricatorMetaMTAMail.php b/src/applications/metamta/storage/PhabricatorMetaMTAMail.php
index c4793ed..9e9c9b9 100644
--- a/src/applications/metamta/storage/PhabricatorMetaMTAMail.php
+++ b/src/applications/metamta/storage/PhabricatorMetaMTAMail.php
@@ -632,6 +632,9 @@ final class PhabricatorMetaMTAMail
         $body .= "\n";
         $body .= pht('(This email was truncated at %d bytes.)', $max);
       }
+
+      var_dump($body);
+
       $mailer->setBody($body);
 
       $html_emails = false;

Then I used bin/worker execute --id <id> to send the message in the foreground:

$ ./bin/worker execute --id 495535
Executing task 495535 (PhabricatorMetaMTAWorker)...
string(1059) "admin created this revision.
admin added a reviewer: dog.
admin updated this object.
Herald added a subscriber: squeakybirdo.
Herald added a reviewer: saurus.

TEST PLAN
  qwer

BRANCH
  eengine15

REVISION DETAIL
  http://local.phacility.com/D98

AFFECTED FILES
  src/applications/conpherence/editor/ConpherenceEditor.php
  src/applications/pholio/editor/PholioMockEditor.php
  src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php

CHANGE DETAILS
  Index: src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php
  ===================================================================
  --- src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php
  +++ src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php
  @@ -775,8 +775,6 @@
           throw new PhabricatorApplicationTransactionValidationException($errors);
         }
   
  -      $file_phids = $this->extractFilePHIDs($object, $xactions);
  -
         if…
(This email was truncated at 1024 bytes.)"

This appeared to truncate the mail body properly.

Do you have any idea what I might need to do differently to reproduce the issue?

I have metamta.differential.inline-patches set to 100
metamta.email-body-limit is set to the Global Default 524288

It did not seem to truncate the message for this one message. After it failed over 250 times the job got removed from the queue. The only thing I can think of is that in my case it involved a binary file that phabricator thought was a text file. I'm not sure if this affects the pipeline at all. Is there any other data I can give?

sadly I can't run this

bin/worker execute --id <id>

since the task has been archived

epriestley claimed this task.

It sounds like this no longer reproduces for you.

Only due to the fact that it failed 250 times and got removed by Phabricator.... I suspect it will happen again the next time that file changes.