diff --git a/src/docs/user/configuration/configuring_file_storage.diviner b/src/docs/user/configuration/configuring_file_storage.diviner index 120c44f61e..8ea60080ac 100644 --- a/src/docs/user/configuration/configuring_file_storage.diviner +++ b/src/docs/user/configuration/configuring_file_storage.diviner @@ -1,198 +1,200 @@ @title Configuring File Storage @group config Setup file storage and support for large files. Overview ======== This document describes how to configure Phabricator to support large file uploads, and how to choose where Phabricator stores files. There are two major things to configure: - set up PHP and your HTTP server to accept large requests; - choose and configure a storage engine. The following sections will guide you through this configuration. How Phabricator Stores Files ============================ Phabricator stores files in "storage engines", which are modular backends that implement access to some storage system (like MySQL, the filesystem, or a cloud storage service like Amazon S3). Phabricator stores large files by breaking them up into many chunks (a few megabytes in size) and storing the chunks in an underlying storage engine. This makes it easier to implement new storage engines and gives Phabricator more flexibility in managing file data. The first section of this document discusses configuring your install so that PHP and your HTTP server will accept requests which are larger than the size of one file chunk. Without this configuration, file chunk data will be rejected. The second section discusses choosing and configuring storage engines, so data is stored where you want it to be. Configuring Upload Limits ========================= File uploads are limited by several pieces of configuration at different layers of the stack. Generally, the minimum value of all the limits is the effective one. To upload large files, you need to increase all the limits to at least **32MB**. This will allow you to upload file chunks, which will let Phabricator store arbitrarily large files. The settings which limit file uploads are: **HTTP Server**: The HTTP server may set a limit on the maximum request size. If you exceed this limit, you'll see a default server page with an HTTP error. These directives limit the total size of the request body, so they must be somewhat larger than the desired maximum filesize. - **Apache**: Apache limits requests with the Apache `LimitRequestBody` directive. - **nginx**: nginx limits requests with the nginx `client_max_body_size` directive. This often defaults to `1M`. - **lighttpd**: lighttpd limits requests with the lighttpd `server.max-request-size` directive. Set the applicable limit to at least **32MB**. Phabricator can not read these settings, so it can not raise setup warnings if they are misconfigured. **PHP**: PHP has several directives which limit uploads. These directives are found in `php.ini`. - **post_max_size**: Maximum POST request size PHP will accept. If you exceed this, Phabricator will give you a useful error. This often defaults to `8M`. Set this to at least `32MB`. Phabricator will give you a setup warning about this if it is set too low. - **memory_limit**: For some uploads, file data will be read into memory before Phabricator can adjust the memory limit. If you exceed this, PHP may give you a useful error, depending on your configuration. It is recommended that you set this to `-1` to disable it. Phabricator will give you a setup warning about this if it is set too low. You may also want to configure these PHP options: - **max_input_vars**: When files are uploaded via HTML5 drag and drop file upload APIs, PHP parses the file body as though it contained normal POST parameters, and may trigger `max_input_vars` if a file has a lot of brackets in it. You may need to set it to some astronomically high value. - **upload_max_filesize**: Maximum file size PHP will accept in a raw file upload. This is not normally used when uploading files via drag-and-drop, but affects some other kinds of file uploads. If you exceed this, Phabricator will give you a useful error. This often defaults to `2M`. Set this to at least `32MB`. Once you've adjusted all this configuration, your server will be able to receive chunk uploads. As long as you have somewhere to store them, this will enable you to store arbitrarily large files. Storage Engines =============== Phabricator supports several different file storage engines: | Engine | Setup | Cost | Notes | |--------|-------|------|-------| | MySQL | Automatic | Free | May not scale well. | | Local Disk | Easy | Free | Does not scale well. | | Amazon S3 | Easy | Cheap | Scales well. | | Custom | Hard | Varies | Implement a custom storage engine. | You can review available storage engines and their configuration by navigating to {nav Applications > Files > Help/Options > Storage Engines} in the web UI. By default, Phabricator is configured to store files up to 1MB in MySQL, and reject files larger than 1MB. To store larger files, you can either: - increase the MySQL limit to at least 8MB; or - configure another storage engine. Doing either of these will enable the chunk storage engine and support for arbitrarily large files. The remaining sections of this document discuss the available storage engines and how to configure them. Engine: MySQL ============= - **Pros**: Low latency, no setup required. - **Cons**: Storing files in a database is a classic bad idea. May become difficult to administrate if you have a large amount of data. MySQL storage is configured by default, for files up to (just under) 1MB. You can configure it with these keys: - `storage.mysql-engine.max-size`: Change the filesize limit. Set to 0 to disable. For most installs, it is reasonable to leave this engine as-is and let small files (like thumbnails and profile images) be stored in MySQL, which is usually the lowest-latency filestore, even if you configure another storage engine. To support large files, increase this limit to at least **8MB**. This will activate chunk storage in MySQL. Engine: Local Disk ================== - **Pros**: Simple to setup. - **Cons**: Doesn't scale to multiple web frontends without NFS. To configure file storage on the local disk, set: - `storage.local-disk.path`: Set to some writable directory on local disk. Make that directory. Engine: Amazon S3 ================= - **Pros**: Scales well. - **Cons**: Slightly more complicated than other engines, not free. To enable file storage in S3, set these keys: - `amazon-s3.access-key`: Your AWS access key. - `amazon-s3.secret-key`: Your AWS secret key. + - `amazon-s3.region`: Your AWS S3 region. + - `amazon-s3.endpoint`: Your AWS S3 endpoint. - `storage.s3.bucket`: S3 bucket name where files should be stored. Testing Storage Engines ======================= You can test that things are correctly configured by dragging and dropping a file onto the Phabricator home page. If engines have been configured properly, the file should upload. Migrating Files Between Engines =============================== If you want to move files between storage engines, you can use the `bin/files` script to perform migrations. For example, suppose you previously used MySQL but recently set up S3 and want to migrate all your files there. First, migrate one file to make sure things work: phabricator/ $ ./bin/files migrate --engine amazon-s3 F12345 If that works properly, you can then migrate everything: phabricator/ $ ./bin/files migrate --engine amazon-s3 --all You can use `--dry-run` to show which migrations would be performed without taking any action. Run `bin/files help` for more options and information. Next Steps ========== Continue by: - returning to the @{article:Configuration Guide}. diff --git a/src/docs/user/configuration/configuring_inbound_email.diviner b/src/docs/user/configuration/configuring_inbound_email.diviner index 7d57f01e3a..4b20881f58 100644 --- a/src/docs/user/configuration/configuring_inbound_email.diviner +++ b/src/docs/user/configuration/configuring_inbound_email.diviner @@ -1,253 +1,254 @@ @title Configuring Inbound Email @group config This document contains instructions for configuring inbound email, so users may interact with some Phabricator applications via email. = Preamble = This can be extremely difficult to configure correctly. This is doubly true if you use a local MTA. There are a few approaches available: | Receive Mail With | Setup | Cost | Notes | |--------|-------|------|-------| | Mailgun | Easy | Cheap | Recommended | | SendGrid | Easy | Cheap | | | Local MTA | Extremely Difficult | Free | Strongly discouraged! | The remainder of this document walks through configuring Phabricator to receive mail, and then configuring your chosen transport to deliver mail to Phabricator. = Configuring Phabricator = By default, Phabricator uses a `noreply@phabricator.example.com` email address as the 'From' (configurable with `metamta.default-address`) and sets 'Reply-To' to the user generating the email (e.g., by making a comment), if the mail was generated by a user action. This means that users can reply (or reply-all) to email to discuss changes, but the conversation won't be recorded in Phabricator and users will not be able to take actions like claiming tasks or requesting changes to revisions. To change this behavior so that users can interact with objects in Phabricator over email, change the configuration key `metamta.reply-handler-domain` to some domain you configure according to the instructions below, e.g. `phabricator.example.com`. Once you set this key, emails will use a 'Reply-To' like `T123+273+af310f9220ad@phabricator.example.com`, which -- when configured correctly, according to the instructions below -- will parse incoming email and allow users to interact with Differential revisions, Maniphest tasks, etc. over email. If you don't want Phabricator to take up an entire domain (or subdomain) you can configure a general prefix so you can use a single mailbox to receive mail on. To make use of this set `metamta.single-reply-handler-prefix` to the prefix of your choice, and Phabricator will prepend this to the 'Reply-To' mail address. This works because everything up to the first (optional) '+' character in an email-address is considered the receiver, and everything after is essentially ignored. You can also set up application email addresses to allow users to create application objects via email. For example, you could configure `bugs@phabricator.example.com` to create a Maniphest task out of any email which is sent to it. To do this, see application settings for a given application at {nav icon=home, name=Home > name=Applications > icon=cog, name=Settings} = Security = The email reply channel is "somewhat" authenticated. Each reply-to address is unique to the recipient and includes a hash of user information and a unique object ID, so it can only be used to update that object and only be used to act on behalf of the recipient. However, if an address is leaked (which is fairly easy -- for instance, forwarding an email will leak a live reply address, or a user might take a screenshot), //anyone// who can send mail to your reply-to domain may interact with the object the email relates to as the user who leaked the mail. Because the authentication around email has this weakness, some actions (like accepting revisions) are not permitted over email. This implementation is an attempt to balance utility and security, but makes some sacrifices on both sides to achieve it because of the difficulty of authenticating senders in the general case (e.g., where you are an open source project and need to interact with users whose email accounts you have no control over). If you leak a bunch of reply-to addresses by accident, you can change `phabricator.mail-key` in your configuration to invalidate all the old hashes. You can also set `metamta.public-replies`, which will change how Phabricator delivers email. Instead of sending each recipient a unique mail with a personal reply-to address, it will send a single email to everyone with a public reply-to address. This decreases security because anyone who can spoof a "From" address can act as another user, but increases convenience if you use mailing lists and, practically, is a reasonable setting for many installs. The reply-to address will still contain a hash unique to the object it represents, so users who have not received an email about an object can not blindly interact with it. If you enable application email addresses, those addresses also use the weaker "From" authentication mechanism. NOTE: Phabricator does not currently attempt to verify "From" addresses because this is technically complex, seems unreasonably difficult in the general case, and no installs have had a need for it yet. If you have a specific case where a reasonable mechanism exists to provide sender verification (e.g., DKIM signatures are sufficient to authenticate the sender under your configuration, or you are willing to require all users to sign their email), file a feature request. = Testing and Debugging Inbound Email = You can use the `bin/mail` utility to test and review inbound mail. This can help you determine if mail is being delivered to Phabricator or not: phabricator/ $ ./bin/mail list-inbound # List inbound messages. phabricator/ $ ./bin/mail show-inbound # Show details about a message. You can also test receiving mail, but note that this just simulates receiving the mail and doesn't send any information over the network. It is primarily aimed at developing email handlers: it will still work properly if your inbound email configuration is incorrect or even disabled. phabricator/ $ ./bin/mail receive-test # Receive test message. Run `bin/mail help ` for detailed help on using these commands. = Mailgun Setup = To use Mailgun, you need a Mailgun account. You can sign up at . Provided you have such an account, configure it like this: - Configure a mail domain according to Mailgun's instructions. - Add a Mailgun route with a `catch_all()` rule which takes the action `forward("https://phabricator.example.com/mail/mailgun/")`. Replace the example domain with your actual domain. + - Set the `mailgun.api-key` config key to your Mailgun API key. = SendGrid Setup = To use SendGrid, you need a SendGrid account with access to the "Parse API" for inbound email. Provided you have such an account, configure it like this: - Configure an MX record according to SendGrid's instructions, i.e. add `phabricator.example.com MX 10 mx.sendgrid.net.` or similar. - Go to the "Parse Incoming Emails" page on SendGrid () and add the domain as the "Hostname". - Add the URL `https://phabricator.example.com/mail/sendgrid/` as the "Url", using your domain (and HTTP instead of HTTPS if you are not configured with SSL). - If you get an error that the hostname "can't be located or verified", it means your MX record is either incorrectly configured or hasn't propagated yet. - Set `metamta.reply-handler-domain` to `phabricator.example.com`" (whatever you configured the MX record for). That's it! If everything is working properly you should be able to send email to `anything@phabricator.example.com` and it should appear in `bin/mail list-inbound` within a few seconds. = Local MTA: Installing Mailparse = If you're going to run your own MTA, you need to install the PECL mailparse extension. In theory, you can do that with: $ sudo pecl install mailparse You may run into an error like "needs mbstring". If so, try: $ sudo yum install php-mbstring # or equivalent $ sudo pecl install -n mailparse If you get a linker error like this: COUNTEREXAMPLE PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/mailparse.so' - /usr/lib64/php/modules/mailparse.so: undefined symbol: mbfl_name2no_encoding in Unknown on line 0 ...you need to edit your php.ini file so that mbstring.so is loaded **before** mailparse.so. This is not the default if you have individual files in `php.d/`. = Local MTA: Configuring Sendmail = Before you can configure Sendmail, you need to install Mailparse. See the section "Installing Mailparse" above. Sendmail is very difficult to configure. First, you need to configure it for your domain so that mail can be delivered correctly. In broad strokes, this probably means something like this: - add an MX record; - make sendmail listen on external interfaces; - open up port 25 if necessary (e.g., in your EC2 security policy); - add your host to /etc/mail/local-host-names; and - restart sendmail. Now, you can actually configure sendmail to deliver to Phabricator. In `/etc/aliases`, add an entry like this: phabricator: "| /path/to/phabricator/scripts/mail/mail_handler.php" If you use the `PHABRICATOR_ENV` environmental variable to select a configuration, you can pass the value to the script as an argument: .../path/to/mail_handler.php This is an advanced feature which is rarely used. Most installs should run without an argument. After making this change, run `sudo newaliases`. Now you likely need to symlink this script into `/etc/smrsh/`: sudo ln -s /path/to/phabricator/scripts/mail/mail_handler.php /etc/smrsh/ Finally, edit `/etc/mail/virtusertable` and add an entry like this: @yourdomain.com phabricator@localhost That will forward all mail to @yourdomain.com to the Phabricator processing script. Run `sudo /etc/mail/make` or similar and then restart sendmail with `sudo /etc/init.d/sendmail restart`. = Local MTA: Configuring Lamson = Before you can configure Lamson, you need to install Mailparse. See the section "Installing Mailparse" above. In contrast to Sendmail, Lamson is relatively easy to configure. It is fairly minimal, and is suitable for a development or testing environment. Lamson listens for incoming SMTP mails and passes the content directly to Phabricator. To get started, follow the provided instructions () to set up an instance. One likely deployment issue is that binding to port 25 requires root privileges. Lamson is capable of starting as root then dropping privileges, but you must supply `-uid` and `-gid` arguments to do so, as demonstrated by Step 8 in Lamson's deployment tutorial (located here: ). The Lamson handler code itself is very concise; it merely needs to pass the content of the email to Phabricator: import logging, subprocess from lamson.routing import route, stateless from lamson import view PHABRICATOR_ROOT = "/path/to/phabricator" PHABRICATOR_ENV = "custom/myconf" LOGGING_ENABLED = True @route("(address)@(host)", address=".+") @stateless def START(message, address=None, host=None): if LOGGING_ENABLED: logging.debug("%s", message.original) process = subprocess.Popen([PHABRICATOR_ROOT + "scripts/mail/mail_handler.php",PHABRICATOR_ENV],stdin=subprocess.PIPE) process.communicate(message.original)