snapshot of https://github.com/zulip/zulip.git @ 48c69911c7dd4e03b20552eb3bd77ad8151c1357 (2026-09-15)

docs/production/upload-backends.md

@@ -0,0 +1,346 @@
+# File upload backends
+
+Zulip in production supports a couple different backends for storing
+files uploaded by users of the Zulip server (messages, profile
+pictures, organization icons, custom emoji, etc.).
+
+The default is the `LOCAL_UPLOADS_DIR` backend, which just stores
+files on disk in the specified directory on the Zulip server.
+Obviously, this backend doesn't work with multiple Zulip servers and
+doesn't scale, but it's great for getting a Zulip server up and
+running quickly. You can later migrate the uploads to S3 by
+[following the instructions here](#migrating-from-local-uploads-to-amazon-s3-backend).
+
+We also support an `S3` backend, which uses the Python `boto` library
+to upload files to Amazon S3 (or an S3-compatible block storage
+provider supported by the `boto` library).
+
+Regardless of the backend you choose, you can configure the maximum
+size of individual uploaded files using the `MAX_FILE_UPLOAD_SIZE`
+[server setting](../production/settings.md). Setting it to 0 disables
+file uploads, and hides the UI for uploading files from the web and
+desktop apps.
+
+## S3 backend configuration
+
+(s3-backend)=
+
+:::{important}
+
+If you are using [Docker](docker.md), see
+{doc}`docker:how-to/compose-settings` for `SETTING_S3_*`
+configuration and {doc}`docker:how-to/compose-secrets` for the
+`s3_key` / `s3_secret_key` secrets. Switching to S3 also keeps
+uploads out of the `/data` volume.
+
+:::
+
+Here, we document the process for configuring Zulip's S3 file upload
+backend. To enable this backend, you need to do the following:
+
+1. In the AWS management console, create a new IAM account (aka API
+   user) for your Zulip server, and two buckets in S3, one for uploaded
+   files included in messages, and another for user avatars. You need
+   two buckets because the "user avatars" bucket is generally configured
+   as world-readable, whereas the "uploaded files" one is not.
+
+1. Set `s3_key` and `s3_secret_key` in /etc/zulip/zulip-secrets.conf
+   to be the S3 access and secret keys for the IAM account.
+   Alternately, if your Zulip server runs on an EC2 instance, set the
+   IAM role for the EC2 instance to the role created in the previous
+   step.
+
+1. Set the `S3_AUTH_UPLOADS_BUCKET` and `S3_AVATAR_BUCKET` settings in
+   `/etc/zulip/settings.py` to be the names of the S3 buckets you
+   created (e.g., `"exampleinc-zulip-uploads"`).
+
+1. Comment out the `LOCAL_UPLOADS_DIR` setting in
+   `/etc/zulip/settings.py` (add a `#` at the start of the line).
+
+1. If you are using a non-AWS block storage provider,
+   you need to set the `S3_ENDPOINT_URL` setting to your
+   endpoint url (e.g., `"https://s3.eu-central-1.amazonaws.com"`).
+
+   For certain AWS regions, you may need to set the `S3_REGION`
+   setting to your default AWS region's code (e.g., `"eu-central-1"`).
+
+1. Non-AWS block storage providers may need `S3_SKIP_CHECKSUM = True`; you
+   should try without this at first, but enable it if you see exceptions
+   involving `XAmzContentSHA256Mismatch`.
+
+1. Finally, restart the Zulip server so that your settings changes
+   take effect
+   (`/home/zulip/deployments/current/scripts/restart-server`).
+
+It's simplest to just do this configuration when setting up your Zulip
+server for production usage. Note that if you had any existing
+uploading files, this process does not upload them to Amazon S3; see
+[migration instructions](#migrating-from-local-uploads-to-amazon-s3-backend)
+below for those steps.
+
+### Google Cloud Platform
+
+In addition to configuring `settings.py` as suggested above:
+
+```python
+S3_AUTH_UPLOADS_BUCKET = "..."
+S3_AVATAR_BUCKET = "..."
+S3_ENDPOINT_URL = "https://storage.googleapis.com"
+S3_SKIP_CHECKSUM = True
+```
+
+...and adding `s3_key` and `s3_secret_key` in `/etc/zulip/zulip-secrets.conf`,
+you will need to also add a `/etc/zulip/gcp_key.json` which contains a [service
+account key][gcp-key] with "Storage Object Admin" permissions on the uploads
+bucket. This is used by the `tusd` chunked upload service when receiving file
+uploads from clients.
+
+[gcp-key]: https://cloud.google.com/iam/docs/keys-create-delete
+
+## S3 local caching
+
+For performance reasons, Zulip stores a cache of recently served user
+uploads on disk locally, even though the durable storage is kept in
+S3. There are a number of parameters which control the size and usage
+of this cache, which is maintained by nginx:
+
+- `s3_memory_cache_size` controls the in-memory size of the cache
+  _index_; the default is 1MB, which is enough to store about 8 thousand
+  entries.
+- `s3_disk_cache_size` controls the on-disk size of the cache
+  _contents_; the default is 200MB.
+- `s3_cache_inactive_time` controls the longest amount of time an
+  entry will be cached since last use; the default is 30 days. Since
+  the contents of the cache are immutable, this serves only as a
+  potential additional limit on the size of the contents on disk;
+  `s3_disk_cache_size` is expected to be the primary control for cache
+  sizing.
+
+These defaults are likely sufficient for small-to-medium deployments.
+Large deployments, or deployments with image-heavy use cases, will
+want to increase `s3_disk_cache_size`, potentially to be several
+gigabytes. `s3_memory_cache_size` should potentially be increased,
+based on estimating the number of files that the larger disk cache
+will hold.
+
+You may also wish to increase the cache sizes if the S3 storage (or
+S3-compatible equivalent) is not closely located to your Zulip server,
+as cache misses will be more expensive.
+
+## nginx DNS nameserver configuration
+
+The S3 cache described above is maintained by nginx. nginx's configuration
+requires an explicitly-set DNS nameserver to resolve the hostname of the S3
+servers; Zulip defaults this value to the first nameserver found in
+`/etc/resolv.conf`, but this resolver can be [adjusted in
+`/etc/zulip/zulip.conf`][s3-resolver] if needed. If you adjust this value, you
+will need to run `/home/zulip/deployments/current/scripts/zulip-puppet-apply` to
+update the nginx configuration for the new value.
+
+[s3-resolver]: system-configuration.md#nameserver
+
+## S3 bucket policy
+
+The best way to do the S3 integration with Amazon is to create a new IAM user
+just for your Zulip server with limited permissions. For both the user uploads
+bucket and the user avatars bucket, you'll need to adjust the [S3 bucket
+policy](https://awspolicygen.s3.amazonaws.com/policygen.html).
+
+The file uploads bucket should have a policy of:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Id": "Policy1468991802320",
+    "Statement": [
+        {
+            "Sid": "Stmt1468991795370",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": [
+                "s3:GetObject",
+                "s3:DeleteObject",
+                "s3:PutObject"
+            ],
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
+        },
+        {
+            "Sid": "Stmt1468991795371",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": "s3:ListBucket",
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE"
+        }
+    ]
+}
+```
+
+The file-uploads bucket should not be world-readable. See the
+[documentation on the Zulip security model](securing-your-zulip-server.md) for
+details on the security model for uploaded files.
+
+However, the avatars bucket is intended to be world-readable, so its
+policy should be:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Id": "Policy1468991802321",
+    "Statement": [
+        {
+            "Sid": "Stmt1468991795380",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": [
+                "s3:GetObject",
+                "s3:DeleteObject",
+                "s3:PutObject"
+            ],
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
+        },
+        {
+            "Sid": "Stmt1468991795381",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": "s3:ListBucket",
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE"
+        },
+        {
+            "Sid": "Stmt1468991795382",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "*"
+            },
+            "Action": "s3:GetObject",
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
+        }
+    ]
+}
+```
+
+## Migrating from local uploads to Amazon S3 backend
+
+As you scale your server, you might want to migrate the uploads from
+your local backend to Amazon S3. Follow these instructions, step by
+step, to do the migration.
+
+1. First, [set up the S3 backend](#s3-backend-configuration) in the settings
+   (all the auth stuff), but leave `LOCAL_UPLOADS_DIR` set -- the
+   migration tool will need that value to know where to find your uploads.
+2. Run `./manage.py transfer_uploads_to_s3`. This will upload all the
+   files from the local uploads directory to Amazon S3. By default,
+   this command runs on 6 parallel processes, since uploading is a
+   latency-sensitive operation. You can control this parameter using
+   the `--processes` option.
+3. Once the transfer script completes, disable `LOCAL_UPLOADS_DIR`, and
+   restart your server (continuing the last few steps of the S3
+   backend setup instructions).
+
+Congratulations! Your uploaded files are now migrated to S3.
+
+**Caveat**: The current version of this tool does not migrate an
+uploaded organization avatar or logo.
+
+## S3 data storage class
+
+In general, uploaded files in Zulip are accessed frequently at first, and then
+age out of frequent access. The S3 backend provides the [S3
+Intelligent-Tiering][s3-it] [storage class][s3-storage-class] which provides
+cheaper storage for less frequently accessed objects, and may provide overall
+cost savings for large deployments.
+
+You can configure Zulip to store uploaded files using Intelligent-Tiering by
+setting `S3_UPLOADS_STORAGE_CLASS` to `INTELLIGENT_TIERING` in `settings.py`.
+This setting can take any of the following [storage class
+value][s3-storage-class-constant] values:
+
+- `STANDARD`
+- `STANDARD_IA`
+- `ONEZONE_IA`
+- `REDUCED_REDUNDANCY`
+- `GLACIER_IR`
+- `INTELLIGENT_TIERING`
+
+Setting `S3_UPLOADS_STORAGE_CLASS` does not affect the storage class of existing
+objects. In order to change those, for example to `INTELLIGENT_TIERING`, perform
+an in-place copy:
+
+    aws s3 cp --storage-class INTELLIGENT_TIERING --recursive \
+        s3://your-bucket-name/ s3://your-bucket-name/
+
+Note that changing the lifecycle of existing objects will incur a [one-time
+lifecycle transition cost][s3-pricing].
+
+[s3-it]: https://aws.amazon.com/s3/storage-classes/intelligent-tiering/
+[s3-storage-class]: https://aws.amazon.com/s3/storage-classes/
+[s3-storage-class-constant]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass
+[s3-pricing]: https://aws.amazon.com/s3/pricing/
+
+## Data export bucket
+
+The [data export process](export-and-import.md#data-export) process, when
+[triggered from the
+UI](https://zulip.com/help/export-your-organization), uploads the
+completed export so it is available to download from the server; this
+is also available [from the command
+line](export-and-import.md#export-your-zulip-data) by passing
+`--upload`. When the S3 backend is used, these uploads are done to S3.
+
+By default, they are uploaded to the bucket with user avatars
+(`S3_AVATAR_BUCKET`), because that bucket is world-readable, allowing
+easy generation of links to download the export.
+
+If you would like to store exports in a dedicated bucket, you can set
+`S3_EXPORT_BUCKET` in your `/etc/zulip/settings.py`. This bucket
+should also be configured like the uploads bucket, only allowing write
+access to the Zulip account, as it will generate links which are valid
+for 1 week at a time:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Id": "Policy1468991802322",
+    "Statement": [
+        {
+            "Sid": "Stmt1468991795390",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": [
+                "s3:GetObject",
+                "s3:DeleteObject",
+                "s3:PutObject"
+            ],
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
+        },
+        {
+            "Sid": "Stmt1468991795391",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": "s3:ListBucket",
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE"
+        }
+    ]
+}
+```
+
+You should copy existing exports to the new bucket. For instance,
+using the [AWS CLI](https://aws.amazon.com/cli/)'s [`aws s3
+sync`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html),
+if the old bucket was named `example-zulip-avatars` and the new export
+bucket is named `example-zulip-exports`:
+
+```
+aws s3 sync s3://example-zulip-avatars/exports/ s3://example-zulip-exports/
+```