snapshot of https://github.com/zulip/zulip.git @ 48c69911c7dd4e03b20552eb3bd77ad8151c1357 (2026-09-15)

docs/production/system-configuration.md

@@ -0,0 +1,546 @@
+## System configuration
+
+The file `/etc/zulip/zulip.conf` is an [INI
+format](https://en.wikipedia.org/wiki/INI_file) configuration file
+used to configure properties of the system and deployment;
+`/etc/zulip/settings.py` is used to [configure the application
+itself](settings.md). The `zulip.conf` sections and settings are
+described below. Changes to `zulip.conf` generally do not take effect
+until you run `zulip-puppet-apply` as root:
+
+```console
+# /home/zulip/deployments/current/scripts/zulip-puppet-apply
+```
+
+The `zulip-puppet-apply` command will display the configuration
+changes it will make and prompt for you to confirm you'd like to make
+those changes, before executing them (if you approve).
+
+:::{important}
+
+If you are using [Docker](docker.md), see
+{doc}`docker:how-to/compose-settings` and
+{doc}`docker:reference/environment-vars` for configuring
+`zulip.conf` via `CONFIG_*` environment variables.
+
+:::
+
+### Truthy values
+
+When a setting refers to "set to true" or "set to false", the values
+`true` and `false` are canonical, but any of the following values will
+be considered "true", case-insensitively:
+
+- 1
+- y
+- t
+- yes
+- true
+- enable
+- enabled
+
+Any other value (including the empty string) is considered false.
+
+### `[machine]`
+
+#### `puppet_classes`
+
+A comma-separated list of the Puppet classes to install on the server.
+The most common is **`zulip::profile::standalone`**, used for a
+stand-alone single-host deployment.
+[Components](../overview/architecture-overview.md#components) of
+that include:
+
+- **`zulip::profile::app_frontend`**
+- **`zulip::profile::memcached`**
+- **`zulip::profile::postgresql`**
+- **`zulip::profile::rabbitmq`**
+- **`zulip::profile::redis`**
+- **`zulip::profile::smokescreen`**
+
+If you are using a [Apache as a single-sign-on
+authenticator](authentication-methods.md#apache-based-sso-with-remote_user),
+you will need to add **`zulip::apache_sso`** to the list.
+
+#### `pgroonga`
+
+Set to true if enabling the [multi-language PGroonga search
+extension](../subsystems/full-text-search.md#multi-language-full-text-search).
+
+#### `timesync`
+
+What time synchronization daemon to use; defaults to `chrony`, but also supports
+`ntpd` and `none`. Installations should not adjust this unless they are aligning
+with a fleet-wide standard of `ntpd`. `none` is only reasonable in containers
+like LXC which do not allow adjustment of the clock; a Zulip server will not
+function correctly without an accurate clock.
+
+### `[deployment]`
+
+#### `deploy_options`
+
+Options passed by `upgrade-zulip` and `upgrade-zulip-from-git` into
+`upgrade-zulip-stage-2`. These might be any of:
+
+- **`--skip-puppet`** skips doing Puppet/apt upgrades. The user will need
+  to run `zulip-puppet-apply` manually after the upgrade.
+- **`--skip-migrations`** skips running database migrations. The
+  user will need to run `./manage.py migrate` manually after the upgrade.
+- **`--skip-purge-old-deployments`** skips purging old deployments;
+  without it, only deployments with the last two weeks are kept.
+
+Generally installations will not want to set any of these options; the
+`--skip-*` options are primarily useful for reducing upgrade downtime
+for servers that are upgraded frequently by core Zulip developers.
+
+#### `git_repo_url`
+
+Default repository URL used when [upgrading from a Git
+repository](upgrade.md#upgrading-from-a-git-repository).
+
+### `[application_server]`
+
+#### `http_only`
+
+If set to true, [configures Zulip to allow HTTP access][using-http];
+use if Zulip is deployed behind a reverse proxy that is handling
+SSL/TLS termination.
+
+[using-http]: reverse-proxies.md#configuring-zulip-to-allow-http
+
+#### `nginx_listen_port`
+
+Set to the port number if you [prefer to listen on a port other than
+443](deployment.md#using-an-alternate-port).
+
+#### `nginx_server_name`
+
+Sets the [`server_name`][nginx_server_name] directive on Zulip's
+nginx `server` blocks; by default it is unset, so they match any
+hostname. Set it if you run other software alongside Zulip whose
+nginx configuration would otherwise claim requests meant for Zulip.
+List every hostname Zulip is served on, separated by spaces,
+including those on its TLS certificate; otherwise automatic
+certificate renewal will fail.
+
+If Zulip is served on more than one hostname, or on a hostname plus
+several subdomains, list them all; nginx's `server_name` supports a
+trailing wildcard for matching subdomains. For example, to serve
+Zulip on both `example.com` and any subdomain of it:
+
+```ini
+[application_server]
+nginx_server_name = example.com *.example.com
+```
+
+[nginx_server_name]: https://nginx.org/en/docs/http/server_names.html
+
+#### `nginx_worker_processes`
+
+Adjusts the [`worker_processes`][nginx_worker_processes] setting in
+the nginx server. This defaults to `auto`, which nginx treats as the
+number of available CPU cores.
+
+[nginx_worker_processes]: https://nginx.org/en/docs/ngx_core_module.html#worker_processes
+
+#### `nginx_worker_connections`
+
+Adjust the [`worker_connections`][nginx_worker_connections] setting in
+the nginx server. This defaults to 10000; increasing it allows more
+concurrent connections per worker process, at the cost of more memory
+consumed by nginx. This number, times the number of worker processes
+(above), should be more than twice the concurrent number of users.
+
+[nginx_worker_connections]: http://nginx.org/en/docs/ngx_core_module.html#worker_connections
+
+#### `queue_workers_multiprocess`
+
+By default, Zulip automatically detects whether the system has enough
+memory to run Zulip queue processors in the higher-throughput but more
+multiprocess mode (or to save 1.5GiB of RAM with the multithreaded
+mode). The calculation is based on whether the system has enough
+memory (currently 3.5GiB) to run a single-server Zulip installation in
+the multiprocess mode.
+
+Set explicitly to true or false to override the automatic
+calculation. This override is useful for Docker systems, where the
+above algorithm will see the host's memory size, not the container's,
+unless a [memory limit is set][docker-memory-limit], as well as when
+using remote servers for PostgreSQL, memcached, Redis, and RabbitMQ.
+
+[docker-memory-limit]: https://docs.docker.com/reference/compose-file/deploy/#memory
+
+#### `dedicated_soft_reactivation_queue`
+
+When a long-term-idle user returns, Zulip backfills the message-history
+rows that were skipped while they were idle; this "soft reactivation"
+can take many seconds. By default it runs in the shared `deferred_work`
+queue, alongside jobs such as data exports that can take minutes. Set
+this to true to process soft reactivations in a dedicated queue (and, in
+multiprocess mode, a dedicated worker process), so they aren't delayed
+behind such jobs. This costs an additional worker process, so it is most
+useful on large servers; smaller servers should leave it disabled.
+
+#### `rolling_restart`
+
+If set to true, when using `./scripts/restart-server` to restart
+Zulip, restart the uwsgi processes one-at-a-time, instead of all at
+once. This decreases the number of 502's served to clients, at the
+cost of slightly increased memory usage, and the possibility that
+different requests will be served by different versions of the code.
+
+#### `service_file_descriptor_limit`
+
+The number of file descriptors which [Supervisor is configured to allow
+processes to use][supervisor-minfds]; defaults to 40000. If your Zulip deployment
+is very large (hundreds of thousands of concurrent users), your Django processes
+hit this limit and refuse connections to clients. Raising it above this default
+may require changing system-level limits, particularly if you are using a
+virtualized environment (e.g., Docker, or Proxmox LXC).
+
+[supervisor-minfds]: http://supervisord.org/configuration.html?highlight=minfds#supervisord-section-values
+
+#### `s3_memory_cache_size`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the in-memory size of the cache _index_; the default is 1MB,
+which is enough to store about 8 thousand entries.
+
+#### `s3_disk_cache_size`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the on-disk size of the cache _contents_; the default is
+200MB.
+
+#### `s3_cache_inactive_time`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the longest amount of time an entry will be cached since last
+use; the default is 30 days. Since the contents of the cache are
+immutable, this serves only as a potential additional limit on the
+size of the contents on disk; `s3_disk_cache_size` is expected to be
+the primary control for cache sizing.
+
+#### `thumbnail_workers`
+
+How many image-thumbnailing workers to run. Defaults to 1; adding more
+workers can prevent the image-thumbnailing queue backlogging when
+large numbers of very large image files are uploaded at once. (When
+backlogged, image previews for images that have not yet been
+thumbnailed will appear as loading spinners).
+
+#### `email_senders_workers`
+
+How many email-sending workers to run. Defaults to 1; adding more
+workers can prevent email-sending queue backlogging when large numbers
+of very large emails are enqueued at once. This is generally only
+necessary on quite large installs.
+
+#### `nameserver`
+
+When the [S3 storage backend][s3-backend] is in use, downloads from S3 are
+proxied from nginx, whose configuration requires an explicit value of a DNS
+nameserver to resolve the S3 server's hostname. Zulip defaults to using the
+resolver found in `/etc/resolv.conf`; this setting overrides any value found
+there.
+
+[s3-backend]: upload-backends.md
+
+#### `uwsgi_listen_backlog_limit`
+
+Override the default uwsgi backlog of 128 connections.
+
+#### `uwsgi_processes`
+
+Override the default `uwsgi` (Django) process count. It defaults to a sliding
+scale between 3 workers for hosts with under 3GB RAM, up to 16 workers for hosts
+with more than 24GB of RAM.
+
+#### `access_log_retention_days`
+
+Number of days of access logs to keep, for both nginx and the application.
+Defaults to 14 days.
+
+#### `katex_server`
+
+Set to a false value to disable the separate service for [rendering math with
+LaTeX](https://zulip.com/help/latex). Disabling this service will save a small
+amount of memory, at the cost of making math blocks significantly slower to
+render, to the point that using dozens of them in a single message may cause
+the message to fail to send.
+
+#### `katex_server_port`
+
+Set to the port number for the KaTeX server; defaults to port 9700.
+
+#### `custom_ca_path`
+
+If you use a custom certificate authority for your authentication
+provider, you will need to provide the certificate of the signing CA
+so Zulip can successfully make requests to it.
+
+Set this to the fully-qualified path to the `.crt` file containing the
+PEM-encoded CA certificate to trust; we suggest storing this in
+`/etc/zulip/`.
+
+Setting this path also means that Zulip will use the operating
+system's CA certificate store, and not its built-in one. There may be
+minor differences in trusted root CA sets.
+
+### `[postgresql]`
+
+#### `effective_io_concurrency`
+
+Override PostgreSQL's [`effective_io_concurrency`
+setting](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-EFFECTIVE-IO-CONCURRENCY).
+
+#### `listen_addresses`
+
+Override PostgreSQL's [`listen_addresses`
+setting](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-LISTEN-ADDRESSES).
+
+#### `random_page_cost`
+
+Override PostgreSQL's [`random_page_cost`
+setting](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-RANDOM-PAGE-COST).
+Zulip defaults this value to 1.1, which is an appropriate value for
+SSDs; if your server uses spinning disks, you should set this back to
+the upstream default of 4.0.
+
+#### `replication_primary`
+
+On the [warm standby replicas](postgresql.md#postgresql-warm-standby), set to the
+hostname of the primary PostgreSQL server that streaming replication
+should be done from.
+
+#### `replication_user`
+
+On the [warm standby replicas](postgresql.md#postgresql-warm-standby), set to the
+username that the host should authenticate to the primary PostgreSQL
+server as, for streaming replication. Authentication will be done
+based on the `pg_hba.conf` file; if you are using password
+authentication, you can set a `postgresql_replication_password` secret
+for authentication.
+
+#### `skip_backups`
+
+If set to as true value, inhibits the nightly [`wal-g` backups][wal-g] which
+would be taken on all non-replicated hosts and [all warm standby
+replicas](postgresql.md#postgresql-warm-standby). This is generally only set if you have
+multiple warm standby replicas, in order to avoid taking multiple backups, one
+per replica.
+
+[wal-g]: export-and-import.md#database-only-backup-tools
+
+#### `backups_disk_concurrency`
+
+Number of concurrent disk reads to use when taking backups. Defaults to 1; you
+may wish to increase this if you are taking backups on a replica, so can afford
+to affect other disk I/O, and have an SSD which is good at parallel random
+reads.
+
+#### `backups_directory`
+
+If S3 secrets are not configured, perform daily database backups to this path on
+disk instead. It should be owned by the `postgres` user.
+
+This option is not recommended for disaster recovery purposes, since unless the
+directory is on a different disk from the database itself, _backups will likely
+also be lost if the database is lost._ This setting can be useful if the path is
+on a NAS mountpoint, or if some other process copies this data off the disk; or
+if backups are purely for point-in-time historical analysis of recent
+application-level data changes.
+
+#### `backups_incremental`
+
+The number of delta (incremental) database backups to take between full backups.
+Defaults to 0 for S3 backups, and 6 for local-disk backups.
+
+#### `backups_storage_class`
+
+What [storage class](https://aws.amazon.com/s3/storage-classes/) to use when
+uploading database backups. Defaults to `STANDARD`, meaning "[S3
+standard][s3-standard]", but many deployments will have overall lower costs if
+"[S3 Standard - Infrequent Access][s3-ia]" is used, via the `STANDARD_IA`
+value. Also supported is "[S3 Reduced Redundancy][s3-rr]", by setting
+`REDUCED_REDUNDANCY`, but this is not suggested for production use.
+
+[s3-standard]: https://aws.amazon.com/s3/storage-classes/#General_purpose
+[s3-ia]: https://aws.amazon.com/s3/storage-classes/#Infrequent_access
+[s3-rr]: https://aws.amazon.com/s3/reduced-redundancy/
+
+#### `backups_compression_method`
+
+What compression method to use when storing backups; defaults to `lz4`, which is
+fast but does not compress particularly well. Other options are `lzma`, `zstd`,
+and `brotli`; `lzma` provides the best (and slowest) compression, while `zstd`
+and `brotli` are middling compromises.
+
+#### `missing_dictionaries`
+
+If set to a true value during initial database creation, uses PostgreSQL's
+standard `pg_catalog.english` text search configuration, rather than Zulip's
+improved set of stopwords. Has no effect after initial database construction.
+
+#### `ssl_ca_file`
+
+Set to the path to the PEM-encoded certificate authority used to
+authenticate client connections.
+
+#### `ssl_cert_file`
+
+Set to the path to the PEM-encoded public certificate used to secure
+client connections.
+
+#### `ssl_key_file`
+
+Set to the path to the PEM-encoded private key used to secure client
+connections.
+
+#### `ssl_mode`
+
+The mode that should be used to verify the server certificate. The
+PostgreSQL default is `prefer`, which provides no security benefit; we
+strongly suggest setting this to `require` or better if you are using
+certificate authentication. See the [PostgreSQL
+documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS)
+for potential values.
+
+#### `version`
+
+The version of PostgreSQL that is in use. Do not set by hand; use the
+[PostgreSQL upgrade tool](upgrade.md#upgrading-postgresql).
+
+### `[memcached]`
+
+#### `memory`
+
+Override the number of megabytes of memory that memcached should be
+configured to consume; defaults to 1/8th of the total server memory.
+
+#### `max_item_size`
+
+Override the maximum size that an item in memcached can store. This defaults to
+1m; adjusting it should only be necessary if your Zulip server has organizations
+which have more than 20k users.
+
+#### `size_reporting`
+
+Set to a true value to enable object size reporting in memcached. This incurs a
+small overhead for every store or delete operation, but allows a
+memcached_exporter to report precise item size distribution.
+
+### `[tornado_sharding]`
+
+Keys in this section are used to configure how many Tornado instances
+are started, and which users are mapped to which of those instances.
+Each Tornado instance listens on a separate port, starting at 9800 and
+proceeding upwards from there. A single Tornado instance can usually
+handle 1000-1500 concurrent active users, depending on message sending
+volume.
+
+Individual organizations may be assigned to ports, either via their
+subdomain names, or their fully-qualified hostname (for [organizations
+using `REALM_HOSTS`](multiple-organizations.md#other-hostnames)):
+
+```ini
+[tornado_sharding]
+9800 = realm-a realm-b
+9801 = realm-c
+9802 = realm-host.example.net
+```
+
+Organizations can also be assigned to ports via regex over their
+fully-qualified hostname:
+
+```ini
+[tornado_sharding]
+9800_regex = ^realm-(a|b)\.example\.com$
+9801_regex = ^other(-option)?\.example.com$
+```
+
+Extremely large organizations can be distributed across multiple
+Tornado shards by joining the ports in the key with `_`:
+
+```ini
+[tornado_sharding]
+9800 = small-realm
+9801_9802 = very-large-realm
+```
+
+After running `scripts/zulip-puppet-apply`, a separate step to run
+`scripts/refresh-sharding-and-restart` is required for any sharding
+changes to take effect.
+
+### `[loadbalancer]`
+
+#### `ips`
+
+Comma-separated list of IP addresses or netmasks of external load balancers
+whose `X-Forwarded-For` and `X-Forwarded-Proto` should be respected. These can
+be individual IP addresses, or CIDR IP address ranges.
+
+#### `rejects_http_requests`
+
+Set to a true value if incoming requests from load loadbalancer's IP addresses
+which do not contain an `X-Forwarded-Proto` should be assumed to have come into
+them over HTTPS. This setting _is a security vulnerability_ unless the load
+balancer unilaterally rejects unencrypted HTTP connections, or responds to them
+with 301 status codes. Note that Zulip's HSTS headers are not sufficient
+protection here, since API clients do not respect them; the load balancer _must
+not_ send any requests to Zulip which came in unencrypted.
+
+### `[http_proxy]`
+
+See "[Customizing the outgoing HTTP
+proxy](deployment.md#customizing-the-outgoing-http-proxy)" for general
+instructions on the outgoing proxy.
+
+#### `host`
+
+The hostname or IP address of an outgoing HTTP `CONNECT`. Defaults to
+`localhost` if unspecified.
+
+#### `port`
+
+The TCP port of the HTTP `CONNECT` proxy on the host specified above.
+Defaults to `4750` if unspecified.
+
+#### `listen_address`
+
+The IP address that Smokescreen should bind to and listen on.
+Defaults to `127.0.0.1`.
+
+#### `enable_for_camo`
+
+Because Camo includes logic to deny access to private subnets, routing
+its requests through Smokescreen is generally not necessary. Set to
+true or false to override the default, which uses the proxy only if
+it is not the default of Smokescreen on a local host.
+
+#### `allow_addresses`, `allow_ranges`, `deny_addresses`, `deny_ranges`
+
+Comma-separated lists of IP addresses or CIDR range rules. All
+private IP addresses (e.g., 127.0.0.0/8, 192.168.0.0/16) are denied by
+default; allow rules override deny rules.
+
+### `[sentry]`
+
+#### `organization`
+
+The Sentry organization used for the [Sentry deploy hook](deployment.md#sentry-deploy-hook).
+
+#### `project`
+
+The Sentry project used for the [Sentry deploy hook](deployment.md#sentry-deploy-hook).
+
+### `[email_gateway]`
+
+#### `listen`
+
+The address for the [incoming email gateway's](email-gateway.md) SMTP
+server to bind to, as a bare port number, or `address:port`; IPv6
+addresses must be enclosed in square brackets (e.g.,
+`[2001:db8::1]:25`). Defaults to listening on port 25 on all
+addresses, both IPv4 and IPv6. This only applies to the local
+delivery mode of the gateway, not the polling mode.