snapshot of https://github.com/zulip/zulip.git @ 48c69911c7dd4e03b20552eb3bd77ad8151c1357 (2026-09-15)

docs/production/ai-integrations.md

@@ -0,0 +1,114 @@
+# AI integrations
+
+Zulip’s [topics](https://zulip.com/help/introduction-to-topics) organize
+conversations within a channel, which is ideal for integrating with AI systems
+and collaborating with AI agents. With Zulip's structure, it's easy to prompt an AI
+with the appropriate context for what you want to accomplish.
+
+You can connect your AI models of choice with Zulip using the [interactive bots
+API](https://zulip.com/api/running-bots), which makes it convenient to have AI
+models participate in conversations in whatever ways your organization finds
+most effective as the technology evolves.
+
+Future Zulip releases will also contain built-in AI features, such as topic
+summarization. A major advantage of self-hosting your team chat system
+in the age of AI is that you maintain full control over your internal
+communications. It’s up to you how you allow third parties and AI models to
+process messages.
+
+Zulip Server 10.x includes a beta topic summarization feature that is available
+for testing and experimentation. We appreciate any
+[feedback](../contributing/suggesting-features.md)
+on your experience, and what configuration options and additional features your
+organization would find useful.
+
+## Built-in AI features
+
+### Data privacy
+
+Making sure customer data is protected is [our highest
+priority](https://zulip.com/security/). We don’t train LLMs on Zulip Cloud
+customer data, and we have [no plans to do
+so](https://blog.zulip.com/2024/05/23/self-hosting-keeps-your-private-data-out-of-ai-models/).
+
+We are committed to keeping Zulip 100% open-source, so the source code that
+defines how data is processed is available for third parties to review and
+audit.
+
+### General configurations
+
+Self-hosted Zulip installations can choose whether to self-host their own AI
+models or use a third-party AI model API provider of their choice. Zulip’s AI
+integrations use the [OpenAI Python SDK](https://github.com/openai/openai-python),
+which can be pointed at any OpenAI-compatible API endpoint, including those
+offered by most popular AI model API providers.
+
+- **Server settings**: You can control costs using `INPUT_COST_PER_GIGATOKEN`,
+  `OUTPUT_COST_PER_GIGATOKEN`, and `MAX_PER_USER_MONTHLY_AI_COST` settings,
+  which let you set a monthly per-user AI usage budget with whatever pricing is
+  appropriate for your selected model.
+- **Organization settings**: Administrators can specify who can use each AI
+  feature that is enabled by the server. The permission can be assigned to any
+  combination of roles, groups, and individual users.
+- **Personal settings**: Users who find AI features intrusive or distracting can
+  hide them from the UI with a **Hide AI features** personal preference setting.
+
+### Topic summarization beta
+
+:::{note}
+
+Topic summarization is not yet available in Zulip Cloud.
+
+:::
+
+The Zulip server supports generating summaries of topics, with convenient
+options for doing so in the web/desktop application’s topic actions menus.
+
+#### How it works
+
+:::{warning}
+
+As with all features powered by LLMs, topic summaries may contain errors and
+hallucinations.
+
+:::
+
+The topic summarization feature uses a Zulip-specific prompt with off-the-shelf
+third-party large language models.
+
+When a user asks for a summary of a given topic, the Zulip server fetches recent
+messages in that conversation that are accessible to the acting user, and sends
+them to the AI model to generate a summary.
+
+Emoji reactions, images, and uploaded files are currently not included in what
+is sent to the AI model, though some LLMs may have features that might follow
+links in content they are asked to summarize. (Note that Zulip’s permissions
+model for uploaded files will prevent the LLM from accessing them unless the
+files have been posted to a channel with the [public access
+option](https://zulip.com/help/public-access-option) enabled.)
+
+#### Enabling topic summarization
+
+:::{important}
+
+If you use a third-party AI platform for topic summarization, you are trusting
+the third party with the security and confidentiality of all the messages that
+are sent for summarization.
+
+:::
+
+Enable topic summarization by configuring `TOPIC_SUMMARIZATION_MODEL`
+and related configuration settings in `/etc/zulip/settings.py`. Topic
+summarization and settings for controlling it will appear in the UI
+only if your server is configured to enable it.
+
+#### Choosing a model
+
+When modeling the pricing for a given model provider, you’ll primarily want to
+look at the cost per input token. Because useful summaries are short compared to
+the messages being summarized, more than 90% of tokens used in generating topic
+summaries end up being input tokens.
+
+Our experience in early 2025 has been that midsize ~70B parameter models
+generate considerably more useful and accurate summaries than smaller ~8B
+parameter models.

docs/production/deployment.md

@@ -0,0 +1,317 @@
+# Deployment options
+
+The default Zulip installation instructions will install a complete
+Zulip server, with all of the services it needs, on a single machine.
+
+For production deployment, however, it's common to want to do
+something more complicated. This page documents the options for doing so.
+
+## Installing Zulip from Git
+
+To install a development version of Zulip from Git, just clone the Git
+repository from GitHub:
+
+```bash
+# First, install Git if you don't have it installed already
+sudo apt install git
+git clone https://github.com/zulip/zulip.git zulip-server-git
+```
+
+and then
+[continue the normal installation instructions](install.md#step-2-install-zulip).
+You can also [upgrade Zulip from Git](upgrade.md#upgrading-from-a-git-repository).
+
+The most common use case for this is upgrading to `main` to get a
+feature that hasn't made it into an official release yet (often
+support for a new base OS release). See [upgrading to
+main][upgrade-to-main] for notes on how `main` works and the
+support story for it, and [upgrading to future
+releases][upgrade-to-future-release] for notes on upgrading Zulip
+afterwards.
+
+In particular, we are always very glad to investigate problems with
+installing Zulip from `main`; they are rare and help us ensure that
+our next major release has a reliable install experience.
+
+[upgrade-to-main]: modify.md#upgrading-to-main
+[upgrade-to-future-release]: modify.md#upgrading-to-future-releases
+
+## Zulip in Docker
+
+See [Running Zulip in Docker](docker.md).
+
+## Zulip installer details
+
+The [Zulip installer](install.md) does the following:
+
+- Creates the `zulip` user, which the various Zulip servers will run as.
+- Creates `/home/zulip/deployments/`, which the Zulip code for this
+  deployment (and future deployments when you upgrade) goes into. At the
+  very end of the install process, the script moves the Zulip code tree
+  it's running from (which you unpacked from a tarball above) to a
+  directory there, and makes `/home/zulip/deployments/current` as a
+  symbolic link to it.
+- Installs Zulip's various dependencies.
+- Configures the various third-party services Zulip uses, including
+  PostgreSQL, RabbitMQ, Memcached and Redis.
+- Initializes Zulip's database.
+
+### Advanced installer options
+
+The Zulip installer supports the following advanced installer options
+as well as those mentioned in the
+[install](install.md#installer-options) documentation:
+
+- `--postgresql-version`: Sets the version of PostgreSQL that will be
+  installed. We currently support PostgreSQL 14, 15, 16, 17, and 18,
+  with 18 being the default.
+
+- `--postgresql-database-name=exampledbname`: With this option, you
+  can customize the default database name. If you do not set this. The
+  default database name will be `zulip`. This setting can only be set
+  on the first install.
+
+- `--postgresql-database-user=exampledbuser`: With this option, you
+  can customize the default database user. If you do not set this. The
+  default database user will be `zulip`. This setting can only be set
+  on the first install.
+
+- `--no-init-db`: This option instructs the installer to not do any
+  database initialization. This should be used when you already have a
+  Zulip database.
+
+- `--no-overwrite-settings`: This option preserves existing
+  `/etc/zulip` configuration files.
+
+[missing-dicts]: system-configuration.md#missing_dictionaries
+
+## Installing on an existing server
+
+Zulip's installation process assumes it is the only application
+running on the server; though installing alongside other applications
+is not recommended, we do have [some notes on the
+process](install-existing-server.md).
+
+## Deployment hooks
+
+Zulip's upgrades have a hook system which allows for arbitrary
+user-configured actions to run before and after an upgrade; see the
+[upgrading documentation](upgrade.md#deployment-hooks) for details on
+how to write your own.
+
+### Zulip message deploy hook
+
+Zulip can use its deploy hooks to send a message immediately before and after
+conducting an upgrade. To configure this:
+
+1. Add `, zulip::hooks::zulip_notify` to the `puppet_classes` line in
+   `/etc/zulip/zulip.conf`
+1. Add a `[zulip_notify]` section to `/etc/zulip/zulip.conf`:
+   ```ini
+   [zulip_notify]
+   bot_email = your-bot@zulip.example.com
+   server = zulip.example.com
+   stream = deployments
+   ```
+1. Add the [api key](https://zulip.com/api/api-keys#get-a-bots-api-key) for the
+   bot user in `/etc/zulip/zulip-secrets.conf` as `zulip_release_api_key`:
+   ```ini
+   # Replace with your own bot's token, found in the Zulip UI
+   zulip_release_api_key = abcd1234E6DK0F7pNSqaMSuzd8C5i7Eu
+   ```
+1. As root, run `/home/zulip/deployments/current/scripts/zulip-puppet-apply`.
+
+### Sentry deploy hook
+
+Zulip can use its deploy hooks to create [Sentry
+releases][sentry-release], which can help associate Sentry [error
+logging][sentry-error] with specific releases. If you are deploying
+Zulip from Git, it can be aware of which Zulip commits are associated
+with the release, and help identify which commits might be relevant to
+an error.
+
+To do so:
+
+1. Enable [Sentry error logging][sentry-error].
+2. Add a new [internal Sentry integration][sentry-internal] named
+   "Release annotator".
+3. Grant the internal integration the [permissions][sentry-perms] of
+   "Admin" on "Release".
+4. Add `, zulip::hooks::sentry` to the `puppet_classes` line in `/etc/zulip/zulip.conf`
+5. Add a `[sentry]` section to `/etc/zulip/zulip.conf`:
+   ```ini
+   [sentry]
+   organization = your-organization-name
+   project = your-project-name
+   ```
+6. Add the [authentication token][sentry-tokens] for your internal Sentry integration
+   to your `/etc/zulip/zulip-secrets.conf`:
+   ```ini
+   # Replace with your own token, found in Sentry
+   sentry_release_auth_token = 6c12f890c1c864666e64ee9c959c4552b3de473a076815e7669f53793fa16afc
+   ```
+7. As root, run `/home/zulip/deployments/current/scripts/zulip-puppet-apply`.
+
+If you are deploying Zulip from Git, you will also need to:
+
+1. In your Zulip project, add the [GitHub integration][sentry-github].
+2. Configure the `zulip/zulip` GitHub project for your Sentry project.
+   You should do this even if you are deploying a private fork of
+   Zulip.
+3. Additionally grant the internal integration "Read & Write" on
+   "Organization"; this is necessary to associate the commits with the
+   release.
+
+[sentry-release]: https://docs.sentry.io/product/releases/
+[sentry-error]: ../subsystems/logging.md#sentry-error-logging
+[sentry-github]: https://docs.sentry.io/product/integrations/source-code-mgmt/github/
+[sentry-internal]: https://docs.sentry.io/product/integrations/integration-platform/internal-integration/
+[sentry-perms]: https://docs.sentry.io/product/integrations/integration-platform/#permissions
+[sentry-tokens]: https://docs.sentry.io/product/integrations/integration-platform/internal-integration#auth-tokens
+
+## Running Zulip's service dependencies on different machines
+
+Zulip has full support for each top-level service living on its own machine.
+
+You can configure remote servers for Memcached, PostgreSQL, RabbitMQ, Redis, and
+Smokescreen in `/etc/zulip/settings.py`; just search for the service name in
+that file and you'll find inline documentation in comments for how to configure
+it.
+
+All puppet modules under `zulip::profile` are allowed to be configured
+stand-alone on a host. You can see most likely manifests you might
+want to choose in the list of includes in [the main manifest for the
+default all-in-one Zulip server][standalone.pp], though it's also
+possible to subclass some of the lower-level manifests defined in that
+directory if you want to customize. A good example of doing this is
+in the [kandra Puppet configuration][zulipchat-puppet] that we use
+as part of managing chat.zulip.org and zulip.com.
+
+For example, to install a Zulip Redis server on a machine, you can run
+the following after unpacking a Zulip production release tarball:
+
+```bash
+./scripts/setup/install --puppet-classes zulip::profile::redis
+```
+
+To run the database on a separate server, including a cloud provider's managed
+PostgreSQL instance (e.g., AWS RDS), or with a warm-standby replica for
+reliability, see our [dedicated PostgreSQL documentation][postgresql].
+
+[standalone.pp]: https://github.com/zulip/zulip/blob/main/puppet/zulip/manifests/profile/standalone.pp
+[zulipchat-puppet]: https://github.com/zulip/zulip/tree/main/puppet/kandra/manifests
+[postgresql]: postgresql.md
+
+## Deploying behind a reverse proxy
+
+See our dedicated page on [reverse proxies][reverse-proxies].
+
+[reverse-proxies]: reverse-proxies.md
+
+## Using an alternate port
+
+If you'd like your Zulip server to use an HTTPS port other than 443, you can
+configure that as follows:
+
+1. Edit `EXTERNAL_HOST` in `/etc/zulip/settings.py`, which controls how
+   the Zulip server reports its own URL, and restart the Zulip server
+   with `/home/zulip/deployments/current/scripts/restart-server`.
+1. Add the following block to `/etc/zulip/zulip.conf`:
+
+   ```ini
+   [application_server]
+   nginx_listen_port = 12345
+   ```
+
+1. As root, run
+   `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This
+   will convert Zulip's main `nginx` configuration file to use your new
+   port.
+
+We also have documentation for a Zulip server [using HTTP][using-http] for use
+behind reverse proxies.
+
+[using-http]: reverse-proxies.md#configuring-zulip-to-allow-http
+
+## Customizing the outgoing HTTP proxy
+
+To protect against [SSRF][ssrf], Zulip routes all outgoing HTTP and
+HTTPS traffic through [Smokescreen][smokescreen], an HTTP `CONNECT`
+proxy; this includes outgoing webhooks, website previews, and mobile
+push notifications.
+
+### IP address rules
+
+By default, Smokescreen denies access to all [non-public IP
+addresses](https://en.wikipedia.org/wiki/Private_network), including
+127.0.0.1, but allows traffic to all public Internet hosts. You can
+[adjust those rules][smokescreen-acls]. For instance, if you have an
+outgoing webhook at `http://10.17.17.17:80/`, you would need to:
+
+1. Add the following block to `/etc/zulip/zulip.conf`, substituting
+   your internal host's IP address:
+
+   ```ini
+   [http_proxy]
+   allow_addresses = 10.17.17.17
+   ```
+
+1. As root, run
+   `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This
+   will reconfigure and restart Zulip.
+
+### Using a different outgoing proxy
+
+To use a custom outgoing proxy:
+
+1. Add the following block to `/etc/zulip/zulip.conf`, substituting in
+   your proxy's hostname/IP and port:
+
+   ```ini
+   [http_proxy]
+   host = 127.0.0.1
+   port = 4750
+   ```
+
+1. As root, run
+   `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This
+   will reconfigure and restart Zulip.
+
+If you wish to disable the outgoing proxy entirely, follow the above
+steps, configuring an empty `host` value. **This is not
+recommended**, as it allows attackers to leverage the Zulip server to
+access internal resources.
+
+### Routing Camo requests through an outgoing proxy
+
+By default, the Camo image proxy will use a custom outgoing proxy if
+one is configured, but will not use the default Smokescreen proxy
+(because Camo includes similar logic to deny access to private
+subnets). You can [override][proxy.enable_for_camo] this logic in
+either direction, if desired.
+
+### Installing Smokescreen on a separate host
+
+If you have a deployment with multiple frontend servers, or wish to
+install Smokescreen on a separate host, you can apply the
+`zulip::profile::smokescreen` Puppet class on that host, and follow
+the above steps, setting the `[http_proxy]` block to point to that
+host.
+
+[ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
+[smokescreen]: https://github.com/stripe/smokescreen
+[proxy.enable_for_camo]: system-configuration.md#enable_for_camo
+[smokescreen-acls]: system-configuration.md#allow_addresses-allow_ranges-deny_addresses-deny_ranges
+
+### S3 file storage requests and outgoing proxies
+
+By default, the [S3 file storage backend][s3] bypasses the Smokescreen
+proxy, because when running on EC2 it may require metadata from the
+IMDS metadata endpoint, which resides on the internal IP address
+169.254.169.254 and would thus be blocked by Smokescreen.
+
+If your S3-compatible storage backend requires use of Smokescreen or
+some other proxy, you can override this default by setting
+`S3_SKIP_PROXY = False` in `/etc/zulip/settings.py`.
+
+[s3]: upload-backends.md#s3-backend-configuration

docs/production/docker.md

@@ -0,0 +1,23 @@
+# Running Zulip in Docker
+
+Zulip provides an {doc}`official Docker image <docker:index>` for
+deployments that prefer to run services in containers. We recommend
+using it only if your organization has a preference for deploying
+this way; the Docker image moderately increases the effort required
+to install, maintain, and upgrade a Zulip installation, compared with
+the [standard installer](install.md).
+
+The Docker image supports two runtimes:
+
+- **Docker Compose** for single-host deployments. See the {doc}`Docker
+Compose getting started guide <docker:how-to/compose-getting-started>`.
+- **Helm charts** for Kubernetes deployments. See the {doc}`Helm
+getting started guide <docker:how-to/helm-getting-started>`.
+
+Zulip's [backup tool](export-and-import.md#backups) supports
+migrating between Docker and a standard installation, so you can
+change your mind later.
+
+For configuration, secrets, networking, upgrades, and other
+operational topics, see the {doc}`Docker documentation
+<docker:index>` in full.

docs/production/email-gateway.md

@@ -0,0 +1,144 @@
+# Incoming email integration
+
+Zulip's incoming email gateway integration makes it possible to send
+messages into Zulip by sending an email. It's highly recommended
+because it enables:
+
+- When users reply to one of Zulip's message notification emails
+  from their email client, the reply can go directly
+  into Zulip.
+- Integrating third-party services that can send email notifications
+  into Zulip. See the [integration
+  documentation](https://zulip.com/integrations/email) for
+  details.
+
+Once this integration is configured, each channel will have a special
+email address displayed on the channel settings page. Emails sent to
+that address will be delivered into the channel.
+
+There are two ways to configure Zulip's email gateway:
+
+1. Local delivery (recommended): A server runs on the Zulip
+   server and passes the emails directly to Zulip.
+1. Polling: A cron job running on the Zulip server checks an IMAP
+   inbox (`username@example.com`) every minute for new emails.
+
+The local delivery configuration is preferred for production because
+it supports nicer looking email addresses and has no cron delay. The
+polling option is convenient for testing/developing this feature
+because it doesn't require a public IP address, setting up MX
+records in DNS, or adjusting firewalls.
+
+:::{note}
+Incoming emails are rate-limited, with the following limits:
+
+- 50 emails per minute.
+- 120 emails per 5 minutes.
+- 600 emails per hour.
+
+:::
+
+## Local delivery setup
+
+:::{important}
+
+If you are using [Docker](docker.md), do not follow the steps below;
+the container is configured differently (port 25 is already exposed,
+and settings are configured via `SETTING_`-prefixed environment
+variables, such as `SETTING_EMAIL_GATEWAY_PATTERN`). Follow
+{doc}`docker:how-to/compose-incoming-email` instead.
+
+:::
+
+Zulip's Puppet configuration provides everything needed to run this
+integration; you just need to enable and configure it as follows.
+
+The main decision you need to make is what email domain you want to
+use for the gateway; for this discussion we'll use
+`emaildomain.example.com`. The email addresses used by the gateway
+will look like `foo@emaildomain.example.com`, so we recommend using
+`EXTERNAL_HOST` here.
+
+We will use `hostname.example.com` as the hostname of the Zulip server
+(this will usually also be the same as `EXTERNAL_HOST`, unless you are
+using an [HTTP reverse proxy][reverse-proxy]).
+
+1. Using your DNS provider, create a DNS MX (mail exchange) record
+   configuring email for `emaildomain.example.com` to be processed by
+   `hostname.example.com`. You can check your work using this command:
+
+   ```console
+   $ dig +short emaildomain.example.com -t MX
+   1 hostname.example.com
+   ```
+
+1. If you have a network firewall enabled, configure it to allow incoming access
+   to port 25 on the Zulip server from the public internet. Other mail servers
+   will need to use it to deliver emails to Zulip.
+
+1. Log in to your Zulip server; the remaining steps all happen there.
+
+1. Add `, zulip::local_mailserver` to `puppet_classes` in
+   `/etc/zulip/zulip.conf`. A typical value after this change is:
+
+   ```ini
+   puppet_classes = zulip::profile::standalone, zulip::local_mailserver
+   ```
+
+1. Run `/home/zulip/deployments/current/scripts/zulip-puppet-apply`
+   (and answer `y`) to apply your new `/etc/zulip/zulip.conf`
+   configuration to your Zulip server.
+
+1. Edit `/etc/zulip/settings.py`, and set `EMAIL_GATEWAY_PATTERN`
+   to `"%s@emaildomain.example.com"`.
+
+1. Restart your Zulip server with
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+Congratulations! The integration should be fully operational.
+
+:::{note}
+The incoming mail server listens on port 25, on all IPv4 and IPv6
+addresses, by default; you can adjust this with the
+[`email_gateway.listen`](system-configuration.md#listen) setting in
+`/etc/zulip/zulip.conf`.
+:::
+
+[reverse-proxy]: reverse-proxies.md
+
+## Polling setup
+
+1. Create an email account dedicated to Zulip's email gateway
+   messages. We assume the address is of the form
+   `username@example.com`. The email provider needs to support the
+   standard model of delivering emails sent to
+   `username+stuff@example.com` to the `username@example.com` inbox.
+
+1. Edit `/etc/zulip/settings.py`, and set `EMAIL_GATEWAY_PATTERN` to
+   `"username+%s@example.com"`.
+
+1. Set up IMAP for your email account and obtain the authentication details.
+   ([Here's how it works with Gmail](https://support.google.com/mail/answer/7126229?hl=en))
+
+1. Configure IMAP access in the appropriate Zulip settings:
+
+   - Login and server connection details in `/etc/zulip/settings.py`
+     in the email gateway integration section (`EMAIL_GATEWAY_LOGIN` and others).
+   - Password in `/etc/zulip/zulip-secrets.conf` as `email_gateway_password`.
+
+1. Test your configuration by sending emails to the target email
+   account and then running the Zulip tool to poll that inbox:
+
+   ```bash
+   su zulip -c '/home/zulip/deployments/current/manage.py email_mirror'
+   ```
+
+1. Once everything is working, install the cron job which will poll
+   the inbox every minute for new messages using the tool you tested
+   in the last step:
+   ```bash
+   cd /home/zulip/deployments/current/
+   sudo cp puppet/zulip/files/cron.d/email-mirror /etc/cron.d/
+   ```
+
+Congratulations! The integration should be fully operational.

docs/production/email.md

@@ -0,0 +1,238 @@
+# Outgoing email
+
+Zulip needs to be able to send email so it can confirm new users'
+email addresses and send notifications.
+
+:::{important}
+
+If you are using [Docker](docker.md), see
+{doc}`docker:how-to/compose-settings` for `SETTING_EMAIL_*`
+configuration and {doc}`docker:how-to/compose-secrets` for the
+`email_password` secret.
+
+:::
+
+## How to configure
+
+1. Identify an outgoing email (SMTP) account where you can have Zulip
+   send mail. If you don't already have one you want to use, see
+   [Email services](#email-services) below.
+
+1. Fill out the section of `/etc/zulip/settings.py` headed "Outgoing
+   email (SMTP) settings". This includes the hostname and typically
+   the port to reach your SMTP provider, and the username to log in to
+   it. If your SMTP server does not require authentication, leave
+   `EMAIL_HOST_USER` empty.
+
+   You'll also want to fill out the noreply email section, taking into
+   account the inline documentation for the proper values for those
+   settings, e.g., `NOREPLY_EMAIL_ADDRESS` and `TOKENIZED_NOREPLY_EMAIL_ADDRESS`.
+   Note that you can also configure `INSTALLATION_NAME` to customize the
+   email sender display name for notification emails from your Zulip
+   server.
+
+1. Put the password for the SMTP user account in
+   `/etc/zulip/zulip-secrets.conf` by setting `email_password`. For
+   example: `email_password = abcd1234`.
+
+   Like any other change to the Zulip configuration, be sure to
+   [restart the server](settings.md) to make your changes take
+   effect.
+
+1. Configure your SMTP server to allows your Zulip server to send
+   emails originating from the email addresses listed in
+   `/etc/zulip/settings.py` as `ZULIP_ADMINISTRATOR`,
+   `NOREPLY_EMAIL_ADDRESS` and if `ADD_TOKENS_TO_NOREPLY_ADDRESS=True`
+   (the default), `TOKENIZED_NOREPLY_EMAIL_ADDRESS`.
+
+   If you don't know how to do this, we recommend using a [free
+   transactional email service](#free-outgoing-email-services); they
+   will guide you through everything you need to do, covering details
+   like configuring DKIM/SPF authentication so your Zulip emails won't
+   be spam filtered.
+
+1. Use Zulip's email configuration test tool, documented in the
+   [Troubleshooting section](#troubleshooting), to verify that your
+   configuration is working.
+
+1. Once your configuration is working, restart the Zulip server with
+   `su zulip -c '/home/zulip/deployments/current/scripts/restart-server'`.
+
+## Email services
+
+### Free outgoing email services
+
+For sending outgoing email from your Zulip server, we highly recommend
+using a "transactional email" service like
+[Mailgun](https://documentation.mailgun.com/docs/mailgun/user-manual/sending-messages/send-smtp),
+[SendGrid](https://www.twilio.com/docs/sendgrid/for-developers/sending-email/integrating-with-the-smtp-api),
+or, for AWS users,
+[Amazon SES](https://docs.aws.amazon.com/ses/latest/dg/send-email-smtp.html).
+These services are designed to send email from servers, and are by far
+the easiest way to get outgoing email working reliably (Mailgun has
+the best documentation).
+
+If you don't have an existing outgoing SMTP provider, don't worry!
+Each of the options we recommend above (as well as dozens of other
+services) have free options. Once you've signed up, you'll want to
+find the service's provided "SMTP credentials", and configure Zulip as
+follows:
+
+- The hostname like `EMAIL_HOST = 'smtp.mailgun.org'` in `/etc/zulip/settings.py`
+- The username like `EMAIL_HOST_USER = 'username@example.com'` in
+  `/etc/zulip/settings.py`.
+- The TLS setting as `EMAIL_USE_TLS = True` in
+  `/etc/zulip/settings.py`, for most providers
+- The port as `EMAIL_PORT = 587` in `/etc/zulip/settings.py`, for most
+  providers
+- The password like `email_password = abcd1234` in `/etc/zulip/zulip-secrets.conf`.
+
+If your SMTP provider uses implicit SSL/TLS on port 465 (and not `STARTTLS` on
+port 587), you need to set `EMAIL_PORT = 465`, as well as replacing
+[`EMAIL_USE_TLS = True`](https://docs.djangoproject.com/en/5.0/ref/settings/#std-setting-EMAIL_USE_TLS)
+with [`EMAIL_USE_SSL = True`](https://docs.djangoproject.com/en/5.0/ref/settings/#std-setting-EMAIL_USE_SSL).
+
+### Using system email
+
+If you'd like to send outgoing email using the local operating
+system's email delivery configuration (e.g., you have `postfix`
+configuration on the system that forwards email sent locally into your
+corporate email system), you will likely need to use something like
+these setting values:
+
+```python
+EMAIL_HOST = 'localhost'
+EMAIL_PORT = 25
+EMAIL_USE_TLS = False
+EMAIL_HOST_USER = ""
+```
+
+We should emphasize that because modern spam filtering is very
+aggressive, you should make sure your downstream email system is
+configured to properly sign outgoing email sent by your Zulip server
+(or check your spam folder) when using this configuration. See
+[documentation on using Django with a local postfix server][postfix-email]
+for additional advice.
+
+[postfix-email]: https://stackoverflow.com/questions/26333009/how-do-you-configure-django-to-send-mail-through-postfix
+
+### Using Gmail for outgoing email
+
+We don't recommend using an inbox product like Gmail for outgoing
+email, because Gmail's anti-spam measures make this annoying. But if
+you want to use a Gmail account to send outgoing email anyway, here's
+how to make it work:
+
+- Create a totally new Gmail account for your Zulip server; you don't
+  want Zulip's automated emails to come from your personal email address.
+- If you're using 2-factor authentication on the Gmail account, you'll
+  need to use an
+  [app-specific password](https://support.google.com/accounts/answer/185833).
+- If you're not using 2-factor authentication, read this Google
+  support answer and configure that account as
+  ["less secure"](https://support.google.com/accounts/answer/6010255);
+  Gmail doesn't allow servers to send outgoing email by default.
+- Note also that the rate limits for Gmail are also quite low
+  (e.g., 100 / day), so it's easy to get rate-limited if your server
+  has significant traffic. For more active servers, we recommend
+  moving to a free account on a transactional email service.
+
+### Logging outgoing email to a file for prototyping
+
+For prototyping, you might want to proceed without setting up an email
+provider. If you want to see the emails Zulip would have sent, you
+can log them to a file instead.
+
+To do so, add these lines to `/etc/zulip/settings.py`:
+
+```python
+EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
+EMAIL_FILE_PATH = '/var/log/zulip/emails'
+```
+
+Then outgoing emails that Zulip would have sent will just be written
+to files in `/var/log/zulip/emails/`.
+
+Remember to delete this configuration (and restart the server) if you
+later set up a real SMTP provider!
+
+## Troubleshooting
+
+You can quickly test your outgoing email configuration using:
+
+```bash
+su zulip -c '/home/zulip/deployments/current/manage.py send_test_email user@example.com'
+```
+
+If it doesn't throw an error, it probably worked; you can confirm by
+checking your email. You should get two emails: One sent by the
+default From address for your Zulip server, and one sent by the
+"noreply" From address.
+
+If it doesn't work, check these common failure causes:
+
+- Your hosting provider may block outgoing SMTP traffic in its default
+  firewall rules. Check whether the port `EMAIL_PORT` is blocked in
+  your hosting provider's firewall.
+
+- Your SMTP server's permissions might not allow the email account
+  you're using to send email from the `noreply` email addresses used
+  by Zulip when sending confirmation emails.
+
+  For security reasons, Zulip sends confirmation emails (used for
+  account creation, etc.) with randomly generated from addresses
+  starting with `noreply-`.
+
+  If necessary, you can set `ADD_TOKENS_TO_NOREPLY_ADDRESS` to `False`
+  in `/etc/zulip/settings.py` (which will cause these confirmation
+  emails to be sent from a consistent `noreply@` address). Disabling
+  `ADD_TOKENS_TO_NOREPLY_ADDRESS` is generally safe if you are not
+  using Zulip's feature that allows anyone to create an account in
+  your Zulip organization if they have access to an email address in a
+  certain domain. See [this article][helpdesk-attack] for details on
+  the security issue with helpdesk software that
+  `ADD_TOKENS_TO_NOREPLY_ADDRESS` helps protect against.
+
+- Make sure you set the password in `/etc/zulip/zulip-secrets.conf`.
+
+- Check the username and password for typos.
+
+- Be sure to restart your Zulip server after editing either
+  `settings.py` or `zulip-secrets.conf`, using
+  `/home/zulip/deployments/current/scripts/restart-server` .
+  Note that the `manage.py` command above will read the latest
+  configuration from the config files, even if the server is still
+  running with an old configuration.
+
+### Advanced troubleshooting
+
+Here are a few final notes on what to look at when debugging why you
+aren't receiving emails from Zulip:
+
+- Most transactional email services have an "outgoing email" log where
+  you can inspect the emails that reached the service, whether an
+  email was flagged as spam, etc.
+
+- Zulip logs an entry in `/var/log/zulip/send_email.log` whenever it
+  attempts to send an email. The log entry includes whether the
+  request succeeded or failed.
+
+- If attempting to send an email throws an exception, a traceback
+  should be in `/var/log/zulip/errors.log`, along with any other
+  exceptions Zulip encounters.
+
+- Zulip's email sending configuration is based on the standard Django
+  [SMTP backend](https://docs.djangoproject.com/en/5.0/topics/email/#smtp-backend)
+  configuration. So if you're having trouble getting your email
+  provider working, you may want to search for documentation related
+  to using your email provider with Django.
+
+  The one thing we've changed from the Django defaults is that we read
+  the email password from the `email_password` entry in the Zulip
+  secrets file, as part of our policy of not having any secret
+  information in the `/etc/zulip/settings.py` file. In other words,
+  if Django documentation references setting `EMAIL_HOST_PASSWORD`,
+  you should instead set `email_password` in
+  `/etc/zulip/zulip-secrets.conf`.
+
+[helpdesk-attack]: https://medium.com/intigriti/how-i-hacked-hundreds-of-companies-through-their-helpdesk-b7680ddc2d4c

docs/production/export-and-import.md

@@ -0,0 +1,662 @@
+# Backups, export and import
+
+Zulip has high quality export and import tools that can be used to
+move data from one Zulip server to another, do backups, compliance
+work, or migrate from your own servers to the hosted Zulip Cloud
+service (or back):
+
+- The [Backup](#backups) tool is designed for exact restoration of a
+  Zulip server's state, for disaster recovery, testing with production
+  data, and hardware migrations.
+
+  We highly recommend this tool in situations where it is applicable,
+  because it is fast, robust, and minimizes disruption for your
+  users. This tool has a few limitations:
+
+  - Backups must be restored on a server running the same Zulip
+    version (most precisely, one where `manage.py showmigrations` has
+    identical output).
+  - Backups must be restored on a server running the same PostgreSQL
+    version. To install Zulip with the same version of PostgreSQL that
+    the backup was taken on, pass the desired version with [the
+    `--postgresql-version` argument][installer-options] when
+    installing. Note that PostgreSQL is easy to [upgrade
+    independently][postgres-upgrade] from the rest of your Zulip
+    installation.
+  - Backups aren't useful for migrating organizations between
+    self-hosting and Zulip Cloud (which may require renumbering all
+    the users/messages/etc.).
+
+  We also document [backup details](#backup-details) for users
+  managing backups manually.
+
+- The [logical data export](#data-export) tool is designed for
+  migrating data between Zulip Cloud and other Zulip servers, as well
+  as various auditing purposes.
+
+  We recommend this tool in cases where the backup tool isn't
+  applicable, including situations where an easily machine-parsable
+  export format is desired. This tool has a few limitations and
+  caveats:
+
+  - Like the backup tool, logical data exports must be imported on a
+    Zulip server running the same Zulip version. However, logical data
+    exports can be imported on Zulip servers running a different
+    PostgreSQL version or hosting a different set of Zulip
+    organizations.
+  - Transferring an organization via the data export tool results in
+    significant user-facing disruption, such as logging all users out of
+    their accounts and requiring them to reset their passwords.
+
+  The logical export tool produces a `.tar.gz` archive with most of
+  the Zulip database data encoded in JSON files–a format shared by our
+  [data import](#import-into-a-new-zulip-server) tools for third-party
+  services like [Slack](https://zulip.com/help/import-from-slack).
+
+- [Compliance exports](#compliance-exports) allow a server
+  administrator to export messages matching a search query.
+
+- Zulip also has an [HTML archive
+  tool](https://github.com/zulip/zulip-archive), which is primarily
+  intended for public archives, but can also be useful to
+  inexpensively preserve public channel conversations when
+  decommissioning a Zulip organization.
+
+- It's possible to set up [PostgreSQL streaming
+  replication](postgresql.md#postgresql-warm-standby)
+  and the [S3 file upload
+  backend](upload-backends.md#s3-backend-configuration)
+  as part of a high availability environment.
+
+[installer-options]: deployment.md#advanced-installer-options
+[postgres-upgrade]: upgrade.md#upgrading-postgresql
+
+## Backups
+
+(backups)=
+
+:::{important}
+
+If you are using [Docker](docker.md), the recommended backup unit
+is a snapshot of the `/data` volume containing a recent `app:backup`
+database dump; see {doc}`docker:reference/data-volume`. The
+`manage.py backup` tool documented below also works for migrating
+between Docker and a standard installation in either direction, but
+is not the recommended mechanism for routine backups of a Docker
+deployment.
+
+:::
+
+The Zulip server has a built-in backup tool:
+
+```bash
+# As the zulip user
+/home/zulip/deployments/current/manage.py backup
+# Or as root
+su zulip -c '/home/zulip/deployments/current/manage.py backup'
+```
+
+The backup tool provides the following options:
+
+- `--output=/tmp/backup.tar.gz`: Filename to write the backup tarball
+  to (default: write to a file in `/tmp`). On success, the
+  console output will show the path to the output tarball.
+- `--skip-db`: Skip backup of the database. Useful if you're using a
+  remote PostgreSQL host with its own backup system and just need to
+  back up non-database state.
+- `--skip-uploads`: If `LOCAL_UPLOADS_DIR` is set, user-uploaded files
+  in that directory will be ignored.
+
+This will generate a `.tar.gz` archive containing all the data stored
+on your Zulip server that would be needed to restore your Zulip
+server's state on another machine perfectly.
+
+### Restoring backups
+
+1. Install the same base OS as the backup was taken on. If you want to [upgrade
+   the OS][upgrade-os], you should do this after restoring the backup.
+
+1. [Install a new Zulip server through Step 3][install-server], with the same
+   version of PostgreSQL that the backup was taken on, by passing the desired
+   version with [the `--postgresql-version` argument][installer-options]. If
+   you want to [upgrade the version of PostgreSQL][upgrade-pg], you should do this after
+   restoring the backup.
+
+1. As root, import the backup:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/restore-backup /path/to/backup
+   ```
+
+When that finishes, your Zulip server should be fully operational again.
+
+[upgrade-os]: upgrade.md#upgrading-the-operating-system
+[upgrade-pg]: upgrade.md#upgrading-postgresql
+
+#### Changing the hostname
+
+It's common, when testing backup restoration, to restore backups with a
+different user-facing hostname than the original server to avoid
+disrupting service (e.g., `zuliptest.example.com` rather than
+`zulip.example.com`).
+
+If you do so, just like any other time you change the hostname, you'll
+need to [update `EXTERNAL_HOST`](settings.md) and then
+restart the Zulip server (after backup restoration completes).
+
+Until you do, your Zulip server will think its user-facing hostname is
+still `zulip.example.com` and will return HTTP `400 BAD REQUEST`
+errors when trying to access it via `zuliptest.example.com`.
+
+#### Changing database settings
+
+If you wish to restore onto a very differently configured host (e.g., with
+`REMOTE_POSTGRES_HOST` set to a different value), you can edit
+`/etc/zulip/settings.py` to configure the host to suit the new host's needs,
+then restore with `--keep-settings`:
+
+```bash
+/home/zulip/deployments/current/scripts/setup/restore-backup --keep-settings /path/to/backup
+```
+
+You can also pass `--keep-zulipconf` if you wish to preserve the local
+`/etc/zulip/zulip.conf`.
+
+#### Inspecting a backup tarball
+
+If you're not sure what versions were in use when a given backup was
+created, you can get that information via the files in the backup
+tarball: `postgres-version`, `os-version`, and `zulip-version`. The
+following command may be useful for viewing these files without
+extracting the entire archive.
+
+```bash
+tar -Oaxf /path/to/archive/zulip-backup-rest.tar.gz zulip-backup/zulip-version
+```
+
+[install-server]: install.md
+
+### What is included
+
+Backups contain everything you need to fully restore your Zulip
+server, including the database, settings, secrets from
+`/etc/zulip`, and user-uploaded files stored on the Zulip server.
+
+The following data is not included in these backup archives,
+and you may want to back up separately:
+
+- The server access/error logs from `/var/log/zulip`. The Zulip
+  server only appends to logs, and they can be very large compared to
+  the rest of the data for a Zulip server.
+
+- Files uploaded with the Zulip
+  [S3 file upload backend](upload-backends.md). We
+  don't include these for two reasons. First, the uploaded file data
+  in S3 can easily be many times larger than the rest of the backup,
+  and downloading it all to a server doing a backup could easily
+  exceed its disk capacity. Additionally, S3 is a reliable persistent
+  storage system with its own high-quality tools for doing backups.
+
+- SSL certificates. These are not included because they are
+  particularly security-sensitive and are either trivially replaced
+  (if generated via Certbot) or provided by the system administrator.
+
+For completeness, Zulip's backups do not include certain highly
+transient state that Zulip doesn't store in a database. For example,
+typing status data, API rate-limiting counters, and RabbitMQ queues
+that are essentially always empty in a healthy server (like outgoing
+emails to send). You can check whether these queues are empty using
+`rabbitmqctl list_queues`.
+
+On a [Docker](docker.md) deployment, the inclusion list is different
+— the backup unit is the `/data` volume rather than a `manage.py
+backup` tarball. See {doc}`docker:reference/data-volume` for the
+layout, and {ref}`docker:compose-volume-snapshot` or
+{ref}`docker:helm-volume-snapshot` for capturing it.
+
+#### Backup details
+
+This section is primarily for users managing backups themselves
+(e.g., if they're using a remote PostgreSQL database with an existing
+backup strategy), and also serves as documentation for what is
+included in the backups generated by Zulip's standard tools. The
+data includes:
+
+- The PostgreSQL database. You can back this up with any standard
+  database export or backup tool; see
+  [below](#database-only-backup-tools) for Zulip's built-in support
+  for continuous point-in-time backups.
+
+- Any user-uploaded files. If you're using S3 as storage for file
+  uploads, this is backed up in S3. But if you have instead set
+  `LOCAL_UPLOADS_DIR`, any files uploaded by users (including avatars)
+  will be stored in that directory and you'll want to back it up.
+
+- Your Zulip configuration including secrets from `/etc/zulip/`.
+  E.g., if you lose the value of `secret_key`, all users will need to
+  log in again when you set up a replacement server since you won't be
+  able to verify their cookies. If you lose `avatar_salt`, any
+  user-uploaded avatars will need to be re-uploaded (since avatar
+  filenames are computed using a hash of `avatar_salt` and user's
+  email), etc.
+
+### Restore from manual backups
+
+To restore from a manual backup, the process is basically the reverse of the above:
+
+- Install new server as normal by downloading a Zulip release tarball
+  and then using `scripts/setup/install`. You should pass
+  `--no-init-db` because we don't need to create a new database.
+
+- Unpack to `/etc/zulip` the `settings.py` and `zulip-secrets.conf` files
+  from your backups.
+
+- Restore your database from the backup.
+
+- Reconfigure rabbitmq to use the password from `secrets.conf`
+  by running, as root, `scripts/setup/configure-rabbitmq`.
+
+- If you're using local file uploads, restore those files to the path
+  specified by `settings.LOCAL_UPLOADS_DIR` and (if appropriate) any
+  logs.
+
+- Start the server using `scripts/restart-server`.
+
+This restoration process can also be used to migrate a Zulip
+installation from one server to another.
+
+We recommend running a disaster recovery test after setting up your backups to
+confirm that your backups are working.
+
+## Data export
+
+Zulip's powerful data export tool is designed to handle migration of a
+Zulip organization between different Zulip installations; as a result,
+these exports contain all non-transient data for a Zulip organization,
+with the exception of secrets, like passwords and API keys.
+
+We recommend instead using the [backup tool](#backups) in all
+scenarios where it is applicable, because this data export process has
+a few downsides in comparison:
+
+- All users will have their passwords randomized and be logged out of
+  their accounts, both on web and mobile clients.
+- All bots and integrations will need to be updated with new API keys.
+- Users, channels, and messages are usually renumbered, which will
+  break most links from external programs referencing these objects.
+
+### Consider upgrading
+
+We recommend [upgrading your Zulip server](../production/upgrade.md)
+to the latest release [maintenance
+release](../overview/release-lifecycle.md), or at least the latest
+maintenance release for your major Zulip version.
+
+**For Zulip Cloud imports**: If you are exporting data from a
+self-hosted version of Zulip for purposes of importing into Zulip
+Cloud, you should first [upgrade your server to the
+`zulip-cloud-current` branch][upgrade-zulip-from-git]:
+
+```bash
+/home/zulip/deployments/current/scripts/upgrade-zulip-from-git zulip-cloud-current
+```
+
+It is not sufficient to be on the latest stable release, because Zulip
+Cloud runs pre-release versions of Zulip that are often several months
+of development ahead of the latest release.
+
+### Preventing changes during the export
+
+For best results, you'll want to shut down access to the organization
+before exporting; so that nobody can send new messages (etc.) while
+you're exporting data. There are two ways to do this:
+
+1. `./scripts/stop-server`, which stops the whole server. This is
+   preferred if you're not hosting multiple organizations, because it has
+   no side effects other than disabling the Zulip server for the
+   duration.
+1. Pass `--deactivate` to `./manage export`, which first deactivates
+   the target organization, logging out all active login sessions and
+   preventing all accounts from logging in or accessing the API. This is
+   preferred for environments like Zulip Cloud where you might want to
+   export a single organization without disrupting any other users, and
+   the intent is to move hosting of the organization (and forcing users
+   to re-log in would be required as part of the hosting migration
+   anyway).
+
+We include both options in the instructions below, commented out so
+that neither runs (using the `# ` at the start of the lines). If
+you'd like to use one of these options, remove the `# ` at the start
+of the lines for the appropriate option.
+
+### Export your Zulip data
+
+Log in to a shell on your Zulip server as the `zulip` user. Run the
+following commands:
+
+```bash
+cd /home/zulip/deployments/current
+# ./scripts/stop-server
+# export DEACTIVATE_FLAG="--deactivate"   # Deactivates the organization
+./manage.py export -r '' $DEACTIVATE_FLAG # Exports the data
+```
+
+(The `-r` option lets you specify the organization to export; `''` is
+the default organization hosted at the Zulip server's root domain.)
+
+This will generate a compressed archive with a name like
+`/tmp/zulip-export-2026-05-25-09-30-45-zcmpxfm6.tar.gz`. The archive contains several JSON
+files (containing the Zulip organization's data) as well as an archive
+of all the organization's uploaded files.
+
+## Import into a new Zulip server
+
+1. [Install a new Zulip server](install.md),
+   **skipping Step 3** (you'll create your Zulip organization via the data
+   import tool instead).
+
+   - Ensure that the Zulip server you're importing into is running the same
+     version of Zulip as the server you're exporting from.
+
+   - For exports created from Zulip Cloud (zulip.com), you need to [upgrade to
+     `zulip-cloud-current`][upgrade-zulip-from-git], which represents the
+     current version that Zulip Cloud is running; this is generally `main`
+     delayed by a week or two. To upgrade to that:
+
+     ```bash
+     /home/zulip/deployments/current/scripts/upgrade-zulip-from-git zulip-cloud-current
+     ```
+
+     It is not sufficient to be on the latest stable release, as
+     zulip.com runs pre-release versions of Zulip that are often
+     several months of development ahead of the latest release.
+
+   - Note that if your server has limited free RAM, you'll want to
+     shut down the Zulip server with `./scripts/stop-server` while
+     you run the import, since our minimal system requirements do not
+     budget extra RAM for running the data import tool.
+
+2. If your new Zulip server is meant to fully replace a previous Zulip
+   server, copying `/etc/zulip/settings.py` and
+   `/etc/zulip/zulip.conf` is safe and recommended, to avoid
+   unnecessarily repeating configuration work.
+
+   Copying `/etc/zulip/zulip-secrets.conf` is also safe and
+   recommended, with the following important exceptions and notes:
+
+   - Copying `avatar_salt` is not recommended. Zulip will
+     automatically rewrite avatars at URLs appropriate for the new
+     user IDs, and using the same avatar salt (and same server URL)
+     post import could result in issues with browsers caching and
+     displaying avatar images improperly for users whose ID was
+     renumbered.
+   - Copying `zulip_org_id` and `zulip_org_key` is recommended to
+     avoid disconnecting your Zulip server from its registration with
+     the [Mobile Push Notifications Service][mobile-push].
+   - If you copy the `rabbitmq_password` secret from
+     `zulip-secrets.conf`, you'll need to run
+     `scripts/setup/configure-rabbitmq` as root to update your local
+     RabbitMQ installation to use the password in your Zulip secrets
+     file.
+   - Copying `camo_key` is required to avoid breaking links from Zulip
+     messages to externally hosted images.
+   - If your Zulip server is on an old Zulip Server release that
+     predates Zulip 5.0, and you use the [Mobile Push Notifications
+     Service][mobile-push], you should upgrade before you do the
+     export/import process if at all possible, and [ask for support][contact-support] if
+     it is not.
+
+3. Log in to a shell on your Zulip server as the `zulip` user. Run the
+   following commands, replacing the filename with the path to your data
+   export tarball:
+
+   ```bash
+   cd ~
+   tar -xf /path/to/export/file/zulip-export-2026-05-25-09-30-45-zcmpxfm6.tar.gz
+   cd /home/zulip/deployments/current
+   ./manage.py import '' ~/zulip-export-2026-05-25-09-30-45-zcmpxfm6
+   ./scripts/start-server
+   ```
+
+This could take several minutes to run depending on how much data you're
+importing.
+
+[contact-support]: https://zulip.com/help/contact-support
+[upgrade-zulip-from-git]: upgrade.md#upgrading-from-a-git-repository
+
+#### Import options
+
+The commands above create an imported organization on the root domain
+(`EXTERNAL_HOST`) of the Zulip installation. You can also import into a
+custom subdomain, e.g., if you already have an existing organization on the
+root domain. Replace the last two lines above with the following, after replacing
+`<subdomain>` with the desired subdomain.
+
+```bash
+./manage.py import <subdomain> ~/zulip-export-2026-05-25-09-30-45-zcmpxfm6
+./scripts/start-server
+```
+
+### Logging in
+
+Once the import completes, all your users will have accounts in your
+new Zulip organization, but those accounts won't have passwords yet
+(since for security reasons, passwords are not exported).
+Your users will need to either authenticate using something like
+Google auth or start by resetting their passwords.
+
+You can use the `./manage.py send_password_reset_email` command to
+send password reset emails to your users. We
+recommend starting with sending one to yourself for testing:
+
+```bash
+./manage.py send_password_reset_email -u username@example.com
+```
+
+and then once you're ready, you can email them to everyone using,
+for example:
+
+```bash
+./manage.py send_password_reset_email -r '' --all-users
+```
+
+(replace `''` with your subdomain if you're using one).
+
+### Deleting and re-importing
+
+If you did a test import of a Zulip organization, you may want to
+delete the test import data from your Zulip server before doing a
+final import. You can **permanently delete** all data from a Zulip
+organization by running (replacing `''` with the subdomain if [you are
+hosting the organization on a subdomain](multiple-organizations.md)):
+
+```bash
+./manage.py delete_realm -r ''
+```
+
+Assuming you're using the
+[local file uploads backend](upload-backends.md), you
+can additionally delete all file uploads, avatars, and custom emoji on
+a Zulip server (across **all organizations**) with the following
+command:
+
+```bash
+rm -rf /home/zulip/uploads/*/*
+```
+
+If you're hosting multiple organizations and would like to remove
+uploads from a single organization, you'll need to access `realm.id`
+in the management shell before deleting the organization from the
+database (this will be `2` for the first organization created on a
+Zulip server, shown in the example below), for example:
+
+```bash
+rm -rf /home/zulip/uploads/*/2/
+```
+
+Once that's done, you can simply re-run the import process.
+
+## Compliance exports
+
+In some circumstances, corporate or legal compliance may require
+performing selective data exports. This can be done with the
+`export_search` command-line tool, which lets you specify the
+following parameters when exporting messages:
+
+- Search keywords in the message text.
+- Message sender or recipient.
+- Time range for when messages were sent.
+
+For example, to search for messages containing the word "wonderland"
+between November 1st and 6th, from `alice@example.com`:
+
+```console
+$ /home/zulip/deployments/current/manage.py export_search --output compliance-export.json
+    -r zulip \
+    --after '2022-11-01 00:00:00' --before '2022-11-06 14:00:00' \
+    --sender alice@example.com \
+    wonderland
+```
+
+The results are written to a JSON or CSV file. The contents of previous versions
+of edited messages are not searched, nor are deleted messages. Attachments
+associated with the resulting messages can optionally also be exported.
+
+See `/home/zulip/deployments/current/manage.py export_search --help`
+for more details on supported options.
+
+## Database-only backup tools
+
+(wal-g)=
+
+The [Zulip-specific backup tool documented above](#backups) is perfect for an
+all-in-one backup solution, and can be used for nightly backups. For
+administrators wanting continuous point-in-time backups, Zulip has built-in
+support for taking daily backup snapshots along with [streaming write-ahead log
+(WAL)][wal] backups using [wal-g](https://github.com/wal-g/wal-g). By default,
+these backups are stored for 30 days.
+
+Note these database backups, by themselves, do not constitute a full
+backup of the Zulip system! [See above](#backup-details) for other
+pieces which are necessary to back up a Zulip system.
+
+Daily full-database backups will be taken at 0200 UTC, and every [WAL][wal]
+archive file will be backed up as it is saved by PostgreSQL; these are written
+every 16KiB of the WAL. This means that if there are periods of slow activity,
+it may be minutes before the backup is saved into S3 -- see
+[`archive_timeout`][archive-timeout] for how to set an upper bound on this.
+
+If you need always-current backup availability, Zulip also has
+[built-in database replication support](postgresql.md#postgresql-warm-standby).
+
+You can (and should) monitor that backups are running regularly, for instance
+via the Prometheus exporter found in
+`puppet/zulip/files/postgresql/wal-g-exporter`
+
+### Streaming backups to S3
+
+This provides a durable and reliable off-host database backup, and we suggest
+this configuration for resilience to disk failures. Because backups are written
+to S3 as the WAL logs are written, this means that an active Zulip server will
+be regularly sending PutObject requests to S3, possibly thousands of times per
+day.
+
+1. Edit `/etc/zulip/zulip-secrets.conf` on the PostgreSQL server to add:
+
+   ```ini
+   s3_region = # region to write to S3; defaults to EC2 host's region
+   s3_backups_key = # aws public key; optional, if access not through role
+   s3_backups_secret_key =  # aws secret key; optional, if access not through role
+   s3_backups_bucket = # name of S3 backup bucket
+   ```
+
+1. Run:
+
+   ```shell
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply
+   ```
+
+You may also want to adjust the
+[concurrency](system-configuration.md#backups_disk_concurrency), [S3 storage
+class](system-configuration.md#backups_storage_class), or [incremental
+backups][incremental] configuration.
+
+### Streaming backups to local disk
+
+As an alternative to storing backups to S3, you can also store backups to a
+local disk. This option is not recommended for disaster recovery purposes,
+since unless the directory is on a different disk from the database itself,
+_backups will likely also be lost if the database is lost._ This setting can be
+useful if the path is on a NAS mountpoint, or if some other process copies this
+data off the disk; or if backups are purely for point-in-time historical
+analysis of recent application-level data changes.
+
+1. Edit `/etc/zulip/zulip.conf` on the PostgreSQL server, and add to the existing
+   `[postgresql]` section:
+
+   ```ini
+    # Adjust this path to your desired storage location; this should be on a
+    # different disk than /var/lib/postgresql/ which stores the database.
+    backups_directory = /srv/zulip-db-backups
+   ```
+
+1. Run:
+
+   ```shell
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply
+   ```
+
+You may also want to adjust the [incremental backups][incremental]
+configuration.
+
+### Restoring from wal-g backups
+
+The following steps will restore the database from the latest
+backup. Note that this process will _delete your current database_.
+
+1. As `root` on your database host, check your list of backups; the
+   most recent will be listed at the bottom, and is what will be
+   restored by the commands below.
+
+   ```shell
+   env-wal-g backup-list --pretty
+   ```
+
+1. Stop Zulip, if it is running. On your application host (which may
+   or may not be different from your database host, depending on your
+   configuration):
+
+   ```shell
+   /home/zulip/deployments/current/scripts/stop-server
+   ```
+
+1. As `root` on your database host, stop PostgreSQL:
+
+   ```shell
+   service postgresql stop
+   ```
+
+1. As `root` on your database host, delete the current database, and
+   restore from the backup. This may take some time, depending on the
+   size of your database and your connection to your backup storage.
+
+   ```shell
+   pg_version=$(crudini --get /etc/zulip/zulip.conf postgresql version)
+   rm -rf "/var/lib/postgresql/$pg_version/main"
+   env-wal-g backup-fetch "/var/lib/postgresql/$pg_version/main" LATEST
+   chown -R postgres.postgres "/var/lib/postgresql/$pg_version/main"
+   touch "/var/lib/postgresql/$pg_version/main/recovery.signal"
+   service postgresql start
+   ```
+
+1. As `root` on your application host, flush caches and start Zulip:
+
+   ```shell
+   /home/zulip/deployments/current/scripts/setup/flush-memcached
+   /home/zulip/deployments/current/scripts/start-server
+   ```
+
+[wal]: https://www.postgresql.org/docs/current/wal-intro.html
+[archive-timeout]: https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-TIMEOUT
+[mobile-push]: ../production/mobile-push-notifications.md
+[incremental]: system-configuration.md#backups_incremental

docs/production/gif-picker-integrations.md

@@ -0,0 +1,148 @@
+# GIF picker integrations
+
+This page documents the server-level configuration required to enable
+GIPHY or Tenor integrations to [add GIFs in your message][help-center-giphy] on
+a self-hosted Zulip server.
+
+:::{note}
+If API keys are configured for multiple integrations, Zulip will prioritize
+Tenor, followed by KLIPY, and then GIPHY.
+:::
+
+## GIPHY
+
+To enable the GIPHY integration, you need to get a beta or production API key
+from [GIPHY](https://developers.giphy.com/).
+
+### Apply for API key
+
+1. [Create a GIPHY account](https://giphy.com/join).
+
+1. Create a GIPHY API key by clicking “Create an App” on the
+   [Developer Dashboard][giphy-dashboard].
+
+1. Choose **SDK** as product type and click **Next Step**.
+
+1. Enter a name and a description for your app and click on **Create
+   New App**. The hostname for your Zulip server is a fine name.
+
+1. You will receive a [rate-limited][giphy-beta-api] beta API key, which
+   may be sufficient for smaller Zulip servers.
+
+1. (_optional_) If the beta API key rate limit is too low for your
+   organization, apply for a production API key by following the
+   instructions provided by GIPHY when you receive your beta API key.
+   Note that when submitting a screenshot to request a production API
+   key, GIPHY expects the screenshot to show the full page (including
+   the URL).
+
+You can then configure your Zulip server to use GIPHY API as
+follows:
+
+1. In `/etc/zulip/settings.py`, enter your GIPHY API key as
+   `GIPHY_API_KEY = "<Your API key from GIPHY>"`.
+
+   GIPHY API keys are not secrets -- GIPHY expects every browser or
+   other client connecting to your Zulip server will receive a copy --
+   which is why they are configured in `settings.py` and not
+   `zulip-secrets.conf`.
+
+1. Restart the Zulip server with
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+Congratulations! You've configured the GIPHY integration for your
+Zulip server. Your users can now use the integration as described in
+[the help center article][help-center-giphy]. (A browser reload may
+be required).
+
+## Tenor
+
+To enable the Tenor GIF integration, you will need to create
+a production API key from Tenor.
+
+### Create a Tenor API key
+
+1. Navigate to the [Google Cloud Console](https://console.cloud.google.com/)
+   and either sign in with an existing Google account or create a new one.
+
+1. Choose an existing Google Cloud project where you want to enable the
+   Tenor API, or create a new project if necessary.
+
+1. In the Google Cloud Console, search for "Tenor API" in the search bar.
+
+1. Select the Tenor API from the results and click "Enable" to activate
+   it for your project.
+
+1. After enabling the API, go to the "Credentials" section within your project.
+
+1. Click "Create credentials" and choose "API Key" from the dropdown menu.
+
+1. While configuring the API key, you can optionally restrict it to only
+   call the Tenor API.
+
+1. Store the newly generated API key.
+
+You can then configure your Zulip server to use the Tenor API
+as follows:
+
+1. In `/etc/zulip/settings.py`, enter your Tenor API key as
+   `TENOR_API_KEY = "<Your API key from the Google Cloud Console>"`.
+
+   Just like GIPHY API keys, Tenor API keys are not secrets -- Tenor expects
+   every browser or other client connecting to your Zulip server will
+   receive a copy -- which is why they are configured in `settings.py` and not
+   `zulip-secrets.conf`.
+
+1. Restart the Zulip server with
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+Congratulations! You've configured the Tenor integration for your
+Zulip server. Your users can now use the integration as described in
+[the help center article][help-center-giphy]. (A browser reload may
+be required).
+
+## KLIPY
+
+To enable the KLIPY GIF integration, you will need to create
+a test or production API key from KLIPY.
+
+### Create a KLIPY API key
+
+1. On the KLIPY Developer page, navigate to the [API Keys section](https://partner.klipy.com/api-keys)
+
+1. Select **Add Platform** and enter the details of your Zulip server.
+
+1. In the **Create API Key** modal, enter the relevant details and select **Create**.
+
+1. You will receive a [rate-limited][klipy-api-key-rate-limit] test API key, which
+   may be sufficient for smaller Zulip servers.
+
+1. (_optional_) If the test API key rate limit is too low for your Zulip server,
+   you can click on the **three-dot button** on the right and select **Request Production**.
+   Enter the necessary details and select **Apply**.
+
+1. Store the newly generated API key.
+
+You can then configure your Zulip server to use the KLIPY API
+as follows:
+
+1. In `/etc/zulip/settings.py`, enter your KLIPY API key as
+   `KLIPY_API_KEY = "<Your API key>"`.
+
+   Just like GIPHY API keys, KLIPY API keys are not secrets -- KLIPY expects
+   every browser or other client connecting to your Zulip server will
+   receive a copy -- which is why they are configured in `settings.py` and not
+   `zulip-secrets.conf`.
+
+1. Restart the Zulip server with
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+Congratulations! You've configured the KLIPY integration for your
+Zulip server. Your users can now use the integration as described in
+[the help center article][help-center-giphy]. (A browser reload may
+be required).
+
+[help-center-giphy]: https://zulip.com/help/animated-gifs-from-giphy
+[giphy-dashboard]: https://developers.giphy.com/dashboard/
+[giphy-beta-api]: https://support.giphy.com/hc/en-us/articles/360035340511
+[klipy-api-key-rate-limit]: https://partner.klipy.com/faqs#4

docs/production/index.md

@@ -0,0 +1,33 @@
+# Zulip in production
+
+```{toctree}
+---
+maxdepth: 3
+---
+
+requirements
+install
+troubleshooting
+management-commands
+settings
+system-configuration
+mobile-push-notifications
+upgrade
+modify
+securing-your-zulip-server
+authentication-methods
+export-and-import
+postgresql
+upload-backends
+ssl-certificates
+email
+deployment
+docker
+reverse-proxies
+multiple-organizations
+email-gateway
+video-calls
+ai-integrations
+gif-picker-integrations
+scim
+```

docs/production/install-existing-server.md

@@ -0,0 +1,108 @@
+---
+orphan: true
+---
+
+# Production installation on an existing server
+
+Here are some tips for installing the latest release of Zulip on a
+production server running Debian or Ubuntu. The Zulip installation
+scripts assume that it has carte blanche to overwrite your
+configuration files in /etc, so we recommend against installing it on
+a server running other nginx or django apps.
+
+But if you do, here are some things you can do that may make it
+possible to retain your existing site. However, this is _NOT_
+recommended, and you may break your server. Make sure you have backups
+and a provisioning script ready to go to wipe and restore your
+existing services if (when) your server goes down.
+
+These instructions are only for experts. If you're not an experienced
+Linux sysadmin, you will have a much better experience if you get a
+dedicated VM to install Zulip on instead (or [use
+zulip.com](https://zulip.com)).
+
+### nginx
+
+Copy your existing nginx configuration to a backup and then merge the
+one created by Zulip into it:
+
+```bash
+sudo cp /etc/nginx/nginx.conf /etc/nginx.conf.before-zulip-install
+sudo curl -fL -o /etc/nginx/nginx.conf.zulip \
+    https://raw.githubusercontent.com/zulip/zulip/main/puppet/zulip/templates/nginx.conf.template.erb
+sudo meld /etc/nginx/nginx.conf /etc/nginx/nginx.conf.zulip  # be sure to merge to the right
+```
+
+Since the file in Zulip is an [ERB Puppet
+template](https://puppet.com/docs/puppet/7/lang_template_erb.html),
+you will also need to replace any `<%= ... %>` sections with
+appropriate content. For instance `<%= @ca_crt %>` should be replaced
+with `/etc/ssl/certs/ca-certificates.crt` on Debian and Ubuntu
+installs.
+
+After the Zulip installation completes, then you can overwrite (or
+merge) your new nginx.conf with the installed one:
+
+```console
+$ sudo meld /etc/nginx/nginx.conf.zulip /etc/nginx/nginx.conf  # be sure to merge to the right
+$ sudo service nginx restart
+```
+
+Zulip's Puppet configuration will change the ownership of
+`/var/log/nginx` so that the `zulip` user can access it. Depending on
+your configuration, this may or may not cause problems.
+
+Depending on how you have configured `nginx` for your other
+services, you may need to set
+[`nginx_server_name`](system-configuration.md#nginx_server_name) in
+`/etc/zulip/zulip.conf`, so that Zulip's `nginx` `server` blocks
+only claim requests for their intended hostnames.
+
+### Puppet
+
+If you have a Puppet server running on your server, you will get an
+error message about not being able to connect to the client during the
+install process:
+
+```console
+puppet-agent[29873]: Could not request certificate: Failed to open TCP connection to puppet:8140
+```
+
+So you'll need to shut down any Puppet servers.
+
+```console
+$ sudo service puppet-agent stop
+$ sudo service puppet stop
+```
+
+### PostgreSQL
+
+Zulip expects to install PostgreSQL 16, and find that listening on
+port 5432; any other version of PostgreSQL that is detected at install
+time will cause the install to abort. If you already have PostgreSQL
+installed, you can pass `--postgresql-version=` to the installer to
+have it use that version. It will replace the package with the latest
+from the PostgreSQL apt repository, but existing data will be
+retained.
+
+If you have an existing PostgreSQL database, note that Zulip will use
+the default `main` as its database name; make sure you're not using
+that.
+
+### Memcached, Redis, and RabbitMQ
+
+Zulip will, by default, configure these services for its use. The
+configuration we use is pretty basic, but if you're using them for
+something else, you'll want to make sure the configurations are
+compatible.
+
+### No uninstall process
+
+We don't provide a convenient way to uninstall a Zulip server.
+
+## No support, but contributions welcome!
+
+Most of the limitations are things we'd accept a pull request to fix;
+we welcome contributions to shrink this list of gotchas. Chat with us
+in the [chat.zulip.org community](https://zulip.com/development-community/) if you're
+interested in helping!

docs/production/install.md

@@ -0,0 +1,204 @@
+# Install a Zulip server
+
+You can choose from several convenient options for hosting Zulip:
+
+- Follow these instructions to **install a self-hosted Zulip server on a system
+  of your choice**.
+- Use Zulip's [Docker image](docker.md)
+- Use a preconfigured
+  [DigitalOcean droplet](https://marketplace.digitalocean.com/apps/zulip?refcode=3ee45da8ee26)
+- Use [Zulip Cloud](https://zulip.com/plans/) hosting. Read our [guide to choosing between Zulip Cloud and
+  self-hosting](https://zulip.com/help/zulip-cloud-or-self-hosting).
+
+To **import data** from [Slack][slack-import], [Mattermost][mattermost-import], [Rocket.Chat][rocketchat-import], [Zulip Cloud][zulip-cloud-import], or [another Zulip
+server][zulip-server-import], follow the linked instructions.
+
+You can **try out Zulip** before setting up your own server by [checking
+it out](https://chat.zulip.org/?show_try_zulip_modal) in the Zulip development community,
+or [creating a free test organization](https://zulip.com/new/) on Zulip Cloud.
+
+:::{note}
+These instructions are for self-hosting Zulip. To
+[contribute](../contributing/contributing.md) to the project, set up the
+[development environment](../development/overview.md).
+:::
+
+## Installation process overview
+
+0. [Set up a base server](#step-0-set-up-a-base-server)
+1. [Download the latest release](#step-1-download-the-latest-release)
+1. [Install Zulip](#step-2-install-zulip)
+1. [Create a Zulip organization, and log in](#step-3-create-a-zulip-organization-and-log-in)
+
+That's it! Once installation is complete, you can
+[configure](settings.md) Zulip to suit your needs.
+
+## Step 0: Set up a base server
+
+Provision and log in to a fresh Ubuntu or Debian system in your preferred
+hosting environment that satisfies the [installation
+requirements](requirements.md) for your expected usage level.
+
+## Step 1: Download the latest release
+
+Download and unpack [the latest server
+release](https://download.zulip.com/server/zulip-server-latest.tar.gz)
+(**Zulip Server {{ LATEST_RELEASE_VERSION }}**) with the following commands:
+
+```bash
+cd $(mktemp -d)
+curl -fLO https://download.zulip.com/server/zulip-server-latest.tar.gz
+tar -xf zulip-server-latest.tar.gz
+```
+
+To verify the download, see [the sha256sums of our release
+tarballs](https://download.zulip.com/server/SHA256SUMS.txt).
+
+## Step 2: Install Zulip
+
+To set up Zulip with the most common configuration, first become the
+`root` user, if you are not already:
+
+```bash
+[ "$(whoami)" != "root" ] && sudo -s
+```
+
+Then, run the installer, providing your email address and the public
+hostname that users will be able to access your server with:
+
+```bash
+./zulip-server-*/scripts/setup/install --push-notifications --certbot \
+    --email=YOUR_EMAIL --hostname=YOUR_HOSTNAME
+```
+
+This command will immediately prompt you to agree to Zulip's [Terms of
+Service][terms], so that your server can be registered for the [Mobile Push
+Notification Service](mobile-push-notifications.md). To skip registering for
+access to push notifications at this time, remove the `--push-notifications`
+flag.
+
+:::{note}
+When registering for push notifications, you can configure whether your server
+will submit aggregate usage statistics. See `--no-submit-usage-statistics`
+[installer option](#installer-options) for details.
+:::
+
+The installer takes a few minutes to run, as it installs Zulip's dependencies. It is
+designed to be idempotent: if the script fails, once you've corrected the cause
+of the failure, you can just rerun the script. For more information, see
+[installer details](deployment.md#zulip-installer-details) and
+[troubleshooting](troubleshooting.md#troubleshooting-the-zulip-installer).
+
+#### Installer options
+
+- `--email=it-team@example.com`: A **real email address for the person
+  or team who maintains the Zulip installation**. Zulip users on your
+  server will see this as the contact email in automated emails, on
+  help pages, on error pages, etc. If you use the [Mobile Push
+  Notification Service](mobile-push-notifications.md), this is used as
+  a point of contact. You can later configure a display name for your
+  contact email with the `ZULIP_ADMINISTRATOR`
+  [setting][doc-settings].
+
+- `--hostname=zulip.example.com`: The user-accessible domain name for this Zulip
+  server, i.e., what users will type in their web browser. This becomes
+  `EXTERNAL_HOST` in the Zulip [settings][doc-settings]. Use Punycode for
+  non-ASCII domains.
+
+- `--certbot`: With this option, the Zulip installer automatically obtains an
+  SSL certificate for the server [using Certbot][doc-certbot], and configures a
+  cron job to renew the certificate automatically. If you prefer to acquire an
+  SSL certificate another way, it's easy to [provide it to
+  Zulip][doc-ssl-manual].
+
+- `--push-notifications`/`--no-push-notifications`: With this option, the Zulip
+  installer registers your server for the [Mobile Push Notification
+  Service](mobile-push-notifications.md), and sets up the initial default
+  configuration. You will be immediately prompted to agree to the [Terms of
+  Service][terms], and your server will be registered at the end of the
+  installation process. You can learn more [about the
+  service](mobile-push-notifications.md) and why it's [necessary for push
+  notifications](mobile-push-notifications.md#why-a-push-notification-service-is-necessary).
+
+- `--no-submit-usage-statistics`: If you enable push notifications, by default
+  your server will submit [basic
+  metadata](mobile-push-notifications.md#uploading-basic-metadata) (required for
+  billing and for determining free plan eligibility), as well as [aggregate
+  usage statistics](mobile-push-notifications.md#uploading-usage-statistics).
+  You can disable submitting usage statistics by passing this flag. If push
+  notifications are not enabled, no data will be submitted, so this flag is
+  redundant.
+
+- `--agree-to-terms-of-service`: If you're using the `--push-notifications` flag,
+  you can pass this additional flag to indicate that you have read and agree to
+  the [Terms of Service][terms].
+  This skips the Terms of Service prompt, allowing for running the installer
+  with `--push-notifications` in scripts without requiring user input.
+
+- `--self-signed-cert`: With this option, the Zulip installer
+  generates a self-signed SSL certificate for the server. This isn't
+  suitable for production use (unless your server is [behind a reverse
+  proxy][reverse-proxy]), but may be convenient for testing.
+
+For advanced installer options, see our [deployment options][doc-deployment-options]
+documentation.
+
+:::{important}
+
+If you are importing data, stop here and return to the import instructions for
+[Slack][slack-import], [Mattermost][mattermost-import],
+[Rocket.Chat][rocketchat-import], [Zulip Cloud][zulip-cloud-import], [a server backup][zulip-backups], or [another Zulip server][zulip-server-import].
+
+:::
+
+[doc-settings]: settings.md
+[doc-certbot]: ssl-certificates.md#certbot-recommended
+[doc-ssl-manual]: ssl-certificates.md#manual-install
+[doc-deployment-options]: deployment.md#advanced-installer-options
+[zulip-backups]: export-and-import.md#backups
+[reverse-proxy]: reverse-proxies.md
+[slack-import]: https://zulip.com/help/import-from-slack
+[mattermost-import]: https://zulip.com/help/import-from-mattermost
+[rocketchat-import]: https://zulip.com/help/import-from-rocketchat
+[zulip-cloud-import]: export-and-import.md#import-into-a-new-zulip-server
+[zulip-server-import]: export-and-import.md#import-into-a-new-zulip-server
+
+## Step 3: Create a Zulip organization, and log in
+
+When the installation process is complete, the install script prints a secure
+one-time-use organization creation link. Open this link in your browser, and
+follow the prompts to set up your organization and your own user account. Your
+Zulip organization is ready to use!
+
+:::{note}
+You can generate a new organization creation link by running `manage.py
+generate_realm_creation_link` on the server. See also our guide on running
+[multiple organizations on the same server](multiple-organizations.md).
+:::
+
+## Getting started with Zulip
+
+To really see Zulip in action, you'll need to get the people you work
+together with using it with you.
+
+- [Set up outgoing email](email.md) so Zulip can confirm new users'
+  email addresses and send notifications.
+- Learn how to [get your organization started][realm-admin-docs] using
+  Zulip at its best.
+
+Learning more:
+
+- Subscribe to the [Zulip announcements email
+  list](https://groups.google.com/g/zulip-announce) for
+  server administrators. This extremely low-traffic list is for
+  important announcements, including [new
+  releases](../overview/release-lifecycle.md) and security issues.
+- Follow us on [Mastodon](https://fosstodon.org/@zulip) or
+  [Bluesky](https://bsky.app/profile/zulip.bsky.social).
+- Learn how to [configure your Zulip server settings](settings.md).
+- Learn about [Backups, export and import](export-and-import.md)
+  and [upgrading](upgrade.md) a production Zulip
+  server.
+
+[realm-admin-docs]: https://zulip.com/help/moving-to-zulip
+[terms]: https://zulip.com/policies/terms

docs/production/maintain-secure-upgrade.md

@@ -0,0 +1,42 @@
+---
+orphan: true
+---
+
+# Maintain, secure, and upgrade
+
+This was once a long page covering a bunch of topics; those topics
+have since all moved to dedicated pages:
+
+### Monitoring
+
+Moved to [Troubleshooting](troubleshooting.md#monitoring).
+
+### Securing your Zulip server
+
+Moved to [Securing your Zulip server](securing-your-zulip-server.md).
+
+### Upgrading
+
+Moved to [Upgrading to a release](upgrade.md#upgrading-to-a-release).
+
+### Upgrading from a Git repository
+
+Moved to [Upgrading from a Git
+repository](upgrade.md#upgrading-from-a-git-repository).
+
+### Upgrading the operating system
+
+Moved to [Upgrading the operating
+system](upgrade.md#upgrading-the-operating-system).
+
+### Scalability
+
+Moved to [Scalability](requirements.md#scalability).
+
+### Management commands
+
+Moved to [Management commands](management-commands.md).
+
+### API and your Zulip URL
+
+Moved to [REST API](https://zulip.com/api/rest).

docs/production/management-commands.md

@@ -0,0 +1,200 @@
+# Management commands
+
+Sometimes, you need to modify or inspect Zulip data from the command
+line. To help with this, Zulip ships with over 100 command-line tools
+implemented using the [Django management commands
+framework][django-management].
+
+Because management commands require server shell access, Zulip Cloud
+users will need to contact support for situations requiring them.
+
+## Running management commands
+
+:::{important}
+
+If you are using [Docker](docker.md), management commands are run
+inside the container; see {doc}`docker:how-to/compose-commands` for
+Docker Compose deployments and {doc}`docker:how-to/helm-commands`
+for Helm deployments.
+
+:::
+
+Start by logging in as the `zulip` user on the Zulip server. Then run
+them as follows:
+
+```bash
+cd /home/zulip/deployments/current
+
+# Start by reading the help
+./manage.py <command_name> --help
+
+# Once you've determined this is the command for you, run it!
+./manage.py <command_name> <args>
+```
+
+A full list of commands is available via `./manage.py help`; you'll
+primarily want to use those in the `[zerver]` section as those are the
+ones specifically built for Zulip.
+
+As a warning, some of them are designed for specific use cases and may
+cause problems if run in other situations. If you're not sure, it's
+worth reading the documentation (or the code, usually available at
+`zerver/management/commands/`; they're generally very simple programs).
+
+### Accessing an organization's `string_id`
+
+Since Zulip supports hosting multiple organizations on a single
+server, many management commands require you specify which
+organization ("realm") you'd like to modify, either via numerical or
+string ID (usually the subdomain).
+
+You can see all the organizations on your Zulip server using
+`./manage.py list_realms`.
+
+```console
+zulip@zulip:~$ /home/zulip/deployments/current/manage.py list_realms
+id    string_id                                name
+--    ---------                                ----
+1     zulipinternal                            None
+2                                              Zulip Community
+```
+
+(Note that every Zulip server has a special `zulipinternal` realm
+containing system-internal bots like `Notification Bot`; you are
+unlikely to ever need to interact with that realm.)
+
+Unless you are
+[hosting multiple organizations on your Zulip server](multiple-organizations.md),
+your single Zulip organization on the root domain will have the empty
+string (`''`) as its `string_id`. So you can run, for example:
+
+```console
+zulip@zulip:~$ /home/zulip/deployments/current/manage.py show_admins -r ''
+```
+
+Otherwise, the `string_id` will correspond to the organization's
+subdomain. E.g., on `it.zulip.example.com`, use
+`/home/zulip/deployments/current/manage.py show_admins -r it`.
+
+## manage.py shell
+
+If you need to query or edit data directly in the Zulip database, the
+best way to do this is with Django's built-in management shell.
+
+You can get an IPython shell with full access to code within the Zulip
+project using `manage.py shell`, e.g., you can do the following to
+change a user's email address:
+
+```console
+$ cd /home/zulip/deployments/current/
+$ ./manage.py shell
+In [1]: user_profile = get_user_profile_by_email("email@example.com")
+In [2]: do_change_user_delivery_email(user_profile, "new_email@example.com")
+```
+
+Any Django tutorial can give you helpful advice on querying and
+formatting data from Zulip's tables for inspection; Zulip's own
+[new feature tutorial](../tutorials/new-feature-tutorial.md) should help
+you understand how the codebase is organized.
+
+We recommend against directly editing objects and saving them using
+Django's `object.save()`. While this will save your changes to the
+database, for most objects, in addition to saving the changes to the
+database, one may also need to flush caches, notify the apps and open
+browser windows, and record the change in Zulip's `RealmAuditLog`
+audit history table. For almost any data change you want to do, there
+is already a function in `zerver.actions` with a name like
+`do_change_full_name` that updates that field and notifies clients
+correctly.
+
+For convenience, Zulip automatically imports `zerver.models`
+into every management shell; if you need to
+access other functions, you'll need to import them yourself.
+
+## Other useful manage.py commands
+
+There are dozens of useful management commands under
+`zerver/management/commands/`. We detail a few here:
+
+- `./manage.py help`: Lists all available management commands.
+- `./manage.py dbshell`: If you're more comfortable with raw SQL than
+  Python, this will open a PostgreSQL SQL shell connected to the Zulip
+  server's database. Beware of changing data; editing data directly
+  with SQL will often not behave correctly because PostgreSQL doesn't
+  know to flush Zulip's caches or notify browsers of changes.
+- `./manage.py send_custom_email`: Can be used to send an email to a set
+  of users. The `--help` documents how to run it from a
+  `manage.py shell` for use with more complex programmatically
+  computed sets of users.
+- `./manage.py send_password_reset_email`: Sends password reset email(s)
+  to one or more users.
+- `./manage.py change_realm_subdomain`: Change subdomain of a realm.
+- `./manage.py change_user_email`: Change a user's email address.
+- `./manage.py change_user_role`: Can change a user's role
+  (easier done [via the
+  UI](https://zulip.com/help/user-roles)) or give bots the
+  `can_forge_sender` permission, which is needed for certain special API features.
+- `./manage.py export_single_user`: does a limited version of the [main
+  export tools](export-and-import.md) containing just
+  the messages accessible by a single user.
+- `./manage.py unarchive_channel`:
+  [Reactivates](https://zulip.com/help/archive-a-channel#unarchiving-archived-channels)
+  an archived channel.
+- `./manage.py reactivate_realm`: Reactivates a realm.
+- `./manage.py deactivate_user`: Deactivates a user. This can be done
+  more easily in Zulip's organization administrator UI.
+- `./manage.py delete_user`: Completely delete a user from the database.
+  For most purposes, deactivating users is preferred, since that does not
+  alter message history for other users.
+  See the `./manage.py delete_user --help` documentation for details.
+- `./manage.py reset_authentication_attempt_count`: If a user failed authentication
+  attempts too many times and further attempts are disallowed by the rate limiter,
+  this can be used to reset the limit.
+
+All of our management commands have internal documentation available
+via `manage.py command_name --help`.
+
+## Custom management commands
+
+Zulip supports several mechanisms for running custom code on a
+self-hosted Zulip server:
+
+- Using an existing [integration][integrations] or writing your own
+  [webhook integration][webhook-integrations] or [bot][writing-bots].
+- Writing a program using the [Zulip API][zulip-api].
+- [Modifying the Zulip server][modifying-zulip].
+- Using the interactive [management shell](#managepy-shell),
+  documented above, for one-time work or prototyping.
+- Writing a custom management command, detailed here.
+
+Custom management commands are Python 3 programs that run inside
+Zulip's context, so that they can access its libraries, database, and
+code freely. They can be the best choice when you want to run custom
+code that is not permitted by Zulip's security model (and thus can't
+be done more easily using the [REST API][zulip-api]) and that you
+might want to run often (and so the interactive `manage.py shell` is
+not suitable, though we recommend using the management shell to
+prototype queries).
+
+Our developer documentation on [writing management
+commands][management-commands-dev] explains how to write them.
+
+Simply writing the command inside a `deployments/` directory is not
+ideal, because a new such directory is created every time you upgrade
+the Zulip server.
+
+Instead, we recommend deploying custom management commands either via
+the [modifying Zulip][modifying-zulip] process or by storing them in
+`/etc/zulip` (so they are included in
+[backups](export-and-import.md#backups)) and then
+symlinking them into
+`/home/zulip/deployments/current/zerver/management/` after each
+upgrade.
+
+[modifying-zulip]: modify.md
+[writing-bots]: https://zulip.com/api/writing-bots
+[integrations]: https://zulip.com/integrations
+[zulip-api]: https://zulip.com/api/rest
+[webhook-integrations]: ../webhooks/incoming-webhooks-overview.md
+[management-commands-dev]: ../subsystems/management-commands.md
+[django-management]: https://docs.djangoproject.com/en/5.0/ref/django-admin/#django-admin-and-manage-py

docs/production/mobile-push-notifications.md

@@ -0,0 +1,547 @@
+# Mobile push notification service
+
+Zulip's iOS and Android [mobile apps](https://zulip.com/apps/) support
+receiving push notifications from Zulip servers to notify users when
+new messages have arrived. This is an important feature for having a
+great mobile app experience.
+
+Google's and Apple's security model for mobile push notifications does not allow
+self-hosted Zulip servers to directly send mobile notifications to the Zulip
+mobile apps. The Zulip Mobile Push Notification Service solves this problem by
+forwarding E2EE mobile push notifications generated by your server to the Zulip
+mobile apps. With Zulip Server 12.0+ and an up-to-date Zulip mobile app, the
+notifications are encrypted end-to-end from your server to your mobile device.
+
+## Signing up
+
+:::{important}
+
+The Zulip Server 10.0+ [installer](install.md#step-2-install-zulip)
+includes a `--push-notifications` flag that automates this
+registration process.
+
+These instructions apply to Zulip 9.0+. If you are running an older
+version of Zulip ([check](https://zulip.com/help/view-zulip-version)
+if you are unsure), see the [Zulip 8.x
+documentation](https://zulip.readthedocs.io/en/8.4/production/mobile-push-notifications.html).
+:::
+
+You can enable the mobile push notification service for your Zulip
+server as follows:
+
+1. Make sure your server has outgoing HTTPS access to the public Internet. If
+   that is restricted by a proxy, you will need to [configure Zulip to use your
+   outgoing HTTP proxy](deployment.md#customizing-the-outgoing-http-proxy)
+   first.
+
+1. Make sure that the `ZULIP_ADMINISTRATOR` setting in your
+   `/etc/zulip/settings.py` file is a real email address which you
+   monitor. If the Mobile Push Notification Service needs to contact
+   you regarding your server, and will use this email address. [See
+   below](#updating-your-servers-registration) for instructions if
+   this contact needs to be updated later.
+
+1. Set `ZULIP_SERVICE_PUSH_NOTIFICATIONS = True` in your
+   `/etc/zulip/settings.py` file. Simply uncomment the appropriate line [in
+   settings.py][update-settings-docs] by deleting the initial `# `.
+
+1. Decide whether to share usage statistics with the Zulip team.
+
+   By default, Zulip installations using the Mobile Push Notification Service
+   submit additional usage statistics that help Zulip's maintainers allocate
+   resources towards supporting self-hosted installations
+   ([details](#uploading-usage-statistics)). You can disable submitting usage
+   statistics now or at any time by setting
+   `ZULIP_SERVICE_SUBMIT_USAGE_STATISTICS=False` in `/etc/zulip/settings.py`
+   (uncomment the appropriate line).
+
+   Note that all systems using the service upload [basic
+   metadata](#uploading-basic-metadata) about the organizations hosted
+   by the installation.
+
+   [update-settings-docs]: ../production/upgrade.md#updating-settingspy-inline-documentation
+
+1. [Restart your Zulip server](settings.md#changing-server-settings) so that
+   your configuration changes take effect.
+
+1. Run the registration command. If you installed Zulip directly on the server
+   (without Docker), run as root:
+
+   ```
+   su zulip -c '/home/zulip/deployments/current/manage.py register_server'
+   ```
+
+   Or if you're {doc}`using Docker <docker:index>`, run:
+
+   ```
+   cd docker-zulip
+   ./manage.py register_server
+   ```
+
+   This command will print the registration data it would send to the Mobile
+   Push Notification Service, ask you to accept the terms of service, and if
+   you accept, register your server. If you have trouble, [contact Zulip
+   support](https://zulip.com/help/contact-support) with the output of this
+   command.
+
+1. Organizations with more than 10 users must upgrade their
+   [plan](https://zulip.com/plans/) in order to access the Mobile Push
+   Notification Service. See [plan management](#plan-management) for details.
+
+1. If you or your users have already set up the Zulip mobile app, you'll each
+   need to log out of the mobile app, and log back in again in order to start
+   getting push notifications.
+
+Congratulations! You've successfully set up the service. You can now test mobile
+push notifications by following [these
+instructions](https://zulip.com/help/mobile-notifications#testing-mobile-notifications).
+
+## Plan management
+
+To access the Mobile Push Notification Service, organizations with more than 10
+users must upgrade to a paid [plan](https://zulip.com/plans/#self-hosted), or
+the free Community plan (if
+[eligible](https://zulip.com/help/self-hosted-billing#free-community-plan)).
+While upgrading your Zulip server to version 8.0+ makes it more convenient to
+manage your plan, the same plans are offered for all Zulip versions.
+
+### Plan management for a Zulip organization
+
+On a self-hosted Zulip server running Zulip 8.0+, [organization
+owners](https://zulip.com/help/user-roles) and billing administrators
+can conveniently access plan management from the Zulip app. See [help center
+documentation](https://zulip.com/help/self-hosted-billing) for detailed
+instructions.
+
+#### Configure who can manage plans and billing
+
+::::{tab-set}
+
+:::{tab-item} Zulip Server 10.0+
+
+Follow [these
+instructions](https://zulip.com/help/self-hosted-billing#configure-who-can-manage-plans-and-billing)
+to configure who can manage plans and billing.
+
+:::
+
+:::{tab-item} Older versions
+
+You can add billing administrators using the `change_user_role` [management
+command][management-commands], passing [the organization's
+`string_id`][accessing-string-id], and the email address of the Zulip user who
+should be added as a billing administrator.
+
+```
+/home/zulip/deployments/current/manage.py change_user_role -r '' username@example.com is_billing_admin
+```
+
+You can remove a user's billing administrator permissions with the `--revoke`
+option:
+
+```
+/home/zulip/deployments/current/manage.py change_user_role --revoke -r '' username@example.com is_billing_admin
+```
+
+:::
+
+::::
+
+[management-commands]: ../production/management-commands.md
+[accessing-string-id]: https://zulip.readthedocs.io/en/stable/production/management-commands.html#accessing-an-organization-s-string-id
+
+### Plan management for an entire Zulip server
+
+Servers running Zulip releases older than Zulip 8.0 can start the plan
+management log in process at
+<https://selfhosting.zulip.com/serverlogin/>. This option is also
+available for Zulip 8.0+ servers, and makes it possible to use a
+single plan for multiple organizations on one installation. See [help
+center documentation](https://zulip.com/help/self-hosted-billing) for
+detailed log in instructions.
+
+You will use your server's `zulip_org_id` and `zulip_org_key` as the username
+and password to access plan management. You can obtain these from
+`/etc/zulip/zulip-secrets.conf` on your Zulip server, or via the following
+commands:
+
+```
+/home/zulip/deployments/current/scripts/get-django-setting ZULIP_ORG_ID
+/home/zulip/deployments/current/scripts/get-django-setting ZULIP_ORG_KEY
+```
+
+## Why a push notification service is necessary
+
+Both Google's and Apple's push notification services have a security
+model that does not support mutually untrusted self-hosted servers
+sending push notifications to the same app. In particular, when an
+app is published to their respective app stores, one must compile into
+the app a secret corresponding to the server that will be able to
+publish push notifications for the app. This means that it is
+impossible for a single app in their stores to receive push
+notifications from multiple, mutually untrusted, servers.
+
+Zulip's solution to this problem is to provide a central push
+notification forwarding service, which allows registered Zulip servers
+to send push notifications to the Zulip app indirectly (through the
+forwarding service).
+
+## Security and privacy
+
+:::{note}
+Use of the push notification bouncer is subject to the Zulip Cloud [Terms of
+Service](https://zulip.com/policies/terms), [Privacy
+Policy](https://zulip.com/policies/privacy) and [Rules of
+Use](https://zulip.com/policies/rules). By using push notifications, you agree
+to these terms.
+:::
+
+Zulip Server 12.0+ uses end-to-end encryption (E2EE) when sending push
+notifications to the Zulip mobile app. E2EE ensures that message
+**content** and **metadata** (including the sender's name, and the
+recipient's name or channel and topic where the message was sent) are
+not visible to Apple, Google, or Zulip's [Mobile Push Notification
+Service](https://zulip.readthedocs.io/en/stable/production/mobile-push-notifications.html).
+
+The Zulip server encrypts the contents of each notification
+with a per-device symmetric key shared between the Zulip server and the
+user's mobile app. The Push Notification Service forwards the encrypted
+payload without being able to decrypt it.
+
+If either the Zulip Server or the mobile app is a version predating
+the 12.0 release (April 2026) that does not support the E2EE protocol,
+the [legacy push notifications protocol](#legacy-protocol) will be
+used. Organization administrators can
+[require](https://zulip.com/help/mobile-notifications#require-end-to-end-encryption-for-mobile-push-notifications)
+E2EE for message content in mobile push notifications.
+
+If you have any questions about the security model, [contact Zulip
+support](https://zulip.com/help/contact-support).
+
+### What Apple, Google, and the Push Notification Service see
+
+For each notification, the Push Notification Service receives and forwards to
+Apple or Google:
+
+- The opaque APNs or FCM token identifying the target device (known to the
+  service from registration, and never seen in plaintext by the Zulip server).
+- An identifier selected by the mobile app (`push_key_id`) that the mobile app
+  uses to select the correct decryption key. This identifier is opaque to
+  the service.
+- The encrypted notification payload (an XSalsa20+Poly1305
+  ciphertext), forwarded to Apple or Google inside the push payload.
+- The delivery priority (higher for a new message, lower for a notification
+  removal).
+
+The notification contents — including the message, sender, recipient,
+channel, and topic — are inside an encrypted ciphertext that Apple,
+Google, and the service cannot decrypt.
+
+All network requests — from Zulip servers to the Push Notification Service,
+and from the Push Notification Service to Apple and Google — are encrypted
+over the wire with SSL/TLS.
+
+### What the Push Notification Service stores
+
+The Push Notification Service stores the following metadata for as long as
+a device's registration is active:
+
+- The APNs and FCM tokens needed to deliver push notifications to iOS and
+  Android devices, one per device registered with your Zulip server.
+- [Basic organization metadata](#uploading-basic-metadata),
+  [optional usage statistics](#uploading-usage-statistics), and aggregate
+  statistics about how many push notifications are sent by each server.
+
+### What the Zulip server sees
+
+The Zulip server is the source of every notification it sends, and
+holds the symmetric key used to encrypt each notification for
+delivery.
+
+E2EE makes notification contents opaque to Apple, Google, and the Push
+Notification Service, but it does **not** protect against a compromised or
+malicious Zulip server. Learn about [Zulip's approach to
+security](https://zulip.com/security/), and [keep your server
+safe](./securing-your-zulip-server.md).
+
+### Limitations
+
+The E2EE protocol has the following known limitations:
+
+- **Incomplete forward secrecy.** A device's symmetric key persists until the
+  mobile app rotates it (generally every 30 days), and is stored by both the
+  mobile app and the Zulip server. An attacker who obtains both a historical
+  capture of encrypted push payloads and the device's current key — for example,
+  by compromising the mobile device or the Zulip server — can decrypt the
+  notifications captured since the key was last rotated.
+
+- **Metadata side channels.** The Push Notification Service, and
+  indirectly Apple and Google, can observe the timing, delivery priority,
+  and size of each push notification. These side channels can reveal
+  coarse patterns of activity even though message contents are encrypted.
+
+### Cryptography
+
+The E2EE protocol encrypts each notification payload with libsodium's
+`crypto_secretbox_easy` (XSalsa20-Poly1305 authenticated encryption) using
+a per-device 256-bit symmetric key and a fresh 192-bit random nonce per
+notification. Initial registration of a device with the Push Notification
+Service uses libsodium's `crypto_box_seal` (Curve25519 + XSalsa20-Poly1305)
+so that the Zulip server can forward the device's APNs/FCM token to the
+bouncer without learning the token itself. A one-byte algorithm prefix on
+each symmetric key reserves space for future cipher migrations.
+
+The code for the push notification forwarding service is 100% open source,
+and is available as part of the [Zulip server project on
+GitHub](https://github.com/zulip/zulip) (specifically,
+[here](https://github.com/zulip/zulip/tree/main/zilencer)).
+
+### Uploading basic metadata
+
+All Zulip installations running Zulip 8.0 or greater that are
+registered for the Mobile Push Notification Service regularly upload
+to the service basic metadata about the organizations hosted by the
+installation. (Older Zulip servers upload these metadata only if
+[uploading usage statistics](#uploading-usage-statistics) is enabled).
+
+Uploaded metadata consists of, for each organization hosted by the
+installation:
+
+- A subset of the basic metadata returned by the unauthenticated [`GET
+/server_settings` API
+  endpoint](https://zulip.com/api/get-server-settings).
+
+  The purpose of that API endpoint is to serve the minimal data
+  needed by the Zulip mobile apps in order to:
+
+  - Verify that a given URL is indeed a valid Zulip server URL
+  - Present a correct login form, offering only the supported features
+    and authentication methods for that organization and Zulip server
+    version.
+
+  Most of the metadata it returns is necessarily displayed to anyone
+  with network access to the Zulip server on the login and signup
+  pages for your Zulip organization as well.
+
+  (Some fields returned by this endpoint, like the organization icon
+  and description, are not included in uploaded metadata.)
+
+- The [organization type](https://zulip.com/help/organization-type)
+  and creation date.
+- The number of user accounts with each role.
+
+Our use of uploaded metadata is governed by the same [Terms of
+Service](https://zulip.com/policies/terms) and [Privacy
+Policy](https://zulip.com/policies/privacy) that covers the Mobile
+Push Notification Service itself.
+
+### Uploading usage statistics
+
+By default, Zulip installations that register for the Mobile Push
+Notification Service upload the following usage statistics. You can
+disable these uploads any time by setting
+`ZULIP_SERVICE_SUBMIT_USAGE_STATISTICS=False` in `/etc/zulip/settings.py`.
+
+- Totals for messages sent and read with subtotals for various
+  combinations of clients and integrations.
+- Totals for active users under a few definitions (1day, 7day, 15day)
+  and related statistics.
+
+Some of the graphs on your server's [usage statistics
+page](https://zulip.com/help/analytics) can be generated from these
+statistics.
+
+When enabled, usage statistics are submitted via an hourly cron
+job. If you'd like to access plan management immediately after
+enabling `SUBMIT_USAGE_STATISTICS=True` (the legacy form of this setting)
+on a pre-8.0 Zulip server, you can run the analytics job manually via:
+
+```
+/home/zulip/deployments/current/manage.py update_analytics_counts
+```
+
+Our use of uploaded usage statistics is governed by the same [Terms of
+Service](https://zulip.com/policies/terms) and [Privacy
+Policy](https://zulip.com/policies/privacy) that covers the Mobile
+Push Notification Service itself.
+
+## Rate limits
+
+The Mobile Push Notification Service API has a very high default rate
+limit of 1000 requests per minute. A Zulip server makes requests to
+this API every time it sends a push notification, which is fairly
+frequent, but we believe it to be unlikely that a self-hosted
+installation will hit this limit.
+
+This limit is primarily intended to protect the service against DoS
+attacks (intentional or otherwise). If you hit this limit or you
+anticipate that your server will require sending more push
+notifications than the limit permits, please [contact
+support](https://zulip.com/help/contact-support).
+
+## Updating your server's registration
+
+Your server's registration includes the server's hostname and contact
+email address (from `EXTERNAL_HOST` and `ZULIP_ADMINISTRATOR` in
+`/etc/zulip/settings.py`, aka the `--hostname` and `--email` options
+in the installer). You can update your server's registration data by
+running `manage.py register_server` again.
+
+If you'd like to rotate your server's API key for this service
+(`zulip_org_key`), you need to use
+`manage.py register_server --rotate-key` option; it will automatically
+generate a new `zulip_org_key` and store that new key in
+`/etc/zulip/zulip-secrets.conf`.
+
+## Moving your registration to a new server
+
+When migrating your Zulip deployment to a new machine, you will likely want to
+retain your original registration and successfully transfer it. This is
+especially important if you have an active plan for the Mobile Push
+Notification Service.
+
+The best way to preserve your registration when moving to a new server is to
+copy over the credentials from the old server. These credentials are stored in
+the `/etc/zulip/zulip-secrets.conf` file, specifically in the `zulip_org_id`
+and `zulip_org_key` fields. After installing Zulip on the new machine, ensure
+that `zulip_org_id` and `zulip_org_key` are set to the same values as on the
+old server.
+
+If you used the [official backup tool](export-and-import.md#backups)
+to restore your Zulip deployment on the new machine, it will have
+automatically transferred all secrets, including the registration
+credentials, correctly.
+
+### Transferring your registration if you lost the original credentials
+
+If you have lost your original credentials, you can still transfer your Zulip
+registration to a new server by following these steps:
+
+1. Ensure Zulip is installed and accessible:
+
+   - Install Zulip on the new machine and ensure it is fully operational.
+   - The server must be accessible on the hostname associated with the original
+     registration, with properly configured SSL certificates.
+   - This process **will not work** if your Zulip server is on a local network
+     or otherwise unreachable from the internet by our Mobile Push Notification
+     Service. If that’s the case, contact
+     [support@zulip.com](mailto:support@zulip.com) for assistance.
+
+1. Run the below command to transfer your registration to the new server. This will
+   execute a verification flow to prove to our Mobile Push Notification Service that
+   you control the hostname and upon success, re-generate the credentials for
+   using the registration and write them to the `/etc/zulip/zulip-secrets.conf` file.
+
+   ```bash
+   /home/zulip/deployments/current/manage.py register_server --registration-transfer
+   ```
+
+   Note that the `zulip_org_key` value changes in the process, and therefore if you
+   still have an old server running using the service, it will lose access upon
+   execution of this command.
+
+1. Apply the changes by restarting the server:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/restart-server
+   ```
+
+   Finally, [verify][verify-push-notifications] that push
+   notifications are working correctly. If you encounter further
+   issues, contact [support@zulip.com](mailto:support@zulip.com).
+
+1. If you store `/etc/zulip/zulip-secrets.conf` secrets externally in
+   an external configuration management tool (Ansible, etc.), or
+   [backups](export-and-import.md#backups), this is a good time to
+   update that configuration.
+
+[verify-push-notifications]: https://zulip.com/help/mobile-notifications#testing-mobile-notifications
+
+## Deactivating your server's registration
+
+If you are deleting your Zulip server or otherwise no longer want to
+use the Mobile Push Notification Service, you can deactivate your server's
+registration.
+
+1. [Cancel any paid
+   plans](https://zulip.com/help/self-hosted-billing#cancel-paid-plan)
+   associated with your server.
+
+1. Run the deregistration command. If you installed Zulip directly on
+   the server (without Docker), run as root:
+
+   ```
+   su zulip -c '/home/zulip/deployments/current/manage.py register_server --deactivate'
+   ```
+
+   Or if you're {doc}`using Docker <docker:index>`, run:
+
+   ```
+   cd docker-zulip
+   ./manage.py register_server --deactivate
+   ```
+
+1. Comment out the
+   `ZULIP_SERVICE_PUSH_NOTIFICATIONS = True` line
+   in your `/etc/zulip/settings.py` file (i.e., add `# ` at the
+   start of the line), and [restart your Zulip
+   server](settings.md#changing-server-settings).
+
+If you ever need to reactivate your server's registration, [contact Zulip
+support](https://zulip.com/help/contact-support).
+
+### Pausing use of the Mobile Push Notification Service
+
+You can temporarily stop using the Mobile Push Notification Service. Comment out
+the `PUSH_NOTIFICATION_BOUNCER_URL = 'https://push.zulipchat.com'` line in your
+`/etc/zulip/settings.py` file (i.e., add `# ` at the start of the line), and
+[restart your Zulip server](settings.md#changing-server-settings). This approach makes it
+easy to start using the service again by uncommenting the same line.
+
+### Legacy protocol
+
+Zulip Servers running 12.0+ sending notifications to a modern mobile
+app version will always use the E2EE protocol. If either the Zulip
+Server or the mobile app is a version predating the 12.0 release
+(April 2026) that does not support the E2EE protocol, the legacy
+non-E2EE protocol will be used.
+
+The central design goal of the legacy Push Notification Service
+service was to avoid any message content being stored or logged by the
+service, even in error cases.
+
+- The Push Notification Service only stores the necessary metadata for
+  delivering the notifications to the appropriate devices and
+  otherwise operating the service:
+  - The APNS/FCM tokens needed to securely send mobile push
+    notifications to iOS and Android devices, one per device
+    registered to be notified by your Zulip server.
+  - User ID numbers generated by your Zulip server, needed to route
+    a given notification to the appropriate set of mobile devices.
+    These user ID numbers are opaque to the Push Notification
+    Service and Kandra Labs.
+  - [Basic organization metadata](#uploading-basic-metadata),
+    [optional usage statistics](#uploading-usage-statistics), and
+    aggregate statistics about how many push notifications are sent by
+    each customer.
+- The Push Notification Service receives (but does not store) the
+  contents of individual mobile push notifications:
+
+  - The numeric message ID generated by your Zulip server.
+  - Metadata on the message's sender (name and avatar URL).
+  - Metadata on the message's recipient (channel name + ID, topic,
+    direct message recipients, etc.).
+  - A timestamp.
+  - The message's content.
+
+  Organization administrators can [require end-to-end encryption for
+  push notifications][require-e2ee]. When this setting is enabled,
+  legacy push notifications are not sent; only clients that support
+  E2EE push notifications will receive push notifications.
+
+  [require-e2ee]: https://zulip.com/help/mobile-notifications#require-end-to-end-encryption-for-mobile-push-notifications
+
+- All of the network requests (both from Zulip servers to the Push
+  Notification Service and from the Push Notification Service to the
+  relevant Google and Apple services) are encrypted over the wire with
+  SSL/TLS.

docs/production/modify.md

@@ -0,0 +1,238 @@
+# Modify Zulip
+
+Zulip is 100% free and open source software, and you're welcome to
+modify it! This section explains how to make and maintain
+modifications in a safe and convenient fashion.
+
+If you do modify Zulip and then report an issue you see in your
+modified version of Zulip, please be responsible about communicating
+that fact:
+
+- Ideally, you'd reproduce the issue in an unmodified version (e.g., in
+  [the Zulip development community](https://zulip.com/development-community/) or on
+  [zulip.com](https://zulip.com)).
+- Where that is difficult or you think it's very unlikely your changes
+  are related to the issue, just mention your changes in the issue report.
+
+If you're looking to modify Zulip by applying changes developed by the
+Zulip core team and merged into `main`, skip to [this
+section](#applying-changes-from-main).
+
+## Making changes
+
+One way to modify Zulip is to just edit files under
+`/home/zulip/deployments/current` and then restart the server. This
+can work OK for testing small changes to Python code or shell scripts.
+But we don't recommend this approach for maintaining changes because:
+
+- You cannot modify JavaScript, CSS, or other frontend files this way,
+  because we don't include them in editable form in our production
+  release tarballs (doing so would make our release tarballs much
+  larger without any runtime benefit).
+- You will need to redo your changes after you next upgrade your Zulip
+  server (or they will be lost).
+- You need to remember to restart the server or your changes won't
+  have effect.
+- Your changes aren't tracked, so mistakes can be hard to debug.
+
+Instead, we recommend the following GitHub-based workflow (see [our
+Git guide][git-guide] if you need a primer):
+
+- Decide where you're going to edit Zulip's code. We recommend [using
+  the Zulip development environment](../development/overview.md) on
+  a desktop or laptop as it will make it extremely convenient for you
+  to test your changes without deploying them in production. But if
+  your changes are small or you're OK with risking downtime, you don't
+  strictly need it; you just need an environment with Git installed.
+- **Important**. Determine what Zulip version you're running on your
+  server. You can check by inspecting `ZULIP_VERSION` in
+  `/home/zulip/deployments/current/version.py` (we'll use `2.0.4`
+  below). If you apply your changes to the wrong version of Zulip,
+  it's likely to fail and potentially cause downtime.
+- [Fork and clone][fork-clone] the [zulip/zulip][] repository on
+  [GitHub](https://github.com).
+- Create a branch (named `acme-branch` below) containing your changes:
+
+```bash
+cd zulip
+git checkout -b acme-branch 2.0.4
+```
+
+- Use your favorite code editor to modify Zulip.
+- Commit your changes and push them to GitHub:
+
+```bash
+git commit -a
+
+# Use `git diff` to verify your changes are what you expect
+git diff 2.0.4 acme-branch
+
+# Push the changes to your GitHub fork
+git push origin +acme-branch
+```
+
+- Log in to your Zulip server and configure and use
+  [upgrade-zulip-from-git][] to install the changes; remember to
+  configure `git_repo_url` to point to your fork on GitHub and run it as
+  `upgrade-zulip-from-git acme-branch`.
+
+This workflow solves all of the problems described above: your change
+will be compiled and installed correctly (restarting the server), and
+your changes will be tracked so that it's convenient to maintain them
+across future Zulip releases.
+
+### Upgrading to future releases
+
+Eventually, you'll want to upgrade to a new Zulip release. If your
+changes were integrated into that Zulip release or are otherwise no
+longer needed, you can just [upgrade as
+usual](upgrade.md#upgrading-to-a-release). If you [upgraded to
+`main`](#upgrading-to-main); review that section again; new
+maintenance releases are likely "older" than your current installation
+and you might need to upgrade to `main` again rather than to the
+new maintenance release.
+
+Otherwise, you'll need to update your branch by rebasing your changes
+(starting from a [clone][fork-clone] of the [zulip/zulip][]
+repository). The example below assumes you have a branch off of 2.0.4
+and want to upgrade to 2.1.0.
+
+```bash
+cd zulip
+git fetch --tags upstream
+git checkout acme-branch
+git rebase --onto 2.1.0 2.0.4
+# Fix any errors or merge conflicts; see Zulip's Git guide for advice
+
+# Use `git diff` to verify your changes are what you expect
+git diff 2.1.0 acme-branch
+
+git push origin +acme-branch
+```
+
+And then use [upgrade-zulip-from-git][] to install your updated
+branch, as before.
+
+### Making changes with docker-zulip
+
+If you are using [our Docker image](docker.md), there are two
+things that are different from the above:
+
+- Because of how container images work, editing files directly is even
+  more precarious, because Docker "boots" from fixed images every
+  time, and restarting the container will your changes.
+- Instead of running `upgrade-zulip-from-git`, you will need to use
+  the {doc}`Docker upgrade workflow <docker:how-to/compose-upgrading>`
+  to build a container image based on your modified version of Zulip.
+
+## Applying changes from `main`
+
+If you are experiencing an issue that has already been fixed by the
+Zulip development community, and you'd like to get the fix now, you
+have a few options. There are two possible ways you might get those
+fixes on your local Zulip server without waiting for an official release.
+
+### Applying a small change
+
+Many bugs have small/simple fixes. In this case, you can use the Git
+workflow [described above](#making-changes), using:
+
+```bash
+git fetch upstream
+git cherry-pick abcd1234
+```
+
+instead of "making changes locally" (where `abcd1234` is the commit ID
+of the change you'd like).
+
+In general, we can't provide unpaid support for issues caused by
+cherry-picking arbitrary commits if the issues don't also affect
+`main` or an official release.
+
+The exception to this rule is when we ask or encourage a user to apply
+a change to their production system to help verify the fix resolves
+the issue for them. You can expect the Zulip community to be
+responsive in debugging any problems caused by a patch we asked
+you to apply.
+
+Also, consider asking whether a small fix that is important to you can
+be added to the current stable release branch (e.g., `2.1.x`). In
+addition to scheduling that change for Zulip's next bug fix release,
+we support changes in stable release branches as though they were
+released.
+
+### Upgrading to `main`
+
+Many Zulip servers (including chat.zulip.org and zulip.com) upgrade to
+`main` on a regular basis to get the latest features. Before doing
+so, it's important to understand how to happily run a server based on
+`main`.
+
+For background, backporting arbitrary patches from `main` to an older
+version requires some care. Common issues include:
+
+- Changes containing database migrations (new files under
+  `*/migrations/`), which includes most new features. We
+  don't support applying database migrations out of order.
+- Changes that are stacked on top of other changes to the same system.
+- Essentially any patch with hundreds of lines of changes will have
+  merge conflicts and require extra work to apply.
+
+While it's possible to backport these sorts of changes, you're
+unlikely to succeed without help from the core team via a support
+contract.
+
+If you need an unreleased feature, the best path is usually to
+upgrade to Zulip `main` using [upgrade-zulip-from-git][]. Before
+upgrading to `main`, make sure you understand:
+
+- In Zulip's version numbering scheme, `main` will always be "newer"
+  than the latest maintenance release (e.g., `3.1` or `2.1.6`) and
+  "older" than the next major release (e.g., `3.0` or `4.0`).
+- The `main` branch is under very active development; dozens of new
+  changes are integrated into it on most days. The `main` branch
+  can have thousands of changes not present in the latest release (all
+  of which will be included in our next major release). On average
+  `main` usually has fewer total bugs than the latest release
+  (because we fix hundreds of bugs in every major release) but it
+  might have some bugs that are more severe than we would consider
+  acceptable for a release.
+- We deploy `main` to chat.zulip.org and zulip.com on a regular
+  basis (often daily), so it's very important to the project that it
+  be stable. Most regressions will be minor UX issues or be fixed
+  quickly, because we need them to be fixed for Zulip Cloud.
+- The development community is very interested in helping debug issues
+  that arise when upgrading from the latest release to `main`, since
+  they provide us an opportunity to fix that category of issue before
+  our next major release. (Much more so than we are in helping folks
+  debug other custom changes). That said, we cannot make any
+  guarantees about how quickly we'll resolve an issue to folks without
+  a formal support contract.
+- We do not support downgrading from `main` to earlier versions, so
+  if downtime for your Zulip server is unacceptable, make sure you
+  have a current
+  [backup](export-and-import.md#backups) in case the
+  upgrade fails.
+- Our changelog contains [draft release
+  notes](../overview/changelog.md) available listing major changes
+  since the last release. The **Upgrade notes** section will always
+  be current, even if some new features aren't documented.
+- Whenever we push a security or maintenance release, the changes in
+  that release will always be merged to `main`; so you can get the
+  security fixes by upgrading to `main`.
+- You can always upgrade from `main` to the next major release when it
+  comes out, using either [upgrade-zulip-from-git][] or the release
+  tarball. So there's no risk of upgrading to `main` resulting in
+  a system that's not upgradeable back to a normal release.
+
+## Contributing patches
+
+Zulip contains thousands of changes submitted by volunteer
+contributors like you. If your changes are likely to be of useful to
+other organizations, consider [contributing
+them](../contributing/contributing.md).
+
+[fork-clone]: ../git/cloning.md#get-zulip-code
+[upgrade-zulip-from-git]: ./upgrade.md#upgrading-from-a-git-repository
+[git-guide]: ../git/index.md
+[zulip/zulip]: https://github.com/zulip/zulip/

docs/production/multiple-organizations.md

@@ -0,0 +1,147 @@
+# Hosting multiple organizations
+
+The vast majority of Zulip servers host just a single organization (or
+"realm", as the Zulip code calls organizations). This article
+documents what's involved in hosting multiple Zulip organizations on a
+single server.
+
+Throughout this article, we'll assume you're working on a Zulip server
+with hostname `zulip.example.com`. You may also find the more
+[technically focused article on realms](../subsystems/realms.md) to be useful
+reading.
+
+## Subdomains
+
+Zulip's approach for supporting multiple organizations on a single
+Zulip server is for each organization to be hosted on its own
+subdomain. E.g., you'd have `org1.zulip.example.com` and
+`org2.zulip.example.com`.
+
+Web security standards mean that one subdomain per organization is
+required to support a user logging into multiple organizations on a
+server at the same time.
+
+When you want to create a new organization, you need to do a few
+things:
+
+- Make sure you have SSL certificates for all of the subdomains you're
+  going to use. If you're using
+  [our Let's Encrypt instructions](ssl-certificates.md), it's easy to
+  just specify multiple subdomains in your certificate request.
+- If necessary, modify your `nginx` configuration to use your new
+  certificates.
+- Use `./manage.py generate_realm_creation_link` again to create your
+  new organization. Review
+  [the install instructions](install.md) if you need a
+  refresher on how this works.
+- If you're planning on using GitHub auth or another social
+  authentication method, review
+  [the notes on `SOCIAL_AUTH_SUBDOMAIN` below](#authentication).
+
+### SSL certificates
+
+You'll need to install an SSL certificate valid for all the
+(sub)domains you're using your Zulip server with. You can get an SSL
+certificate covering several domains for free by using
+[our Certbot wrapper tool](ssl-certificates.md#after-zulip-is-already-installed),
+though if you're going to host a large number of organizations, you
+may want to get a wildcard certificate. You can also get a wildcard
+certificate for
+[free using Certbot](https://community.letsencrypt.org/t/getting-wildcard-certificates-with-certbot/56285),
+but because of the stricter security checks for acquiring a wildcard
+cert, it isn't possible for a generic script like `setup-certbot` to
+create it for you; you'll have to do some manual steps with your DNS
+provider.
+
+### Other hostnames
+
+If you'd like to use hostnames that are not subdomains of each other,
+you can set the `REALM_HOSTS` setting in `/etc/zulip/settings.py` to a
+Python dictionary, like this:
+
+```python
+REALM_HOSTS = {
+    "mysubdomain": "hostname.example.com",
+}
+```
+
+This will make `hostname.example.com` the hostname for the realm that
+would, without this configuration, have been
+`mysubdomain.zulip.example.com`. To create your new realm on
+`hostname.example.com`, one should enter `mysubdomain` as the
+"subdomain" for the new realm.
+
+The value you choose for `mysubdomain` will not be displayed to users;
+the main constraint is that it will be impossible to create a
+different realm on `mysubdomain.zulip.example.com`.
+
+In a future version of Zulip, we expect to move this configuration
+into the database.
+
+### The root domain
+
+Most Zulip servers host a single Zulip organization on the root domain
+(e.g., `zulip.example.com`). The way this is implemented internally
+involves the organization having the empty string (`''`) as its
+"subdomain".
+
+You can mix having an organization on the root domain and some others
+on subdomains (e.g., `subdivision.zulip.example.com`), but this only
+works well if there are no users in common between the two
+organizations, because the auth cookies for the root domain are
+visible to the subdomain (so it's not possible for a single
+browser/client to be logged into both). So we don't recommend that
+configuration.
+
+### Changing subdomains
+
+You can [change the subdomain][help-center-change-url] for an existing
+organization using a [management command][management-commands]. Be
+sure you understand the implications of changing the organization URL
+before doing so, as it can be disruptive to users.
+
+[management-commands]: ../production/management-commands.md
+[help-center-change-url]: https://zulip.com/help/change-organization-url
+
+### Authentication
+
+Many of Zulip's supported authentication methods (Google, GitHub,
+SAML, etc.) can require providing the third-party authentication
+provider with a whitelist of callback URLs to your Zulip server (or
+even a single URL). For those vendors that support a whitelist, you
+can provide the callback URLs for each of your Zulip organizations.
+
+The cleaner solution is to register a special subdomain, e.g.,
+`auth.zulip.example.com` with the third-party provider, and then set
+`SOCIAL_AUTH_SUBDOMAIN = 'auth'` in `/etc/zulip/settings.py`, so that
+Zulip knows to use that subdomain for these authentication callbacks.
+
+### The system bot realm
+
+This is very much an implementation detail, but worth documenting to
+avoid confusion as to why there's an extra realm when inspecting the
+Zulip database.
+
+Every Zulip server comes with 1 realm that isn't created by users: the
+`zulipinternal` realm. By default, this realm only contains the Zulip "system
+bots". You can get a list of these on your system via
+`./scripts/get-django-setting INTERNAL_BOTS`, but this is where bots
+like "Notification Bot", "Welcome Bot", etc. exist. In the future,
+we're considering moving these bots to exist in every realm, so that
+we wouldn't need the system realm anymore.
+
+### Migrating / troubleshooting
+
+If you're migrating from a configuration using the root domain to one
+with realms hosted on subdomains, be sure to clear cookies in any
+browsers that were logged in on the root domain; otherwise, those
+browsers will experience weird/confusing redirects.
+
+## Open realm creation
+
+Installations like [Zulip Cloud](https://zulip.com/plans/) that wish to
+allow anyone on the Internet to create new Zulip organizations can do
+so by setting `OPEN_REALM_CREATION = True` in
+`/etc/zulip/settings.py`. Note that offering Zulip hosting to anyone
+on the Internet entails significant responsibility around security,
+abuse/spam, legal issues like GDPR/CCPA compliance, and more.

docs/production/password-strength.md

@@ -0,0 +1,59 @@
+---
+orphan: true
+---
+
+# Password strength
+
+When a user tries to set a password, we use [zxcvbn][zxcvbn] to check
+that it isn't a weak one.
+
+See discussion in [our main docs for server
+admins](authentication-methods.md#email-and-password). This doc explains in
+more detail how we set the default threshold (`PASSWORD_MIN_GUESSES`) we use.
+
+First, read the doc section there. (It's short.)
+
+Then, the CACM article ["Passwords and the Evolution of Imperfect
+Authentication"][bhos15] is comprehensive, educational, and readable,
+and is especially recommended.
+
+The CACM article is convincing that password requirements should be
+set to make passwords withstand an online attack, but not an offline
+one. Offline attacks are much less common, and there is a wide gap in
+the level of password strength required to beat them vs that for
+online attacks -- and therefore in the level of user frustration that
+such a requirement would cause.
+
+On top of that, estimating strength rapidly becomes more expensive
+at high levels, in both space (for lists of tokens to try) and time.
+As a result, in order to fit in a few MB of download and a few ms of
+check time, zxcvbn focuses on the range of online attacks, for the
+upper limit of which it uses 10^6 (apparently based on the offhand
+estimate of "perhaps one million guesses" in the CACM article.)
+
+Figure 3 of [the zxcvbn paper][zxcvbn-paper] shows that in fact
+overestimation (allowing a weak password) sharply degrades at 100k
+guesses, while underestimation (rejecting a strong password) jumps up
+just after 10k guesses, and grows steadily thereafter.
+
+Moreover, the [Yahoo study][bon12] shows that resistance to even 1M
+guesses is more than nearly half of users accomplish with a freely
+chosen password, and 100k is too much for about 20%. (See Figure 6.)
+It doesn't make sense for a Zulip server to try to educate or push so
+many users far beyond the security practices they're accustomed to; in
+the few environments where users can be expected to work much harder
+for security, local server admins can raise the threshold accordingly.
+Or, more likely, they already have a single-sign-on system in use for
+most everything else in their organization, and will disable password
+auth in Zulip entirely in favor of using that.
+
+Our threshold of 10k guesses provides significant protection against
+online attacks, and quite strong protection with appropriate
+rate-limiting. On the other hand it stays within the range where
+zxcvbn rarely underestimates the strength of a password too severely,
+and only about 10% of users do worse than this without prompting.
+
+[zxcvbn]: https://github.com/dropbox/zxcvbn
+[bhos15]: https://www.cl.cam.ac.uk/~fms27/papers/2015-BonneauHerOorSta-passwords.pdf
+[zxcvbn-paper]: https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_wheeler.pdf
+[bon12]: https://ieeexplore.ieee.org/document/6234435

docs/production/postgresql-support-table.md

@@ -0,0 +1,9 @@
+| Zulip Server version | Supported versions of PostgreSQL |
+| -------------------- | -------------------------------- |
+| 7.x                  | 12, 13, 14, 15                   |
+| 8.x                  | 12, 13, 14, 15, 16               |
+| 9.x                  | 12, 13, 14, 15, 16               |
+| 10.x                 | 13, 14, 15, 16, 17               |
+| 11.x                 | 14, 15, 16, 17                   |
+| 12.x                 | 14, 15, 16, 17, 18               |
+| 13.x (unreleased)    | 14, 15, 16, 17, 18               |

docs/production/postgresql.md

@@ -0,0 +1,243 @@
+# PostgreSQL database details
+
+Zulip supports a range of PostgreSQL versions:
+
+```{include} postgresql-support-table.md
+
+```
+
+We recommend that installations [upgrade to the latest
+PostgreSQL][upgrade-postgresql] supported by their version of Zulip.
+
+[upgrade-postgresql]: upgrade.md#upgrading-postgresql
+
+## Separate PostgreSQL database
+
+:::{important}
+
+If you are using [Docker](docker.md), see
+{doc}`docker:how-to/compose-existing-services` for using an external
+PostgreSQL with Docker Compose, or
+{doc}`docker:how-to/helm-existing-services` for the Helm chart's
+`externalPostgresql` configuration.
+
+:::
+
+It is possible to run Zulip against a PostgreSQL database which is not on the
+primary application server. There are two possible flavors of this -- using a
+managed PostgreSQL instance from a cloud provider, or separating the PostgreSQL
+server onto a separate (but still Zulip-managed) server for scaling purposes.
+
+### Cloud-provider-managed PostgreSQL (e.g., Amazon RDS)
+
+You can use a database-as-a-service like Amazon RDS for the Zulip database. The
+experience is slightly degraded, in that most providers don't include useful
+dictionary files in their installations, and don't provide a way to provide them
+yourself. [Full-text search][fts] will be less useful, due to the inferior
+stemming rules that the built-in dictionaries provide.
+
+[fts]: ../subsystems/full-text-search.md
+
+#### Step 1: Choose database name and username
+
+Zulip defaults to a database user named `zulip` and a database named `zulip`.
+It does support alternate values for those -- for instance, if required by
+convention of your central database server.
+
+#### Step 1: Set up Zulip
+
+Follow the [standard install instructions](install.md), with modified `install`
+arguments -- providing an updated `--puppet-classes`, as well as the database
+and user names, and the version of the remote PostgreSQL server.
+
+```bash
+./zulip-server-*/scripts/setup/install --certbot \
+    --email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
+    --puppet-classes=zulip::profile::standalone_nodb,zulip::process_fts_updates \
+    --postgresql-database-name=zulip \
+    --postgresql-database-user=zulip \
+    --postgresql-version=YOUR_SERVERS_POSTGRESQL_VERSION
+```
+
+#### Step 3: Create the PostgreSQL database
+
+Access an administrative `psql` shell on your PostgreSQL database, and
+run the commands in `scripts/setup/create-db.sql`. This will:
+
+- Create a database called `zulip` with `C.UTF-8` collation.
+- Create a user called `zulip` with full rights on that database.
+
+If you cannot run that SQL directly, or need to use different database
+or usernames, you should perform the equivalent actions.
+
+Depending on how authentication works for your PostgreSQL installation, you may
+also need to set a password for the Zulip user, generate a client certificate,
+or similar; consult the documentation for your database provider for the
+available options.
+
+#### Step 4: Configure Zulip to use the PostgreSQL database
+
+In `/etc/zulip/settings.py` on your Zulip server, configure the
+following settings with details for how to connect to your PostgreSQL
+server. Your database provider should provide these details.
+
+- `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server.
+- `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server.
+- `REMOTE_POSTGRES_SSLMODE`: [SSL Mode][ssl-mode] used to connect to the server.
+
+[ssl-mode]: https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION
+
+If you're using password authentication, you should specify the
+password of the `zulip` user in /etc/zulip/zulip-secrets.conf as
+follows:
+
+```ini
+postgres_password = abcd1234
+```
+
+Now complete the installation by running the following commands.
+
+```bash
+# Ask Zulip installer to initialize the PostgreSQL database.
+su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'
+
+# And then generate a realm creation link:
+su zulip -c '/home/zulip/deployments/current/manage.py generate_realm_creation_link'
+```
+
+### Remote PostgreSQL database
+
+This assumes two servers; one hosting the PostgreSQL database, and one hosting
+the remainder of the Zulip services.
+
+#### Step 1: Set up Zulip
+
+Follow the [standard install instructions](install.md), with modified `install`
+arguments:
+
+```bash
+./zulip-server-*/scripts/setup/install --certbot \
+    --email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
+    --puppet-classes=zulip::profile::standalone_nodb
+```
+
+#### Step 2: Create the PostgreSQL database server
+
+On the host that will run PostgreSQL, download the Zulip tarball and install
+just the PostgreSQL server part:
+
+```bash
+./zulip-server-*/scripts/setup/install \
+    --puppet-classes=zulip::profile::postgresql
+
+./zulip-server-*/scripts/setup/create-database
+```
+
+You will need to [configure `/etc/postgresql/*/main/pg_hba.conf`][pg-hba] to
+allow connections to the `zulip` database as the `zulip` user from the
+application frontend host. How you configure this is up to you (i.e. password
+authentication, certificates, etc), and is outside the scope of this document.
+
+[pg-hba]: https://www.postgresql.org/docs/current/auth-pg-hba-conf.html
+
+#### Step 3: Configure Zulip to use the PostgreSQL database
+
+In `/etc/zulip/settings.py` on your Zulip server, configure the following
+settings with details for how to connect to your PostgreSQL server.
+
+- `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server.
+- `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server; this is likely `5432`
+- `REMOTE_POSTGRES_SSLMODE`: [SSL Mode][ssl-mode] used to connect to the server.
+
+If you're using password authentication, you should specify the
+password of the `zulip` user in /etc/zulip/zulip-secrets.conf as
+follows:
+
+```ini
+postgres_password = abcd1234
+```
+
+Set the remote server's PostgreSQL version in `/etc/zulip/zulip.conf`:
+
+```ini
+[postgresql]
+# Set this to match the version running on your remote PostgreSQL server
+version = 18
+```
+
+Now complete the installation by running the following commands.
+
+```bash
+# Ask Zulip installer to initialize the PostgreSQL database.
+su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'
+
+# And then generate a realm creation link:
+su zulip -c '/home/zulip/deployments/current/manage.py generate_realm_creation_link'
+```
+
+## PostgreSQL warm standby
+
+Zulip's configuration allows for [warm standby database replicas][warm-standby]
+as a disaster recovery solution; see the linked PostgreSQL documentation for
+details on this type of deployment. Zulip's configuration can, but is not
+required to, build on top of `wal-g`, our [streaming database backup
+solution][wal-g], for ease of establishing base images without incurring load on
+the primary.
+
+Warm standby replicas should configure the hostname of their primary replica,
+and username to use for replication, in `/etc/zulip/zulip.conf`:
+
+```ini
+[postgresql]
+replication_user = replicator
+replication_primary = hostname-of-primary.example.com
+```
+
+The `postgres` user on the replica will need to be able to authenticate as the
+`replication_user` user, which may require further configuration of
+`pg_hba.conf` and client certificates on the replica. If you are using password
+authentication, you can set a `postgresql_replication_password` secret in
+`/etc/zulip/zulip-secrets.conf`.
+
+Use `zulip-puppet-apply` to rebuild the PostgreSQL configuration with those
+values. If [wal-g][wal-g] is used, use `env-wal-g backup-fetch` to fetch the
+latest copy of the base backup; otherwise, use [`pg_basebackup`][pg_basebackup].
+The PostgreSQL replica should then begin live-replicating from the primary.
+
+In the event of a primary failure, use [`pg_ctl promote`][promote] on the warm
+standby to promote it to primary. As with all database promotions, care should
+be taken to ensure that the old primary cannot come back online, and leave the
+cluster with two diverging timelines.
+
+To make fail-over to the warm-standby faster, without requiring a restart of
+Zulip services, you can configure Zulip with a comma-separated list of remote
+PostgreSQL servers to connect to; it will choose the first which accepts writes
+(i.e. is not a read-only replica). In the event that the primary fails, it will
+repeatedly retry the list, in order, until the replica is promoted and becomes
+writable. To configure this, in `/etc/zulip/settings.py`, set:
+
+```python3
+REMOTE_POSTGRES_HOST = 'primary-database-host,warm-standby-host'
+```
+
+[warm-standby]: https://www.postgresql.org/docs/current/warm-standby.html
+[wal-g]: export-and-import.md#database-only-backup-tools
+[pg_basebackup]: https://www.postgresql.org/docs/current/app-pgbasebackup.html
+[promote]: https://www.postgresql.org/docs/current/app-pg-ctl.html
+
+## PostgreSQL vacuuming alerts
+
+The `autovac_freeze` PostgreSQL alert from `check_postgres` is particularly
+important. This alert indicates that the age (in terms of number of
+transactions) of the oldest transaction id (XID) is getting close to the
+`autovacuum_freeze_max_age` setting. When the oldest XID hits that age,
+PostgreSQL will force a VACUUM operation, which can often lead to sudden
+downtime until the operation finishes. If it did not do this and the age of the
+oldest XID reached 2 billion, transaction id wraparound would occur and there
+would be data loss. To clear the nagios alert, perform a `VACUUM` in each
+indicated database as a database superuser (i.e. `postgres`).
+
+See [the PostgreSQL documentation][vacuum] for more details on PostgreSQL
+vacuuming.
+
+[vacuum]: http://www.postgresql.org/docs/current/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND

docs/production/requirements.md

@@ -0,0 +1,254 @@
+# Requirements and scalability
+
+To run a Zulip server, you will need:
+
+- A dedicated machine or VM
+- A supported OS:
+  - Ubuntu 22.04
+  - Ubuntu 24.04
+  - Ubuntu 26.04
+  - Debian 12
+  - Debian 13
+- A supported CPU architecture:
+  - x86-64
+  - aarch64
+- At least 2 GB RAM
+  - If you have < 5 GB RAM, we require some swap space; we recommend configuring
+    2 GB of swap
+  - If you expect 100+ users: 4 GB RAM, and 2 CPUs
+- 10GB free space (i.e. approximately a 25GB total disk, given OS requirements)
+- A hostname in DNS
+- Credentials for sending email
+
+For details on each of these requirements, see below.
+
+[upgrade-from-git]: upgrade.md#upgrading-from-a-git-repository
+
+## Server
+
+#### General
+
+The installer expects Zulip to be the **only thing** running on the
+system; it will install system packages with `apt` (like nginx,
+PostgreSQL, and Redis) and configure them for its own use. We strongly
+recommend using either a fresh machine instance in a cloud provider, a
+fresh VM, [our Docker image](docker.md), or a dedicated
+machine. If you decide to disregard our advice and use a server that
+hosts other services, we can't support you, but [we do have some notes
+on issues you'll encounter](install-existing-server.md).
+
+#### Operating system
+
+Ubuntu 22.04, Ubuntu 24.04, Ubuntu 26.04, Debian 12, and Debian 13
+are supported for running Zulip in production. You can also
+run Zulip on other platforms that support Docker using
+[our Docker image](docker.md).
+
+We recommend installing on the newest supported OS release you're
+comfortable with, to save a bit of future work [upgrading the
+operating system][upgrade-os].
+
+If you're using Ubuntu, the
+[Ubuntu universe repository][ubuntu-repositories] must be
+[enabled][enable-universe], which is usually just:
+
+```bash
+sudo add-apt-repository universe
+sudo apt update
+```
+
+[upgrade-os]: upgrade.md#upgrading-the-operating-system
+[ubuntu-repositories]: https://help.ubuntu.com/community/Repositories/Ubuntu
+[enable-universe]: https://help.ubuntu.com/community/Repositories/CommandLine#Adding_the_Universe_and_Multiverse_Repositories
+
+#### Hardware specifications
+
+- CPU and memory: For installations with 100+ users you'll need a
+  minimum of **2 CPUs** and **4 GB RAM**. For installations with fewer
+  users, 1 CPU and 2 GB RAM with 2 GB of swap is sufficient. We
+  recommend against using highly CPU-limited servers like the AWS `t2`
+  style instances for organizations with hundreds of users (active or
+  no).
+
+- Disk space: You'll need at least 10 GB of dedicated free disk space
+  for a server with dozens of users. We recommend using an SSD and
+  avoiding cloud storage backends that limit the IOPS per second,
+  since the disk is primarily used for the Zulip database.
+
+See our [documentation on scalability](#scalability) below for advice
+on hardware requirements for larger organizations.
+
+#### Network and security specifications
+
+- Incoming HTTPS access (usually port 443, though this is
+  [configurable](deployment.md#using-an-alternate-port))
+  from the networks where your users are (usually, the public
+  Internet).
+- Incoming port 80 access (optional). Zulip only serves content over
+  HTTPS, and will redirect HTTP requests to HTTPS.
+- Incoming port 25 if you plan to enable Zulip's [incoming email
+  integration](email-gateway.md).
+- Incoming port 4369 should be protected by a firewall to prevent
+  exposing `epmd`, an Erlang service which does not support binding
+  only to localhost. Leaving this exposed will allow unauthenticated
+  remote users to determine that the server is running RabbitMQ, and
+  on which port, though no further information is leaked.
+- Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so
+  that Zulip can properly manage image and website previews and mobile
+  push notifications. Outgoing Internet access is not required if you
+  [disable those
+  features](https://zulip.com/help/image-video-and-website-previews).
+- Outgoing SMTP access (usually port 587) to your [SMTP
+  server](email.md) so that Zulip can send emails.
+- A domain name (e.g., `zulip.example.com`) that your users will use to
+  access the Zulip server. In order to generate valid SSL
+  certificates [with Certbot][doc-certbot], and to enable other
+  services such as Google authentication, public DNS name is simpler,
+  but Zulip can be configured to use a non-public domain or even an IP
+  address as its external hostname (though we don't recommend that
+  configuration).
+- Zulip supports [running behind a reverse proxy][reverse-proxy].
+- Zulip configures [Smokescreen, an outgoing HTTP
+  proxy][smokescreen-proxy], to protect against [SSRF attacks][ssrf],
+  which prevents user from making the Zulip server make requests to
+  private resources. If your network has its own outgoing HTTP proxy,
+  Zulip supports using that instead.
+
+Zulip does not, itself, require SSH, but most installations will also require
+access to incoming port 22 for SSH access for remote access.
+
+[ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
+[smokescreen-proxy]: deployment.md#customizing-the-outgoing-http-proxy
+[reverse-proxy]: reverse-proxies.md
+
+## Credentials needed
+
+#### SSL certificate
+
+Your Zulip server will need an SSL certificate for the domain name it
+uses. For most Zulip servers, the recommended (and simplest) way to
+get this is to just [use the `--certbot` option][doc-certbot] in the
+Zulip installer, which will automatically get a certificate for you
+and keep it renewed.
+
+For test installations, an even simpler alternative is always
+available: [the `--self-signed-cert` option][doc-self-signed] in the
+installer.
+
+If you'd rather acquire an SSL certificate another way, see our [SSL
+certificate documentation](ssl-certificates.md).
+
+[doc-certbot]: ssl-certificates.md#certbot-recommended
+[doc-self-signed]: ssl-certificates.md#self-signed-certificate
+
+#### Outgoing email
+
+- Outgoing email (SMTP) credentials that Zulip can use to send
+  outgoing emails to users (e.g., email address confirmation emails
+  during the signup process, message notification emails, password
+  reset, etc.). If you don't have an existing outgoing SMTP solution,
+  read about
+  [free outgoing SMTP options and options for prototyping](email.md#free-outgoing-email-services).
+
+Once you have met these requirements, see [full instructions for installing
+Zulip in production](install.md).
+
+## Scalability
+
+This section details some basic guidelines for running a Zulip server
+for larger organizations (especially >1000 users or 500+ daily active
+users). These guidelines are conservative, since they are intended to
+be sufficient for a wide range of possible usage patterns that may not
+be applicable to your installation.
+
+Zulip's resource needs depend mainly on 3 parameters:
+
+- daily active users (e.g., number of employees if everyone's an
+  employee)
+- total user accounts (can be much larger)
+- message volume.
+
+In the following, we discuss a configuration with at most two types of
+servers: application servers (running Django, Tornado, RabbitMQ,
+Redis, Memcached, etc.) and database servers. Of the application
+server services, Django dominates the resource requirements. One can
+run every service on its own system (as [our Docker
+deployment](docker.md) does) but for most use cases, there's little
+scalability benefit to doing so. See [deployment
+options](deployment.md) for details on installing Zulip with a
+dedicated database server.
+
+- **Dedicated database**. For installations with hundreds of daily
+  active users, we recommend using a [remote PostgreSQL
+  database](postgresql.md), but it's not required.
+
+- **RAM:** We recommend more RAM for larger installations:
+
+  - With 25+ daily active users, 4 GB of RAM.
+  - With 100+ daily active users, 8 GB of RAM.
+  - With 400+ daily active users, 16 GB of RAM for the Zulip
+    application server, plus 16 GB for the database.
+  - With 2000+ daily active users 32 GB of RAM, plus 32 GB for the
+    database.
+  - Roughly linear scaling beyond that.
+
+- **CPU:** The Zulip application server's CPU usage is heavily
+  optimized due to extensive work on optimizing the performance of
+  requests for latency reasons. Because most servers with sufficient
+  RAM have sufficient CPU resources, CPU requirements are rarely an
+  issue. For larger installations with a dedicated database, we
+  recommend high-CPU instances for the application server and a
+  database-optimized (usually low CPU, high memory) instance for the
+  database.
+
+- **Disk for application server:** We recommend using [the S3 file
+  uploads backend][s3-uploads] to store uploaded files at scale. With
+  the S3 backend configuration, we recommend 50 GB of disk for the OS,
+  Zulip software, logs and scratch/free space. Because uploaded files
+  are cached locally, you may need more disk space if you make heavy
+  use of uploaded files.
+
+- **Disk for database:** SSD disk is highly recommended. For
+  installations where most messages have <100 recipients, 10 GB per 1M
+  messages of history is sufficient plus 1 GB per 1000 users is
+  sufficient. If most messages are to public channels with 10K+ users
+  subscribed (like on chat.zulip.org), add 20 GB per (1000 user
+  accounts) per (1M messages to public channels).
+
+- **Example:** When
+  [the Zulip development community](https://zulip.com/development-community/) server
+  had 12K user accounts (~300 daily actives) and 800K messages of
+  history (400K to public channels), it was a default configuration
+  single-server installation with 16 GB of RAM, 4 cores (essentially
+  always idle), and its database was using about 100 GB of disk.
+
+- **Disaster recovery:** One can easily run a warm spare application
+  server and a warm spare database (using [PostgreSQL warm standby
+  replicas][streaming-replication]). Make sure the warm spare
+  application server has copies of `/etc/zulip` and you're either
+  syncing `LOCAL_UPLOADS_DIR` or using the [S3 file uploads
+  backend][s3-uploads].
+
+- **Sharding:** For servers with several thousand daily active users,
+  Zulip supports [sharding its real-time-push Tornado
+  service][tornado-sharding], both by realm/organization (for hosting many
+  organizations) and by user ID (for hosting single very large
+  organizations).
+
+  Care must be taken when dividing traffic for a single Zulip realm
+  between multiple Zulip application servers, which is why we
+  recommend a hot spare over load-balancing for most installations
+  desiring extra redundancy.
+
+If you have scalability questions or are unsure whether Zulip is a fit
+for your use case, contact [Zulip sales or support][contact-support]
+for assistance.
+
+For readers interested in technical details around what features
+impact Zulip's scalability, this [performance and scalability design
+document](../subsystems/performance.md) may also be of interest.
+
+[s3-uploads]: upload-backends.md#s3-backend-configuration
+[streaming-replication]: postgresql.md#postgresql-warm-standby
+[contact-support]: https://zulip.com/help/contact-support
+[tornado-sharding]: system-configuration.md#tornado_sharding

docs/production/reverse-proxies.md

@@ -0,0 +1,288 @@
+## Reverse proxies
+
+Zulip is designed to support being run behind a reverse proxy server.
+This section contains notes on the configuration required with
+variable reverse proxy implementations.
+
+### Installer options
+
+If your Zulip server will not be on the public Internet, we recommend,
+installing with the `--self-signed-cert` option (rather than the
+`--certbot` option), since Certbot requires the server to be on the
+public Internet.
+
+#### Configuring Zulip to allow HTTP
+
+Zulip requires clients to connect to Zulip servers over the secure
+HTTPS protocol; the insecure HTTP protocol is not supported. However,
+we do support using a reverse proxy that speaks HTTPS to clients and
+connects to the Zulip server over HTTP; this can be secure when the
+Zulip server is not directly exposed to the public Internet.
+
+After installing the Zulip server as [described
+above](#installer-options), you can configure Zulip to accept HTTP
+requests from a reverse proxy as follows:
+
+1. Add the following block to `/etc/zulip/zulip.conf`:
+
+   ```ini
+   [application_server]
+   http_only = true
+   ```
+
+1. As root, run
+   `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This
+   will convert Zulip's main `nginx` configuration file to allow HTTP
+   instead of HTTPS.
+
+1. Finally, restart the Zulip server, using
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+Note that Zulip must be able to accurately determine if its connection to the
+client was over HTTPS or not; if you enable `http_only`, it is very important
+that you correctly configure Zulip to trust the `X-Forwarded-Proto` header from
+its proxy (see the next section), or clients may see infinite redirects.
+
+#### Configuring Zulip to trust proxies
+
+:::{important}
+
+If you are using [Docker](docker.md), see {ref}`docker:proxies` for
+the equivalent `LOADBALANCER_IPS` / `TRUST_GATEWAY_IP` mechanism.
+
+:::
+
+Before placing Zulip behind a reverse proxy, it needs to be configured to trust
+the client IP addresses that the proxy reports via the `X-Forwarded-For` header,
+and the protocol reported by the `X-Forwarded-Proto` header. This is important
+to have accurate IP addresses in server logs, as well as in notification emails
+which are sent to end users. Zulip doesn't default to trusting all
+`X-Forwarded-*` headers, because doing so would allow clients to spoof any IP
+address, and claim connections were over a secure connection when they were not;
+we specify which IP addresses are the Zulip server's incoming proxies, so we
+know which `X-Forwarded-*` headers to trust.
+
+1. Determine the IP addresses of all reverse proxies you are setting up, as seen
+   from the Zulip host. Depending on your network setup, these may not be the
+   same as the public IP addresses of the reverse proxies. These can also be IP
+   address ranges, as expressed in CIDR notation.
+
+1. Add the following block to `/etc/zulip/zulip.conf`.
+
+   ```ini
+   [loadbalancer]
+   # Use the IP addresses you determined above, separated by commas.
+   ips = 192.168.0.100
+   ```
+
+1. Reconfigure Zulip with these settings. As root, run
+   `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This will
+   adjust Zulip's `nginx` configuration file to accept the `X-Forwarded-For`
+   header when it is sent from one of the reverse proxy IPs.
+
+1. Finally, restart the Zulip server, using
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+### nginx configuration
+
+Below is a working example of a full nginx configuration. It assumes
+that your Zulip server sits at `https://10.10.10.10:443`; see
+[above](#configuring-zulip-to-allow-http) to switch to HTTP.
+
+1. Follow the instructions to [configure Zulip to trust
+   proxies](#configuring-zulip-to-trust-proxies).
+
+1. Configure the root `nginx.conf` file. We recommend using
+   `/etc/nginx/nginx.conf` from your Zulip server for our recommended
+   settings. E.g., if you don't set `client_max_body_size`, it won't be
+   possible to upload large files to your Zulip server.
+
+1. Configure the `nginx` site-specific configuration (in
+   `/etc/nginx/sites-available`) for the Zulip app. The following
+   example is a good starting point:
+
+   ```nginx
+   server {
+           listen 80;
+           listen [::]:80;
+           location / {
+                   return 301 https://$host$request_uri;
+           }
+   }
+
+   server {
+           listen                  443 ssl http2;
+           listen                  [::]:443 ssl http2;
+           server_name             zulip.example.com;
+
+           ssl_certificate         /etc/letsencrypt/live/zulip.example.com/fullchain.pem;
+           ssl_certificate_key     /etc/letsencrypt/live/zulip.example.com/privkey.pem;
+
+           location / {
+                   proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
+                   proxy_set_header        X-Forwarded-Proto $scheme;
+                   proxy_set_header        Host $host;
+                   proxy_http_version      1.1;
+                   proxy_buffering         off;
+                   proxy_read_timeout      20m;
+                   proxy_pass              https://10.10.10.10:443;
+           }
+   }
+   ```
+
+   Don't forget to update `server_name`, `ssl_certificate`,
+   `ssl_certificate_key` and `proxy_pass` with the appropriate values
+   for your deployment.
+
+### Apache2 configuration
+
+Below is a working example of a full Apache2 configuration. It assumes
+that your Zulip server sits at `https://internal.zulip.hostname:443`.
+Note that if you wish to use SSL to connect to the Zulip server,
+Apache requires you use the hostname, not the IP address; see
+[above](#configuring-zulip-to-allow-http) to switch to HTTP.
+
+1. Follow the instructions to [configure Zulip to trust
+   proxies](#configuring-zulip-to-trust-proxies).
+
+1. Set `USE_X_FORWARDED_HOST = True` in `/etc/zulip/settings.py` and
+   restart Zulip.
+
+1. Enable some required Apache modules:
+
+   ```bash
+   a2enmod ssl proxy proxy_http headers rewrite
+   ```
+
+1. Create an Apache2 virtual host configuration file, similar to the
+   following. Place it the appropriate path for your Apache2
+   installation and enable it (e.g., if you use Debian or Ubuntu, then
+   place it in `/etc/apache2/sites-available/zulip.example.com.conf`
+   and then run
+   `a2ensite zulip.example.com && systemctl reload apache2`):
+
+   ```apache
+   <VirtualHost *:80>
+       ServerName zulip.example.com
+       RewriteEngine On
+       RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
+   </VirtualHost>
+
+   <VirtualHost *:443>
+     ServerName zulip.example.com
+
+     RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
+
+     RewriteEngine On
+     RewriteRule /(.*)           https://internal.zulip.hostname:443/$1 [P,L]
+
+     <Location />
+       Require all granted
+       ProxyPass https://internal.zulip.hostname:443/ timeout=1200
+     </Location>
+
+     SSLEngine on
+     SSLProxyEngine on
+     SSLCertificateFile /etc/letsencrypt/live/zulip.example.com/fullchain.pem
+     SSLCertificateKeyFile /etc/letsencrypt/live/zulip.example.com/privkey.pem
+     # This file can be found in ~zulip/deployments/current/puppet/zulip/files/nginx/dhparam.pem
+     SSLOpenSSLConfCmd DHParameters "/etc/nginx/dhparam.pem"
+     SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
+     SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
+     SSLHonorCipherOrder off
+     SSLSessionTickets off
+     Header set Strict-Transport-Security "max-age=31536000"
+   </VirtualHost>
+   ```
+
+   Don't forget to update `ServerName`, `RewriteRule`, `ProxyPass`,
+   `SSLCertificateFile`, and `SSLCertificateKeyFile` as are
+   appropriate for your deployment.
+
+### HAProxy configuration
+
+Below is a working example of a HAProxy configuration. It assumes that
+your Zulip server sits at `https://10.10.10.10:443`; see
+[above](#configuring-zulip-to-allow-http) to switch to HTTP.
+
+1. Follow the instructions to [configure Zulip to trust
+   proxies](#configuring-zulip-to-trust-proxies).
+
+1. Configure HAProxy. The below is a minimal `frontend` and `backend`
+   configuration:
+
+   ```text
+   frontend zulip
+       mode http
+       bind *:80
+       bind *:443 ssl crt /etc/ssl/private/zulip-combined.crt
+       http-request redirect scheme https code 301 unless { ssl_fc }
+       http-request set-header X-Forwarded-Proto http unless { ssl_fc }
+       http-request set-header X-Forwarded-Proto https if { ssl_fc }
+       default_backend zulip
+
+   backend zulip
+       mode http
+       timeout server 20m
+       server zulip 10.10.10.10:443 check ssl ca-file /etc/ssl/certs/ca-certificates.crt
+   ```
+
+   Don't forget to update `bind *:443 ssl crt` and `server` as is
+   appropriate for your deployment.
+
+### Other proxies
+
+If you're using another reverse proxy implementation, there are few
+things you need to be careful about when configuring it:
+
+1. Configure your reverse proxy (or proxies) to correctly maintain the
+   `X-Forwarded-For` HTTP header, which is supposed to contain the series
+   of IP addresses the request was forwarded through. Additionally,
+   [configure Zulip to respect the addresses sent by your reverse
+   proxies](#configuring-zulip-to-trust-proxies). You can verify
+   your work by looking at `/var/log/zulip/server.log` and checking it
+   has the actual IP addresses of clients, not the IP address of the
+   proxy server.
+
+1. Configure your reverse proxy (or proxies) to correctly maintain the
+   `X-Forwarded-Proto` HTTP header, which is supposed to contain either `https`
+   or `http` depending on the connection between your browser and your
+   proxy. This will be used by Django to perform CSRF checks regardless of your
+   connection mechanism from your proxy to Zulip. Note that the proxies _must_
+   set the header, overriding any existing values, not add a new header.
+
+   If your proxy _cannot_ set the `X-Forwarded-Proto` header, you can opt to do
+   all HTTP-to-HTTPS redirection at the load-balancer level, and set
+   [`loadbalancer.rejects_http_requests` in `zulip.conf`][no-proto-header]; but
+   note the important security caveats for that in its documentation.
+
+1. Configure your proxy to pass along the `Host:` header as was sent
+   from the client, not the internal hostname as seen by the proxy.
+   If this is not possible, you can set `USE_X_FORWARDED_HOST = True`
+   in `/etc/zulip/settings.py`, and pass the client's `Host` header to
+   Zulip in an `X-Forwarded-Host` header.
+
+1. Ensure your proxy doesn't interfere with Zulip's use of
+   long-polling for real-time push from the server to your users'
+   browsers. This [nginx code snippet][nginx-proxy-longpolling-config]
+   does this.
+
+   The key configuration options are:
+
+   - `proxy_read_timeout 1200;`. It's critical that this be significantly above
+     60s, but the precise value isn't important. This is most important for the
+     events API, but must be applied to all endpoints.
+   - `proxy_buffering off`. If you don't do this, your `nginx` proxy may return
+     occasional 502 errors to clients using Zulip's events API.
+
+1. The other tricky failure mode we've seen with `nginx` reverse
+   proxies is that they can load-balance between the IPv4 and IPv6
+   addresses for a given hostname. This can result in mysterious errors
+   that can be quite difficult to debug. Be sure to declare your
+   `upstreams` equivalent in a way that won't do load-balancing
+   unexpectedly (e.g., pointing to a DNS name that you haven't configured
+   with multiple IPs for your Zulip machine; sometimes this happens with
+   IPv6 configuration).
+
+[no-proto-header]: system-configuration.md#rejects_http_requests
+[nginx-proxy-longpolling-config]: https://github.com/zulip/zulip/blob/main/puppet/zulip/files/nginx/zulip-include-common/proxy_longpolling

docs/production/scim.md

@@ -0,0 +1,111 @@
+# SCIM provisioning
+
+Zulip has beta support for user provisioning and deprovisioning via
+the SCIM protocol. In SCIM, a third-party SCIM Identity Provider (IdP)
+acts as the SCIM client, connecting to the service provider (your Zulip
+server).
+
+See the [SCIM help center page](https://zulip.com/help/scim) for
+documentation on SCIM in [Zulip Cloud](https://zulip.com) as well as
+detailed documentation for how to configure some SCIM IdP providers.
+
+Synchronizing groups via SCIM is currently not supported.
+
+## Server configuration
+
+:::{important}
+
+If you are using [Docker](docker.md), set `SCIM_CONFIG` via the
+`SETTING_SCIM_CONFIG` environment variable; see
+{doc}`docker:how-to/compose-settings`.
+
+:::
+
+The Zulip server-side configuration is straightforward:
+
+1. Pick a client name for your SCIM client. This name is internal to
+   your Zulip configuration, so the name of your IdP provider is a
+   good choice. We'll use `okta` in the examples below.
+
+1. Configure the Zulip server by adding a `SCIM_CONFIG` block to your
+   `/etc/zulip/settings.py`:
+
+   ```python
+   SCIM_CONFIG = {
+       "subdomain": {
+           "bearer_token": "<secret token>",
+           "scim_client_name": "okta",
+           "name_formatted_included": False,
+       }
+   }
+   ```
+
+   The `bearer_token` should contain a secure, secret token that you
+   generate. You can use any secure password generation tools for this,
+   such as the `apg` command included by default in some Linux distributions.
+   For example, `apg -m20` will generate some passwords of minimum length 20
+   for you.
+
+   Make sure to restart the server after editing your settings, by running
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+   The SCIM IdP will authenticate its requests to your Zulip server by
+   sending an `Authorization` header like this:
+   `Authorization: Bearer <secret token>`. `name_formatted_included` needs to be set
+   to `False` for Okta. It tells Zulip whether the IdP includes
+   `name.formatted` in its `User` representation.
+
+1. Now you can proceed to [configuring your SCIM IdP](https://zulip.com/help/scim).
+   Use the value `Bearer <secret token>` using the `bearer_token` you've generated
+   earlier as the `API token` that the SCIM IdP will ask for when configuring
+   authentication details.
+
+## Additional options
+
+- To enable the creation of guest accounts without automatically subscribing them to any initial channels,
+  add `"create_guests_without_streams": True` to your client's config dict in `SCIM_CONFIG`.
+  This option is particularly useful for organizations that prefer to manually invite new guest users
+  to the appropriate channels.
+
+  Example configuration with the additional option:
+
+  ```python
+  SCIM_CONFIG = {
+     "subdomain": {
+        "bearer_token": "<secret token>",
+        "scim_client_name": "okta",
+        "name_formatted_included": False,
+        "create_guests_without_streams": True,
+     }
+  }
+  ```
+
+- To sync custom profile fields from the SCIM IdP, add
+  `"custom_profile_field_map"` to your client's config dict. This is a
+  dictionary mapping Zulip custom profile field names to SCIM
+  attribute names. A custom profile field is represented for this feature
+  by its user-facing name in lowercase with spaces replaced by underscores
+  (e.g., "Phone number" is represented by `phone_number`).
+
+  Example configuration with custom profile field syncing:
+
+  ```python
+  SCIM_CONFIG = {
+     "subdomain": {
+        "bearer_token": "<secret token>",
+        "scim_client_name": "okta",
+        "name_formatted_included": False,
+        "custom_profile_field_map": {
+            # Phone number
+            "phone_number": "phoneNumber",
+            # Birthday
+            "birthday": "birthday",
+        },
+     }
+  }
+  ```
+
+  With this configuration, when the SCIM IdP sends a `phoneNumber`
+  attribute in a User request, its value will be synced to the "Phone
+  number" custom profile field in Zulip. The custom profile fields
+  referenced must already exist in the Zulip organization settings.

docs/production/securing-your-zulip-server.md

@@ -0,0 +1,161 @@
+# Securing your Zulip server
+
+This page offers practical information on how to secure your Zulip server. For a
+deeper understanding of the security model, take a look at Zulip's [security
+overview](https://zulip.com/security/).
+
+Here are some best practices for keeping your Zulip server secure:
+
+1. [Limit shell access to a small set of trusted individuals.](#1-limit-shell-access-to-a-small-set-of-trusted-individuals)
+2. [Consider requiring authentication with single sign-on (SSO).](#2-consider-requiring-authentication-with-single-sign-on-sso)
+3. [Teach users how to protect their account.](#3-teach-users-how-to-protect-their-account)
+4. [Become familiar with Zulip's access management model.](#4-become-familiar-with-zulips-access-management-model)
+5. [Understand security for user-uploaded content and user-generated requests.](#5-understand-security-for-user-uploaded-content-and-user-generated-requests)
+6. [Understand Zulip's rate-limiting system.](#6-understand-zulips-rate-limiting-system)
+
+If you believe you've identified a security issue, please report it to Zulip's
+security team at [security@zulip.com](mailto:security@zulip.com) as soon as
+possible, so that we can address it and make a responsible disclosure.
+
+## 1. Limit shell access to a small set of trusted individuals.
+
+Anyone with root access to a Zulip application server or Zulip database server,
+or with access to the `zulip` user on a Zulip application server, has _complete
+control over the Zulip installation_ and all of its data (so they can read
+messages, modify history, etc.). This means that _only trusted individuals_
+should have shell access to your Zulip server.
+
+## 2. Consider requiring authentication with single sign-on (SSO).
+
+The preferred way to log in to Zulip is using a single sign-on (SSO)
+solution like Google authentication, LDAP, or similar, but Zulip
+also supports password authentication. See [the authentication
+methods documentation](authentication-methods.md) for
+details on Zulip's available authentication methods.
+
+## 3. Teach users how to protect their account.
+
+Every Zulip user has an API key, which can be used to do essentially everything
+that users can do when they're logged in. Make sure users know to immediately
+[reset their API key and password](https://zulip.com/help/protect-your-account)
+if their credentials are compromised (e.g., their cell phone is lost or stolen).
+
+## 4. Become familiar with Zulip's access management model.
+
+Zulip's help center offers a detailed overview of Zulip's permissions management
+system. Reading the following guides will give you a good starting point:
+
+- [Channel types and permissions](https://zulip.com/help/channel-permissions)
+- [User roles](https://zulip.com/help/user-roles)
+- [User groups](https://zulip.com/help/user-groups)
+- [Restrict message editing and deletion](https://zulip.com/help/restrict-message-editing-and-deletion)
+- [Bots overview](https://zulip.com/help/bots-overview)
+
+## 5. Understand security for user-uploaded content and user-generated requests.
+
+- Zulip supports user-uploaded files. Ideally they should be hosted
+  from a separate domain from the main Zulip server to protect against
+  various same-domain attacks (e.g., zulip-user-content.example.com).
+
+  We support two ways of hosting them: the basic `LOCAL_UPLOADS_DIR`
+  file storage backend, where they are stored in a directory on the
+  Zulip server's filesystem, and the S3 backend, where the files are
+  stored in Amazon S3. It would not be difficult to add additional
+  supported backends should there be a need; see
+  `zerver/lib/upload.py` for the full interface.
+
+  For both backends, the URLs used to access uploaded files are long,
+  random strings, providing one layer of security against unauthorized
+  users accessing files uploaded in Zulip (an authorized user would
+  need to share the URL with an unauthorized user in order for the
+  file to be accessed by the unauthorized user. Of course, any
+  such authorized user could have just downloaded and sent the file
+  instead of the URL, so this is arguably pretty good protection.)
+
+  However, to help protect against accidental sharing of URLs to
+  restricted files (e.g., by forwarding a missed-message email or leaks
+  involving the Referer header), every access to an uploaded file has
+  access control verified (confirming that the browser is logged into
+  a Zulip account that has received the uploaded file in question).
+
+- Zulip supports using the [go-camo][go-camo] image proxy to proxy content like
+  inline image previews, that can be inserted into the Zulip message feed by
+  other users. This ensures that clients do not make requests to external
+  servers to fetch images, improving privacy.
+
+- By default, Zulip will provide image previews inline in the body of
+  messages when a message contains a link to an image. You can
+  control this using the `INLINE_IMAGE_PREVIEW` setting.
+
+- Zulip may make outgoing HTTP connections to other servers in a
+  number of cases:
+
+  - Outgoing webhook bots (creation of which can be restricted)
+  - Inline image previews in messages (enabled by default, but can be disabled)
+  - Inline webpage previews and embeds (must be configured to be enabled)
+  - Twitter message previews (must be configured to be enabled)
+  - BigBlueButton, Constructor Groups, Nextcloud Talk and Zoom API requests
+    (must be configured to be enabled)
+  - Mobile push notifications (must be configured to be enabled)
+
+- Notably, these first 3 features give end users (limited) control to cause
+  the Zulip server to make HTTP requests on their behalf. Because of this,
+  Zulip routes all outgoing HTTP requests [through
+  Smokescreen][smokescreen-setup] to ensure that Zulip cannot be
+  used to execute [SSRF attacks][ssrf] against other systems on an
+  internal corporate network. The default Smokescreen configuration
+  denies access to all non-public IP addresses, including 127.0.0.1.
+
+  The Camo image server does not, by default, route its traffic
+  through Smokescreen, since Camo includes logic to deny access to
+  private subnets; this can be [overridden][proxy.enable_for_camo].
+
+[go-camo]: https://github.com/cactus/go-camo
+[ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
+[smokescreen-setup]: deployment.md#customizing-the-outgoing-http-proxy
+[proxy.enable_for_camo]: system-configuration.md#enable_for_camo
+
+## 6. Understand Zulip's rate-limiting system.
+
+Zulip has built-in rate limiting of login attempts, all access to the
+API, as well as certain other types of actions that may be involved in
+abuse. For example, the email confirmation flow, by its nature, needs
+to allow sending an email to an email address that isn't associated
+with an existing Zulip account. Limiting the ability of users to
+trigger such emails helps prevent bad actors from damaging the spam
+reputation of a Zulip server by sending confirmation emails to random
+email addresses.
+
+The default rate limiting rules for a Zulip server will change as we improve
+the product. A server administrator can browse the current rules using
+`/home/zulip/deployments/current/scripts/get-django-setting
+RATE_LIMITING_RULES`; or with comments by reading
+`DEFAULT_RATE_LIMITING_RULES` in `zproject/default_settings.py`.
+
+Server administrators can tweak rate limiting in the following ways in
+`/etc/zulip/settings.py`:
+
+- The `RATE_LIMITING` setting can be set to `False` to completely
+  disable all rate-limiting.
+- The `RATE_LIMITING_RULES` setting can be used to override specific
+  rules. See the comment in the file for more specific details on how
+  to do it. After changing the setting, we recommend using
+  `/home/zulip/deployments/current/scripts/get-django-setting
+RATE_LIMITING_RULES` to verify your changes. You can then restart
+  the Zulip server with `scripts/restart-server` to have the new
+  configuration take effect.
+- The `RATE_LIMIT_TOR_TOGETHER` setting can be set to `True` to group all
+  known exit nodes of [TOR](https://www.torproject.org/) together for purposes
+  of IP address limiting. Since traffic from a client using TOR is distributed
+  across its exit nodes, without enabling this setting, TOR can otherwise be
+  used to avoid IP-based rate limiting. The updated list of TOR exit nodes
+  is refetched once an hour.
+- If a user runs into the rate limit for login attempts, a server
+  administrator can clear this state using the
+  `manage.py reset_authentication_attempt_count`
+  [management command][management-commands].
+
+See also our [API documentation on rate limiting][rate-limit-api].
+
+[management-commands]: ../production/management-commands.md
+[rate-limit-api]: https://zulip.com/api/rest-error-handling#rate-limit-exceeded

docs/production/settings.md

@@ -0,0 +1,103 @@
+# Server configuration
+
+This page explains the configuration options in `/etc/zulip/settings.py`, the
+main configuration file used by system administrators to configure their Zulip
+server.
+
+[Organization administrators][user-roles] can also [configure][realm-admin-docs]
+many options for a Zulip organization from the web or desktop app. See [system
+and deployment configuration documentation](system-configuration.md) for advanced
+configuration of the various services that make up a complete Zulip installation
+(`/etc/zulip/zulip.conf`).
+
+[realm-admin-docs]: https://zulip.com/help/moving-to-zulip
+[user-roles]: https://zulip.com/help/user-roles
+
+:::{important}
+
+If you are using [Docker](docker.md), see
+{doc}`docker:how-to/compose-settings` for configuring `settings.py`
+via `SETTING_*` environment variables, or
+{doc}`docker:how-to/compose-manual-configuration` to edit it
+directly.
+
+:::
+
+## Server settings overview
+
+The Zulip server self-documents more than a hundred settings in the
+organized comments in `/etc/zulip/settings.py`. You can read [the
+latest version of the settings.py template][settings-py-template] in a
+browser.
+
+[settings-py-template]: https://github.com/zulip/zulip/blob/main/zproject/prod_settings_template.py
+
+Important settings in `/etc/zulip/settings.py` include:
+
+- The mandatory `EXTERNAL_HOST` and `ZULIP_ADMINISTRATOR` settings,
+  which are prefilled by the [installer](install.md).
+- [Authentication methods](authentication-methods.md), including data
+  synchronization options like LDAP and SCIM.
+- The [email gateway](email-gateway.md), which lets
+  users send emails into Zulip.
+- [Video and voice call integrations](video-calls.md).
+- How the server should store [uploaded files](upload-backends.md).
+
+## Changing server settings
+
+To change any of the settings in `/etc/zulip/settings.py`, modify and save the
+file on your Zulip server, and restart the server with the following command:
+
+```bash
+su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
+```
+
+If you have questions about how to configure your server, best-effort community
+support is available in the [Zulip development community][chat-zulip-org].
+Contact [sales@zulip.com](mailto:sales@zulip.com) to learn about paid support
+options.
+
+[chat-zulip-org]: https://zulip.com/development-community/
+
+## Customizing user onboarding
+
+### Navigation tour video
+
+New users are offered a [2-minute video
+tour](https://static.zulipchat.com/static/navigation-tour-video/zulip-10.mp4),
+which shows where to find everything to get started with Zulip.
+
+You can modify `NAVIGATION_TOUR_VIDEO_URL` in `/etc/zulip/settings.py` in order
+to host the official video on your network, or to provide your own introductory
+video with information on how your organization uses Zulip. A value of `None`
+disables the navigation tour video experience.
+
+### Terms of Service and Privacy policy
+
+:::{important}
+
+If you are using this feature, please make sure the name of your organization
+appears prominently in all documents, to avoid confusion with policies for Zulip
+Cloud.
+
+:::
+
+Zulip lets you configure your server's Terms of Service and
+Privacy Policy pages.
+
+Policy documents are stored as Markdown files in the configured
+`POLICIES_DIRECTORY`. We recommend using `/etc/zulip/policies` as the directory,
+so that your policies are naturally backed up with the server's other
+configurations.
+
+To provide Terms of Service and a Privacy Policy for your users, place Markdown
+files named `terms.md` and `privacy.md` in the configured directory, and set
+`TERMS_OF_SERVICE_VERSION` to `1.0` to enable this feature.
+
+You can put additional files in the same directory to document other
+policies; if you do so, you may want to:
+
+- Create a Markdown file `sidebar_index.md` listing the pages in your
+  policies site; this generates the policies site navigation.
+- Create a Markdown file `missing.md` with custom content for 404s in
+  this directory.

docs/production/ssl-certificates.md

@@ -0,0 +1,244 @@
+# Installing SSL certificates
+
+To keep your communications secure, Zulip runs over HTTPS only.
+You'll need an SSL/TLS certificate.
+
+Fortunately, since about 2017, new options can make getting and
+maintaining a genuine, trusted-by-browsers certificate no longer the
+chore (nor expense) that it used to be.
+
+:::{important}
+
+If you are using [Docker](docker.md), see
+{doc}`docker:how-to/compose-ssl` for the `CERTIFICATES` mechanism,
+or {doc}`docker:how-to/helm-ssl` for Helm's Ingress-based TLS.
+
+:::
+
+## Manual install
+
+If you already have an SSL certificate, just install (or symlink) its
+files into place at the following paths:
+
+- `/etc/ssl/private/zulip.key` for the private key
+- `/etc/ssl/certs/zulip.combined-chain.crt` for the certificate.
+
+Your certificate file should contain not only your own certificate but
+its **full chain, including any intermediate certificates** used by
+your certificate authority (CA). See the [nginx
+documentation][nginx-chains] for details on what this means. If
+you're missing part of the chain, your server may work with some
+browsers, but not others and not the Zulip mobile and desktop apps.
+The desktop apps support [configuring a custom CA][desktop-certs] to
+allow validation of certificates generated by an internal CA.
+
+[nginx-chains]: https://nginx.org/en/docs/http/configuring_https_servers.html#chains
+
+### Testing
+
+Just trying in a browser is not an adequate test, because some
+browsers ignore errors that others don't.
+
+Two good tests include:
+
+- If your server is accessible from the public Internet, use the [SSL
+  Labs tester][ssllabs-tester]. Be sure to check for "Chain issues";
+  if any, your certificate file is missing intermediate certificates.
+
+- Alternatively, run a command like `curl -SsI https://zulip.example.com`
+  (using your server's URL) from a machine that can reach your server.
+  Make sure that on the same machine,
+  `curl -SsI https://incomplete-chain.badssl.com` gives an error;
+  `curl` on some machines, including Macs, will accept incomplete
+  chains.
+
+[ssllabs-tester]: https://www.ssllabs.com/ssltest/analyze.html
+
+## Certbot (recommended)
+
+[Let's Encrypt](https://letsencrypt.org/) is a free, completely
+automated CA launched in 2016 to help make HTTPS routine for the
+entire Web. Zulip offers a simple automation for
+[Certbot](https://certbot.eff.org/), a Let's Encrypt client, to get
+SSL certificates from Let's Encrypt and renew them automatically.
+
+We recommend most Zulip servers use Certbot. You'll want something
+else if:
+
+- you have an existing workflow for managing SSL certificates
+  that you prefer;
+- you need wildcard certificates (support from Let's Encrypt released
+  in [March 2018][letsencrypt-wildcard]); or
+- your Zulip server is not on the public Internet. (In this case you
+  can [still use Certbot][certbot-manual-mode], but it's less
+  convenient; and you'll want to ignore Zulip's automation.)
+
+[letsencrypt-wildcard]: https://certbot.eff.org/faq#does-let-s-encrypt-issue-wildcard-certificates
+[certbot-manual-mode]: https://certbot.eff.org/docs/using.html#manual
+
+### At initial Zulip install
+
+To enable the Certbot automation when first installing Zulip, just
+pass the `--certbot` flag when [running the install script][doc-install-script].
+
+The `--hostname` and `--email` options are required when using
+`--certbot`. You'll need the hostname to be a real DNS name, and the
+Zulip server machine to be reachable by that name from the public
+Internet.
+
+If you need to configure a multiple domain certificate, you can generate
+one as described in the section below after installing Zulip.
+
+[doc-install-script]: install.md#step-2-install-zulip
+
+### After Zulip is already installed
+
+To enable the Certbot automation on an already-installed Zulip
+server, run the following commands:
+
+```bash
+sudo -s  # If not already root
+/home/zulip/deployments/current/scripts/setup/setup-certbot --email=EMAIL HOSTNAME [HOSTNAME2...]
+```
+
+where HOSTNAME is the domain name users see in their browser when
+using the server (e.g., `zulip.example.com`), and EMAIL is a contact
+address for the server admins. Additional hostnames can also be
+specified to issue a certificate for multiple domains.
+
+### How it works
+
+When the Certbot automation in Zulip is first enabled, by either
+method, it creates an account for the server at the Let's Encrypt CA;
+requests a certificate for the given hostname; proves to the CA that
+the server controls the website at that hostname; and is then given a
+certificate. (For details, refer to
+[Let's Encrypt](https://letsencrypt.org/how-it-works/).)
+
+### Renewal
+
+Let's Encrypt certificates expire after 90 days. Short expiration
+periods are good for security, but they also mean that it's important
+to automatically renew them to avoid regular maintenance work.
+
+Zulip configures automatic renewal for you. As a result, a Zulip
+server configured with Certbot does not require any ongoing work to
+maintain a current valid SSL certificate.
+
+The `certbot` package configures a systemd timer (similar to a cron
+job) that will renew any Certbot certificates that are due for
+renewal. The renewal process repeats the Certbot proof-of-control
+process, receives the new certificate from Certbot, installs the new
+certificate, and then reloads `nginx`.
+
+#### Troubleshooting
+
+If your Certbot certificate expires, it is usually because of firewall
+rules preventing the Certbot renewal process (which is essentially
+identical to the initial certificate request process) from
+working. You can debug interactively by running the command from the
+cron job, `/usr/bin/certbot renew`, as `root`.
+
+## Self-signed certificate
+
+If you aren't able to use Certbot, you can generate a self-signed SSL
+certificate. This can be convenient for testing, but isn't
+recommended for production, as it is insecure. The Zulip desktop and
+mobile apps will not connect to a server if they cannot validate its
+SSL certificate. The desktop apps support [configuring a custom
+certificate authority][desktop-certs] to allow validation of an
+internal certificate.
+
+To generate a self-signed certificate when first installing Zulip,
+just pass the `--self-signed-cert` flag when
+[running the install script][doc-install-script].
+
+To generate a self-signed certificate for an already-installed Zulip
+server, run the following commands:
+
+```bash
+sudo -s  # If not already root
+/home/zulip/deployments/current/scripts/setup/generate-self-signed-cert HOSTNAME
+```
+
+where HOSTNAME is the domain name (or IP address) to use on the
+generated certificate.
+
+After replacing the certificates, you need to reload `nginx` by
+running the following as `root`:
+
+```bash
+service nginx reload
+```
+
+[desktop-certs]: https://zulip.com/help/custom-certificates
+
+## Troubleshooting
+
+### The Android app can't connect to the server
+
+This is most often caused by an incomplete certificate chain. See
+discussion in the [Manual install](#manual-install) section above.
+
+### The iOS app can't connect to the server
+
+This can be caused by a server set up to support only TLS 1.1 or
+older (including TLS 1.0, SSL 3, or SSL 2.)
+
+TLS 1.2 has been a standard for over 10 years, and all modern web
+server software supports it. Starting in early 2020, all major
+browsers [will _require_ TLS 1.2 or later][tls12-required-news], and
+will refuse to connect over TLS 1.1 or older. And on iOS, Apple [has
+since iOS 9][apple-ats] required TLS 1.2 for all connections made by
+apps, unless the app specifically opts into lower security.
+
+[tls12-required-news]: https://www.thesslstore.com/blog/apple-microsoft-google-disable-tls-1-0-tls-1-1/
+[apple-ats]: https://developer.apple.com/library/archive/releasenotes/General/WhatsNewIniOS/Articles/iOS9.html
+
+If your server is reachable from the public Internet, a convenient way
+to check what TLS versions it supports is the [SSL Labs
+tester][ssllabs-tester].
+
+To resolve this issue, update your server to support TLS 1.2,
+and preferably also TLS 1.3. For nginx, see [the `ssl_protocols`
+directive][nginx-doc-protocols] in your configuration.
+
+[nginx-doc-protocols]: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols
+
+### The Android app connects to the server on some devices but not others
+
+An issue on Android 7.0 ([report][android7.0-tls-issue],
+[description][android7.0-tls-issue-so]) in the system TLS/SSL stack,
+which the Zulip app relies on, makes it finicky about the server's TLS
+configuration.
+
+[android7.0-tls-issue]: https://issuetracker.google.com/issues/37122132#comment13
+[android7.0-tls-issue-so]: https://stackoverflow.com/a/42047877
+
+The issue is that Android 7.0 supports only the curve `secp256r1` when
+doing elliptic-curve cryptography for TLS, and not other curves like
+`secp384r1` or `secp512r1`. If your server's TLS/SSL configuration
+offers only other curves, then Android 7.0 clients will be unable to
+connect.
+
+By default `nginx` (and therefore a Zulip server) offers the
+`secp256r1` curve among others, and so everything works. You can
+control the offered curves with `ssl_ecdh_curve` in the `nginx`
+configuration on your server. See [nginx docs][nginx-doc-curve] for
+details.
+
+[nginx-doc-curve]: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ecdh_curve
+
+Two signs for diagnosing this issue in contrast to some other root
+cause:
+
+- This issue affects only Android 7.0; it's fixed in Android 7.1.1 and
+  later.
+
+- If your server is reachable from the public Internet, use the [SSL
+  Labs tester][ssllabs-tester]. Under "Cipher Suites" you may see
+  lines beginning with `TLS_ECDHE`, for cipher suites which use
+  elliptic-curve cryptography. These lines will have further text
+  like `ECDH secp256r1` or `ECDH secp384r1`, which identifies specific
+  elliptic curves your server offers to use. This issue applies if
+  your server does not offer `secp256r1`.

docs/production/system-configuration.md

@@ -0,0 +1,546 @@
+## System configuration
+
+The file `/etc/zulip/zulip.conf` is an [INI
+format](https://en.wikipedia.org/wiki/INI_file) configuration file
+used to configure properties of the system and deployment;
+`/etc/zulip/settings.py` is used to [configure the application
+itself](settings.md). The `zulip.conf` sections and settings are
+described below. Changes to `zulip.conf` generally do not take effect
+until you run `zulip-puppet-apply` as root:
+
+```console
+# /home/zulip/deployments/current/scripts/zulip-puppet-apply
+```
+
+The `zulip-puppet-apply` command will display the configuration
+changes it will make and prompt for you to confirm you'd like to make
+those changes, before executing them (if you approve).
+
+:::{important}
+
+If you are using [Docker](docker.md), see
+{doc}`docker:how-to/compose-settings` and
+{doc}`docker:reference/environment-vars` for configuring
+`zulip.conf` via `CONFIG_*` environment variables.
+
+:::
+
+### Truthy values
+
+When a setting refers to "set to true" or "set to false", the values
+`true` and `false` are canonical, but any of the following values will
+be considered "true", case-insensitively:
+
+- 1
+- y
+- t
+- yes
+- true
+- enable
+- enabled
+
+Any other value (including the empty string) is considered false.
+
+### `[machine]`
+
+#### `puppet_classes`
+
+A comma-separated list of the Puppet classes to install on the server.
+The most common is **`zulip::profile::standalone`**, used for a
+stand-alone single-host deployment.
+[Components](../overview/architecture-overview.md#components) of
+that include:
+
+- **`zulip::profile::app_frontend`**
+- **`zulip::profile::memcached`**
+- **`zulip::profile::postgresql`**
+- **`zulip::profile::rabbitmq`**
+- **`zulip::profile::redis`**
+- **`zulip::profile::smokescreen`**
+
+If you are using a [Apache as a single-sign-on
+authenticator](authentication-methods.md#apache-based-sso-with-remote_user),
+you will need to add **`zulip::apache_sso`** to the list.
+
+#### `pgroonga`
+
+Set to true if enabling the [multi-language PGroonga search
+extension](../subsystems/full-text-search.md#multi-language-full-text-search).
+
+#### `timesync`
+
+What time synchronization daemon to use; defaults to `chrony`, but also supports
+`ntpd` and `none`. Installations should not adjust this unless they are aligning
+with a fleet-wide standard of `ntpd`. `none` is only reasonable in containers
+like LXC which do not allow adjustment of the clock; a Zulip server will not
+function correctly without an accurate clock.
+
+### `[deployment]`
+
+#### `deploy_options`
+
+Options passed by `upgrade-zulip` and `upgrade-zulip-from-git` into
+`upgrade-zulip-stage-2`. These might be any of:
+
+- **`--skip-puppet`** skips doing Puppet/apt upgrades. The user will need
+  to run `zulip-puppet-apply` manually after the upgrade.
+- **`--skip-migrations`** skips running database migrations. The
+  user will need to run `./manage.py migrate` manually after the upgrade.
+- **`--skip-purge-old-deployments`** skips purging old deployments;
+  without it, only deployments with the last two weeks are kept.
+
+Generally installations will not want to set any of these options; the
+`--skip-*` options are primarily useful for reducing upgrade downtime
+for servers that are upgraded frequently by core Zulip developers.
+
+#### `git_repo_url`
+
+Default repository URL used when [upgrading from a Git
+repository](upgrade.md#upgrading-from-a-git-repository).
+
+### `[application_server]`
+
+#### `http_only`
+
+If set to true, [configures Zulip to allow HTTP access][using-http];
+use if Zulip is deployed behind a reverse proxy that is handling
+SSL/TLS termination.
+
+[using-http]: reverse-proxies.md#configuring-zulip-to-allow-http
+
+#### `nginx_listen_port`
+
+Set to the port number if you [prefer to listen on a port other than
+443](deployment.md#using-an-alternate-port).
+
+#### `nginx_server_name`
+
+Sets the [`server_name`][nginx_server_name] directive on Zulip's
+nginx `server` blocks; by default it is unset, so they match any
+hostname. Set it if you run other software alongside Zulip whose
+nginx configuration would otherwise claim requests meant for Zulip.
+List every hostname Zulip is served on, separated by spaces,
+including those on its TLS certificate; otherwise automatic
+certificate renewal will fail.
+
+If Zulip is served on more than one hostname, or on a hostname plus
+several subdomains, list them all; nginx's `server_name` supports a
+trailing wildcard for matching subdomains. For example, to serve
+Zulip on both `example.com` and any subdomain of it:
+
+```ini
+[application_server]
+nginx_server_name = example.com *.example.com
+```
+
+[nginx_server_name]: https://nginx.org/en/docs/http/server_names.html
+
+#### `nginx_worker_processes`
+
+Adjusts the [`worker_processes`][nginx_worker_processes] setting in
+the nginx server. This defaults to `auto`, which nginx treats as the
+number of available CPU cores.
+
+[nginx_worker_processes]: https://nginx.org/en/docs/ngx_core_module.html#worker_processes
+
+#### `nginx_worker_connections`
+
+Adjust the [`worker_connections`][nginx_worker_connections] setting in
+the nginx server. This defaults to 10000; increasing it allows more
+concurrent connections per worker process, at the cost of more memory
+consumed by nginx. This number, times the number of worker processes
+(above), should be more than twice the concurrent number of users.
+
+[nginx_worker_connections]: http://nginx.org/en/docs/ngx_core_module.html#worker_connections
+
+#### `queue_workers_multiprocess`
+
+By default, Zulip automatically detects whether the system has enough
+memory to run Zulip queue processors in the higher-throughput but more
+multiprocess mode (or to save 1.5GiB of RAM with the multithreaded
+mode). The calculation is based on whether the system has enough
+memory (currently 3.5GiB) to run a single-server Zulip installation in
+the multiprocess mode.
+
+Set explicitly to true or false to override the automatic
+calculation. This override is useful for Docker systems, where the
+above algorithm will see the host's memory size, not the container's,
+unless a [memory limit is set][docker-memory-limit], as well as when
+using remote servers for PostgreSQL, memcached, Redis, and RabbitMQ.
+
+[docker-memory-limit]: https://docs.docker.com/reference/compose-file/deploy/#memory
+
+#### `dedicated_soft_reactivation_queue`
+
+When a long-term-idle user returns, Zulip backfills the message-history
+rows that were skipped while they were idle; this "soft reactivation"
+can take many seconds. By default it runs in the shared `deferred_work`
+queue, alongside jobs such as data exports that can take minutes. Set
+this to true to process soft reactivations in a dedicated queue (and, in
+multiprocess mode, a dedicated worker process), so they aren't delayed
+behind such jobs. This costs an additional worker process, so it is most
+useful on large servers; smaller servers should leave it disabled.
+
+#### `rolling_restart`
+
+If set to true, when using `./scripts/restart-server` to restart
+Zulip, restart the uwsgi processes one-at-a-time, instead of all at
+once. This decreases the number of 502's served to clients, at the
+cost of slightly increased memory usage, and the possibility that
+different requests will be served by different versions of the code.
+
+#### `service_file_descriptor_limit`
+
+The number of file descriptors which [Supervisor is configured to allow
+processes to use][supervisor-minfds]; defaults to 40000. If your Zulip deployment
+is very large (hundreds of thousands of concurrent users), your Django processes
+hit this limit and refuse connections to clients. Raising it above this default
+may require changing system-level limits, particularly if you are using a
+virtualized environment (e.g., Docker, or Proxmox LXC).
+
+[supervisor-minfds]: http://supervisord.org/configuration.html?highlight=minfds#supervisord-section-values
+
+#### `s3_memory_cache_size`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the in-memory size of the cache _index_; the default is 1MB,
+which is enough to store about 8 thousand entries.
+
+#### `s3_disk_cache_size`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the on-disk size of the cache _contents_; the default is
+200MB.
+
+#### `s3_cache_inactive_time`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the longest amount of time an entry will be cached since last
+use; the default is 30 days. Since the contents of the cache are
+immutable, this serves only as a potential additional limit on the
+size of the contents on disk; `s3_disk_cache_size` is expected to be
+the primary control for cache sizing.
+
+#### `thumbnail_workers`
+
+How many image-thumbnailing workers to run. Defaults to 1; adding more
+workers can prevent the image-thumbnailing queue backlogging when
+large numbers of very large image files are uploaded at once. (When
+backlogged, image previews for images that have not yet been
+thumbnailed will appear as loading spinners).
+
+#### `email_senders_workers`
+
+How many email-sending workers to run. Defaults to 1; adding more
+workers can prevent email-sending queue backlogging when large numbers
+of very large emails are enqueued at once. This is generally only
+necessary on quite large installs.
+
+#### `nameserver`
+
+When the [S3 storage backend][s3-backend] is in use, downloads from S3 are
+proxied from nginx, whose configuration requires an explicit value of a DNS
+nameserver to resolve the S3 server's hostname. Zulip defaults to using the
+resolver found in `/etc/resolv.conf`; this setting overrides any value found
+there.
+
+[s3-backend]: upload-backends.md
+
+#### `uwsgi_listen_backlog_limit`
+
+Override the default uwsgi backlog of 128 connections.
+
+#### `uwsgi_processes`
+
+Override the default `uwsgi` (Django) process count. It defaults to a sliding
+scale between 3 workers for hosts with under 3GB RAM, up to 16 workers for hosts
+with more than 24GB of RAM.
+
+#### `access_log_retention_days`
+
+Number of days of access logs to keep, for both nginx and the application.
+Defaults to 14 days.
+
+#### `katex_server`
+
+Set to a false value to disable the separate service for [rendering math with
+LaTeX](https://zulip.com/help/latex). Disabling this service will save a small
+amount of memory, at the cost of making math blocks significantly slower to
+render, to the point that using dozens of them in a single message may cause
+the message to fail to send.
+
+#### `katex_server_port`
+
+Set to the port number for the KaTeX server; defaults to port 9700.
+
+#### `custom_ca_path`
+
+If you use a custom certificate authority for your authentication
+provider, you will need to provide the certificate of the signing CA
+so Zulip can successfully make requests to it.
+
+Set this to the fully-qualified path to the `.crt` file containing the
+PEM-encoded CA certificate to trust; we suggest storing this in
+`/etc/zulip/`.
+
+Setting this path also means that Zulip will use the operating
+system's CA certificate store, and not its built-in one. There may be
+minor differences in trusted root CA sets.
+
+### `[postgresql]`
+
+#### `effective_io_concurrency`
+
+Override PostgreSQL's [`effective_io_concurrency`
+setting](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-EFFECTIVE-IO-CONCURRENCY).
+
+#### `listen_addresses`
+
+Override PostgreSQL's [`listen_addresses`
+setting](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-LISTEN-ADDRESSES).
+
+#### `random_page_cost`
+
+Override PostgreSQL's [`random_page_cost`
+setting](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-RANDOM-PAGE-COST).
+Zulip defaults this value to 1.1, which is an appropriate value for
+SSDs; if your server uses spinning disks, you should set this back to
+the upstream default of 4.0.
+
+#### `replication_primary`
+
+On the [warm standby replicas](postgresql.md#postgresql-warm-standby), set to the
+hostname of the primary PostgreSQL server that streaming replication
+should be done from.
+
+#### `replication_user`
+
+On the [warm standby replicas](postgresql.md#postgresql-warm-standby), set to the
+username that the host should authenticate to the primary PostgreSQL
+server as, for streaming replication. Authentication will be done
+based on the `pg_hba.conf` file; if you are using password
+authentication, you can set a `postgresql_replication_password` secret
+for authentication.
+
+#### `skip_backups`
+
+If set to as true value, inhibits the nightly [`wal-g` backups][wal-g] which
+would be taken on all non-replicated hosts and [all warm standby
+replicas](postgresql.md#postgresql-warm-standby). This is generally only set if you have
+multiple warm standby replicas, in order to avoid taking multiple backups, one
+per replica.
+
+[wal-g]: export-and-import.md#database-only-backup-tools
+
+#### `backups_disk_concurrency`
+
+Number of concurrent disk reads to use when taking backups. Defaults to 1; you
+may wish to increase this if you are taking backups on a replica, so can afford
+to affect other disk I/O, and have an SSD which is good at parallel random
+reads.
+
+#### `backups_directory`
+
+If S3 secrets are not configured, perform daily database backups to this path on
+disk instead. It should be owned by the `postgres` user.
+
+This option is not recommended for disaster recovery purposes, since unless the
+directory is on a different disk from the database itself, _backups will likely
+also be lost if the database is lost._ This setting can be useful if the path is
+on a NAS mountpoint, or if some other process copies this data off the disk; or
+if backups are purely for point-in-time historical analysis of recent
+application-level data changes.
+
+#### `backups_incremental`
+
+The number of delta (incremental) database backups to take between full backups.
+Defaults to 0 for S3 backups, and 6 for local-disk backups.
+
+#### `backups_storage_class`
+
+What [storage class](https://aws.amazon.com/s3/storage-classes/) to use when
+uploading database backups. Defaults to `STANDARD`, meaning "[S3
+standard][s3-standard]", but many deployments will have overall lower costs if
+"[S3 Standard - Infrequent Access][s3-ia]" is used, via the `STANDARD_IA`
+value. Also supported is "[S3 Reduced Redundancy][s3-rr]", by setting
+`REDUCED_REDUNDANCY`, but this is not suggested for production use.
+
+[s3-standard]: https://aws.amazon.com/s3/storage-classes/#General_purpose
+[s3-ia]: https://aws.amazon.com/s3/storage-classes/#Infrequent_access
+[s3-rr]: https://aws.amazon.com/s3/reduced-redundancy/
+
+#### `backups_compression_method`
+
+What compression method to use when storing backups; defaults to `lz4`, which is
+fast but does not compress particularly well. Other options are `lzma`, `zstd`,
+and `brotli`; `lzma` provides the best (and slowest) compression, while `zstd`
+and `brotli` are middling compromises.
+
+#### `missing_dictionaries`
+
+If set to a true value during initial database creation, uses PostgreSQL's
+standard `pg_catalog.english` text search configuration, rather than Zulip's
+improved set of stopwords. Has no effect after initial database construction.
+
+#### `ssl_ca_file`
+
+Set to the path to the PEM-encoded certificate authority used to
+authenticate client connections.
+
+#### `ssl_cert_file`
+
+Set to the path to the PEM-encoded public certificate used to secure
+client connections.
+
+#### `ssl_key_file`
+
+Set to the path to the PEM-encoded private key used to secure client
+connections.
+
+#### `ssl_mode`
+
+The mode that should be used to verify the server certificate. The
+PostgreSQL default is `prefer`, which provides no security benefit; we
+strongly suggest setting this to `require` or better if you are using
+certificate authentication. See the [PostgreSQL
+documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS)
+for potential values.
+
+#### `version`
+
+The version of PostgreSQL that is in use. Do not set by hand; use the
+[PostgreSQL upgrade tool](upgrade.md#upgrading-postgresql).
+
+### `[memcached]`
+
+#### `memory`
+
+Override the number of megabytes of memory that memcached should be
+configured to consume; defaults to 1/8th of the total server memory.
+
+#### `max_item_size`
+
+Override the maximum size that an item in memcached can store. This defaults to
+1m; adjusting it should only be necessary if your Zulip server has organizations
+which have more than 20k users.
+
+#### `size_reporting`
+
+Set to a true value to enable object size reporting in memcached. This incurs a
+small overhead for every store or delete operation, but allows a
+memcached_exporter to report precise item size distribution.
+
+### `[tornado_sharding]`
+
+Keys in this section are used to configure how many Tornado instances
+are started, and which users are mapped to which of those instances.
+Each Tornado instance listens on a separate port, starting at 9800 and
+proceeding upwards from there. A single Tornado instance can usually
+handle 1000-1500 concurrent active users, depending on message sending
+volume.
+
+Individual organizations may be assigned to ports, either via their
+subdomain names, or their fully-qualified hostname (for [organizations
+using `REALM_HOSTS`](multiple-organizations.md#other-hostnames)):
+
+```ini
+[tornado_sharding]
+9800 = realm-a realm-b
+9801 = realm-c
+9802 = realm-host.example.net
+```
+
+Organizations can also be assigned to ports via regex over their
+fully-qualified hostname:
+
+```ini
+[tornado_sharding]
+9800_regex = ^realm-(a|b)\.example\.com$
+9801_regex = ^other(-option)?\.example.com$
+```
+
+Extremely large organizations can be distributed across multiple
+Tornado shards by joining the ports in the key with `_`:
+
+```ini
+[tornado_sharding]
+9800 = small-realm
+9801_9802 = very-large-realm
+```
+
+After running `scripts/zulip-puppet-apply`, a separate step to run
+`scripts/refresh-sharding-and-restart` is required for any sharding
+changes to take effect.
+
+### `[loadbalancer]`
+
+#### `ips`
+
+Comma-separated list of IP addresses or netmasks of external load balancers
+whose `X-Forwarded-For` and `X-Forwarded-Proto` should be respected. These can
+be individual IP addresses, or CIDR IP address ranges.
+
+#### `rejects_http_requests`
+
+Set to a true value if incoming requests from load loadbalancer's IP addresses
+which do not contain an `X-Forwarded-Proto` should be assumed to have come into
+them over HTTPS. This setting _is a security vulnerability_ unless the load
+balancer unilaterally rejects unencrypted HTTP connections, or responds to them
+with 301 status codes. Note that Zulip's HSTS headers are not sufficient
+protection here, since API clients do not respect them; the load balancer _must
+not_ send any requests to Zulip which came in unencrypted.
+
+### `[http_proxy]`
+
+See "[Customizing the outgoing HTTP
+proxy](deployment.md#customizing-the-outgoing-http-proxy)" for general
+instructions on the outgoing proxy.
+
+#### `host`
+
+The hostname or IP address of an outgoing HTTP `CONNECT`. Defaults to
+`localhost` if unspecified.
+
+#### `port`
+
+The TCP port of the HTTP `CONNECT` proxy on the host specified above.
+Defaults to `4750` if unspecified.
+
+#### `listen_address`
+
+The IP address that Smokescreen should bind to and listen on.
+Defaults to `127.0.0.1`.
+
+#### `enable_for_camo`
+
+Because Camo includes logic to deny access to private subnets, routing
+its requests through Smokescreen is generally not necessary. Set to
+true or false to override the default, which uses the proxy only if
+it is not the default of Smokescreen on a local host.
+
+#### `allow_addresses`, `allow_ranges`, `deny_addresses`, `deny_ranges`
+
+Comma-separated lists of IP addresses or CIDR range rules. All
+private IP addresses (e.g., 127.0.0.0/8, 192.168.0.0/16) are denied by
+default; allow rules override deny rules.
+
+### `[sentry]`
+
+#### `organization`
+
+The Sentry organization used for the [Sentry deploy hook](deployment.md#sentry-deploy-hook).
+
+#### `project`
+
+The Sentry project used for the [Sentry deploy hook](deployment.md#sentry-deploy-hook).
+
+### `[email_gateway]`
+
+#### `listen`
+
+The address for the [incoming email gateway's](email-gateway.md) SMTP
+server to bind to, as a bare port number, or `address:port`; IPv6
+addresses must be enclosed in square brackets (e.g.,
+`[2001:db8::1]:25`). Defaults to listening on port 25 on all
+addresses, both IPv4 and IPv6. This only applies to the local
+delivery mode of the gateway, not the polling mode.

docs/production/troubleshooting.md

@@ -0,0 +1,353 @@
+# Troubleshooting and monitoring
+
+This page offers detailed guidance for troubleshooting and monitoring your Zulip
+installation. If you suspect that you have encountered a bug, or are otherwise
+unable to resolve an issue with your Zulip installation, best-effort community
+support is available in the [Zulip development community ][chat-zulip-org]. We
+provide free, interactive support for the vast majority of questions about
+running a Zulip server.
+
+To report a problem or ask a question, please start a new topic in the
+[#production help][production-help] channel in the [Zulip development community
+][chat-zulip-org]:
+
+- Describe what you are trying to do and any problems you've encountered.
+- Provide the relevant logs, such as the full traceback from the bottom
+  of `/var/log/zulip/errors.log`, or the installation script logs at
+  `/var/log/zulip/install.log`. Please post logging output using [code
+  blocks][code-block], not screenshots.
+- Be sure to include what version of Zulip Server you are running, or between
+  which versions you are upgrading.
+
+[chat-zulip-org]: https://zulip.com/development-community/
+[production-help]: https://chat.zulip.org/#narrow/channel/31-production-help
+[code-block]: https://zulip.com/help/code-blocks
+
+Contact [sales@zulip.com](mailto:sales@zulip.com) if you'd like to
+learn about paid support options, including phone support from Zulip's
+core engineering team.
+
+## Overview and resources
+
+If you encounter issues while self-hosting Zulip, the first thing to
+do is look at Zulip's logs, which are located in `/var/log/zulip/`.
+
+That directory contains one log file for each service, plus
+`errors.log` (has all errors and the first place you should check),
+`server.log` (has logs from the Django and Tornado servers), and
+`workers.log` (has combined logs from the queue workers). Zulip also
+provides a [tool to search through `server.log`][log-search].
+
+Zulip uses [Supervisor](http://supervisord.org/index.html) to monitor
+and control its many Python services. Read the next section, [Using
+supervisorctl](#using-supervisorctl), to learn how to use the
+Supervisor client to monitor and manage services.
+
+If you haven't already, now might be a good time to read Zulip's [architectural
+overview](../overview/architecture-overview.md), particularly the
+[Components](../overview/architecture-overview.md#components) section. This will help you
+understand the many services Zulip uses.
+
+The section [troubleshooting services](#troubleshooting-services)
+on this page includes details about how to fix common issues with Zulip services.
+
+[log-search]: ../subsystems/logging.md#searching-backend-log-files
+
+## Using supervisorctl
+
+:::{important}
+
+If you are using [Docker](docker.md), `supervisorctl` runs inside
+the container; see {doc}`docker:how-to/compose-commands` and
+{doc}`docker:how-to/helm-commands` for shell access and log
+retrieval.
+
+:::
+
+To see what Zulip-related services are configured to
+use Supervisor, look at `/etc/supervisor/conf.d/zulip.conf` and
+`/etc/supervisor/conf.d/zulip-db.conf`.
+
+Use the supervisor client `supervisorctl` to list the status of, stop, start,
+and restart various services.
+
+### Checking status with `supervisorctl status`
+
+You can check if the Zulip application is running using:
+
+```bash
+supervisorctl status
+```
+
+When everything is running as expected, you will see something like this:
+
+```console
+process-fts-updates                                             RUNNING   pid 11392, uptime 19:40:06
+smokescreen                                                     RUNNING   pid 3113, uptime 29 days, 21:58:32
+zulip-django                                                    RUNNING   pid 11441, uptime 19:39:57
+zulip-tornado                                                   RUNNING   pid 11397, uptime 19:40:03
+zulip_deliver_scheduled_emails                                  RUNNING   pid 10289, uptime 19:41:04
+zulip_deliver_scheduled_messages                                RUNNING   pid 10294, uptime 19:41:02
+zulip-workers:zulip_events_deferred_work                        RUNNING   pid 10314, uptime 19:41:00
+zulip-workers:zulip_events_digest_emails                        RUNNING   pid 10339, uptime 19:40:57
+zulip-workers:zulip_events_email_mirror                         RUNNING   pid 10751, uptime 19:40:52
+zulip-workers:zulip_events_email_senders                        RUNNING   pid 10769, uptime 19:40:49
+zulip-workers:zulip_events_embed_links                          RUNNING   pid 11035, uptime 19:40:46
+zulip-workers:zulip_events_embedded_bots                        RUNNING   pid 11139, uptime 19:40:43
+zulip-workers:zulip_events_invites                              RUNNING   pid 11261, uptime 19:40:36
+zulip-workers:zulip_events_missedmessage_emails                 RUNNING   pid 11346, uptime 19:40:21
+zulip-workers:zulip_events_missedmessage_mobile_notifications   RUNNING   pid 11351, uptime 19:40:19
+zulip-workers:zulip_events_outgoing_webhooks                    RUNNING   pid 11358, uptime 19:40:17
+zulip-workers:zulip_events_user_activity                        RUNNING   pid 11365, uptime 19:40:14
+zulip-workers:zulip_events_user_activity_interval               RUNNING   pid 11376, uptime 19:40:11
+```
+
+If you see any services showing a status other than `RUNNING`, or you
+see an uptime under 5 seconds (which indicates it's crashing
+immediately after startup and repeatedly restarting), that service
+isn't running. If you don't see relevant logs in
+`/var/log/zulip/errors.log`, check the log file declared via
+`stdout_logfile` for that service's entry in
+`/etc/supervisor/conf.d/zulip.conf` for details. Logs only make it to
+`/var/log/zulip/errors.log` once a service has started fully.
+
+### Restarting services with `supervisorctl restart`
+
+After you change configuration in `/etc/zulip/settings.py` or fix a
+misconfiguration, you will often want to restart the Zulip
+application. In order to restart all of Zulip's services, you can use:
+
+```bash
+/home/zulip/deployments/current/scripts/restart-server
+```
+
+If you want to restart just one of them, you can use
+`supervisorctl`:
+
+```bash
+# You can use this for any service found in `supervisorctl list`
+supervisorctl restart zulip-django
+```
+
+:::{warning}
+A configuration file might be used by multiple services, so generally
+`scripts/restart-server` is the correct tool to use for reloading
+purposes. Only use `supervisorctl restart` for an individual service
+if you're confident that this specific service requires restarting.
+In particular, it is not the right tool for applying settings changes
+in `/etc/zulip/settings.py` and may cause inconsistent behavior.
+:::
+
+### Stopping services with `supervisorctl stop`
+
+Similarly, while stopping all of Zulip is best done by running
+`scripts/stop-server`, you can stop individual Zulip services using:
+
+```bash
+# You can use this for any service found in `supervisorctl list`
+supervisorctl stop zulip-django
+```
+
+## Troubleshooting services
+
+The Zulip application uses several major open source services to store
+and cache data, queue messages, and otherwise support the Zulip
+application:
+
+- PostgreSQL
+- RabbitMQ
+- nginx
+- Redis
+- memcached
+
+If one of these services is not installed or functioning correctly,
+Zulip will not work. Below we detail some common configuration
+problems and how to resolve them:
+
+- If your browser reports no web server is running, that is likely
+  because nginx is not configured properly and thus failed to start.
+  nginx will fail to start if you configured SSL incorrectly or did
+  not provide SSL certificates. To fix this, configure them properly
+  and then run:
+
+  ```bash
+  service nginx restart
+  ```
+
+- If your host is being port scanned by unauthorized users, you may see
+  messages in `/var/log/zulip/server.log` like
+
+  ```text
+  2017-02-22 14:11:33,537 ERROR Invalid HTTP_HOST header: '10.2.3.4'. You may need to add u'10.2.3.4' to ALLOWED_HOSTS.
+  ```
+
+  Django uses the hostnames configured in `ALLOWED_HOSTS` to identify
+  legitimate requests and block others. When an incoming request does
+  not have the correct HTTP Host header, Django rejects it and logs the
+  attempt. For more on this issue, see the [Django release notes on Host header
+  poisoning](https://www.djangoproject.com/weblog/2013/feb/19/security/#s-issue-host-header-poisoning)
+
+- An AMQPConnectionError traceback or error running rabbitmqctl
+  usually means that RabbitMQ is not running; to fix this, try:
+  ```bash
+  service rabbitmq-server restart
+  ```
+  If RabbitMQ fails to start, the problem is often that you are using
+  a virtual machine with broken DNS configuration; you can often
+  correct this by configuring `/etc/hosts` properly.
+
+### Restrict unattended upgrades
+
+:::{important}
+We recommend that you disable or limit Ubuntu's unattended-upgrades
+to skip some server packages. With unattended upgrades enabled but
+not limited, the moment a new PostgreSQL release is published, your
+Zulip server will have its PostgreSQL server upgraded (and thus
+restarted). If you do disable unattended-upgrades, do not forget to
+regularly install apt upgrades manually!
+:::
+
+Restarting one of the system services that Zulip uses (PostgreSQL,
+memcached, Redis, or RabbitMQ) will drop the connections that
+Zulip processes have to the service, resulting in future operations on
+those connections throwing errors.
+
+Zulip is designed to recover from system service downtime by creating
+new connections once the system service is back up, so the Zulip
+outage will end once the system service finishes restarting. But
+you'll get a bunch of error emails during the system service outage
+whenever one of the Zulip server's ~20 workers attempts to access the
+system service.
+
+An unplanned outage will also result in an annoying (and potentially
+confusing) trickle of error emails over the following hours or days.
+These emails happen because a worker only learns its connection was
+dropped when it next tries to access the connection (at which point
+it'll send an error email and make a new connection), and several
+workers are commonly idle for periods of hours or days at a time.
+
+You can prevent this trickle when doing a planned upgrade by
+restarting the Zulip server with
+`/home/zulip/deployments/current/scripts/restart-server` after
+installing system package updates to PostgreSQL, memcached,
+RabbitMQ, or Redis.
+
+You can ensure that the `unattended-upgrades` package never upgrades
+PostgreSQL, memcached, Redis, or RabbitMQ, by configuring in
+`/etc/apt/apt.conf.d/50unattended-upgrades`:
+
+```text
+// Python regular expressions, matching packages to exclude from upgrading
+Unattended-Upgrade::Package-Blacklist {
+    "libc\d+";
+    "memcached$";
+    "nginx-full$";
+    "postgresql-\d+$";
+    "postgresql-\d+-pgdg-pgroonga$";
+    "rabbitmq-server$";
+    "redis-server$";
+    "supervisor$";
+};
+```
+
+## Monitoring
+
+Chat is mission-critical to many organizations. This section contains
+advice on monitoring your Zulip server to minimize downtime.
+
+First, we should highlight that Zulip sends Django error emails to
+`ZULIP_ADMINISTRATOR` for any backend exceptions. A properly
+functioning Zulip server shouldn't send any such emails, so it's worth
+reporting/investigating any that you do see.
+
+Beyond that, the most important monitoring for a Zulip server is
+standard stuff:
+
+- Basic host health monitoring for issues running out of disk space,
+  especially for the database and where uploads are stored.
+- Service uptime and standard monitoring for the [services Zulip
+  depends on](#troubleshooting-services). Most monitoring software
+  has standard plugins for nginx, PostgreSQL, Redis, RabbitMQ,
+  and memcached, and those will work well with Zulip.
+- `supervisorctl status` showing all services `RUNNING`.
+- Checking for processes being OOM killed.
+
+Beyond that, Zulip ships a few application-specific end-to-end health
+checks. The Nagios plugins `check_send_receive_time`,
+`check_rabbitmq_queues`, and `check_rabbitmq_consumers` are generally
+sufficient to point to the cause of any Zulip production issue. See
+the next section for details.
+
+### Nagios configuration
+
+The complete Nagios configuration (sans secret keys) used to
+monitor zulip.com is available under `puppet/kandra` in the
+Zulip Git repository (those files are not installed in the release
+tarballs).
+
+The Nagios plugins used by that configuration are installed
+automatically by the Zulip installation process in subdirectories
+under `/usr/lib/nagios/plugins/`. The following is a summary of the
+useful Nagios plugins included with Zulip and what they check:
+
+Application server and queue worker monitoring:
+
+- `check_send_receive_time`: Sends a test message through the system
+  between two bot users to check that end-to-end message sending
+  works. An effective end-to-end check for Zulip's Django and Tornado
+  systems being healthy.
+- `check_rabbitmq_consumers` and `check_rabbitmq_queues`: Effective
+  checks for Zulip's RabbitMQ-based queuing systems being healthy.
+- `check_worker_memory`: Monitors for memory leaks in queue workers.
+
+Database monitoring:
+
+- `check_fts_update_log`: Checks whether full-text search updates are
+  being processed properly or getting backlogged.
+- `check_postgres`: General checks for database health.
+- `check_postgresql_replication_lag`: Checks whether PostgreSQL streaming
+  replication is up to date.
+
+Standard server monitoring:
+
+- `check_debian_packages`: Checks whether the system is behind on
+  `apt upgrade`.
+
+If you're using these plugins, bug reports and pull requests to make
+it easier to monitor Zulip and maintain it in production are
+encouraged!
+
+## Memory leak mitigation
+
+As a measure to mitigate the potential impact of any future memory
+leak bugs in one of the Zulip daemons, Zulip service automatically
+restarts itself every Sunday early morning. See
+`/etc/cron.d/restart-zulip` for the precise configuration.
+
+## Troubleshooting the Zulip installer
+
+:::{important}
+
+The Zulip installer is designed to be idempotent: if the script fails, once
+you've corrected the cause of the failure, you can just rerun the script.
+
+:::
+
+The install script automatically logs a transcript to
+`/var/log/zulip/install.log`. In case of failure, you might find the
+log handy for resolving the issue. Please include a copy of this log
+file in any bug reports.
+
+If you get an error after `scripts/setup/install` completes, check the bottom of
+`/var/log/zulip/errors.log` for a traceback, and consult [the rest of this
+page](#troubleshooting-and-monitoring) for advice on how to debug or get help.
+
+### The `zulip` user's password.
+
+By default, the `zulip` user doesn't
+have a password, and is intended to be accessed by `su zulip` from the
+`root` user (or via SSH keys or a password, if you want to set those
+up, but that's up to you as the system administrator). Most people
+who are prompted for a password when running `su zulip` turn out to
+already have switched to the `zulip` user earlier in their session,
+and can just skip that step.

docs/production/upgrade.md

@@ -0,0 +1,929 @@
+# Upgrade Zulip
+
+This page explains how to upgrade a Zulip server, including:
+
+- [Upgrading to a release](#upgrading-to-a-release)
+- [Upgrading from a Git repository](#upgrading-from-a-git-repository)
+- [Updating `settings.py` inline documentation](#updating-settingspy-inline-documentation)
+- [Troubleshooting and rollback](#troubleshooting-and-rollback)
+- [Preserving local changes to service configuration files](#preserving-local-changes-to-service-configuration-files)
+- [Upgrading PostgreSQL](#upgrading-postgresql)
+- [Upgrading the operating system](#upgrading-the-operating-system)
+
+We recommend always running [the latest Zulip server
+release](../overview/release-lifecycle.md#stable-releases). We work extremely
+hard to make sure these releases are stable and have no regressions, and that
+the upgrade process Just Works.
+
+If you do have any questions or problems with your upgrade process, best-effort
+community support is available in the [Zulip development
+community](https://zulip.com/development-community/).
+
+:::{note}
+
+For professional support, upgrade to [Zulip
+Business](https://zulip.com/plans/#self-hosted), or reach out to
+[sales@zulip.com](mailto:sales@zulip.com).
+
+:::
+
+## Upgrading to a release
+
+:::{important}
+
+Be sure to follow the additional instructions if you're {doc}`using
+Docker <docker:how-to/compose-upgrading>`, have [patched
+Zulip](modify.md), or have [modified Zulip-managed configuration
+files](#preserving-local-changes-to-service-configuration-files).
+
+:::
+
+To upgrade to a new Zulip release:
+
+1. Read the [upgrade notes](../overview/changelog.md#upgrade-notes) for all
+   releases between your current release and the one you're upgrading to.
+
+1. [Download](https://download.zulip.com/server/) the appropriate release
+   tarball. You can get the latest release (**Zulip Server
+   {{ LATEST_RELEASE_VERSION }}**) with the following command:
+
+   ```bash
+   curl -fLO https://download.zulip.com/server/zulip-server-latest.tar.gz
+   ```
+
+1. Log in to your Zulip, and run as root:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-latest.tar.gz
+   ```
+
+Once the Zulip upgrade is complete, you may also wish to [upgrade the version of
+PostgreSQL](#upgrading-postgresql).
+
+### What to expect during an upgrade
+
+The upgrade process will:
+
+1. Run `apt-get upgrade`.
+1. Install new versions of Zulip's dependencies (mainly Python packages).
+1. Shut down the Zulip service.
+1. Run a `puppet apply`.
+1. Run any database migrations.
+1. Bring the Zulip service back up on the new version.
+
+Upgrading will result in brief downtime for the service, which should
+be under 30 seconds unless there is an expensive database migration
+involved. Such migrations will be documented in the [release
+notes](../overview/changelog.md), and can usually can be avoided with
+some care. If downtime is problematic for your organization,
+consider testing the upgrade on a
+[backup](export-and-import.md#backups) in advance,
+doing the final upgrade at off hours, or arranging support.
+
+:::{note}
+
+If you run into any issues or need to roll back the upgrade, see the
+[troubleshooting guide](#troubleshooting-and-rollback).
+
+:::
+
+## Upgrading from a Git repository
+
+:::{important}
+
+If your deployment is using Docker, please follow {ref}`these
+instructions <docker:how-to/compose-upgrading:upgrading from a git repository>`.
+
+:::
+
+Zulip supports upgrading a production installation to any commit in a Git
+repository. This lets you get unreleased features and bugfixes, or [maintain a
+fork](modify.md#making-changes) of Zulip.
+
+### Upgrading to an unreleased version of Zulip
+
+The [git versions](../overview/release-lifecycle.md#git-versions) documentation
+describes some branches you may choose to upgrade to, depending on your
+requirements.
+
+If you are considering upgrading to `main`, see our [upgrading to `main`
+guide](modify.md#upgrading-to-main) for detailed information. You can also
+[apply a small change](modify.md#applying-changes-from-main) to get a fix for an
+issue that matters for your organization.
+
+To upgrade to a branch from Git, simply run:
+
+```bash
+# Upgrade to an official release
+/home/zulip/deployments/current/scripts/upgrade-zulip-from-git 11.5
+
+# Upgrade to a maintenance branch
+/home/zulip/deployments/current/scripts/upgrade-zulip-from-git 11.x
+
+# Upgrade to the Zulip Cloud branch
+/home/zulip/deployments/current/scripts/upgrade-zulip-from-git zulip-cloud-current
+
+# Upgrade to the `main` branch
+/home/zulip/deployments/current/scripts/upgrade-zulip-from-git main
+```
+
+Zulip will automatically fetch the relevant Git commit and upgrade to
+that version of Zulip.
+
+In addition to the steps described [above](#what-to-expect-during-an-upgrade),
+the upgrade process will also build Zulip's frontend assets using `webpack`.
+
+#### Upgrading from another repository
+
+By default, this uses the main upstream Zulip server repository, but
+you can configure any other Git repository by adding a section like
+this to `/etc/zulip/zulip.conf`:
+
+```ini
+[deployment]
+git_repo_url = https://github.com/zulip/zulip.git
+```
+
+## Updating `settings.py` inline documentation
+
+Zulip installations often upgrade many times over their lifetime, and we strive
+to keep all configuration files backwards-compatible. The
+`/etc/zulip/settings.py` file is not automatically modified during upgrades.
+
+After an upgrade, new features may thus be missing from that file, as it is
+based on a template provided by an earlier version of Zulip.
+
+After upgrading across major versions of Zulip Server, we recommend
+comparing your `/etc/zulip/settings.py` file to the current settings
+template, which can be found in
+`/home/zulip/deployments/current/zproject/prod_settings_template.py`. We
+suggest using that updated template to update
+`/etc/zulip/settings.py`:
+
+1. Copy the current `settings.py` to make a backup (especially if you
+   do not have a recent [complete backup][backups]), and make a copy
+   of the current template:
+
+   ```bash
+   cp -a /etc/zulip/settings.py ~/zulip-settings-backup.py
+   cp -a /home/zulip/deployments/current/zproject/prod_settings_template.py /etc/zulip/settings-new.py
+   ```
+
+1. Open both `/etc/zulip/settings.py` and `/etc/zulip/settings-new.py`
+   files in an editor; for each setting set in `settings.py`, find its
+   section in `/etc/zulip/settings-new.py` and copy the setting from
+   `settings.py` into there.
+
+   To help with this process, the following tool finds the most likely version
+   of the template that your `/etc/zulip/settings.py` was installed from, and
+   shows the modifications you've made:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/compare-settings-to-template
+   ```
+
+   If there are settings which you cannot find documented in
+   `/etc/zulip/settings-new.py`, check the [changelog][changelog] to see
+   if they have been removed.
+
+1. Overwrite the configuration with the updated file, and restart the
+   server to pick up the updates; this should be a no-op, but it is
+   much better to discover immediately if it is not:
+
+   ```bash
+   cp -a /etc/zulip/settings-new.py /etc/zulip/settings.py
+   su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
+   ```
+
+[backups]: export-and-import.md#backups
+[changelog]: ../overview/changelog.md
+
+## Troubleshooting and rollback
+
+The upgrade scripts are idempotent, so there's no harm in trying again
+after resolving an issue. The most common causes of errors are:
+
+- Networking issues (e.g., your Zulip server doesn't have reliable
+  Internet access or needs a proxy set up). Fix the networking issue
+  and try again.
+- Especially when using `upgrade-zulip-from-git`, systems with the
+  minimal RAM for running Zulip can run into out-of-memory issues
+  during the upgrade process (generally `tools/webpack` is the step
+  that fails). You can get past this by shutting down the Zulip
+  server with `./scripts/stop-server` to free up RAM before running
+  the upgrade process.
+
+Useful logs are available in a few places:
+
+- The Zulip upgrade scripts log all output to
+  `/var/log/zulip/upgrade.log`.
+- The Zulip server logs all Internal Server Errors to
+  `/var/log/zulip/errors.log`.
+
+See also the general Zulip server [troubleshooting guide](troubleshooting.md).
+
+### Rolling back to a prior version
+
+This rollback process is intended for minor releases (e.g., `9.4` to
+`9.3`); a more complicated process is required to roll back database
+migrations before downgrading to an older major release.
+
+The Zulip upgrade process works by creating a new deployment under
+`/home/zulip/deployments/` containing a complete copy of the Zulip server code,
+and then moving the symlinks at `/home/zulip/deployments/{current,last,next}`
+as part of the upgrade process.
+
+This means that if the new version isn't working,
+you can quickly downgrade to the old version by running
+`/home/zulip/deployments/last/scripts/restart-server`, or to an
+earlier previous version by running
+`/home/zulip/deployments/DATE/scripts/restart-server`. The
+`restart-server` script stops any running Zulip server, and starts
+the version corresponding to the `restart-server` path you call.
+
+## Deployment hooks
+
+Zulip's upgrades have a hook system which allows for arbitrary
+user-configured actions to run before and after an upgrade.
+
+Files in the `/etc/zulip/pre-deploy.d` and `/etc/zulip/post-deploy.d`
+directories are inspected for files ending with `.hook`, just before and after
+the critical period when the server is restarted. Files are called from the
+working directory of the new version in alphabetical order, with
+environment variables as described below. If any of them exit with non-0 exit
+code, the upgrade will abort.
+
+The hook is run with the following environment variables set:
+
+- `ZULIP_OLD_VERSION`: The version being upgraded from, which may either be a
+  release name (e.g., `10.0` or `10.0-beta1`) or the output from `git describe`
+  (e.g., `10.0-beta1-2-abcd158b18f2`).
+- `ZULIP_NEW_VERSION`: The version being upgraded to, in the same format as
+  `ZULIP_OLD_VERSION`.
+
+If the upgrade is upgrading between [versions in `git`][upgrade-from-git], then
+the following environment variables will also be present:
+
+- `ZULIP_OLD_COMMIT`: The full commit hash of the version being upgraded from.
+- `ZULIP_NEW_COMMIT`: The full commit hash of the version being upgraded to.
+- `ZULIP_OLD_MERGE_BASE_COMMIT`: The full commit hash of the merge-base of the
+  version being upgraded from, and the public branch in
+  [`zulip/zulip`][zulip/zulip]. This will be the closest commit in standard
+  Zulip Server to the version being upgraded from.
+- `ZULIP_NEW_MERGE_BASE_COMMIT`: The full commit hash of the merge-base of the
+  version being upgraded to, and the public branch in
+  [`zulip/zulip`][zulip/zulip]. This will be the closest commit in standard
+  Zulip Server to the version being upgraded to.
+
+See the [deploy documentation](deployment.md#deployment-hooks) for
+hooks included with Zulip.
+
+[upgrade-from-git]: #upgrading-from-a-git-repository
+[zulip/zulip]: https://github.com/zulip/zulip/
+
+## Preserving local changes to service configuration files
+
+:::{warning}
+If you have modified service configuration files installed by
+Zulip (e.g., the nginx configuration), the Zulip upgrade process will
+overwrite your configuration when it does the `puppet apply`.
+:::
+
+You can test whether any files will be overwritten assuming no upstream changes
+to the configuration using `scripts/zulip-puppet-apply` (without the `-f`
+option), which will do a test Puppet run and output and changes it would make.
+Using this list, you can save a copy of any files that you've modified, do the
+upgrade, and then restore your configuration.
+
+That said, Zulip's configuration files are designed to be flexible
+enough for a wide range of installations, from a small self-hosted
+system to Zulip Cloud. Before making local changes to a configuration
+file, first check whether there's an option supported by
+`/etc/zulip/zulip.conf` for the customization you need. And if you
+need to make local modifications, please report the issue so that we
+can look into making the Zulip Puppet configuration flexible enough to
+handle your setup.
+
+### nginx configuration changes
+
+If you need to extend Zulip's `nginx` configuration, there are a few different
+include directories you can use, in different [contexts][context]:
+
+- `/etc/nginx/conf.d` is in the [`http` context][http-context]
+- `/etc/nginx/zulip-include/app.d` is in the [`server` context][server-context]
+  for the public-facing server
+- `/etc/nginx/zulip-include/localhost.d` is in the [`server`
+  context][server-context] for the server listening on `127.0.0.1:80`, which is
+  used for internal inter-process communication
+
+[context]: http://nginx.org/en/docs/beginners_guide.html#conf_structure
+[http-context]: http://nginx.org/en/docs/http/ngx_http_core_module.html#http
+[server-context]: http://nginx.org/en/docs/http/ngx_http_core_module.html#server
+
+## Upgrading PostgreSQL
+
+:::{important}
+
+If you are using [Docker](docker.md), PostgreSQL is upgraded via
+image bumps rather than the `upgrade-postgresql` script; see
+{doc}`docker:how-to/compose-upgrading` and
+{doc}`docker:reference/versioning`.
+
+:::
+
+The major version of PostgreSQL is upgraded separately from the Zulip
+server version. Further, the version of PostgreSQL included with a
+Zulip server is not linked to that of the host OS; the Zulip installer
+uses the latest available version of PostgreSQL at installation time
+(currently, version 16).
+
+The following table details which versions each major Zulip Server
+version supports:
+
+```{include} postgresql-support-table.md
+
+```
+
+To upgrade the version of PostgreSQL on the Zulip server:
+
+1. Upgrade your Zulip server, at least to the latest Zulip maintenance
+   release for your major Zulip version (e.g., upgrade 9.1 to
+   9.4). This ensures you're using the most robust version of the
+   PostgreSQL upgrade tool.
+
+1. Stop the server, as the `zulip` user:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/stop-server
+   ```
+
+1. Take a backup, in case of any problems:
+
+   ```bash
+   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/postgresql-upgrade.backup.tar.gz
+   ```
+
+1. As root, run the database upgrade tool:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/upgrade-postgresql
+   ```
+
+1. As the `zulip` user, start the server again:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/start-server
+   ```
+
+You should now be able to navigate to the Zulip server's URL and
+confirm everything is working correctly.
+
+## Upgrading the operating system
+
+:::{important}
+
+If you are using [Docker](docker.md), the operating system is part
+of the container image and is upgraded with it; the instructions
+below do not apply.
+
+:::
+
+When you upgrade the operating system on which Zulip is installed
+(e.g., Ubuntu 22.04 to Ubuntu 24.04), you need to take some additional
+steps to update your Zulip installation, documented below.
+
+The steps are largely the same for the various OS upgrades aside from
+the versions of PostgreSQL, so you should be able to adapt these
+instructions for other supported platforms.
+
+### Upgrading from Ubuntu 22.04 Jammy to 24.04 Noble
+
+1. Upgrade your server to the latest Zulip `8.x` release (at
+   least 8.3, which adds support for Ubuntu 24.04).
+
+1. As the Zulip user, stop the Zulip server and run the following
+   to back up the system:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/stop-server
+   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
+   ```
+
+1. One of Zulip's dependencies, RabbitMQ, is used to store deferred work
+   in queues. RabbitMQ's Ubuntu packaging has [problems][ubuntu-rabbitmq-bug]
+   upgrading from version 3.9 in Ubuntu 22.04 to 3.12 in Ubuntu
+   24.04. To work around this bug, you'll need to uninstall
+   `rabbitmq-server`, purging its database, before upgrading the OS;
+   the steps after the OS upgrade will reinstall the new version and
+   configure it properly. You can do this uninstallation process
+   safely via the following process:
+
+   1. As root, run:
+      ```bash
+      rabbitmqctl list_queues
+      ```
+      to check whether any of Zulip's RabbitMQ queues contain
+      unprocessed events.
+   1. If any queues contain events, you can run as the `zulip` user
+      ```bash
+      /home/zulip/deployments/current/manage.py process_queue --all
+      ```
+      ...which will start all workers consuming any remaining events.
+      You should cancel this (with ^C) once `rabitmqctl list_queues`
+      shows that no queues contain events anymore. You can also decide
+      to skip this step if you're OK losing a bit of data of the
+      relevant type.
+   1. As root, run `apt purge rabbitmq-server` to remove the RabbitMQ
+      package, including, critically, its database and configuration
+      state, which would otherwise cause installation of the Ubuntu
+      24.04 package to crash.
+
+1. Switch to the root user and upgrade the operating system, following
+   the prompts until it completes successfully:
+
+   ```bash
+   sudo -i # Or otherwise get a root shell
+   do-release-upgrade
+   ```
+
+   When `do-release-upgrade` asks you how to upgrade configuration
+   files for services that Zulip manages like Redis, PostgreSQL,
+   nginx, and memcached, the best choice is `N` to keep the
+   currently installed version. But it's not important; the next
+   step will re-install Zulip's configuration in any case.
+
+   The `do-release-upgrade` tool will complete by prompting you to
+   restart the system; press `N`, as we will do so later.
+
+1. Next, we need to reinstall the current version of Zulip, which
+   among other things will recompile Zulip's Python module
+   dependencies for your new version of Python and rewrite Zulip's
+   full-text search indexes to work with the upgraded dictionary
+   packages. This will also take care of re-installing and re-configuring
+   RabbitMQ which we removed earlier.
+
+   ```bash
+   rm -rf /srv/zulip-venv-cache/* /home/zulip/deployments/current/.venv /root/.cache/uv
+   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
+       /home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
+   ```
+
+   This process may show a dialog about a "pending kernel upgrade",
+   which can safely be ignored. It may also prompt about "daemons
+   using outdated libraries"; you should select "cancel".
+
+1. As root, upgrade the database to the latest version of PostgreSQL:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/upgrade-postgresql
+   ```
+
+1. As root, restart the server:
+
+   ```bash
+   reboot
+   ```
+
+You should now be able to navigate to your Zulip server's URL and
+confirm everything is working correctly.
+
+[ubuntu-rabbitmq-bug]: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/2074309
+
+### Upgrading from Ubuntu 20.04 Focal to 22.04 Jammy
+
+1. Upgrade your server to the latest Zulip `5.x` release (at
+   least 5.3, which adds support for Ubuntu 22.04 and above).
+
+2. As the Zulip user, stop the Zulip server and run the following
+   to back up the system:
+
+   ```bash
+   supervisorctl stop all
+   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
+   ```
+
+3. Switch to the root user and upgrade the operating system, following
+   the prompts until it completes successfully:
+
+   ```bash
+   sudo -i # Or otherwise get a root shell
+   do-release-upgrade
+   ```
+
+   When `do-release-upgrade` asks you how to upgrade configuration
+   files for services that Zulip manages like Redis, PostgreSQL,
+   nginx, and memcached, the best choice is `N` to keep the
+   currently installed version. But it's not important; the next
+   step will re-install Zulip's configuration in any case.
+
+   The `do-release-upgrade` tool will complete by prompting you to
+   restart the system; press `N`, as we will do so later.
+
+4. Next, we need to reinstall the current version of Zulip, which
+   among other things will recompile Zulip's Python module
+   dependencies for your new version of Python and rewrite Zulip's
+   full-text search indexes to work with the upgraded dictionary
+   packages:
+
+   ```bash
+   rm -rf /srv/zulip-venv-cache/*
+   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
+       /home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
+   ```
+
+   This process may show a dialog about a "pending kernel upgrade",
+   which can safely be ignored. It may also prompt about "daemons
+   using outdated libraries"; you should select "cancel".
+
+5. As root, upgrade the database to the latest version of PostgreSQL:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/upgrade-postgresql
+   ```
+
+6. As root, restart the server:
+
+   ```bash
+   reboot
+   ```
+
+You should now be able to navigate to your Zulip server's URL and
+confirm everything is working correctly.
+
+### Upgrading from Ubuntu 18.04 Bionic to 20.04 Focal
+
+1. Upgrade your server to the latest Zulip `3.x` or `4.x` release (at
+   least 3.0, which adds support for Ubuntu 20.04). You can only
+   upgrade to Zulip 5.0 and newer after completing this process, since
+   newer releases don't support Ubuntu 18.04.
+
+2. As the Zulip user, stop the Zulip server and run the following
+   to back up the system:
+
+   ```bash
+   supervisorctl stop all
+   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
+   ```
+
+3. Switch to the root user and upgrade the operating system, following
+   the prompts until it completes successfully:
+
+   ```bash
+   sudo -i # Or otherwise get a root shell
+   do-release-upgrade
+   ```
+
+   When `do-release-upgrade` asks you how to upgrade configuration
+   files for services that Zulip manages like Redis, PostgreSQL,
+   nginx, and memcached, the best choice is `N` to keep the
+   currently installed version. But it's not important; the next
+   step will re-install Zulip's configuration in any case.
+
+   The `do-release-upgrade` tool will complete by prompting you to
+   restart the system; press `N`, as we will do so later.
+
+4. Next, we need to reinstall the current version of Zulip, which
+   among other things will recompile Zulip's Python module
+   dependencies for your new version of Python and rewrite Zulip's
+   full-text search indexes to work with the upgraded dictionary
+   packages:
+
+   ```bash
+   rm -rf /srv/zulip-venv-cache/*
+   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
+       /home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
+   ```
+
+5. As root, upgrade the database to the latest version of PostgreSQL:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/upgrade-postgresql
+   ```
+
+6. Finally, Ubuntu 20.04 has a different version of the low-level
+   glibc library, which affects how PostgreSQL orders text data (known
+   as "collations"); this corrupts database indexes that rely on
+   collations. Regenerate the affected indexes by running:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
+   ```
+
+7. As root, restart the server:
+
+   ```bash
+   reboot
+   ```
+
+8. [Upgrade from Ubuntu 20.04 to
+   22.04](#upgrading-from-ubuntu-2004-focal-to-2204-jammy), so that
+   you are running a supported operating system.
+
+### Upgrading from Ubuntu 16.04 Xenial to 18.04 Bionic
+
+1. Upgrade your server to the latest Zulip `2.1.x` release. You can
+   only upgrade to Zulip 3.0 and newer after completing this process,
+   since newer releases don't support Ubuntu 16.04 Xenial.
+
+2. Same as for Ubuntu 18.04 to 20.04.
+
+3. Same as for Ubuntu 18.04 to 20.04.
+
+4. As root, upgrade the database installation and OS configuration to
+   match the new OS version:
+
+   ```bash
+   touch /usr/share/postgresql/10/pgroonga_setup.sql.applied
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply -f
+   pg_dropcluster 10 main --stop
+   systemctl stop postgresql
+   pg_upgradecluster 9.5 main
+   pg_dropcluster 9.5 main
+   apt remove postgresql-9.5
+   systemctl start postgresql
+   systemctl restart memcached
+   ```
+
+5. Finally, we need to reinstall the current version of Zulip, which
+   among other things will recompile Zulip's Python module
+   dependencies for your new version of Python:
+
+   ```bash
+   rm -rf /srv/zulip-venv-cache/*
+   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
+       /home/zulip/deployments/current/ --ignore-static-assets
+   ```
+
+   This will finish by restarting your Zulip server; you should now
+   be able to navigate to its URL and confirm everything is working
+   correctly.
+
+6. [Upgrade to the latest `4.x` release](#upgrading-to-a-release).
+
+7. As root, verify the contents of the full-text indexes:
+
+   ```bash
+   /home/zulip/deployments/current/manage.py audit_fts_indexes
+   ```
+
+8. [Upgrade from Ubuntu 18.04 to
+   20.04](#upgrading-from-ubuntu-1804-bionic-to-2004-focal), the next
+   in chain of upgrades leading to a supported operating system.
+
+### Upgrading from Debian 12 to 13
+
+1. Upgrade your server to the latest `11.x` release.
+
+1. As the Zulip user, stop the Zulip server and run the following
+   to back up the system:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/stop-server
+   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
+   ```
+
+1. One of Zulip's dependencies, RabbitMQ, is used to store deferred work
+   in queues. RabbitMQ's Debian packaging has [problems][debian-rabbitmq-bug]
+   upgrading from version 3.10 in Debian 12 to 4.0 in Debian 13. To
+   work around this bug, you'll need to uninstall `rabbitmq-server`,
+   purging its database, before upgrading the OS; the steps after the
+   OS upgrade will reinstall the new version and configure it
+   properly. You can do this uninstallation process safely via the
+   following process:
+
+   1. As root, run:
+      ```bash
+      rabbitmqctl list_queues
+      ```
+      to check whether any of Zulip's RabbitMQ queues contain
+      unprocessed events.
+   1. If any queues contain events, you can run as the `zulip` user
+      ```bash
+      /home/zulip/deployments/current/manage.py process_queue --all
+      ```
+      ...which will start all workers consuming any remaining events.
+      You should cancel this (with ^C) once `rabitmqctl list_queues`
+      shows that no queues contain events anymore. You can also decide
+      to skip this step if you're OK losing a bit of data of the
+      relevant type.
+   1. As root, run `apt purge rabbitmq-server` to remove the RabbitMQ
+      package, including, critically, its database and configuration
+      state, which would otherwise cause installation of the Debian 13
+      package to crash.
+
+1. Follow [Debian's instructions to upgrade the OS][trixie-upgrade].
+
+   [trixie-upgrade]: https://www.debian.org/releases/trixie/release-notes/upgrading.en.html
+
+   When prompted for you how to upgrade configuration
+   files for services that Zulip manages like Redis, PostgreSQL,
+   nginx, and memcached, the best choice is `N` to keep the
+   currently installed version. But it's not important; the next
+   step will re-install Zulip's configuration in any case.
+
+1. Next, we need to reinstall the current version of Zulip, which
+   among other things will recompile Zulip's Python module
+   dependencies for your new version of Python and rewrite Zulip's
+   full-text search indexes to work with the upgraded dictionary
+   packages. This will also take care of re-installing and re-configuring
+   RabbitMQ which we removed earlier.
+
+   ```bash
+   rm -rf /srv/zulip-venv-cache/* /home/zulip/deployments/current/.venv /root/.cache/uv
+   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
+       /home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
+   ```
+
+1. As an additional step, you can also [upgrade the PostgreSQL version](#upgrading-postgresql).
+
+1. As root, restart the server:
+
+   ```bash
+   reboot
+   ```
+
+You should now be able to navigate to your Zulip server's URL and
+confirm everything is working correctly.
+
+[debian-rabbitmq-bug]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165
+
+### Upgrading from Debian 11 to 12
+
+1. Upgrade your server to the latest `7.x` release.
+
+2. As the Zulip user, stop the Zulip server and run the following
+   to back up the system:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/stop-server
+   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
+   ```
+
+3. Follow [Debian's instructions to upgrade the OS][bookworm-upgrade].
+
+   [bookworm-upgrade]: https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.html
+
+   When prompted for you how to upgrade configuration
+   files for services that Zulip manages like Redis, PostgreSQL,
+   nginx, and memcached, the best choice is `N` to keep the
+   currently installed version. But it's not important; the next
+   step will re-install Zulip's configuration in any case.
+
+4. As root, run the following steps to regenerate configurations
+   for services used by Zulip:
+
+   ```bash
+   apt remove upstart -y
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply -f
+   ```
+
+5. Reinstall the current version of Zulip, which among other things
+   will recompile Zulip's Python module dependencies for your new
+   version of Python:
+
+   ```bash
+   rm -rf /srv/zulip-venv-cache/*
+   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
+       /home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
+   ```
+
+   This will finish by restarting your Zulip server; you should now
+   be able to navigate to its URL and confirm everything is working
+   correctly.
+
+6. As an additional step, you can also [upgrade the PostgreSQL version](#upgrading-postgresql).
+
+### Upgrading from Debian 10 to 11
+
+1. Upgrade your server to the latest `5.x` release. You can only
+   upgrade to Zulip Server 6.0 and newer after completing this
+   process, since newer releases don't support Debian 10.
+
+2. As the Zulip user, stop the Zulip server and run the following
+   to back up the system:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/stop-server
+   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
+   ```
+
+3. Follow [Debian's instructions to upgrade the OS][bullseye-upgrade].
+
+   [bullseye-upgrade]: https://www.debian.org/releases/bullseye/amd64/release-notes.en.txt
+
+   When prompted for you how to upgrade configuration
+   files for services that Zulip manages like Redis, PostgreSQL,
+   nginx, and memcached, the best choice is `N` to keep the
+   currently installed version. But it's not important; the next
+   step will re-install Zulip's configuration in any case.
+
+4. As root, run the following steps to regenerate configurations
+   for services used by Zulip:
+
+   ```bash
+   apt remove upstart -y
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply -f
+   ```
+
+5. Reinstall the current version of Zulip, which among other things
+   will recompile Zulip's Python module dependencies for your new
+   version of Python:
+
+   ```bash
+   rm -rf /srv/zulip-venv-cache/*
+   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
+       /home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
+   ```
+
+   This will finish by restarting your Zulip server; you should now
+   be able to navigate to its URL and confirm everything is working
+   correctly.
+
+6. Debian 11 has a different version of the low-level glibc
+   library, which affects how PostgreSQL orders text data (known as
+   "collations"); this corrupts database indexes that rely on
+   collations. Regenerate the affected indexes by running:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
+   ```
+
+7. As an additional step, you can also [upgrade the PostgreSQL version](#upgrading-postgresql).
+
+8. [Upgrade from Debian 11 to 12](#upgrading-from-debian-11-to-12),
+   so that you are running a supported operating system.
+
+### Upgrading from Debian 9 to 10
+
+1. Upgrade your server to the latest Zulip `2.1.x` release. You can
+   only upgrade to Zulip 3.0 and newer after completing this process,
+   since newer releases don't support Debian 9.
+
+2. As the Zulip user, stop the Zulip server and run the following
+   to back up the system:
+
+   ```bash
+   supervisorctl stop all
+   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
+   ```
+
+3. Follow [Debian's instructions to upgrade the OS][debian-upgrade-os].
+
+   [debian-upgrade-os]: https://web.archive.org/web/20230314235744id_/https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.html
+
+   When prompted for you how to upgrade configuration
+   files for services that Zulip manages like Redis, PostgreSQL,
+   nginx, and memcached, the best choice is `N` to keep the
+   currently installed version. But it's not important; the next
+   step will re-install Zulip's configuration in any case.
+
+4. As root, upgrade the database installation and OS configuration to
+   match the new OS version:
+
+   ```bash
+   apt remove upstart -y
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply -f
+   pg_dropcluster 11 main --stop
+   systemctl stop postgresql
+   pg_upgradecluster -m upgrade 9.6 main
+   pg_dropcluster 9.6 main
+   apt remove postgresql-9.6
+   systemctl start postgresql
+   service memcached restart
+   ```
+
+5. Finally, we need to reinstall the current version of Zulip, which
+   among other things will recompile Zulip's Python module
+   dependencies for your new version of Python:
+
+   ```bash
+   rm -rf /srv/zulip-venv-cache/*
+   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
+       /home/zulip/deployments/current/ --ignore-static-assets
+   ```
+
+   This will finish by restarting your Zulip server; you should now
+   be able to navigate to its URL and confirm everything is working
+   correctly.
+
+6. [Upgrade to the latest `5.x` release](#upgrading-to-a-release), now
+   that your server is running a supported operating system.
+
+7. Debian 10 has a different version of the low-level glibc
+   library, which affects how PostgreSQL orders text data (known as
+   "collations"); this corrupts database indexes that rely on
+   collations. Regenerate the affected indexes by running:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
+   ```
+
+8. As root, finish by verifying the contents of the full-text indexes:
+
+   ```bash
+   /home/zulip/deployments/current/manage.py audit_fts_indexes
+   ```
+
+9. [Upgrade from Debian 10 to 11](#upgrading-from-debian-10-to-11),
+   the next in chain of upgrades leading to a supported operating
+   system.

docs/production/upload-backends.md

@@ -0,0 +1,346 @@
+# File upload backends
+
+Zulip in production supports a couple different backends for storing
+files uploaded by users of the Zulip server (messages, profile
+pictures, organization icons, custom emoji, etc.).
+
+The default is the `LOCAL_UPLOADS_DIR` backend, which just stores
+files on disk in the specified directory on the Zulip server.
+Obviously, this backend doesn't work with multiple Zulip servers and
+doesn't scale, but it's great for getting a Zulip server up and
+running quickly. You can later migrate the uploads to S3 by
+[following the instructions here](#migrating-from-local-uploads-to-amazon-s3-backend).
+
+We also support an `S3` backend, which uses the Python `boto` library
+to upload files to Amazon S3 (or an S3-compatible block storage
+provider supported by the `boto` library).
+
+Regardless of the backend you choose, you can configure the maximum
+size of individual uploaded files using the `MAX_FILE_UPLOAD_SIZE`
+[server setting](../production/settings.md). Setting it to 0 disables
+file uploads, and hides the UI for uploading files from the web and
+desktop apps.
+
+## S3 backend configuration
+
+(s3-backend)=
+
+:::{important}
+
+If you are using [Docker](docker.md), see
+{doc}`docker:how-to/compose-settings` for `SETTING_S3_*`
+configuration and {doc}`docker:how-to/compose-secrets` for the
+`s3_key` / `s3_secret_key` secrets. Switching to S3 also keeps
+uploads out of the `/data` volume.
+
+:::
+
+Here, we document the process for configuring Zulip's S3 file upload
+backend. To enable this backend, you need to do the following:
+
+1. In the AWS management console, create a new IAM account (aka API
+   user) for your Zulip server, and two buckets in S3, one for uploaded
+   files included in messages, and another for user avatars. You need
+   two buckets because the "user avatars" bucket is generally configured
+   as world-readable, whereas the "uploaded files" one is not.
+
+1. Set `s3_key` and `s3_secret_key` in /etc/zulip/zulip-secrets.conf
+   to be the S3 access and secret keys for the IAM account.
+   Alternately, if your Zulip server runs on an EC2 instance, set the
+   IAM role for the EC2 instance to the role created in the previous
+   step.
+
+1. Set the `S3_AUTH_UPLOADS_BUCKET` and `S3_AVATAR_BUCKET` settings in
+   `/etc/zulip/settings.py` to be the names of the S3 buckets you
+   created (e.g., `"exampleinc-zulip-uploads"`).
+
+1. Comment out the `LOCAL_UPLOADS_DIR` setting in
+   `/etc/zulip/settings.py` (add a `#` at the start of the line).
+
+1. If you are using a non-AWS block storage provider,
+   you need to set the `S3_ENDPOINT_URL` setting to your
+   endpoint url (e.g., `"https://s3.eu-central-1.amazonaws.com"`).
+
+   For certain AWS regions, you may need to set the `S3_REGION`
+   setting to your default AWS region's code (e.g., `"eu-central-1"`).
+
+1. Non-AWS block storage providers may need `S3_SKIP_CHECKSUM = True`; you
+   should try without this at first, but enable it if you see exceptions
+   involving `XAmzContentSHA256Mismatch`.
+
+1. Finally, restart the Zulip server so that your settings changes
+   take effect
+   (`/home/zulip/deployments/current/scripts/restart-server`).
+
+It's simplest to just do this configuration when setting up your Zulip
+server for production usage. Note that if you had any existing
+uploading files, this process does not upload them to Amazon S3; see
+[migration instructions](#migrating-from-local-uploads-to-amazon-s3-backend)
+below for those steps.
+
+### Google Cloud Platform
+
+In addition to configuring `settings.py` as suggested above:
+
+```python
+S3_AUTH_UPLOADS_BUCKET = "..."
+S3_AVATAR_BUCKET = "..."
+S3_ENDPOINT_URL = "https://storage.googleapis.com"
+S3_SKIP_CHECKSUM = True
+```
+
+...and adding `s3_key` and `s3_secret_key` in `/etc/zulip/zulip-secrets.conf`,
+you will need to also add a `/etc/zulip/gcp_key.json` which contains a [service
+account key][gcp-key] with "Storage Object Admin" permissions on the uploads
+bucket. This is used by the `tusd` chunked upload service when receiving file
+uploads from clients.
+
+[gcp-key]: https://cloud.google.com/iam/docs/keys-create-delete
+
+## S3 local caching
+
+For performance reasons, Zulip stores a cache of recently served user
+uploads on disk locally, even though the durable storage is kept in
+S3. There are a number of parameters which control the size and usage
+of this cache, which is maintained by nginx:
+
+- `s3_memory_cache_size` controls the in-memory size of the cache
+  _index_; the default is 1MB, which is enough to store about 8 thousand
+  entries.
+- `s3_disk_cache_size` controls the on-disk size of the cache
+  _contents_; the default is 200MB.
+- `s3_cache_inactive_time` controls the longest amount of time an
+  entry will be cached since last use; the default is 30 days. Since
+  the contents of the cache are immutable, this serves only as a
+  potential additional limit on the size of the contents on disk;
+  `s3_disk_cache_size` is expected to be the primary control for cache
+  sizing.
+
+These defaults are likely sufficient for small-to-medium deployments.
+Large deployments, or deployments with image-heavy use cases, will
+want to increase `s3_disk_cache_size`, potentially to be several
+gigabytes. `s3_memory_cache_size` should potentially be increased,
+based on estimating the number of files that the larger disk cache
+will hold.
+
+You may also wish to increase the cache sizes if the S3 storage (or
+S3-compatible equivalent) is not closely located to your Zulip server,
+as cache misses will be more expensive.
+
+## nginx DNS nameserver configuration
+
+The S3 cache described above is maintained by nginx. nginx's configuration
+requires an explicitly-set DNS nameserver to resolve the hostname of the S3
+servers; Zulip defaults this value to the first nameserver found in
+`/etc/resolv.conf`, but this resolver can be [adjusted in
+`/etc/zulip/zulip.conf`][s3-resolver] if needed. If you adjust this value, you
+will need to run `/home/zulip/deployments/current/scripts/zulip-puppet-apply` to
+update the nginx configuration for the new value.
+
+[s3-resolver]: system-configuration.md#nameserver
+
+## S3 bucket policy
+
+The best way to do the S3 integration with Amazon is to create a new IAM user
+just for your Zulip server with limited permissions. For both the user uploads
+bucket and the user avatars bucket, you'll need to adjust the [S3 bucket
+policy](https://awspolicygen.s3.amazonaws.com/policygen.html).
+
+The file uploads bucket should have a policy of:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Id": "Policy1468991802320",
+    "Statement": [
+        {
+            "Sid": "Stmt1468991795370",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": [
+                "s3:GetObject",
+                "s3:DeleteObject",
+                "s3:PutObject"
+            ],
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
+        },
+        {
+            "Sid": "Stmt1468991795371",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": "s3:ListBucket",
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE"
+        }
+    ]
+}
+```
+
+The file-uploads bucket should not be world-readable. See the
+[documentation on the Zulip security model](securing-your-zulip-server.md) for
+details on the security model for uploaded files.
+
+However, the avatars bucket is intended to be world-readable, so its
+policy should be:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Id": "Policy1468991802321",
+    "Statement": [
+        {
+            "Sid": "Stmt1468991795380",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": [
+                "s3:GetObject",
+                "s3:DeleteObject",
+                "s3:PutObject"
+            ],
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
+        },
+        {
+            "Sid": "Stmt1468991795381",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": "s3:ListBucket",
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE"
+        },
+        {
+            "Sid": "Stmt1468991795382",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "*"
+            },
+            "Action": "s3:GetObject",
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
+        }
+    ]
+}
+```
+
+## Migrating from local uploads to Amazon S3 backend
+
+As you scale your server, you might want to migrate the uploads from
+your local backend to Amazon S3. Follow these instructions, step by
+step, to do the migration.
+
+1. First, [set up the S3 backend](#s3-backend-configuration) in the settings
+   (all the auth stuff), but leave `LOCAL_UPLOADS_DIR` set -- the
+   migration tool will need that value to know where to find your uploads.
+2. Run `./manage.py transfer_uploads_to_s3`. This will upload all the
+   files from the local uploads directory to Amazon S3. By default,
+   this command runs on 6 parallel processes, since uploading is a
+   latency-sensitive operation. You can control this parameter using
+   the `--processes` option.
+3. Once the transfer script completes, disable `LOCAL_UPLOADS_DIR`, and
+   restart your server (continuing the last few steps of the S3
+   backend setup instructions).
+
+Congratulations! Your uploaded files are now migrated to S3.
+
+**Caveat**: The current version of this tool does not migrate an
+uploaded organization avatar or logo.
+
+## S3 data storage class
+
+In general, uploaded files in Zulip are accessed frequently at first, and then
+age out of frequent access. The S3 backend provides the [S3
+Intelligent-Tiering][s3-it] [storage class][s3-storage-class] which provides
+cheaper storage for less frequently accessed objects, and may provide overall
+cost savings for large deployments.
+
+You can configure Zulip to store uploaded files using Intelligent-Tiering by
+setting `S3_UPLOADS_STORAGE_CLASS` to `INTELLIGENT_TIERING` in `settings.py`.
+This setting can take any of the following [storage class
+value][s3-storage-class-constant] values:
+
+- `STANDARD`
+- `STANDARD_IA`
+- `ONEZONE_IA`
+- `REDUCED_REDUNDANCY`
+- `GLACIER_IR`
+- `INTELLIGENT_TIERING`
+
+Setting `S3_UPLOADS_STORAGE_CLASS` does not affect the storage class of existing
+objects. In order to change those, for example to `INTELLIGENT_TIERING`, perform
+an in-place copy:
+
+    aws s3 cp --storage-class INTELLIGENT_TIERING --recursive \
+        s3://your-bucket-name/ s3://your-bucket-name/
+
+Note that changing the lifecycle of existing objects will incur a [one-time
+lifecycle transition cost][s3-pricing].
+
+[s3-it]: https://aws.amazon.com/s3/storage-classes/intelligent-tiering/
+[s3-storage-class]: https://aws.amazon.com/s3/storage-classes/
+[s3-storage-class-constant]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass
+[s3-pricing]: https://aws.amazon.com/s3/pricing/
+
+## Data export bucket
+
+The [data export process](export-and-import.md#data-export) process, when
+[triggered from the
+UI](https://zulip.com/help/export-your-organization), uploads the
+completed export so it is available to download from the server; this
+is also available [from the command
+line](export-and-import.md#export-your-zulip-data) by passing
+`--upload`. When the S3 backend is used, these uploads are done to S3.
+
+By default, they are uploaded to the bucket with user avatars
+(`S3_AVATAR_BUCKET`), because that bucket is world-readable, allowing
+easy generation of links to download the export.
+
+If you would like to store exports in a dedicated bucket, you can set
+`S3_EXPORT_BUCKET` in your `/etc/zulip/settings.py`. This bucket
+should also be configured like the uploads bucket, only allowing write
+access to the Zulip account, as it will generate links which are valid
+for 1 week at a time:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Id": "Policy1468991802322",
+    "Statement": [
+        {
+            "Sid": "Stmt1468991795390",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": [
+                "s3:GetObject",
+                "s3:DeleteObject",
+                "s3:PutObject"
+            ],
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
+        },
+        {
+            "Sid": "Stmt1468991795391",
+            "Effect": "Allow",
+            "Principal": {
+                "AWS": "ARN_PRINCIPAL_HERE"
+            },
+            "Action": "s3:ListBucket",
+            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE"
+        }
+    ]
+}
+```
+
+You should copy existing exports to the new bucket. For instance,
+using the [AWS CLI](https://aws.amazon.com/cli/)'s [`aws s3
+sync`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html),
+if the old bucket was named `example-zulip-avatars` and the new export
+bucket is named `example-zulip-exports`:
+
+```
+aws s3 sync s3://example-zulip-avatars/exports/ s3://example-zulip-exports/
+```

docs/production/video-calls.md

@@ -0,0 +1,242 @@
+# Video call providers
+
+Zulip makes it convenient to [start a
+call](https://zulip.com/help/start-a-call) with the click of a button, using the
+call provider of your choice. The call providers
+supported by Zulip are:
+
+- [Jitsi Meet](https://zulip.com/integrations/jitsi), a fully-encrypted,
+  100% open source video conferencing solution.
+- [Zoom](https://zulip.com/integrations/zoom)
+- [BigBlueButton](https://zulip.com/integrations/big-blue-button)
+- [Constructor Groups](https://zulip.com/integrations/constructor-groups)
+- [Nextcloud Talk](https://zulip.com/integrations/nextcloud-talk)
+- [Webex](https://zulip.com/integrations/webex)
+
+By default, Zulip uses the [cloud version of Jitsi Meet](https://meet.jit.si/)
+as its call provider. This page documents the configurations required to support
+other [video call integration
+options](https://zulip.com/help/configure-call-provider) on a self-hosted Zulip
+server.
+
+:::{note}
+You can disable the video and voice call buttons for your organization by
+[setting the call
+provider](https://zulip.com/help/configure-call-provider)
+to "None".
+:::
+
+## Jitsi
+
+You can configure Zulip to use a self-hosted
+instance of Jitsi Meet by providing the URL of your self-hosted Jitsi Meet
+server [in organization
+settings](https://zulip.com/help/configure-call-provider#use-a-self-hosted-instance-of-jitsi-meet).
+No server configuration changes are required.
+
+## Zoom
+
+To use a [Zoom](https://zoom.us) integration on a self-hosted
+installation, you'll need to register a custom Zoom application for
+your Zulip server. Zulip supports two types of custom Zoom apps:
+
+- [Server to Server OAuth app](#server-to-server-oauth-app): Easiest to set up,
+  but requires users to be part of the Zoom organization that created the
+  application in order to create calls.
+
+- [General OAuth app](#general-oauth-app): Recommended for settings where the
+  limitations of the Server to Server OAuth app are problematic (e.g., this is
+  used by Zulip Cloud).
+
+### Server to Server OAuth app
+
+This Zoom application type, introduced in Zulip 10.0, is easiest to
+set up, and is ideal for most installations that self-host Zulip. To
+[create Zoom meeting links in Zulip
+messages](https://zulip.com/help/start-a-call#start-a-call) using this
+integration, users will need to be members of your Zoom
+organization and use the same email address in Zulip that they have
+registered with Zoom.
+
+You can set up this integration as follows:
+
+1. Select [**Build App**](https://marketplace.zoom.us/develop/create)
+   at the Zoom Marketplace. Create a **Server to Server OAuth App**.
+
+1. Choose an app name such as "ExampleCorp Zulip".
+
+1. In the **Information** tab:
+
+   - Add a short description and company name.
+   - Add a name and email for the developer contact information.
+
+1. In the **Scopes** tab, add the `meeting:write:meeting:admin` and
+   `meeting:write:meeting:master` scopes.
+
+1. In the **Activation** tab, activate your app. You can now
+   [configure your Zulip server](#configure-your-zulip-server)
+   to use the app.
+
+### General OAuth app
+
+This Zoom application type is more flexible than the server-to-server
+integration. However, current Zoom policy requires all General OAuth
+apps to go through the full Zoom Marketplace review process, even for
+[unlisted
+apps](https://developers.zoom.us/docs/platform/key-concepts/#private-vs-beta-vs-published-vs-unlisted-apps)
+that will only be used by a single customer. As a result, you have to
+do quite a bit of publishing overhead work in order to create this
+type of Zoom application for your Zulip server.
+
+1. Select [**Build App**](https://marketplace.zoom.us/develop/create)
+   at the Zoom Marketplace. Create a **General App**.
+
+1. In the **Basic Information** tab:
+
+   - Choose an app name such as "ExampleCorp Zulip".
+   - Select **User-managed app**.
+   - In the **OAuth Information** section, set the **OAuth Redirect URL**
+     to `https://zulip.example.com/calls/zoom/complete` (replacing
+     `zulip.example.com` by your main Zulip hostname).
+
+<!--
+If we ever need to increase scopes in the future, we should also include the scopes
+required to retrieve a user's details via this
+endpoint: https://developers.zoom.us/docs/api/users/#tag/users/get/users/{userId}
+
+This will help us implement a deauthorization workflow for Zoom app integrations,
+where we delete the user's zoom_token from the database upon receiving a request
+from Zoom’s deauthorization webhook, based on the user's zoom_id, which is part
+of the deauthorization request payload.
+
+Details: https://chat.zulip.org/#narrow/channel/49-development-help/topic/What's.20the.20use.20of.20.60.2Fcalls.2Fzoom.2Fdeauthorize.60.20endpoint/with/2296326
+ -->
+
+1. In the **Scopes** tab, add the `meeting:write:meeting` scope.
+
+1. Switch to the **Production** tab and complete the information needed
+   for the [Zoom App Marketplace Review
+   Process](https://developers.zoom.us/docs/distribute/app-review-process/)
+   in the **App Listing** and **Technical Design** tabs.
+
+1. [Submit your app for
+   review](https://developers.zoom.us/docs/build-flow/submitting-apps-for-review/).
+   Select the **Publish the app by myself** option on the **App Submission**
+   page so that the app will be
+   [unlisted](https://developers.zoom.us/docs/build-flow/publishing-your-apps/#unlisted-apps).
+
+1. Once your app has been approved by the Zoom app review team, then
+   you can proceed to [configure your Zulip server](#configure-your-zulip-server)
+   to use the app.
+
+### Configure your Zulip server
+
+1. In `/etc/zulip/zulip-secrets.conf`, set `video_zoom_client_secret`
+   to be your app's "Client Secret".
+
+1. In `/etc/zulip/settings.py`, set `VIDEO_ZOOM_CLIENT_ID` to your
+   app's "Client ID". Some Zoom enterprise customers that don't use
+   `zoom.us` will need to set `VIDEO_ZOOM_API_URL` for the Zoom API
+   server and `VIDEO_ZOOM_OAUTH_URL` for your instance's
+   authorization. If you're using a Zoom [Server to Server OAuth
+   app](#server-to-server-oauth-app), set
+   `VIDEO_ZOOM_SERVER_TO_SERVER_ACCOUNT_ID` to be your app's "Account
+   ID".
+
+1. Restart the Zulip server with
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+This enables Zoom support in your Zulip server. Finally, [configure Zoom as the
+video call
+provider](https://zulip.com/help/configure-call-provider)
+in the Zulip organizations where you want to use it.
+
+## BigBlueButton
+
+To use the [BigBlueButton](https://bigbluebutton.org/) video call
+integration on a self-hosted Zulip installation, you'll need to have a
+BigBlueButton server (version 2.4+) and configure it:
+
+1. Get the Shared Secret using the `bbb-conf --secret` command on your
+   BigBlueButton Server. See also [the BigBlueButton
+   documentation](https://docs.bigbluebutton.org/administration/customize/#extract-the-shared-secret).
+
+2. Get the URL to your BigBlueButton API. The URL has the form of
+   `https://bigbluebutton.example.com/bigbluebutton/` and can also be
+   found using the `bbb-conf --secret` command.
+
+You can then configure your Zulip server to use that BigBlueButton
+Server as follows:
+
+1. In `/etc/zulip/zulip-secrets.conf`, set `big_blue_button_secret`
+   to be your BigBlueButton Server's shared secret.
+
+2. In `/etc/zulip/settings.py`, set `BIG_BLUE_BUTTON_URL` to your
+   BigBlueButton Server's API URL.
+
+3. Restart the Zulip server with
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+This enables BigBlueButton support in your Zulip server. Finally, [configure
+BigBlueButton as the video call
+provider](https://zulip.com/help/configure-call-provider)
+in the Zulip organizations where you want to use it.
+
+## Constructor Groups
+
+To use the [Constructor Groups](https://constructor.tech/products/learning/groups)
+video call integration on a self-hosted Zulip installation, you'll need to
+have a Constructor Groups account. See documentation to configure the Constructor
+Groups video call integration [here](https://zulip.com/integrations/constructor-groups).
+
+## Nextcloud Talk
+
+To use the [Nextcloud Talk](https://nextcloud.com/talk/) video call
+integration on a self-hosted Zulip installation, you'll need to have access
+to a Nextcloud server (version 22+) with the Talk app enabled. See
+documentation to configure the Nextcloud Talk video call integration
+[here](https://zulip.com/integrations/nextcloud-talk).
+
+## Webex
+
+To use the [Webex](https://www.webex.com/) integration on a self-hosted
+Zulip installation, you'll need to register a custom Webex integration.
+
+For users in paid Webex organizations, which have the ability to create
+public rooms that are visible to every member within that organization,
+an ad-hoc meeting is created in Webex. Otherwise, for users of free
+Webex organizations, a personal room meeting is created in Webex.
+
+### Create a custom Webex integration
+
+1. Visit the [New Integration](https://developer.webex.com/my-apps/new/integration)
+   page on the Webex developer portal.
+
+1. Select "No" for "Will this integration use a mobile SDK?".
+
+1. Fill in the **Integration name**, **Icon** and **App Hub Description**
+   according to your preferences.
+
+1. For **Redirect URI(s)**, enter `https://zulip.example.com/calls/webex/complete`
+   replacing `zulip.example.com` with your Zulip organization's URL.
+
+1. For **Scopes**, select `spark:all`, `meeting:schedules_read` and
+   `meeting:schedules_write`, and select **Add Integration**.
+
+1. _optional_: You can submit the integration to the [Webex App Hub](https://apphub.webex.com/)
+   by following [this documentation](https://developer.webex.com/create/docs/app-hub-submission-process).
+
+### Configure your Zulip server
+
+1. In `/etc/zulip/zulip-secrets.conf`, set `video_webex_client_secret`
+   to your Webex integration's "Client Secret".
+
+1. In `/etc/zulip/settings.py`, set `VIDEO_WEBEX_CLIENT_ID` to your
+   Webex integration's "Client ID".
+
+1. Restart the Zulip server with
+   `/home/zulip/deployments/current/scripts/restart-server`.
+
+This enables Webex support in your Zulip server. Finally, [configure Webex
+as the video call provider](https://zulip.com/help/configure-call-provider)
+in the Zulip organization where you want to use it.