snapshot of https://github.com/zulip/zulip.git @ 48c69911c7dd4e03b20552eb3bd77ad8151c1357 (2026-09-15)

docs/overview/architecture-overview.md

@@ -0,0 +1,261 @@
+# Zulip architectural overview
+
+## Key codebases
+
+The main Zulip codebase is at <https://github.com/zulip/zulip>. It
+contains the Zulip backend (written in Python 3.x and Django), the
+web app (written in JavaScript and TypeScript) and our library of
+incoming webhook [integrations](https://zulip.com/integrations/)
+with other services and applications (see [the directory structure
+guide](../subsystems/directory-structure.md)).
+
+[Zulip Flutter](https://github.com/zulip/zulip-flutter) is the official
+mobile Zulip client supporting both iOS and Android, built with Flutter,
+and [Zulip Desktop](https://github.com/zulip/zulip-desktop) is the
+official Zulip desktop client for macOS, Linux, and Windows.
+[Zulip Terminal](https://github.com/zulip/zulip-terminal) is our
+official terminal-based client.
+
+We also maintain several separate repositories for integrations and
+other glue code: [Python API
+bindings](https://github.com/zulip/python-zulip-api); [JavaScript API
+bindings](https://github.com/zulip/zulip-js); a [Hubot
+adapter](https://github.com/zulip/hubot-zulip); integrations with
+[Jenkins](https://github.com/zulip/zulip-jenkins-plugin),
+[Puppet](https://github.com/matthewbarr/puppet-zulip),
+[Redmine](https://github.com/zulip/zulip-redmine-plugin), and
+[Trello](https://github.com/zulip/trello-to-zulip);
+and [many more](https://github.com/zulip/).
+
+We use [Weblate](https://hosted.weblate.org/projects/zulip/) to do
+translations.
+
+In this overview, we'll mainly discuss the core Zulip server and web
+application.
+
+## Usage assumptions and concepts
+
+Zulip is a real-time team chat application meant to provide a great
+experience for a wide range of organizations, from companies to
+volunteer projects to groups of friends, ranging in size from a small
+team to tens of thousands of users. It has [hundreds of
+features](https://zulip.com/features/) both large and small, and
+supports dedicated apps for iOS, Android, Linux, Windows, and macOS,
+all modern web browsers, several cross-protocol chat clients, and
+numerous dedicated [Zulip API](https://zulip.com/api/) clients
+(e.g., bots).
+
+A server can host multiple Zulip _realms_ (organizations), each on its
+own (sub)domain. While most installations host only one organization, some
+host thousands, such as zulip.com. Each organization is a private
+chamber with its own users, channels, customizations, and so on. This
+means that one person might be a user of multiple Zulip realms. The
+administrators of an organization have a great deal of control over
+who can register an account, what permissions new users have, etc. For
+more on security considerations and options, see our [guide on securing
+your Zulip server][security-guide], [security overview][security-overview],
+and the [Zulip help center](https://zulip.com/help/).
+
+[security-overview]: https://zulip.com/security/
+[security-guide]: ../production/securing-your-zulip-server.md
+
+## Components
+
+![architecture-simple](../images/architecture_simple.png)
+
+### Django and Tornado
+
+Zulip is primarily implemented in the
+[Django](https://www.djangoproject.com/) Python web framework. We
+also make use of [Tornado](https://www.tornadoweb.org) for the
+real-time push system.
+
+Django is the main web application server; Tornado runs the
+server-to-client real-time push system. The app servers are configured
+by the Supervisor configuration (which explains how to start the server
+processes; see "Supervisor" below) and the nginx configuration (which
+explains which HTTP requests get sent to which app server).
+
+Tornado is an asynchronous server and is meant specifically to hold
+open tens of thousands of long-lived (long-polling)
+connections -- that is to say, routes that maintain a persistent
+connection from every running client. For this reason, it's
+responsible for event (message) delivery, but not much else. We try to
+avoid any blocking calls in Tornado because we don't want to delay
+delivery to thousands of other connections (as this would make Zulip
+very much not real-time). For instance, we avoid doing cache or
+database queries inside the Tornado code paths, since those blocking
+requests carry a very high performance penalty for a single-threaded,
+asynchronous server system. (In principle, we could do non-blocking
+requests to those services, but the Django-based database libraries we
+use in most of our codebase don't support that, and in any case,
+our architecture doesn't require Tornado to do that).
+
+The parts that are activated relatively rarely (e.g., when people type or
+click on something) are processed by the Django application server.
+
+There is detailed documentation on the
+[real-time push and event queue system](../subsystems/events-system.md); most of
+the code is in `zerver/tornado`.
+
+#### HTML templates, JavaScript, etc.
+
+Zulip's HTML is primarily implemented using two types of HTML
+templates: backend templates (powered by the [Jinja2][] template
+engine used for logged-out ("portico") pages and the web app's base
+content) and frontend templates (powered by [Handlebars][]) used for
+live-rendering HTML from JavaScript for things like the main message
+feed.
+
+For more details on the frontend, see our documentation on
+[translation](../translating/translating.md),
+[templates](../subsystems/html-css.md#html-templates),
+[directory structure](../subsystems/directory-structure.md), and
+[the static asset pipeline](../subsystems/html-css.md#static-asset-pipeline).
+
+[jinja2]: http://jinja.pocoo.org/
+[handlebars]: https://handlebarsjs.com/
+
+### nginx
+
+nginx is the front-end web server to all Zulip traffic; it serves static
+assets and proxies to Django and Tornado. It handles HTTP requests
+according to the rules laid down in the many config files found in
+`puppet/zulip/files/nginx/` and `puppet/zulip/templates/nginx/`.
+
+`puppet/zulip/files/nginx/zulip-include-frontend/app` is the most
+important of these files. It explains what happens when requests come in
+from outside.
+
+- In production, all requests to URLs beginning with `/static/` are
+  served from the corresponding files in `/home/zulip/prod-static/`,
+  and the production build process (`tools/build-release-tarball`)
+  compiles, minifies, and installs the static assets into the
+  `prod-static/` tree form. In development, files are served directly
+  from `/static/` in the Git repository.
+- Requests to `/json/events` and `/api/v1/events`, i.e. the
+  real-time push system, are sent to the Tornado server.
+- Requests to all other paths are sent to the Django app running via
+  `uWSGI` via `unix:/home/zulip/deployments/uwsgi-socket`.
+- By default (i.e. if `LOCAL_UPLOADS_DIR` is set), nginx will serve
+  user-uploaded content like avatars, custom emoji, and uploaded
+  files. However, one can configure Zulip to store these in a cloud
+  storage service like Amazon S3 instead.
+
+Note that we do not use `nginx` in the development environment, opting
+for a simple Tornado-based proxy instead.
+
+### Supervisor
+
+We use [supervisord](http://supervisord.org/) to start server processes,
+restart them automatically if they crash, and direct logging.
+
+The config file is
+`puppet/zulip/templates/supervisor/zulip.conf.template.erb`. This
+is where Tornado and Django are set up, as well as a number of background
+processes that process event queues. We use event queues for the kinds
+of tasks that are best run in the background because they are
+expensive (in terms of performance) and don't have to be synchronous
+--- e.g., sending emails or updating analytics. Also see [the queuing
+guide](../subsystems/queuing.md).
+
+### memcached
+
+memcached is used to cache database model
+objects. `zerver/lib/cache.py` and `zerver/lib/cache_helpers.py`
+manage putting things into memcached, and invalidating the cache when
+values change. The memcached configuration is in
+`puppet/zulip/templates/memcached.conf.template.erb`. See our
+[caching guide](../subsystems/caching.md) to learn how this works in
+detail.
+
+### Redis
+
+Redis is used for a few very short-term data stores, primarily
+our rate-limiting system.
+
+Redis is configured in `puppet/zulip/templates/zulip-redis.template.erb` and
+the main contents are the following, which turns off persistence:
+
+```text
+# Disable saving to disk to optimize performance
+save ""
+```
+
+People often wonder if we could replace memcached with Redis (or
+replace RabbitMQ with Redis, with some loss of functionality).
+
+The answer is likely yes, but it wouldn't improve Zulip.
+Operationally, our current setup is likely easier to develop and run
+in production than a pure Redis system would be. Meanwhile, the
+perceived benefit for using Redis is usually to reduce memory
+consumption by running fewer services, and no such benefit would
+materialize:
+
+- Our cache uses significant memory, but that memory usage would be
+  essentially the same with Redis as it is with memcached.
+- All of these services have low minimum memory requirements, and in
+  fact our applications for Redis and RabbitMQ do not use significant
+  memory even at scale.
+- We would likely need to run multiple Redis services (with different
+  configurations) in order to ensure the pure LRU use case (memcached)
+  doesn't push out data that we want to persist until expiry
+  (Redis-based rate limiting) or until consumed (RabbitMQ-based
+  queuing of deferred work).
+
+### RabbitMQ
+
+RabbitMQ is a queueing system. Its config files live in
+`puppet/zulip/files/rabbitmq`. Initial configuration happens in
+`scripts/setup/configure-rabbitmq`.
+
+We use RabbitMQ for queuing expensive work (e.g., sending emails
+triggered by a message, push notifications, some analytics, etc.) that
+require reliable delivery but which we don't want to do on the main
+thread. It's also used for communication between the application server
+and the Tornado push system.
+
+Two simple wrappers around `pika` (the Python RabbitMQ client) are in
+`zerver/lib/queue.py`. There's an asynchronous client for use in
+Tornado and a more general client for use elsewhere. Most of the
+processes started by Supervisor are queue processors that continually
+pull things out of a RabbitMQ queue and handle them; they are defined
+in `zerver/worker/`.
+
+Also see [the queuing guide](../subsystems/queuing.md).
+
+### PostgreSQL
+
+PostgreSQL is the database that stores all persistent data, that is,
+data that's expected to live beyond a user's current session.
+Starting with Zulip 3.0, new Zulip installations will install modern
+PostgreSQL release rather than using the version included with the
+operating system.
+
+In production, PostgreSQL is installed with a default configuration. The
+directory that would contain configuration files
+(`puppet/zulip/files/postgresql`) has only a utility script and a custom
+list of stopwords used by a PostgreSQL extension.
+
+In a development environment, configuration of that PostgreSQL
+extension is handled by `tools/postgresql-init-dev-db` (invoked by
+`tools/provision`). That file also manages setting up the
+development PostgreSQL user.
+
+`tools/provision` also invokes `tools/rebuild-dev-database`
+to create the actual database with its schema.
+
+### Nagios
+
+Nagios is an optional component used for notifications to the system
+administrator, e.g., in case of outages.
+
+`puppet/zulip/manifests/nagios_plugins.pp` installs Nagios plugins from
+`puppet/zulip/files/nagios_plugins/`.
+
+This component is intended to install Nagios plugins intended to be run
+on a Nagios server; most of the Zulip Nagios plugins are intended to be
+run on the Zulip servers themselves, and are included with the relevant
+component of the Zulip server (e.g.,
+`puppet/zulip/manifests/app_frontend_base.pp` installs a few under
+`/usr/lib/nagios/plugins/zulip_app_frontend`).

docs/overview/index.md

@@ -0,0 +1,13 @@
+# Overview
+
+```{toctree}
+---
+maxdepth: 3
+---
+
+readme
+architecture-overview
+release-lifecycle
+roadmap
+changelog
+```

docs/overview/readme.md

@@ -0,0 +1 @@
+../../README.md

docs/overview/release-lifecycle.md

@@ -0,0 +1,225 @@
+# Release lifecycle
+
+This page provides an overview of Zulip server and client app release
+lifecycles. As discussed below, we highly recommend running [the latest Zulip
+server release](#stable-releases). We work extremely hard to make sure these
+releases are stable and have no regressions, and that the [upgrade
+process](../production/upgrade.md) Just Works.
+
+New server releases are announced via the low-traffic [zulip-announce email
+list](https://groups.google.com/g/zulip-announce). Please subscribe to get
+notified about new security releases.
+
+## Server and web app versions
+
+The Zulip server and web app are developed together in the [Zulip
+server repository][zulip-server].
+
+### Stable releases
+
+:::{note}
+
+The first digit of the Zulip server release version is its major release series
+(e.g., 9.4 is part of the 9.x series).
+
+:::
+
+Organizations self-hosting Zulip primarily use stable releases (e.g., Zulip {{
+LATEST_RELEASE_VERSION }}).
+
+- [New major releases][blog-major-releases], like Zulip 9.0, are published twice
+  a year, and contain hundreds of features, bug fixes, and improvements to Zulip's
+  internals.
+
+- New maintenance releases, like 9.4, are published roughly once a month.
+  Maintenance releases are designed to have no risky changes and be easy to
+  reverse, to minimize stress for administrators.
+
+When upgrading to a new major release series, we recommend always upgrading to
+the latest maintenance release in that series, so that you use the latest
+version of the upgrade code.
+
+[blog-major-releases]: https://blog.zulip.com/tag/major-releases/
+
+#### Security releases
+
+When we discover a security issue in Zulip, we publish a security and
+bug fix release, transparently documenting the issue using the
+industry-standard [CVE advisory process](https://cve.mitre.org/).
+
+When new security releases are published, we simultaneously publish the fixes to
+the `main` branch and the release branch for the current major release series.
+
+See also our [security overview][security-overview], and our [guide on securing
+your Zulip server][securing-your-zulip-server].
+
+[security-overview]: https://zulip.com/security/
+[securing-your-zulip-server]: ../production/securing-your-zulip-server.md
+
+### Git versions
+
+Many Zulip servers run versions from Git that have not been published
+in a stable release.
+
+- You can [upgrade to `main`][upgrading-to-main] for the latest changes.
+- We maintain Git branches with names like `9.x` containing backported
+  commits from `main` that we plan to include in the next maintenance
+  release. Self-hosters can [upgrade][upgrade-from-git] to these
+  stable release branches to get bug fixes staged for the next stable
+  release (which is very useful when you reported a bug whose fix we
+  choose to backport). We support these branches as though they were a
+  stable release.
+- The bleeding-edge server for the [Zulip development community][chat-zulip-org]
+  at <https://chat.zulip.org> runs the `chat.zulip.org` branch. It's upgraded
+  to `main` several times a week. We also often "test deploy" changes not yet in
+  `main` to this branch, to facilitate design feedback.
+- [Zulip Cloud](https://zulip.com) runs the `zulip-cloud-current` branch plus
+  some cherry-picked changes. This branch is usually delayed by one to two weeks
+  from `main` to allow for recent changes to be validated further prior to being
+  deployed to customers.
+- You can also run [a fork of Zulip][fork-zulip] on top of any of
+  these branches.
+
+[upgrade-from-git]: ../production/upgrade.md#upgrading-from-a-git-repository
+
+### What version am I running?
+
+The Zulip web app displays the current server version [in the gear
+menu](https://zulip.com/help/view-zulip-version); it's also available [via the
+API](https://zulip.com/api/get-server-settings).
+
+### Versioned documentation
+
+To make sure you can access documentation for your server version, [the help
+center](https://zulip.com/help/), [API documentation](https://zulip.com/api/),
+and [integrations documentation](https://zulip.com/integrations/) are
+distributed with the Zulip server itself (e.g.,
+`https://zulip.example.com/help/`).
+
+This ReadTheDocs documentation has a widget in the top left corner
+that lets you view the documentation for other versions.
+
+## Client apps
+
+Zulip's official client apps support all Zulip server versions
+released in the **previous 18 months**.
+
+The official client apps are designed to be compatible with
+intermediate Zulip server Git commits between the oldest supported
+server release and the current `main` branch.
+
+This allows server administrators to take advantage of the ability to
+upgrade to [Git versions](#git-versions) without breaking clients.
+
+[The API changelog](https://zulip.com/api/changelog) details all
+changes to the API, to make it easy for third-party client developers
+to maintain a similar level of compatibility.
+
+### Mobile app
+
+The Zulip mobile apps release new versions from the development
+branch frequently (usually every couple weeks). Except when fixing a
+critical bug, releases are first published to our [beta
+channels][mobile-beta].
+
+Mobile and desktop client apps update automatically, unless a user disables
+automatic updates.
+
+### Desktop app
+
+The Zulip [desktop app](https://zulip.com/apps/) is implemented in
+[Electron][electron], the browser-based desktop application framework used by
+essentially all modern chat applications. The Zulip UI in these apps is served
+from the Zulip server (and thus can vary between tabs when it is connected to
+organizations hosted by different servers).
+
+The desktop app automatically updates soon after each new release. New desktop
+app releases rarely contain new features, because the desktop app tab inherits
+its features from the Zulip server/web app. However, it is important to upgrade
+because they often contain important security or OS compatibility fixes from the
+upstream Chromium project. Be sure to leave automatic updates enabled, or
+otherwise arrange to promptly upgrade after new security releases.
+
+The Zulip server supports blocking access or displaying a warning to
+users attempting to access the server with extremely old or known
+insecure versions of the Zulip desktop and mobile apps, with an error
+message telling the user to upgrade.
+
+### Terminal app
+
+The beta Zulip [terminal app](https://github.com/zulip/zulip-terminal)
+is designed to support the same range of server versions targeted by
+other client apps.
+
+However, we do not support running old versions of the terminal app
+against the latest Zulip server. This means that terminal app users
+will sometimes need to upgrade to the latest version after their Zulip
+Server is upgraded to a new major release.
+
+## Server and client app compatibility
+
+Zulip is designed to make sure you can always run the [latest server
+release](#server-and-web-app-versions) (and we highly recommend that you do
+so!). We therefore generally do not backport changes to previous stable
+release series, except in rare cases involving a security issue or
+critical bug discovered just after publishing a major release.
+
+The Zulip server preserves backwards compatibility in its API to support
+versions of the mobile and desktop apps released in the last 12 months. Because
+these clients auto-update, the vast majority of active clients will have upgraded
+by the time we desupport a version.
+
+As noted [above](#client-apps), Zulip's official client apps support
+all Zulip server versions released in the **previous 18 months**.
+
+### Upgrade nag
+
+The Zulip web app will display a banner warning users of a server running a
+Zulip release that is more than 18 months old, and is no longer officially
+supported by mobile and desktop apps. The nag will appear only to organization
+administrators starting a month before the deadline; after that, it will appear
+for all users on the server.
+
+You can adjust the deadline for your installation by setting, for
+example, `SERVER_UPGRADE_NAG_DEADLINE_DAYS = 30 * 21` in
+`/etc/zulip/settings.py`, and [restarting the server](../production/settings.md).
+
+:::{warning}
+
+Servers older than 18 months are likely to be vulnerable to security bugs in
+Zulip or its upstream dependencies.
+
+:::
+
+## Operating system support
+
+For platforms we support, like Debian and Ubuntu, Zulip aims to
+support all versions of the upstream operating systems that are fully
+supported by the vendor. We document how to correctly [upgrade the
+operating system][os-upgrade] for a Zulip server, including how to
+correctly chain upgrades when the latest Zulip release no longer
+supports your OS.
+
+Note that we consider [Ubuntu interim releases][ubuntu-release-cycle],
+which only have 8 months of security support, to be betas, not
+releases, and do not support them in production.
+
+[ubuntu-release-cycle]: https://ubuntu.com/about/release-cycle
+
+## API bindings
+
+The [Zulip API](https://zulip.com/api/) bindings, and related projects like the
+[Python](https://zulip.com/api/configuring-python-bindings) and
+[JavaScript](https://github.com/zulip/zulip-js#readme) bindings, are released
+independently as needed.
+
+[electron]: https://www.electronjs.org/
+[upgrading-to-main]: ../production/modify.md#upgrading-to-main
+[os-upgrade]: ../production/upgrade.md#upgrading-the-operating-system
+[chat-zulip-org]: https://zulip.com/development-community/
+[fork-zulip]: ../production/modify.md
+[zulip-server]: https://github.com/zulip/zulip
+[mobile-beta]: https://zulip.com/help/mobile-app-install-guide#install-a-beta-release
+[label-blocker]: https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22priority%3A+blocker%22
+[label-high]: https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22priority%3A+high%22
+[label-help-wanted]: https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22

docs/overview/roadmap.md

@@ -0,0 +1,47 @@
+# Roadmap
+
+We welcome participation from our user community in influencing the Zulip
+roadmap. If a bug or missing feature is causing significant pain for you, we'd
+love to hear from you, either in
+[chat.zulip.org](https://zulip.com/development-community/) or on the relevant
+GitHub issue.
+
+Please an include an explanation of your use case: such details can be extremely
+helpful in designing appropriately general solutions, and also helps us identify
+cases where an existing solution can solve your problem. See our guide on
+[suggesting features and improvements](../contributing/suggesting-features.md)
+for more details.
+
+## Server and web app roadmap
+
+The Zulip server project uses GitHub projects and labels to structure
+communication about priorities:
+
+- We use a [GitHub project
+  board](https://github.com/orgs/zulip/projects/9/views/13) to publicly track
+  goals for major releases. The items with the "Done" status will be included in
+  the next major release. Otherwise, the project board should be seen a list of
+  priorities being _considered_ for the release, not a guarantee that features
+  will be included. As the release date approaches, features that will not make
+  it into the release are dropped from the project board on an ongoing basis.
+
+- The [high priority][label-high] label tags issues that we consider important.
+  It is reviewed in the planning stage of the release cycle to identify
+  priorities for the next release.
+
+- The [help wanted][label-help-wanted] label tags issues that are open for
+  contributions.
+
+The Zulip community feels strongly that all the little issues are, in
+aggregate, just as important as the big things. Many resolved issues
+are never explicitly tagged as release goals.
+
+[label-high]: https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3A%22priority%3A+high%22
+[label-help-wanted]: https://github.com/zulip/zulip/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22
+
+## Mobile app roadmap
+
+We use a [GitHub project
+board](https://github.com/orgs/zulip/projects/5/views/4) to publicly track
+milestones for Zulip's [next-generation mobile
+app](https://blog.zulip.com/2024/12/12/new-flutter-mobile-app-beta/).