snapshot of https://github.com/zulip/zulip.git @ 48c69911c7dd4e03b20552eb3bd77ad8151c1357 (2026-09-15)

docs/subsystems/accessibility.md

@@ -0,0 +1,84 @@
+# Accessibility
+
+## Guidelines
+
+In order to accommodate all users, Zulip strives to implement accessibility
+best practices in its user interface. There are many aspects to accessibility;
+here are some of the more important ones to keep in mind.
+
+- All images should have alternative text attributes for the benefit of users
+  who cannot see them (this includes users who are utilizing a voice interface
+  to free up their eyes to look at something else instead).
+- The entire application should be usable via a keyboard (many users are unable
+  to use a mouse, and many accessibility aids emulate a keyboard).
+- Text should have good enough contrast against the background to enable
+  even users with moderate visual impairment to be able to read it.
+- [ARIA](https://www.w3.org/WAI/intro/aria) (Accessible Rich Internet
+  Application) attributes should be used appropriately to enable screen
+  readers and other alternative interfaces to navigate the application
+  effectively.
+
+There are many different standards for accessibility, but the most relevant
+one for Zulip is the W3C's [WCAG](https://www.w3.org/TR/WCAG20/) (Web Content
+Accessibility Guidelines), currently at version 2.0. Whenever practical, we
+should strive for compliance with the AA level of this specification.
+(The W3C itself
+[recommends not trying](https://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html#uc-conf-req1-head)
+to comply with the highest AAA level for an entire web site or application,
+as it is not possible for some content.)
+
+## Tools
+
+There are tools available to automatically audit a web page for compliance
+with many of the WCAG guidelines. Here are some of the more useful ones:
+
+- [Accessibility Developer Tools][chrome-webstore]
+  This open-source Chrome extension from Google adds an accessibility audit to
+  the "Audits" tab of the Chrome Developer Tools. The audit is performed
+  against the page's DOM via JavaScript, allowing it to identify some issues
+  that a static HTML inspector would miss.
+- [axe](https://www.deque.com/products/axe/) An open-source Chrome and Firefox
+  extension which runs a somewhat different set of checks than Google's Chrome
+  extension.
+- [Wave](https://wave.webaim.org/) This web application takes a URL and loads
+  it in a frame, reporting on all the issues it finds with links to more
+  information. Has the advantage of not requiring installation, but requires
+  a URL which can be directly accessed by an external site.
+- [Web Developer](https://chrispederick.com/work/web-developer/) This browser
+  extension has many useful features, including a convenient link for opening
+  the current URL in Wave to get an accessibility report.
+
+Note that these tools cannot catch all possible accessibility problems, and
+sometimes report false positives as well. They are a useful aid in quickly
+identifying potential problems and checking for regressions, but their
+recommendations should not be blindly obeyed.
+
+## GitHub issues
+
+Problems with Zulip's accessibility should be reported as
+[GitHub issues](https://github.com/zulip/zulip/issues) with the "accessibility"
+label. This label can be added by entering the following text in a separate
+comment on the issue:
+
+> @zulipbot add "accessibility"
+
+If you want to help make Zulip more accessible, here is a list of the
+[currently open accessibility issues][accessibility-issues].
+
+## Additional resources
+
+For more information about making Zulip accessible to as many users as
+possible, the following resources may be useful.
+
+- [Font Awesome accessibility guide](https://fontawesome.com/how-to-use/on-the-web/other-topics/accessibility),
+  which is especially helpful since Zulip uses Font Awesome for its icons.
+- [Web Content Accessibility Guidelines (WCAG) 2.0](https://www.w3.org/TR/WCAG/)
+- [WAI-ARIA](https://www.w3.org/WAI/intro/aria) - Web Accessibility Initiative
+  Accessible Rich Internet Application Suite
+- [WebAIM](https://webaim.org/) - Web Accessibility in Mind
+- The [MDN page on accessibility](https://developer.mozilla.org/en-US/docs/Web/Accessibility)
+- The [Open edX Accessibility Guidelines][openedx-guidelines] for developers
+
+[chrome-webstore]: https://chrome.google.com/webstore/detail/accessibility-developer-t/fpkknkljclfencbdbgkenhalefipecmb
+[openedx-guidelines]: https://docs.openedx.org/en/latest/developers/concepts/accessibility.html
+[accessibility-issues]: https://github.com/zulip/zulip/issues?q=is%3Aissue+is%3Aopen+label%3Aaccessibility

docs/subsystems/analytics.md

@@ -0,0 +1,227 @@
+# Analytics
+
+Zulip has a cool analytics system for tracking various useful statistics
+that currently power the `/stats` page, and over time will power other
+features, like showing usage statistics for the various channels. It is
+designed around the following goals:
+
+- Minimal impact on scalability and service complexity.
+- Well-tested so that we can count on the results being correct.
+- Efficient to query so that we can display data in-app (e.g., on the channels
+  page) with minimum impact on the overall performance of those pages.
+- Storage size smaller than the size of the main Message/UserMessage
+  database tables, so that we can store the data in the main PostgreSQL
+  database rather than using a specialized database platform.
+
+There are a few important things you need to understand in order to
+effectively modify the system.
+
+## Analytics backend overview
+
+There are three main components:
+
+- models: The `UserCount`, `StreamCount`, `RealmCount`, and `InstallationCount`
+  tables (`analytics/models.py`) collect and store time series data.
+- stat definitions: The `CountStat` objects in the `COUNT_STATS` dictionary
+  (`analytics/lib/counts.py`) define the set of stats Zulip collects.
+- accounting: The `FillState` table (`analytics/models.py`) keeps track of what
+  has been collected for which `CountStat`.
+
+The next several sections will dive into the details of these components.
+
+## The `*Count` database tables
+
+The Zulip analytics system is built around collecting time series data in a
+set of database tables. Each of these tables has the following fields:
+
+- property: A human readable string uniquely identifying a `CountStat`
+  object. Example: `"active_users_audit:is_bot:hour"` or
+  `"messages_sent:client:day"`.
+- subgroup: Almost all `CountStat` objects are further sliced by subgroup. For
+  `"active_users_audit:is_bot:day"`, this column will be `False` for measurements of
+  humans, and `True` for measurements of bots. For `"messages_sent:client:day"`,
+  this column is the client_id of the client under consideration.
+- end_time: A datetime indicating the end of a time interval. It will be on
+  an hour (or UTC day) boundary for stats collected at hourly (or daily)
+  frequency. The time interval is determined by the `CountStat`.
+- various "id" fields: Foreign keys into `Realm`, `UserProfile`, `Stream`, or
+  nothing. E.g., the `RealmCount` table has a foreign key into `Realm`.
+- value: The integer counts. For `"active_users_audit:is_bot:hour"` in the
+  `RealmCount` table, this is the number of active humans or bots (depending
+  on subgroup) in a particular realm at a particular `end_time`. For
+  `"messages_sent:client:day"` in the `UserCount` table, this is the number of
+  messages sent by a particular user, from a particular client, on the day
+  ending at `end_time`.
+
+There are four tables: `UserCount`, `StreamCount`, `RealmCount`, and
+`InstallationCount`. Every `CountStat` is initially collected into `UserCount`,
+`StreamCount`, or `RealmCount`. Every stat in `UserCount` and `StreamCount` is
+aggregated into `RealmCount`, and then all stats are aggregated from
+`RealmCount` into `InstallationCount`. So for example,
+`"messages_sent:client:day"` has rows in `UserCount` corresponding to
+`(user, end_time, client)` triples. These are summed to rows in `RealmCount`
+corresponding to triples of `(realm, end_time, client)`. And then these are
+summed to rows in `InstallationCount` with totals for pairs of `(end_time, client)`.
+
+Note: In most cases, we do not store rows with value 0. See
+[Performance strategy](#performance-strategy) below.
+
+## CountStats
+
+`CountStat` objects declare what analytics data should be generated and stored. The
+`CountStat` class definition and instances live in `analytics/lib/counts.py`.
+These declarations specify at a high level which tables should be populated
+by the system and with what data.
+
+## The FillState table
+
+The default Zulip production configuration runs a cron job once an hour that
+updates the `*Count` tables for each of the `CountStat` objects in the `COUNT_STATS`
+dictionary. The `FillState` table simply keeps track of the last `end_time` that
+we successfully updated each stat. It also enables the analytics system to
+recover from errors (by retrying) and to monitor that the cron job is
+running and running to completion.
+
+## Performance strategy
+
+An important consideration with any analytics system is performance, since
+it's easy to end up processing a huge amount of data inefficiently and
+needing a system like Hadoop to manage it. For the built-in analytics in
+Zulip, we've designed something lightweight and fast that can be available
+on any Zulip server without any extra dependencies through the carefully
+designed set of tables in PostgreSQL.
+
+This requires some care to avoid making the analytics tables larger than the
+rest of the Zulip database or adding a ton of computational load, but with
+careful design, we can make the analytics system very low cost to operate.
+Also, note that a Zulip application database has 2 huge tables: `Message` and
+`UserMessage`, and everything else is small and thus not performance or
+space-sensitive, so it's important to optimize how many expensive queries we
+do against those 2 tables.
+
+There are a few important principles that we use to make the system
+efficient:
+
+- Not repeating work to keep things up to date (via `FillState`)
+- Storing data in the `*Count` tables to avoid our endpoints hitting the core
+  `Message`/`UserMessage` tables is key, because some queries could take minutes
+  to calculate. This allows any expensive operations to run offline, and
+  then the endpoints to server data to users can be fast.
+- Doing expensive operations inside the database, rather than fetching data
+  to Python and then sending it back to the database (which can be far
+  slower if there's a lot of data involved). The Django ORM currently
+  doesn't support the `"insert into .. select"` type SQL query that's needed
+  for this, which is why we use raw database queries (which we usually avoid
+  in Zulip) rather than the ORM.
+- Aggregating where possible to avoid unnecessary queries against the
+  `Message` and `UserMessage` tables. E.g., rather than querying the `Message`
+  table both to generate sent message counts for each realm and again for
+  each user, we just query for each user, and then add up the numbers for
+  the users to get the totals for the realm.
+- Not storing rows when the value is 0. An hourly user stat would otherwise
+  collect 24 \* 365 \* roughly .5MB per db row = 4GB of data per user per
+  year, most of whose values are 0. A related note is to be cautious about
+  adding queries that are typically non-0 instead of being typically 0.
+
+## Backend testing
+
+There are a few types of automated tests that are important for this sort of
+system:
+
+- Most important: Tests for the code path that actually populates data into
+  the analytics tables. These are most important, because it can be very
+  expensive to fix bugs in the logic that generates these tables (one
+  basically needs to regenerate all of history for those tables), and these
+  bugs are hard to discover. It's worth taking the time to think about
+  interesting corner cases and add them to the test suite.
+- Tests for the backend views code logic for extracting data from the
+  database and serving it to clients.
+
+For manual backend testing, it sometimes can be valuable to use
+`./manage.py dbshell` to inspect the tables manually to check that
+things look right; but usually anything you feel the need to check
+manually, you should add some sort of assertion for to the backend
+analytics tests, to make sure it stays that way as we refactor.
+
+## LoggingCountStats
+
+The system discussed above is designed primarily around the technical
+problem of showing useful analytics about things where the raw data is
+already stored in the database (e.g., `Message`, `UserMessage`). This is great
+because we can always backfill that data to the beginning of time, but of
+course sometimes one wants to do analytics on things that aren't worth
+storing every data point for (e.g., activity data, request performance
+statistics, etc.). There is currently a reference implementation of a
+`LoggingCountStat` that shows how to handle such a situation.
+
+## Analytics UI development and testing
+
+### Setup and testing
+
+The main testing approach for the `/stats` page UI is manual testing.
+For most UI testing, you can visit `/stats/realm/analytics` while
+logged in as Iago (this is the server administrator view of stats for
+a given realm). The only piece that you can't test here is the "Me"
+buttons, which won't have any data. For those, you can instead log in
+as the `shylock@analytics.ds` in the `analytics` realm and visit
+`/stats` there (which is only a bit more work). Note that the
+`analytics` realm is a shell with no channels, so you'll only want to
+use it for testing the graphs.
+
+If you're adding a new stat/table, you'll want to edit
+`analytics/management/commands/populate_analytics_db.py` and add code
+to generate fake data of the form needed for your new stat/table;
+you'll then run `./manage.py populate_analytics_db` before looking at
+the updated graphs.
+
+### Adding or editing /stats graphs
+
+The relevant files are:
+
+- `analytics/views/stats.py`: All chart data requests from the /stats page call
+  get_chart_data in this file.
+- `web/src/stats/stats.ts`: The JavaScript and Plotly code.
+- `templates/analytics/stats.html`
+- `web/styles/stats.css` and `web/styles/legacy_portico.css`: We are in the
+  process of re-styling this page to use in-app CSS instead of the portico CSS,
+  but there is currently still a lot of portico influence.
+- `analytics/urls.py`: Has the URL routes; it's unlikely you will have to
+  modify this, including for adding a new graph.
+
+Most of the code is self-explanatory, and for adding say a new graph, the
+answer to most questions is to copy what the other graphs do. It is easy
+when writing this sort of code to have a lot of semi-repeated code blocks
+(especially in `stats.ts`); it's good to do what you can to reduce this.
+
+Tips and tricks:
+
+- Use `$.get` to fetch data from the backend. You can grep through `stats.ts`
+  to find examples of this.
+- The Plotly documentation is at
+  <https://plot.ly/javascript/> (check out the full reference, event
+  reference, and function reference). The documentation pages seem to work
+  better in Chrome than in Firefox, though this hasn't been extensively
+  verified.
+- Unless a graph has a ton of data, it is typically better to just redraw it
+  when something changes (e.g., in the various aggregation click handlers)
+  rather than to use retrace or relayout or do other complicated
+  things. Performance on the `/stats` page is nice but not critical, and we've
+  run into a lot of small bugs when trying to use Plotly's retrace/relayout.
+- There is a way to access raw d3 functionality through Plotly, though it
+  isn't documented well.
+- `'paper'` as a Plotly option refers to the bounding box of the graph (or
+  something related to that).
+- You can't right click and inspect the elements of a Plotly graph (e.g., the
+  bars in a bar graph) in your browser, since there is an interaction layer
+  on top of it. But if you hunt around the document tree you should be able
+  to find it.
+
+### /activity page
+
+- There's a somewhat less developed `/activity` page, for server
+  administrators, showing data on all the realms on a server. To
+  access it, you need to have the `is_staff` bit set on your
+  `UserProfile` object. You can set it using `manage.py shell` and
+  editing the `UserProfile` object directly. A great future project is
+  to clean up that page's data sources, and make this a documented
+  interface.

docs/subsystems/billing.md

@@ -0,0 +1,219 @@
+# Billing (Development)
+
+This section deals with developing and testing the billing system.
+
+## Common setup
+
+- Create a Stripe account
+  - Make sure that the country of your Stripe account is set to USA when
+    you create the account.
+  - You might need to use a VPN for this.
+- Ensure that the [API version](https://stripe.com/docs/api/versioning) of
+  your Stripe account is same as `STRIPE_API_VERSION` defined in
+  `corporate/lib/stripe.py`. You can upgrade to a higher version from
+  the Stripe dashboard.
+- Set the private API key.
+  - Go to <https://dashboard.stripe.com/test/apikeys>
+  - Double-check that you're viewing test API keys (not live keys) to avoid
+    actual charges while testing code.
+  - Add `stripe_secret_key` to `zproject/dev-secrets.conf`.
+
+## Manual testing
+
+Manual testing involves testing the various flows like upgrade, card change,
+etc. through the browser. This is the bare minimum testing that you need to
+do when you review a billing PR or when you are working on adding a new
+feature to billing.
+
+### Setup
+
+Apart from the common setup mentioned above, you also need to set up your
+development environment to receive webhook events from Stripe.
+
+- Install the Stripe CLI locally by following the instructions
+  [here](https://stripe.com/docs/stripe-cli).
+- Log in to Stripe CLI using the command `stripe login`.
+- You can get Stripe CLI to forward all Stripe webhook events to our local
+  webhook endpoint using the following command:
+  `stripe listen --forward-to http://localhost:9991/stripe/webhook/`
+- Note that the webhook secret key needs to be updated every 90 days following
+  the steps [here](https://stripe.com/docs/stripe-cli#install).
+- Wait for the `stripe listen` command in the previous step to output the
+  webhook signing secret.
+  - The signing secret would be used by our billing system to verify that
+    the events received by our webhook endpoint are sent by Stripe and not
+    by an intruder. In production, there is no Stripe CLI, so the step for
+    configuring this is a bit different. See Stripe's documentation on
+    [taking webhooks live](https://stripe.com/docs/webhooks/go-live) for
+    more details.
+- Copy the webhook signing secret and set it as `stripe_webhook_endpoint_secret`
+  in `zproject/dev-secrets.conf`.
+- Your development environment is now all set to receive webhook events from
+  Stripe.
+- With `tools/run-dev` stopped, you can run `./manage.py
+populate_billing_realms` to populate different billing states, both
+  Cloud and self-hosted, with various initial plans and billing schedules.
+- Feel free to modify `populate_billing_realms` to add more states if they
+  seem useful in your testing. After running the command, you will see a list of
+  populated organizations.
+- Populated Cloud-style `Realms` can be accessed as follows:
+  - Logout and go to `localhost:9991/devlogin`.
+  - Select the realm from the `Realms` dropdown you wist to test.
+  - Login as the only available user.
+  - Go to `/billing`.
+- Populated `RemoteZulipServer` customers can be accessed by going to
+  `http://selfhosting.zulipdev.com:9991/serverlogin/` and providing the
+  credentials in the login form for the server state you wish to
+  test. The credentials are printed in the terminal by `./manage.py
+populate_billing_realms`.
+- Populated `RemoteRealm` customers can be accessed simply by following
+  their links printed in the terminal in the `./manage.py
+populate_billing_realms` output.
+
+### Test card numbers
+
+Stripe provides various card numbers to test for specific responses from Stripe.
+The most commonly used ones are mentioned in below wherever appropriate. The full
+list is available [here](https://stripe.com/docs/testing#cards).
+
+### Flows to test
+
+There are various flows that you can test manually from the browser. Here are
+a few things to keep in mind while conducting these tests manually:
+
+- The flows work from start to end as expected.
+- We show appropriate success and error messages to users.
+- Charges are made or not made (free trial) as expected. You can verify this
+  through the Stripe dashboard. However, this is not super important since
+  our automated tests take care of such granular testing for us.
+- Renewals can be tested by calling `./manage.py invoice_plans --date
+2024-04-30T08:12:53` -- this will run invoicing, including
+  end-of-cycle updates, as though the current date is as specified.
+
+#### Upgrading a Zulip Cloud organization
+
+Here are some flows to test when upgrading a Zulip Cloud organization:
+
+- When free trials are not enabled, i.e. `CLOUD_FREE_TRIAL_DAYS` is not set
+  to any value in `dev_settings.py` (aka the default). You can
+  double-check that the setting is disabled by verifying
+  `./scripts/get-django-setting CLOUD_FREE_TRIAL_DAYS` returns 0.
+
+  - Using a valid card number like `4242 4242 4242 4242`, the
+    official Visa example credit card number.
+  - Using an invalid card number like `4000000000000341`, which will add the card
+    to the customer account but the charge will fail.
+    - Retry the upgrade after adding a new card by clicking on the retry upgrade
+      link.
+    - Retry the upgrade from scratch.
+
+- Upgrade an organization when free trials are enabled. The free
+  trials setting has been (possibly permanently) disabled in
+  production for some time now, so testing this code path is not a
+  priority. You can set `CLOUD_FREE_TRIAL_DAYS` to any number greater than
+  `0` in `dev_settings.py` to enable free trials. There are two
+  different flows to test here:
+  - Right after the organization is created by following the instructions in the
+    onboarding page.
+    - Make sure that after the upgrade is complete the billing page shows a link to
+      go to the organization.
+  - By manually going to the `/billing` page and upgrading the organization.
+
+#### Upgrading a remote Zulip organization
+
+Here are some flows to test when upgrading a remote Zulip organization:
+
+- Free trial for remote organizations is enabled by default by setting
+  `SELF_HOSTING_FREE_TRIAL_DAYS` to `30` days. You can change this
+  value and other settings for your development environment only in
+  `zproject/custom_dev_settings.py`, or secrets in
+  `zproject/dev-secrets.conf`. Note that this only provides free trail
+  for the basic plan.
+
+  - Using a valid card number like `4242 4242 4242 4242`, the
+    official Visa example credit card number.
+  - Using an invalid card number like `4000000000000341`, which will add the card
+    to the customer account but the charge will fail.
+    - Retry the upgrade after adding a new card by clicking on the retry upgrade
+      link.
+    - Retry the upgrade from scratch.
+
+- Try upgrading to Zulip Business using `Pay by card` as described above or
+  `Pay by Invoice`.
+
+#### Changing the card
+
+The following flow should be tested when updating cards in our billing system:
+
+- Go to the `/billing` page of an organization that has already been upgraded
+  using a card. Try changing the card to another valid card such as
+  `5555555555554444`.
+  - Make sure that the flow completes without any errors and that the new card
+    details are now shown on the billing page instead of the older card.
+- You can also try adding a card number that results in it getting attached to
+  the customer's account but charges fail. However, to test this, you need pending
+  invoices since we try to charge for pending invoices when the card is updated.
+  This is tested in our automated tests so it is not strictly necessary to test this
+  manually.
+
+## Upgrading Stripe API versions
+
+Stripe makes pretty regular updates to their API. The process for upgrading
+our code is:
+
+- Go to the [Stripe Dashboard](https://dashboard.stripe.com/developers) in
+  your Stripe account.
+- Upgrade the API version.
+- Run `tools/test-backend --generate-stripe-fixtures --parallel=1 corporate/`.
+- Fix any failing tests, and manually look through `git diff` to understand
+  the changes. Ensure that there are no material changes.
+- Update the value of `STRIPE_API_VERSION` in `corporate/lib/stripe.py`.
+- Commit the changes, and open a PR.
+- Ask Tim Abbott to upgrade the API version on the
+  [Stripe Dashboard](https://dashboard.stripe.com/developers) for Zulip's official
+  Stripe account.
+
+We currently aren't set up to do version upgrades where there are breaking
+changes, though breaking changes should be unlikely given the parts of the
+product we use. The main remaining work for handling breaking version upgrades
+is ensuring that we set the stripe version in our API calls.
+Stripe's documentation for
+[Upgrading your API version](https://stripe.com/docs/upgrades#how-can-i-upgrade-my-api)
+has some additional information.
+
+## Writing tests
+
+Writing new tests is fairly easy. Billing tests live in
+`corporate/tests/`: `test_stripe` covers the Zulip Cloud billing
+flows, `test_stripe_remote_realm` and `test_stripe_remote_server`
+cover self-hosted realms and servers, `test_billing_lib` covers the
+helpers in `corporate/lib/stripe.py`, and `test_sponsorship` covers
+sponsorship requests. If you need to do API calls to stripe, wrap the test
+function in `@mock_stripe` and run `tools/test-backend TEST_NAME
+--generate-stripe-fixtures`. It will run all your calls and generate
+fixtures for any API calls to stripe, so that they can be used to
+consistently run that test offline. You can then commit the new test
+fixtures along with your code changes.
+
+Regenerating the fixtures for all of our existing billing tests is
+expensive, in that it creates extremely large diffs from editing
+dates/IDs that grow the zulip/zulip Git repository and make PRs harder
+to read, both visually and by making the GitHub UI very slow.
+
+So you should generally aim to only (re)generate fixtures where it's
+necessary, such as when we change how we're calling some Stripe APIs
+or adding new tests.
+
+So you'll usually want to pass `--generate-stripe-fixtures` only when
+running the tests for a specific set of tests whose behavior you know
+that you changed. Once you've committed those changes, you can verify
+that everything would pass if new fixtures were generated as follows:
+
+- Run `tools/test-backend corporate/ --generate-stripe-fixtures`.
+- If it passes, you can just run `git reset --hard` to drop the
+  unnecessary fixture updates.
+- If it fails, you can do the same, but then rerun the tests that
+  failed with `--generate-stripe-fixtures` as you debug them.
+- In either case, you can skip the diffs for any unexpected changes in
+  payloads before dropping them, though it's pretty painful to do so
+  given how many files have IDs change.

docs/subsystems/caching.md

@@ -0,0 +1,266 @@
+# Caching in Zulip
+
+Like any product with good performance characteristics, Zulip makes
+extensive use of caching. This article talks about our caching
+strategy, focusing on how we use `memcached` (since it's the thing
+people generally think about when they ask about how a server does
+caching).
+
+## Backend caching with memcached
+
+On the backend, Zulip uses `memcached`, a popular key-value store, for
+caching. Our `memcached` caching helps let us optimize Zulip's
+performance and scalability, since we often avoid overhead related
+to database requests. With Django a typical trivial query can
+often take 3-10x as long as a memcached fetch.
+
+We use Django's built-in caching integration to manage talking to
+memcached, and then a small application-layer library
+(`zerver/lib/cache.py`).
+
+It's common for projects using a caching system like `memcached` to
+either have the codebase littered with explicit requests to interact
+with the cache (or flush data from a cache), or (worse) be littered
+with weird bugs that disappear after you flush memcached.
+
+Caching bugs are a pain to track down, because they generally require
+an extra and difficult-to-guess step to reproduce (namely, putting the
+wrong data into the cache).
+
+So we've designed our backend to ensure that if we write a small
+amount of Zulip's core caching code correctly, then the code most developers
+naturally write will both benefit from caching and not create any cache
+consistency problems.
+
+The overall result of this design is that for many places in the
+Zulip's Django codebase, all one needs to do is call the standard
+accessor functions for data (like `get_user` to fetch
+user objects, or, for view code, functions like
+`access_stream_by_id`, which checks permissions), and everything will
+work great. The data fetches automatically benefit from `memcached`
+caching, since those accessor methods have already been written to
+transparently use Zulip's memcached caching system, and the developer
+doesn't need to worry about whether the data returned is up-to-date:
+it is. In the following sections, we'll talk about how we make this
+work.
+
+As a side note, the policy of using these accessor functions wherever
+possible is a good idea, regardless of caching, because the functions
+also generally take care of details you might not think about
+(e.g., case-insensitive matching of channel names or email addresses).
+It's amazing how slightly tricky logic that's duplicated in several
+places invariably ends up buggy in some of those places, and in
+aggregate we call these accessor functions hundreds of times in
+Zulip. But the caching is certainly a nice bonus.
+
+### The core implementation
+
+The `get_user` function is a pretty typical piece of code using this
+framework; as you can see, it's very little code on top of our
+`cache_with_key` decorator:
+
+```python
+def user_profile_by_email_realm_id_cache_key(email: str, realm_id: int) -> str:
+    return f"user_profile:{hashlib.sha1(email.strip().encode()).hexdigest()}:{realm_id}"
+
+def user_profile_by_email_realm_cache_key(email: str, realm: "Realm") -> str:
+    return user_profile_by_email_realm_id_cache_key(email, realm.id)
+
+@cache_with_key(user_profile_by_email_realm_cache_key, timeout=3600 * 24 * 7)
+def get_user(email: str, realm: Realm) -> UserProfile:
+    # A small amount of complexity, of prefetching additional relationd objects,
+    # has been trimmed here
+    return UserProfile.objects.get(email__iexact=email.strip(), realm=realm)
+```
+
+This decorator implements a pretty classic caching paradigm:
+
+- The `user_profile_by_email_realm_id_cache_key` function defines a
+  unique map from a canonical form of its arguments to a string. These
+  strings are namespaced (the `user_profile:` part) so that they won't
+  overlap with other caches, and encode the arguments so that two uses
+  of this cache won't overlap. In this case, a hash of the email
+  address and realm ID are those canonicalized arguments. (The
+  `make_safe_digest` is important to ensure we don't send special
+  characters to memcached). And we have two versions, depending
+  whether the caller has access to a `Realm` or just a `realm_id`.
+- When `get_user` is called, `cache_with_key` will compute the key,
+  and do a Django `cache_get` query for the key (which goes to
+  memcached). If the key is in the cache, it just returns the value.
+  Otherwise, it fetches the value from the database (using the actual
+  code in the body of `get_user`), and then stores the value back to
+  that memcached key before returning the result to the caller.
+- Cache entries expire after the timeout; in this case, a week.
+  Though in frequently deployed environments like chat.zulip.org,
+  often cache entries will stop being used long before that, because
+  `KEY_PREFIX` is rotated every time we deploy to production; see
+  below for details.
+
+We use this decorator in about 30 places in Zulip, and it saves a
+huge amount of otherwise very self-similar caching code.
+
+### Cautions
+
+The one thing to be really careful with in using `cache_with_key` is
+that if an item is in the cache, the body of `get_user` (above) is
+never called. This means some things that might seem like clever code
+reuse are actually a really bad idea. For example:
+
+- Don't add a `get_active_user` function that uses the same cache key
+  function as `get_user` (but with a different query that filters our
+  deactivated users). If one called `get_active_user` to access a
+  deactivated user, the right thing would happen, but if you called
+  `get_user` to access that user first, then the `get_active_user`
+  function would happily return the user from the cache, without ever
+  doing your more restrictive query.
+
+So remember: Use separate cache key functions for different data sets,
+even if they feature the same objects.
+
+### Cache invalidation after writes
+
+The caching strategy described above works pretty well for anything
+where the state it's storing is immutable (i.e. never changes). With
+mutable state, one needs to do something to ensure that the Python
+processes don't end up fetching stale data from the cache after a
+write to the database.
+
+We handle this using Django's longstanding
+[post_save signals][post-save-signals] feature. Django signals let
+you configure some code to run every time Django does something (for
+`post_save`, right after any write to the database using Django's
+`.save()`).
+
+There's a handful of lines in `zerver/models/*.py` like these that
+configure this:
+
+```python
+post_save.connect(flush_realm, sender=Realm)
+post_save.connect(flush_user_profile, sender=UserProfile)
+```
+
+Once this `post_save` hook is registered, whenever one calls
+`user_profile.save(...)` with a UserProfile object in our Django
+project, Django will call the `flush_user_profile` function. Zulip is
+systematic about using the standard Django `.save()` function for
+modifying `user_profile` objects (and passing the `update_fields`
+argument to `.save()` consistently, which encodes which fields on an
+object changed). This means that all we have to do is write those
+cache-flushing functions correctly, and people writing Zulip code
+won't need to think about (or even know about!) the caching.
+
+Each of those flush functions basically just computes the list of
+cache keys that might contain data that was modified by the
+`.save(...)` call (based on the object changed and the `update_fields`
+data), and then sends a bulk delete request to `memcached` to remove
+those keys from the cache (if present).
+
+Maintaining these flush functions requires some care (every time we
+add a new cache, we need to look through them), but overall it's a
+pretty simple algorithm: If the changed data appears in any form in a
+given cache key, that cache key needs to be cleared. E.g., the
+`active_user_ids_cache_key` cache for a realm needs to be flushed
+whenever a new user is created in that realm, or user is
+deactivated/reactivated, even though it's just a list of IDs and thus
+doesn't explicitly contain the `is_active` flag.
+
+Once you understand how that works, it's pretty easy to reason about
+when a particular flush function should clear a particular cache; so
+the main thing that requires care is making sure we remember to reason
+about that when changing cache semantics.
+
+But the overall benefit of this cache system is that almost all the
+code in Zulip just needs to modify Django model objects and call
+`.save()`, and the caching system will do the right thing.
+
+### Production deployments and database migrations
+
+When upgrading a Zulip server, it's important to avoid having one
+version of the code interact with cached objects from another version
+that has a different data layout. In Zulip, we avoid this through
+some clever caching strategies. Each "deployment directory" for Zulip
+in production has inside it a `var/remote_cache_prefix` file,
+containing a cache prefix (`KEY_PREFIX` in the code) that is
+automatically appended to the start of any cache keys accessed by that
+deployment directory (this is all handled internally by
+`zerver/lib/cache.py`).
+
+This completely solves the problem of potentially having contamination
+from inconsistent versions of the source code / data formats in the cache.
+
+### Automated testing and memcached
+
+For Zulip's `test-backend` unit tests, we use the same strategy. In
+particular, we just edit `KEY_PREFIX` before each unit test; this
+means each of the thousands of test cases in Zulip has its own
+independent memcached key namespace on each run of the unit tests. As
+a result, we never have to worry about memcached caching causing
+problems across multiple tests.
+
+This is a really important detail. It makes it possible for us to do
+assertions in our tests on the number of database queries or memcached
+queries that are done as part of a particular function/route, and have
+those checks consistently get the same result (those tests are great
+for catching bugs where we accidentally do database queries in a
+loop). And it means one can debug failures in the test suite without
+having to consider the possibility that memcached is somehow confusing
+the situation.
+
+Further, this `KEY_PREFIX` model means that running the backend tests
+won't potentially conflict with whatever you're doing in a Zulip
+development environment on the same machine, which also saves a ton of
+time when debugging, since developers don't need to think about things
+like whether some test changed Hamlet's email address and that's why
+login is broken.
+
+More full-stack test suites like `test-js-with-puppeteer` or `test-api`
+use a similar strategy (set a random `KEY_PREFIX` at the start of the
+test run).
+
+### Manual testing and memcached
+
+Zulip's development environment will automatically flush (delete all
+keys in) `memcached` when provisioning and when starting `run-dev`.
+You can run the server with that behavior disabled using
+`tools/run-dev --no-clear-memcached`.
+
+### Performance
+
+One thing be careful about with memcached queries is to avoid doing
+them in loops (the same applies for database queries!). Instead, one
+should use a bulk query. We have a fancy function,
+`generate_bulk_cached_fetch`, which is super magical and handles this
+for us, with support for a bunch of fancy features like marshalling
+data before/after going into the cache (e.g., to compress `message`
+objects to minimize data transfer between Django and memcached).
+
+## In-process caching in Django
+
+We generally try to avoid in-process backend caching in Zulip's Django
+codebase, because every Zulip production installation involves
+multiple servers. We do have a few, however:
+
+- `@cache_for_current_request`: We use this decorator to
+  cache values in memory during the lifetime of a request. We use this
+  for linkifiers and display recipients. The middleware knows how to
+  flush the relevant in-memory caches at the start of a request.
+- Caches of various data, like the `SourceMap` object, that are
+  expensive to construct, not needed for most requests, and don't
+  change once a Zulip server has been deployed in production.
+
+## Browser caching of state
+
+Zulip makes extensive use of caching of data in the browser and mobile
+apps; details like which users exist, with metadata like names and
+avatars, similar details for channels, recent message history, etc.
+
+This data is fetched in the `/register` endpoint (or `page_params`
+for the web app), and kept correct over time. The key to keeping these
+state up to date is Zulip's
+[real-time events system](events-system.md), which
+allows the server to notify clients whenever state that might be
+cached by clients is changed. Clients are responsible for handling
+the events, updating their state, and rerendering any UI components
+that might display the modified state.
+
+[post-save-signals]: https://docs.djangoproject.com/en/5.0/ref/signals/#post-save

docs/subsystems/client.md

@@ -0,0 +1,38 @@
+# Clients in Zulip
+
+`zerver.models.Client` is Zulip's analogue of the HTTP User-Agent
+header (and is populated from User-Agent). It exists for use in
+analytics and other places to provide human-readable summary data
+about "which Zulip client" was used for an operation (e.g., was it the
+Android app, the desktop app, or a bot?).
+
+In general, it shouldn't be used for anything controlling the behavior
+of Zulip; it's primarily intended to assist debugging.
+
+## Analytics
+
+A `Client` is used to sort messages into client categories such as
+`ZulipElectron` on the `/stats` page. For more information see,
+[Analytics](analytics.md).
+
+## Integrations
+
+Generally, integrations in Zulip should declare a unique User-Agent,
+so that it's easy to figure out which integration is involved when
+debugging an issue. For incoming webhook integrations, we do that
+conveniently via the auth decorators (as we will describe shortly);
+other integrations generally should set the first User-Agent element
+on their HTTP requests to something of the form
+ZulipIntegrationName/1.2 so that they are categorized properly.
+
+The `webhook_view` auth decorator, used for most incoming
+webhooks, accepts the name of the integration as an argument and uses
+it to generate a client name that it adds to the `request_notes`
+object that can be accessed with the `request` (Django
+[HttpRequest](https://docs.djangoproject.com/en/5.0/ref/request-response/#django.http.HttpRequest))
+object via `zerver.lib.request.get_request_notes(request)`.
+
+In most integrations, `request_notes.client` is then passed to
+`check_send_webhook_message`, where it is used to keep track of which client
+sent the message (which in turn is used by analytics). For more
+information, see [the incoming webhook walkthrough](../webhooks/incoming-webhooks-walkthrough.md).

docs/subsystems/database-concurrency.md

@@ -0,0 +1,129 @@
+# Database concurrency
+
+## select_for_update()
+
+When a transaction does a read-then-write sequence where correctness depends on the
+state it read, it needs a row lock to serialize that operation with other
+writers. Without a lock, another transaction can commit between the read and
+write, making the transaction have stale information. Two requests/workers can
+then apply conflicting updates.
+
+Django exposes a way to control the appropriate row locks via
+[`QuerySet.select_for_update()`][django-select-for-update-doc]. This produces
+either `SELECT ... FOR NO KEY UPDATE` or `SELECT ... FOR UPDATE` queries,
+depending on the `no_key` boolean argument passed to `select_for_update()`. The
+key decision at each call site is choosing the appropriate lock strength.
+
+[django-select-for-update-doc]: https://docs.djangoproject.com/en/5.0/ref/models/querysets/#select-for-update
+
+### What select_for_update() does
+
+`select_for_update()` issues a `SELECT ... FOR ...` query that locks the
+selected rows until the surrounding transaction commits or rolls back. A
+transaction trying to take a conflicting lock on those rows will block.
+The behavior can be modified to either fail immediately if the lock can't be taken
+or skip the rows we cannot lock, via the `nowait=True` and `skip_locked=True` arguments,
+respectively.
+
+This is the right tool for code paths where correctness depends on reading a
+stable value and then writing based on that value in the same transaction.
+This protection only works if other code paths that mutate the same data also
+use appropriate row locks.
+
+Additionally, `select_for_update()` should be used to ensure a consistent
+order of lock acquisition to prevent deadlocks. In particular, for these two
+example classes of situations:
+
+- If two concurrent transactions attempt to update overlapping sets of rows,
+  inconsistent ordering can lead to a deadlock. For example, if the first
+  transaction locks rows in the order `id=1`, `id=2` and the second transaction
+  locks them in the order `id=2`, `id=1`, they deadlock with the first
+  transaction waiting for the lock on `id=2` to be released, while the second
+  transaction waits for the lock on `id=1`. By using `select_for_update()` with
+  a consistent `order_by()`, we can ensure both transactions acquire locks in
+  the same order, preventing such deadlocks.
+- If two concurrent transactions update rows in two tables in an inconsistent
+  order, they can deadlock in a similar way. For example, if the first
+  transaction updates a `Message` row, followed by some related `UserMessage`
+  rows, while the second transaction updates the `UserMessage` rows, followed
+  by the `Message` row, they can deadlock. The first transaction can end up
+  stuck waiting for the lock on `UserMessage` rows, while the second
+  transaction waits for the `Message` row lock. To prevent this, both
+  transactions should use `select_for_update()` on the intended `Message` at
+  the beginning - ensuring they execute serially and preventing the deadlock
+  risk.
+
+Because the lock lives for the duration of the transaction, we should keep the
+`transaction.atomic()` block small once the lock is acquired.
+
+As a project policy, Zulip requires an explicit `no_key=` kwarg on every
+`select_for_update()` call, enforced by semgrep.
+
+### Lock strengths (`no_key`)
+
+Django's `select_for_update(no_key=...)` maps to PostgreSQL row-level locks:
+
+- `no_key=True` takes a `FOR NO KEY UPDATE` lock.
+- `no_key=False` takes a `FOR UPDATE` lock.
+
+`FOR UPDATE` is stronger. In particular, it conflicts with the `FOR KEY SHARE` lock,
+which PostgreSQL acquires for foreign-key checks when inserting/updating
+referencing rows. Therefore, `FOR UPDATE` on a parent row will block inserts in other
+tables that reference that row.
+
+`FOR NO KEY UPDATE` still protects against concurrent writes to the locked row,
+but does not block `FOR KEY SHARE`. This avoids unnecessary cross-table
+contention and is usually the right default, unless the transaction has the possibility
+of deleting some of the selected rows.
+
+See [PostgreSQL's row-level lock documentation][postgres-row-level-lock-doc] for
+further detail.
+
+### Choosing `no_key=True` vs `no_key=False`
+
+Use `no_key=True` when:
+
+- You are updating fields other than FK-referenced key columns (in Zulip
+  today, this is effectively just the row's primary key column).
+- You are serializing access to prevent lost updates/races.
+- The row itself will not be deleted and those key columns are not changing.
+
+Use `no_key=False` when:
+
+- You will delete the locked row (or call helper code that may delete it).
+- You will change key columns that are referenced by foreign keys (as noted
+  above, in Zulip today, this is just the row's primary key column).
+- You intentionally need to block FK-reference creation to that row while the
+  transaction is in progress.
+
+If using `no_key=False`, add a short comment explaining why the stronger lock is
+required.
+
+### Example patterns
+
+Typical update path (`no_key=True`):
+
+```python
+with transaction.atomic():
+    row = Model.objects.select_for_update(no_key=True).get(id=row_id)
+    row.some_number = row.some_number + 1
+    row.save(update_fields=["some_number"])
+```
+
+Delete path (`no_key=False`):
+
+```python
+with transaction.atomic():
+    row = Model.objects.select_for_update(no_key=False).get(id=row_id)
+    maybe_delete_row(row)
+```
+
+When locking rows from a queryset with joins, use `of=("self",)` unless you
+intentionally need to lock related tables too.
+
+```python
+rows = Message.objects.select_related(*Message.DEFAULT_SELECT_RELATED) \
+    .select_for_update(of=("self",), no_key=True)
+```
+
+[postgres-row-level-lock-doc]: https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-ROWS

docs/subsystems/dependencies.md

@@ -0,0 +1,273 @@
+# Provisioning and third-party dependencies
+
+Zulip is a large project, with well over 100 third-party dependencies,
+and managing them well is essential to the quality of the project. In
+this document, we discuss the various classes of dependencies that
+Zulip has, and how we manage them. Zulip's dependency management has
+some really nice properties:
+
+- **Fast provisioning**. When switching to a different commit in the
+  Zulip project with the same dependencies, it takes under 5 seconds
+  to re-provision a working Zulip development environment after
+  switching. If there are new dependencies, one only needs to wait to
+  download the new ones, not all the pre-existing dependencies.
+- **Consistent provisioning**. Every time a Zulip development or
+  production environment is provisioned/installed, it should end up
+  using the exactly correct versions of all major dependencies.
+- **Low maintenance burden**. To the extent possible, we want to
+  avoid manual work and keeping track of things that could be
+  automated. This makes it easy to keep running the latest versions
+  of our various dependencies.
+
+The purpose of this document is to detail all of Zulip's third-party
+dependencies and how we manage their versions.
+
+## Provisioning
+
+We refer to "provisioning" as the process of installing and
+configuring the dependencies of a Zulip development environment. It's
+done using `tools/provision`, and the output is conveniently logged by
+`var/log/provision.log` to help with debugging. Provisioning makes
+use of a lot of caching. Some of those caches are not immune to being
+corrupted if you mess around with files in your repository a lot. We
+have `tools/provision --force` to (still fairly quickly) rerun most
+steps that would otherwise have been skipped due to caching.
+
+In the Vagrant development environment, `vagrant provision` will run
+the provision script; `vagrant up` will boot the machine, and will
+also run an initial provision the first time only.
+
+### PROVISION_VERSION
+
+In `version.py`, we have a special parameter, `PROVISION_VERSION`,
+which is used to help ensure developers don't spend time debugging
+test/linter/etc. failures that actually were caused by the developer
+rebasing and forgetting to provision". `PROVISION_VERSION` has a
+format of `(x, y)`; when `x` doesn't match the value from the last time
+the user provisioned, or `y` is higher than the value from last
+time, most Zulip tools will crash early and ask the user to provision.
+This has empirically made a huge impact on how often developers spend
+time debugging a "weird failure" after rebasing that had an easy
+solution. (Of course, the other key part of achieving this is all the
+work that goes into making sure that `provision` reliably leaves the
+development environment in a good state.)
+
+`PROVISION_VERSION` must be manually updated when making changes that
+require re-running provision, so don't forget about it!
+
+## Philosophy on adding third-party dependencies
+
+In the Zulip project, we take a pragmatic approach to third-party
+dependencies. Overall, if a third-party project does something well
+that Zulip needs to do (and has an appropriate license), we'd love to
+use it rather than reinventing the wheel. If the third-party project
+needs some small changes to work, we prefer to make those changes and
+contribute them upstream. When the upstream maintainer is slow to
+respond, we may use a fork of the dependency until the code is merged
+upstream; as a result, we usually have a few packages in
+`pyproject.toml` that are installed from a GitHub URL.
+
+What we look for in choosing dependencies is whether the project is
+well-maintained. Usually one can tell fairly quickly from looking at
+a project's issue tracker how well-managed it is: a quick look at how
+the issue tracker is managed (or not) and the test suite is usually
+enough to decide if a project is going to be a high-maintenance
+dependency or not. That said, we do still take on some smaller
+dependencies that don't have a well-managed project, if we feel that
+using the project will still be a better investment than writing our
+own implementation of that project's functionality. We've adopted a
+few projects in the past that had a good codebase but whose maintainer
+no longer had time for them.
+
+One case where we apply added scrutiny to third-party dependencies is
+JS libraries. They are a particularly important concern because we
+want to keep the Zulip web app's JS bundle small, so that Zulip
+continues to load quickly on systems with low network bandwidth.
+We'll look at large JS libraries with much greater scrutiny for
+whether their functionality justifies their size than Python
+dependencies, since an extra 50KB of code usually doesn't matter in
+the backend, but does in JavaScript.
+
+## System packages
+
+For the third-party services like PostgreSQL, Redis, nginx, and RabbitMQ
+that are documented in the
+[architecture overview](../overview/architecture-overview.md), we rely on the
+versions of those packages provided alongside the Linux distribution
+on which Zulip is deployed. Because Zulip
+[only supports Debian or Ubuntu in production](../production/requirements.md),
+this usually means `apt`, though we do support
+[other platforms in development](../development/setup-advanced.md). Since
+we don't control the versions of these dependencies, we avoid relying
+on specific versions of these packages wherever possible.
+
+The exact lists of `apt` packages needed by Zulip are maintained in a
+few places:
+
+- For production, in our Puppet configuration, `puppet/zulip/`, using
+  the `Package` and `SafePackage` directives.
+- For development, in `SYSTEM_DEPENDENCIES` in `tools/lib/provision.py`.
+- The packages needed to build a Zulip virtualenv, in
+  `VENV_DEPENDENCIES` in `scripts/lib/setup_venv.py`. These are
+  separate from the rest because (1) we may need to install a
+  virtualenv before running the more complex scripts that, in turn,
+  install other dependencies, and (2) because that list is shared
+  between development and production.
+
+We also rely on the PGroonga PPA for the PGroonga PostgreSQL
+extension, used by our [full-text search](full-text-search.md).
+
+## Python packages
+
+Zulip uses the version of Python itself provided by the host OS for
+the Zulip server. We currently support Python 3.10 and newer, with
+Ubuntu 22.04 being the platform requiring 3.10 support. The comments
+in `.github/workflows/zulip-ci.yml` document the Python versions used
+by each supported platform.
+
+We manage third-party Python packages using [uv](https://docs.astral.sh/uv/),
+with our requirements listed in
+[pyproject.toml](https://docs.astral.sh/uv/concepts/projects/layout/#the-pyprojecttoml),
+and locked versions stored in
+[`uv.lock`](https://docs.astral.sh/uv/concepts/projects/layout/#the-lockfile).
+
+- **Scripts**. Often, we want a script running in production to use
+  the Zulip virtualenv. To make that work without a lot of duplicated
+  code, we have a helpful function,
+  `scripts.lib.setup_path.setup_path`, which on import will put the
+  currently running Python script into the Zulip virtualenv. This is
+  called by `./manage.py` to ensure that our Django code always uses
+  the correct virtualenv as well.
+- **Mypy type checker**. Because we're using mypy in a strict mode,
+  when you add use of a new Python dependency, you usually need to
+  either adds stubs to the `stubs/` directory for the library, or edit
+  `pyproject.toml` in the root of the Zulip project to configure
+  `ignore_missing_imports` for the new library. See
+  [our mypy docs][mypy-docs] for more details.
+
+[mypy-docs]: ../testing/mypy.md
+
+## JavaScript and other frontend packages
+
+We use the same set of strategies described for Python dependencies
+for most of our JavaScript dependencies, so we won't repeat the
+reasoning here.
+
+- We use [pnpm][], a `pip`-like tool for JavaScript, to download most
+  JavaScript dependencies. pnpm talks to the standard [npm][]
+  repository. We use the standard `package.json` file to declare our
+  direct dependencies, with sections for development and
+  production. pnpm takes care of pinning the versions of indirect
+  dependencies in the `pnpm-lock.yaml` file; `pnpm install` updates the
+  `pnpm-lock.yaml` file.
+- `tools/update-prod-static`. This process is discussed in detail in
+  the [static asset pipeline](html-css.md#static-asset-pipeline)
+  article, but we don't use the `node_modules` directories directly in
+  production. Instead, static assets are compiled using our static
+  asset pipeline and it is the compiled assets that are served
+  directly to users. As a result, we don't ship the `node_modules`
+  directory in a Zulip production release tarball, which is a good
+  thing, because doing so would more than double the size of a Zulip
+  release tarball.
+- **Checked-in packages**. In contrast with Python, we have a few
+  JavaScript dependencies that we have copied into the main Zulip
+  repository under `web/third`, often with patches. These date
+  from an era before `npm` existed. It is a project goal to eliminate
+  these checked-in versions of dependencies and instead use versions
+  managed by the npm repositories.
+
+## Node.js and pnpm
+
+Node.js is installed by `scripts/lib/install-node` to
+`/srv/zulip-node` and symlinked to `/usr/local/bin/node`. A pnpm
+symlink at `/usr/local/bin/pnpm` is managed by
+[Corepack](https://nodejs.org/api/corepack.html).
+
+We don't do anything special to try to manage multiple versions of
+Node.js. (Previous versions of Zulip installed multiple versions of
+Node.js using the third-party `nvm` installer, but the current version
+no longer uses `nvm`; if it’s present in `/usr/local/nvm` where
+previous versions installed it, it will now be removed.)
+
+## ShellCheck and shfmt
+
+In the development environment, the `tools/setup/install-shellcheck`
+and `tools/setup/install-shfmt` scripts download binaries for
+ShellCheck and shfmt from GitHub, check them against a known hash, and
+install them to `/usr/local/bin`. These tools are run as part of the
+[linting system](../testing/linters.md).
+
+## Puppet packages
+
+Third-party puppet modules are downloaded from the Puppet Forge into
+subdirectories under `/srv/zulip-puppet-cache`, hashed based on their
+versions; the latest is always symlinked as
+`/srv/zulip-puppet-cache/current`. `zulip-puppet-apply` installs
+these dependencies immediately before they are needed.
+
+## Other third-party and generated files
+
+In this section, we discuss the other third-party dependencies,
+generated code, and other files whose original primary source is not
+the Zulip server repository, and how we provision and otherwise
+maintain them.
+
+### Emoji
+
+Zulip uses the [iamcal emoji data package][iamcal] for its emoji data
+and sprite sheets. We download this dependency using `npm`, and then
+have a tool, `tools/setup/build_emoji`, which reformats the emoji data
+into the files under `static/generated/emoji`. Those files are in
+turn used by our [Markdown processor](markdown.md) and
+`tools/update-prod-static` to make Zulip's emoji work in the various
+environments where they need to be displayed.
+
+Since processing emoji is a relatively expensive operation, as part of
+optimizing provisioning, we use the same caching strategy for the
+compiled emoji data as we use for virtualenvs and `node_modules`
+directories, with `scripts/lib/clean_emoji_cache.py` responsible for
+garbage-collection. This caching and garbage-collection is required
+because a correct emoji implementation involves over 1000 small image
+files and a few large ones. There is a more extended article on our
+[emoji infrastructure](emoji.md).
+
+### Translations data
+
+Zulip's [translations infrastructure](../translating/translating.md) generates
+several files from the source data, which we manage similar to our
+emoji, but without the caching (and thus without the
+garbage-collection). New translations data is downloaded from
+Transifex and then compiled to generate both the production locale
+files and also language data in `locale/language*.json` using
+`manage.py compilemessages`, which extends the default Django
+implementation of that tool.
+
+### Pygments data
+
+The list of languages supported by our Markdown syntax highlighting
+comes from the [pygments][] package. `tools/setup/build_pygments_data` is
+responsible for generating `web/generated/pygments_data.json` so that
+our JavaScript Markdown processor has access to the supported list.
+
+## Modifying provisioning
+
+When making changes to Zulip's provisioning process or dependencies,
+usually one needs to think about making changes in 3 places:
+
+- `tools/lib/provision.py`. This is the main provisioning script,
+  used by most developers to maintain their development environment.
+- `docs/development/dev-setup-non-vagrant.md`. This is our "manual installation"
+  documentation. Strategically, we'd like to move the support for more
+  versions of Linux from here into `tools/lib/provision.py`.
+- Production. Our tools for compiling/generating static assets need
+  to be called from `tools/update-prod-static`, which is called by
+  `tools/build-release-tarball` (for doing Zulip releases) as well as
+  `tools/upgrade-zulip-from-git` (for deploying a Zulip server off of
+  `main`).
+
+[virtualenv]: https://virtualenv.pypa.io/en/stable/
+[virtualenv-clone]: https://github.com/edwardgeorge/virtualenv-clone/
+[pnpm]: https://pnpm.io/
+[npm]: https://npmjs.com/
+[iamcal]: https://github.com/iamcal/emoji-data
+[pygments]: https://pygments.org/

docs/subsystems/directory-structure.md

@@ -0,0 +1,186 @@
+# Directory structure
+
+This page documents the directory structure of the [Zulip server and
+web app project](https://github.com/zulip/zulip). You may also find
+the [new application feature
+tutorial](../tutorials/new-feature-tutorial.md) helpful for
+understanding the flow through these files.
+
+### Core Python files
+
+Zulip uses the [Django web
+framework](https://docs.djangoproject.com/en/5.0/), so a lot of these
+paths will be familiar to Django developers.
+
+- `zproject/urls.py` Main
+  [Django routes file](https://docs.djangoproject.com/en/5.0/topics/http/urls/).
+  Defines which URLs are handled by which view functions or templates.
+
+- `zerver/models/*.py`
+  [Django models](https://docs.djangoproject.com/en/5.0/topics/db/models/)
+  files. Defines Zulip's database tables.
+
+- `zerver/lib/*.py` Most library code.
+
+- `zerver/actions/*.py` Most code doing writes to user-facing
+  database tables lives here. In particular, we have a policy that
+  all code calling `send_event_on_commit` to trigger [pushing data to
+  clients](../subsystems/events-system.md) must live here.
+
+- `zerver/views/*.py` Most [Django views](https://docs.djangoproject.com/en/5.0/topics/http/views/).
+
+- `zerver/webhooks/` Webhook views and tests for [Zulip's incoming webhook integrations](../webhooks/incoming-webhooks-overview.md).
+
+- `zerver/tornado/views.py` Tornado views.
+
+- `zerver/worker/` [Queue workers](../subsystems/queuing.md).
+
+- `zerver/lib/markdown/` [Backend Markdown processor](../subsystems/markdown.md).
+
+- `zproject/backends.py` [Authentication backends](https://docs.djangoproject.com/en/5.0/topics/auth/customizing/).
+
+---
+
+### HTML templates
+
+See [our docs](../subsystems/html-css.md) for details on Zulip's
+templating systems.
+
+- `templates/zerver/` For [Jinja2](http://jinja.pocoo.org/) templates
+  for the backend (for zerver app; logged-in content is in `templates/zerver/app`).
+
+- `web/templates/` [Handlebars](https://handlebarsjs.com/) templates for the frontend.
+
+---
+
+### JavaScript, TypeScript, and other frontend assets
+
+- `web/src/` Zulip's own JavaScript and TypeScript sources.
+
+- `web/styles/` Zulip's own CSS.
+
+- `web/images/` Images bundled with webpack.
+
+- `static/images/` Images served directly to the web.
+
+- `web/third/` Third-party JavaScript and CSS that has been vendored.
+
+- `node_modules/` Third-party JavaScript installed via pnpm.
+
+- `web/icons/` Icons placed in this directory are compiled
+  into an icon font.
+
+---
+
+### Tests
+
+- `zerver/tests/` Backend tests.
+
+- `web/tests/` Node Frontend unit tests.
+
+- `web/e2e-tests/` Puppeteer frontend integration tests.
+
+- `tools/test-*` Developer-facing test runner scripts.
+
+---
+
+### Management commands
+
+These are distinguished from scripts, below, by needing to run a
+Django context (i.e. with database access).
+
+- `zerver/management/commands/`
+  [Management commands](../subsystems/management-commands.md) one might run at a
+  production deployment site (e.g., scripts to change a value or
+  deactivate a user properly).
+
+- `zilencer/management/commands/` includes some dev-specific
+  commands such as `populate_db`, which are not included in
+  the production distribution.
+
+---
+
+### Scripts
+
+- `scripts/` Scripts that production deployments might run manually
+  (e.g., `restart-server`).
+
+- `scripts/lib/` Scripts that are needed on production deployments but
+  humans should never run directly.
+
+- `scripts/setup/` Scripts that production deployments will only run
+  once, during installation.
+
+- `tools/` Scripts used only in a Zulip development environment.
+  These are not included in production release tarballs for Zulip, so
+  that we can include scripts here one wouldn't want someone to run in
+  production accidentally (e.g., things that delete the Zulip database
+  without prompting).
+
+- `tools/setup/` Subdirectory of `tools/` for things only used during
+  the development environment setup process.
+
+- `tools/ci/` Subdirectory of `tools/` for things only used to
+  set up and run our tests in CI. Actual test suites should
+  go in `tools/`.
+
+---
+
+### API and bots
+
+- See the [Zulip API repository](https://github.com/zulip/python-zulip-api).
+  Zulip's Python API bindings, a number of Zulip integrations and
+  bots, and a framework for running and testing Zulip bots, used to be
+  developed in the main Zulip server repo but are now in their own repo.
+
+- `templates/zerver/integrations/` (within `templates/zerver/`, above).
+  Documentation for these integrations.
+
+---
+
+### Production Puppet configuration
+
+This is used to deploy essentially all configuration in production.
+
+- `puppet/zulip/` For configuration for production deployments.
+
+- `puppet/zulip/manifests/profile/standalone.pp` Main manifest for Zulip standalone deployments.
+
+---
+
+### Additional Django apps
+
+- `confirmation` Email confirmation system.
+
+- `analytics` Analytics for the Zulip server administrator (needs work to
+  be useful to normal Zulip sites).
+
+- `corporate` The old Zulip.com website. Not included in production
+  distribution.
+
+- `zilencer` Primarily used to hold management commands that aren't
+  used in production. Not included in production distribution.
+
+---
+
+### Jinja2 compatibility files
+
+- `zproject/jinja2/__init__.py` Jinja2 environment.
+
+---
+
+### Translation files
+
+- `locale/` Backend (Django) and frontend translation data files.
+
+---
+
+### Documentation
+
+- `docs/` Source for this documentation.
+
+---
+
+You can consult the repository's `.gitattributes` file to see exactly
+which components are excluded from production releases (release
+tarballs are generated using `tools/build-release-tarball`).

docs/subsystems/django-upgrades.md

@@ -0,0 +1,36 @@
+# Upgrading Django
+
+This article documents notes on the process for upgrading Zulip to
+new major versions of Django. Here are the steps:
+
+- Carefully read the Django upstream changelog, and `git grep` to
+  check if we're using anything deprecated or significantly modified
+  and put them in an issue (and then starting working through them).
+  Also, note any new features we might want to use after the upgrade,
+  and open an issue listing them;
+  [example](https://github.com/zulip/zulip/issues/2564).
+- Start submitting PRs to do any deprecation-type migrations that work
+  on both the old and new version of Django. The goal here is to have
+  the actual cutover commit be as small as possible, and to test as
+  much of the changes for the migration as we can independently from
+  the big cutover.
+- Check the version support of the third-party Django packages we use
+  (`git grep django pyproject.toml` to see a list), upgrade any as
+  needed and file bugs upstream for any that lack support. Look into
+  fixing said bugs.
+- Look at the pieces of Django code that we've copied and then
+  adapted, and confirm whether Django has any updates to the modified
+  code we should apply. Partial list:
+  - `CursorDebugWrapper`, which we have a modified version of in
+    `zerver/lib/db.py`. See
+    [the issue for contributing this upstream](https://github.com/zulip/zulip/issues/974)
+  - `PasswordResetForm` and any other forms we import from
+    `django.contrib.auth.forms` in `zerver/forms.py` (which has all of
+    our Django forms).
+  - Our AsyncDjangoHandler class has some code copied from the core
+    Django handlers code; look at whether that code was changed in
+    Django upstream.
+  - Our `FilteredManagementUtility` in `manage.py`, which forks the
+    management command discovery code.
+  - `zerver/management/commands/change_password.py` is forked from the
+    upstream `changepassword.py`.

docs/subsystems/email.md

@@ -0,0 +1,167 @@
+# Email
+
+This page has developer documentation on the Zulip email system. If you're
+trying to configure your server to send email, you might be looking for our
+guide to [sending outgoing email](../production/email.md). If you're trying to
+configure an email integration to receive incoming email (e.g., so that users
+can reply to message notification emails via email), you might be interested in
+our instructions for
+[setting up an email integration](https://zulip.com/integrations/email).
+
+On to the documentation. Zulip's email system is fairly straightforward,
+with only a few things you need to know to get started.
+
+- All email templates are in `templates/zerver/emails/`. Each email has three
+  template files: `<template_prefix>.subject.txt`, `<template_prefix>.txt`, and
+  `<template_prefix>.html`. Email templates, along with all other templates
+  in the `templates/` directory, are Jinja2 templates.
+- Most of the CSS and HTML layout for emails is in `email_base_default.html`. Note
+  that email has to ship with all of its CSS and HTML, so nothing in
+  `static/` is useful for an email. If you're adding new CSS or HTML for an
+  email, there's a decent chance it should go in `email_base_default.html`.
+- All email is eventually sent by `zerver.lib.send_email.send_email`. There
+  are several other functions in `zerver.lib.send_email`, but all of them
+  eventually call the `send_email` function. The most interesting one is
+  `send_future_email`. The `ScheduledEmail` entries are eventually processed
+  by a supervisor job that runs `zerver/management/commands/deliver_scheduled_emails.py`.
+- Always use `user_profile.delivery_email`, not `user_profile.email`,
+  when passing data into the `send_email` library. The
+  `user_profile.email` field may not always be valid.
+- A good way to find a bunch of example email pathways is to `git grep` for
+  `zerver/emails` in the `zerver/` directory.
+
+One slightly complicated decision you may have to make when adding an email
+is figuring out how to schedule it. There are 3 ways to schedule email.
+
+- Send it immediately, in the current Django process, e.g., by calling
+  `send_email` directly. An example of this is the `confirm_registration`
+  email.
+- Add it to a queue. An example is the `invitation` email.
+- Send it (approximately) at a specified time in the future, using
+  `send_future_email`. An example is the `onboarding_zulip_topics` email.
+
+Email takes about a quarter second per email to process and send. Generally
+speaking, if you're sending just one email, doing it in the current process
+is fine. If you're sending emails in a loop, you probably want to send it
+from a queue. Documentation on our queueing system is available
+[here](queuing.md).
+
+## Development and testing
+
+All the emails sent in the development environment can be accessed by
+visiting `/emails` in the browser. The way that this works is that
+we've set the email backend (aka what happens when you call the email
+`.send()` method in Django) in the development environment to be our
+custom backend, `EmailLogBackEnd`. It does the following:
+
+- Logs any sent emails to `var/log/email_content.log`. This log is
+  displayed by the `/emails` endpoint
+  (e.g., http://zulip.zulipdev.com:9991/emails).
+- Print a friendly message on console advertising `/emails` to make
+  this nice and discoverable.
+
+### Testing in a real email client
+
+You can also forward all the emails sent in the development
+environment to an email account of your choice by clicking on
+**Forward emails to an email account** on the `/emails` page. This
+feature can be used for testing how the emails gets rendered by
+actual email clients. This is important because web email clients
+have limited CSS functionality, autolinkify things, and otherwise
+mutate the HTML email one can see previewed on `/emails`.
+
+To do this sort of testing, you need to set up an outgoing SMTP
+provider. Our production advice for
+[Gmail](../production/email.md#using-gmail-for-outgoing-email) and
+[transactional email
+providers](../production/email.md#free-outgoing-email-services) are
+relevant; you can ignore the Gmail warning as Gmail's rate limits are
+appropriate for this sort of low-volume testing.
+
+Once you have the login credentials of the SMTP provider, since there
+is not `/etc/zulip/settings.py` in development, configure it using the
+following keys in `zproject/dev-secrets.conf`
+
+- `email_host` - SMTP hostname.
+- `email_port` - SMTP port.
+- `email_host_user` - Username of the SMTP user
+- `email_password` - Password of the SMTP user.
+- `email_use_tls` - Set to `true` for most providers. Else, don't set any value.
+
+Here is an example of how `zproject/dev-secrets.conf` might look if
+you are using Gmail.
+
+```ini
+email_host = smtp.gmail.com
+email_port = 587
+email_host_user = username@gmail.com
+email_use_tls = true
+
+# This is different from your Gmail password if you have 2FA enabled for your Google account.
+# See the configuring Gmail to send email section above for more details
+email_password = gmail_password
+```
+
+### Notes
+
+- Images won't be displayed in a real email client unless you change
+  the `images_base_url` used for emails to a public URL such as
+  `https://chat.zulip.org/static/images/emails` (image links to
+  `localhost:9991` aren't allowed by modern email providers). See
+  `zproject/email_backends.py` for more details.
+
+- While running the backend test suite, we use
+  `django.core.mail.backends.locmem.EmailBackend` as the email
+  backend. The `locmem` backend stores messages in a special attribute
+  of the django.core.mail module, "outbox". The outbox attribute is
+  created when the first message is sent. It’s a list with an
+  EmailMessage instance for each message that would be sent.
+
+## Email templates
+
+Zulip's email templates live under `templates/zerver/emails`. Email
+templates are a messy problem, because on the one hand, you want nice,
+readable markup and styling, but on the other, email clients have very
+limited CSS support and generally require us to inject any CSS we're
+using in the emails into the email as inline styles. And then you
+also need both plain-text and HTML emails. We solve these problems
+using a combination of the
+[css-inline](https://github.com/Stranger6667/css-inline) library and having
+two copies of each email (plain-text and HTML).
+
+So, for each email, there are two source templates: the `.txt` version
+(for plain-text format) as well as a `.html` template. The `.txt` version
+is used directly, while `.html` is processed by `css-inline`, which injects
+the CSS we use for styling our emails (`templates/zerver/emails/email.css`)
+into the templates just before sending an email.
+
+While this model is great for the markup side, it isn't ideal for
+[translations](../translating/translating.md). The Django
+translation system works with exact strings, and having different new
+markup can require translators to re-translate strings, which can
+result in problems like needing 2 copies of each string (one for
+plain-text, one for HTML). Re-translating these strings is
+relatively easy in Transifex, but annoying.
+
+So when writing email templates, we try to translate individual
+sentences that are shared between the plain-text and HTML content
+rather than larger blocks that might contain markup; this allows
+translators to not have to deal with multiple versions of each string
+in our emails.
+
+One can test whether you did the translating part right by running
+`manage.py makemessages` and then searching
+for the strings in `locale/en/LC_MESSAGES/django.po`; if there
+are multiple copies or they contain CSS colors, you did it wrong.
+
+A final note for translating emails is that strings that are sent to
+user accounts (where we know the user's language) are higher-priority
+to translate than things sent to an email address (where we don't).
+E.g., for password reset emails, it makes sense for the code path for
+people with an actual account can be tagged for translation, while the
+code path for the "you don't have an account email" might not be,
+since we might not know what language to use in the second case.
+
+Future work in this space could be to actually generate the plain-text
+versions of emails from the `.html` markup, so that we don't
+need to maintain two copies of each email's text.

docs/subsystems/emoji.md

@@ -0,0 +1,186 @@
+# Emoji
+
+Emoji seem like a simple idea, but there's actually a ton of
+complexity that goes into an effective emoji implementation. This
+document discusses a number of these issues.
+
+Currently, Zulip supports these three display formats for emoji:
+
+- Google
+- Twitter
+- Plain text
+
+## Emoji codes
+
+The Unicode standard has various ranges of characters set aside for
+emoji. So you can put emoji in your terminal using actual Unicode
+characters like 😀 and 👍. If you paste those into Zulip, Zulip will
+render them as the corresponding emoji image.
+
+However, the Unicode committee did not standardize on a set of
+human-readable names for emoji. So, for example, when using the
+popular `:` based style for entering emoji from the keyboard, we have
+to decide whether to use `:angry:` or `:angry_face:` to represent an
+angry face. Different products use different approaches, but for
+purposes like emoji pickers or autocomplete, you definitely want to
+pick exactly one of these names, since otherwise users will always be
+seeing duplicates of a given emoji next to each other.
+
+Picking which emoji name to use is surprisingly complicated! See the
+section on [picking emoji names](#picking-emoji-names) below.
+
+### Custom emoji
+
+Zulip supports custom user-uploaded emoji. We manage those by having
+the name of the emoji be its "emoji code", and using an emoji_type
+field to keep track of it. We are in the progress of migrating Zulip
+to refer to these emoji only by ID, which is a requirement for being
+able to support deprecating old realm emoji in a sensible way.
+
+## Tooling
+
+We use the [iamcal emoji data package][iamcal] to provide sprite
+sheets and individual images for our emoji, as well as a data set of
+emoji categories, code points, etc. The sprite sheets are used
+by the Zulip web app to display emoji in messages, emoji reactions,
+etc. However, we can't use the sprite sheets in some contexts, such
+as missed-message and digest emails, that need to have self-contained
+assets. For those, we use individual emoji files under
+`static/generated/emoji`. The structure of that repository contains
+both files named after the Unicode representation of emoji (as actual
+image files) as well as symlinks pointing to those emoji.
+
+We need to maintain those both for the names used in the iamcal emoji
+data set as well as our old emoji data set (`emoji_map.json`). Zulip
+has a tool, `tools/setup/emoji/build_emoji`, that combines the
+`emoji.json` file from iamcal with the old `emoji_map.json` data set
+to construct the various symlink farms and output files described
+below that support our emoji experience.
+
+The `build_emoji` tool generates the set of files under
+`static/generated/emoji` (or really, it generates the
+`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
+`static/generated/emoji` is a symlink to that tree; we do this in
+order to cache old versions to make provisioning and production
+deployments super fast in the common case that we haven't changed the
+emoji tooling). See [our dependencies document](dependencies.md)
+for more details on this strategy.
+
+The emoji tree generated by this process contains several import elements:
+
+- `emoji_codes.json`: A set of mappings used by the Zulip frontend to
+  understand what Unicode emoji exist and what their shortnames are,
+  used for autocomplete, emoji pickers, etc. This has been
+  deduplicated using the logic in
+  `tools/setup/emoji/emoji_setup_utils.py` to generally only have
+  `:angry:` and not also `:angry_face:`, since having both is ugly and
+  pointless for purposes like autocomplete and emoji pickers.
+- `images/emoji/unicode/*.png`: A farm of emoji
+- `images/emoji/*.png`: A farm of symlinks from emoji names to the
+  `images/emoji/unicode/` tree. This is used to serve individual emoji
+  images, as well as for the
+  [backend Markdown processor](markdown.md) to know which emoji
+  names exist and what Unicode emoji / images they map to. In this
+  tree, we currently include all of the emoji in `emoji-map.json`;
+  this means that if you send `:angry_face:`, it won't autocomplete,
+  but will still work (but not in previews).
+- Some CSS and PNGs for the emoji spritesheets, used in Zulip for
+  emoji pickers where we would otherwise need to download over 1000 of
+  individual emoji images (which would cause a browser performance
+  problem). We have multiple spritesheets: one for each emoji
+  provider that we support (Google, Twitter, EmojiOne, and Apple.).
+
+[iamcal]: https://github.com/iamcal/emoji-data
+
+## Picking emoji names
+
+I think it is fair to say Zulip has by far the best set of emoji names of
+any product at the time of the writing of this document. If you find an
+emoji name you don't like, or think is missing, please let us know!
+
+The following set of considerations is not comprehensive, but has a few
+principles that were applied to the current set of names. We use (strong),
+(medium), and (weak) denote how strong a consideration it is.
+
+- Even with over 1000 symbols, emoji feels surprisingly sparse as a language,
+  and more often than not, if you search for something, you don't find an
+  appropriate emoji for it. So a primary goal for our set of names is to
+  maximize the number of situations in which the user finds an emoji that
+  feels appropriate. (strong)
+
+- Conversely, we remove generic words that will gum up the typeahead. So
+  `:outbox:` instead of `:outbox_tray:`. Each word should count. (medium)
+
+- We aim for the set of names to be as widely culturally applicable as
+  possible, even if the glyphs are not. So `:statue:` instead of
+  `:new_york:` for the statue of liberty, and `:tower:` instead of
+  `:tokyo_tower:`. (strong)
+
+- We remove unnecessary gender descriptions. So `:ok_signal:` instead of
+  `:ok_woman:`. (strong)
+
+- We don't add names that could be inappropriate in school or work
+  environments, even if the use is common on the internet. For example, we
+  have not added `:butt:` for `:peach:`, or `:cheers:` for
+  `:beers:`. (strong)
+
+- Names should be compatible with the four emoji sets we support, but don't
+  have to be compatible with any other emoji set. (medium)
+
+- We try not to use a creative canonical_name for emoji that are likely to
+  be familiar to a large subset of users. This largely applies to certain
+  faces. (medium)
+
+- The set of names should be compatible with the iamcal, gemoji, and Unicode
+  names. Compatible here means that if there is an emoji name a user knows
+  from one of those sets, and the user searches for the key word of that
+  name, they will get an emoji in our set. It is okay if this emoji has a
+  slightly different name or codepoint from the names/codepoints in the
+  other sets. (weak)
+
+Much of the work of picking names went into the first bullet above: making
+the emoji language less sparse. Some tricks and heuristics that were used
+for that:
+
+- There are many near duplicates, like `:dog:` and `:dog_face:`, or
+  `:mailbox:`, `:mailbox_with_mail:`, and `:mailbox_with_no_mail:`. In these
+  cases we repurpose the duplicates to be as useful as we can, like `:dog:`
+  and `:puppy:`, and `:mailbox:`, `:unread_mail:`, `:inbox_zero:` for the
+  ones above. There isn't a ton of flexibility, since we can't change the
+  glyphs. But in most cases we have been able to come up with something.
+
+- Many emoji have commonly understood meanings among people that use emoji a
+  lot, and there are websites and articles that document some of these
+  meanings. A commonly understood meaning can be a great thing to add as an
+  alternate name, since often it is a sign that the meaning is addressing a
+  real gap in the emoji system.
+
+- Many emoji names are unnecessarily specific in iamcal/etc, like
+  `:flower_playing_cards:`, `:izakaya_lantern:`, or `:amphora:`. Renaming
+  them to `:playing_cards:`, `:lantern:`, and `:vase:` makes them more
+  widely usable. In such cases we often keep the specific name as an
+  alternate.
+
+- If there are natural things someone might type, like `:happy:`, we try to
+  find an emoji to match. This extends to things that someone might not
+  think to type, but as soon as someone in the organization discovers it
+  could get wide use, like `:working_on_it:`. Good future work would be to
+  collect (by survey or tooling) things people type into the emoji picker
+  typeahead on chat.zulip.org, and find ways to add those names as
+  alternates.
+
+Other notes
+
+- Occasionally there are near duplicates where we don't have ideas for
+  useful names for the second one. In that case we sometimes remove the
+  emoji rather than have two nearly identical glyphs in the emoji picker and
+  typeahead. For instance, we kept `:spiral_notepad:` and dropped
+  `:spiral_calendar_pad:`. If the concepts are near duplicates but the sets
+  of glyphs look very different, we'll find two names that allow them both
+  to stay.
+
+- We removed many of the moons and clocks, to make the typeahead experience
+  better when searching for something that catches all the moons or all the
+  clocks. We kept all the squares and diamonds and other shapes, even though
+  they have the same problem, since they are commonly used to make emoji art
+  on Twitter, and could conceivably be used the same way on Zulip.

docs/subsystems/events-system.md

@@ -0,0 +1,463 @@
+# Real-time push and events
+
+Zulip's "events system" is the server-to-client push system that
+powers our real-time sync. This document explains how it works; to
+read an example of how a complete feature using this system works,
+check out the
+[new application feature tutorial](../tutorials/new-feature-tutorial.md).
+
+Any single-page web application like Zulip needs a story for how
+changes made by one client are synced to other clients, though having
+a good architecture for this is particularly important for a chat tool
+like Zulip, since the state is constantly changing. When we talk
+about clients, think a browser tab, mobile app, or API bot that needs
+to receive updates to the Zulip data. The simplest example is a new
+message being sent by one client; other clients must be notified in
+order to display the message. But a complete application like Zulip
+has dozens of different types of data that need to be synced to other
+clients, whether it be new channels, changes in a user's name or
+avatar, settings changes, etc. In Zulip, we call these updates that
+need to be sent to other clients **events**.
+
+An important thing to understand when designing such a system is that
+events need to be synced to every client that has a copy of the old
+data if one wants to avoid clients displaying inaccurate data to
+users. So if a user has two browser windows open and sends a message,
+every client controlled by that user as well as any recipients of the
+message, including both of those two browser windows, will receive
+that event. (Technically, we don't need to send events to the client
+that triggered the change, but this approach saves a bunch of
+unnecessary duplicate UI update code, since the client making the
+change can just use the same code as every other client, maybe plus a
+little notification that the operation succeeded).
+
+Architecturally, there are a few things needed to make a successful
+real-time sync system work:
+
+- **Generation**. Generating events when changes happen to data, and
+  determining which users should receive each event.
+- **Delivery**. Efficiently delivering those events to interested
+  clients, ideally in an exactly-once fashion.
+- **UI updates**. Updating the UI in the client once it has received
+  events from the server.
+
+Reactive JavaScript libraries like React and Vue can help simplify the
+last piece, but there aren't good standard systems for doing
+generation and delivery, so we have to build them ourselves.
+
+This document discusses how Zulip solves the generation and delivery
+problems in a scalable, correct, and predictable way.
+
+## Generation system
+
+Zulip's generation system is built around a Python function,
+`send_event_on_commit(realm, event, users)`. It accepts the realm (used for
+sharding), the event data structure (just a Python dictionary with
+some keys and value; `type` is always one of the keys but the rest
+depends on the specific event) and a list of user IDs for the users
+whose clients should receive the event. In special cases such as
+message delivery, the list of users will instead be a list of dicts
+mapping user IDs to user-specific data like whether that user was
+mentioned in that message. The data passed to `send_event_on_commit` are
+simply marshalled as JSON and placed in the `notify_tornado` RabbitMQ
+queue to be consumed by the delivery system.
+
+Usually, this list of users is one of 3 things:
+
+- A single user (e.g., for user-level settings changes).
+- Everyone in the realm (e.g., for organization-level settings changes,
+  like new realm emoji).
+- Everyone who would receive a given message (for messages, emoji
+  reactions, message editing, etc.); i.e. the subscribers to a channel
+  or the people on a direct message thread.
+
+It is the responsibility of the caller of `send_event_on_commit` to choose
+the list of user IDs correctly. There can be security problems if, for
+example, an event containing direct message content is sent to the
+entire organization. However, if an event isn't sent to enough clients,
+there will likely be user-visible real-time sync bugs.
+
+As indicated in the name, `send_event_on_commit` is intended to be
+called inside the database transaction that is changing the state
+itself; this design ensures that the event is only sent if the
+transaction successfully commits.
+
+Most of the hard work in event generation is about defining consistent
+event dictionaries that are clear, readable, will be useful to the
+wide range of possible clients, and make it easy for developers.
+
+## Delivery system
+
+Zulip's event delivery (real-time push) system is based on Tornado,
+which is ideal for handling a large number of open requests. Details
+on Tornado are available in the
+[architecture overview](../overview/architecture-overview.md), but in short it
+is good at holding open a large number of connections for a long time.
+The complete system is about 2000 lines of code in `zerver/tornado/`,
+primarily `zerver/tornado/event_queue.py`.
+
+Zulip's event delivery system is based on "long-polling"; basically
+clients make `GET /json/events` calls to the server, and the server
+doesn't respond to the request until it has an event to deliver to the
+client. This approach is reasonably efficient and works everywhere
+(unlike websockets, which have a decreasing but nonzero level of
+client compatibility problems).
+
+For each connected client, the **event queue server** maintains an
+**event queue**, which contains any events that are to be delivered to
+that client which have not yet been acknowledged by that client.
+Ignoring the subtle details around error handling, the protocol is
+pretty simple; when a client does a `GET /json/events` call, the
+server checks if there are any events in the queue. If there are, it
+returns the events immediately. If there aren't, it records that
+queue as having a waiting client (often called a `handler` in the
+code).
+
+When it pulls an event off the `notify_tornado` RabbitMQ queue, it
+simply delivers the event to each queue associated with one of the
+target users. If the queue has a waiting client, it breaks the
+long-poll connection by returning an HTTP response to the waiting
+client request. If there is no waiting client, it simply pushes the
+event onto the queue.
+
+When starting up, each client makes a `POST /json/register` to the
+server, which creates a new event queue for that client and returns the
+`queue_id` as well as an initial `last_event_id` to the client (it can
+also, optionally, fetch the initial data to save an RTT and avoid
+races; see the below section on initial data fetches for details on
+why this is useful). Once the event queue is registered, the client
+can just do an infinite loop calling `GET /json/events` with those
+parameters, updating `last_event_id` each time to acknowledge any
+events it has received (see `call_on_each_event` in the
+[Zulip Python API bindings][api-bindings-code] for a complete example
+implementation). When handling each `GET /json/events` request, the
+queue server can safely delete any events that have an event ID less
+than or equal to the client's `last_event_id` (event IDs are just a
+counter for the events a given queue has received.)
+
+If network failures were impossible, the `last_event_id` parameter in
+the protocol would not be required, but it is important for enabling
+exactly-once delivery in the presence of potential failures. (Without
+it, the queue server would have to delete events from the queue as
+soon as it attempted to send them to the client; if that specific HTTP
+response didn't reach the client due to a network TCP failure, then
+those events could be lost).
+
+[api-bindings-code]: https://github.com/zulip/python-zulip-api/blob/main/zulip/zulip/__init__.py
+
+The queue servers are a very high-traffic system, processing at a
+minimum one request for every message delivered to every Zulip client.
+Additionally, as a workaround for low-quality NAT servers that kill
+HTTP connections that are open without activity for more than 60s, the
+queue servers also send a heartbeat event to each queue at least once
+every 45s or so (if no other events have arrived in the meantime).
+
+To avoid a large memory and other resource leak, the queues are
+garbage collected after (by default) 10 minutes of inactivity from a
+client, under the theory that the client has likely gone off the
+Internet (or no longer exists) access; this happens constantly. If
+the client returns, it will receive a "queue not found" error when
+requesting events; its handler for this case should just restart the
+client / reload the browser so that it refetches initial data the same
+way it would on startup. Since clients have to implement their
+startup process anyway, this approach adds minimal technical
+complexity to clients. A nice side effect is that if the event queue
+server (which stores queues in memory) were to crash and lose
+its data, clients would recover, just as if they had lost Internet
+access briefly (there is some DoS risk to manage, though).
+
+Note that the garbage-collection system has hooks that are important
+for the implementation of [notifications](notifications.md).
+
+(The event queue server is designed to save any event queues to disk
+and reload them when the server is restarted, and catches exceptions
+carefully, so such incidents are very rare, but it's nice to have a
+design that handles them without leaving broken out-of-date clients
+anyway).
+
+## The initial data fetch
+
+When a client starts up, it usually wants to get 2 things from the
+server:
+
+- The "current state" of various pieces of data, e.g., the current
+  settings, set of users in the organization (for typeahead), channel,
+  messages, etc. (aka the "initial state").
+- A subscription to receive updates to those data when they are
+  changed by a client (aka an event queue).
+
+Ideally, one would get those two things atomically, i.e. if some other
+user changes their name, either the name change happens after the
+fetch (and thus the old name is in the initial state and there will be
+an event in the queue for the name change) or before (the new name is
+in the initial state, and there is no event for that name change in
+the queue).
+
+Achieving this atomicity goals means we save a huge amount of work
+that the N clients for Zulip don't need to worry about a wide range of
+potential rare and hard to reproduce race conditions; we just have to
+implement things correctly once in the Zulip server.
+
+This is quite challenging to do technically, because fetching the
+initial state for a complex web application like Zulip might involve
+dozens of queries to the database, caches, etc. over the course of
+100ms or more, and it is thus nearly impossible to do all of those
+things together atomically. So instead, we use a more complicated
+algorithm that can produce the atomic result from non-atomic
+subroutines. Here's how it works when you make a `register` API
+request; the logic is in `zerver/views/events_register.py` and
+`zerver/lib/events.py`. The request is directly handled by Django:
+
+- Django makes an HTTP request to Tornado, requesting that a new event
+  queue be created, and records its queue ID.
+- Django does all the various database/cache/etc. queries to fetch the
+  data, non-atomically, from the various data sources (see
+  the `fetch_initial_state_data` function).
+- Django makes a second HTTP request to Tornado, requesting any events
+  that had been added to the Tornado event queue since it
+  was created.
+- Finally, Django "applies" the events (see the `apply_events`
+  function) to the initial state that it fetched. E.g., for a name
+  change event, it finds the user data in the `realm_user` data
+  structure, and updates it to have the new name.
+
+### Testing
+
+The design above achieves everything we desire, at the cost that we need to
+write a correct `apply_events` function. This is a difficult function to
+implement correctly, because the situations that it handles almost never
+happen (being race conditions) during manual testing. Fortunately, we have
+a protocol for testing `apply_events` in our automated backend tests.
+
+#### Overview
+
+Once you are completely confident that an "action function" works correctly
+in terms of "normal" operation (which typically involves writing a
+full-stack test for the corresponding POST/GET operation), then you will be
+ready to write a test in `test_events.py`.
+
+The actual code for a `test_events` test can be quite concise:
+
+```python
+def test_default_streams_events(self) -> None:
+    stream = get_stream("Scotland", self.user_profile.realm)
+    events = self.verify_action(lambda: do_add_default_stream(stream))
+    check_default_streams("events[0]", events[0])
+    # (some details omitted)
+```
+
+The real trick is debugging these tests.
+
+The test example above has three things going on:
+
+- Set up some data (`get_stream`)
+- Call `verify_action` with an action function (`do_add_default_stream`)
+- Use a schema checker to validate data (`check_default_streams`)
+
+#### verify_action
+
+All the heavy lifting that pertains to `apply_events` happens within the
+call to `verify_action`, which is a test helper in the `BaseAction` class
+within `test_events.py`.
+
+The `verify_action` function simulates the possible race condition in
+order to verify that the `apply_events` logic works correctly in the
+context of some action function. To use our concrete example above,
+we are seeing that applying the events from the
+`do_add_default_stream` action inside of `apply_events` to a stale
+copy of your state results in the same state dictionary as doing the
+action and then fetching a fresh copy of the state.
+
+In particular, `verify_action` does the following:
+
+- Call `fetch_initial_state_data` to get the current state.
+- Call the action function (e.g., `do_add_default_stream`).
+- Capture the events generated by the action function.
+- Check the events generated are documented in the [OpenAPI
+  schema](../documentation/api.md) defined in
+  `zerver/openapi/zulip.yaml`.
+- Call `apply_events(state, events)`, to get the resulting "hybrid state".
+- Call `fetch_initial_state_data` again to get the "normal state".
+- Compare the two results.
+
+In the event that you wrote the `apply_events` logic correctly the
+first time, then the two states will be identical, and the
+`verify_action` call will succeed and return the events that came from
+the action.
+
+Often you will get the `apply_events` logic wrong at first, which will
+cause `verify_action` to fail. To help you debug, it will print a diff
+between the "hybrid state" and the "normal state" obtained from
+calling `fetch_initial_state_data` after the changes. If you encounter a
+diff like this, you may be in for a challenging debugging exercise. It
+will be helpful to re-read this documentation to understand the rationale
+behind the `apply_events` function. It may also be helpful to read the code
+for `verify_action` itself. Finally, you may want to ask for help on chat.
+
+Before we move on to the next step, it's worth noting that `verify_action`
+only has one required parameter, which is the action function. We
+typically express the action function as a lambda, so that we
+can pass in arguments:
+
+```python
+events = self.verify_action(lambda: do_add_default_stream(stream))
+```
+
+There are some notable optional parameters for `verify_action`:
+
+- `state_change_expected` must be set to `False` if your action
+  doesn't actually require state changes for some reason; otherwise,
+  `verify_action` will complain that your test doesn't really
+  exercise any `apply_events` logic. Typing notifications (which
+  are ephemeral) are a common place where we use this.
+
+- `num_events` will tell `verify_action` how many events the
+  `hamlet` user will receive after the action (the default is 1).
+
+- parameters such as `client_gravatar` and `slim_presence` get
+  passed along to `fetch_initial_state_data` (and it's important
+  to test both boolean values of these parameters for relevant
+  actions).
+
+For advanced use cases of `verify_action`, we highly recommend reading
+the code itself in `BaseAction` (in `test_events.py`).
+
+#### Schema checking
+
+The `test_events.py` system has two forms of schema checking. The
+first is verifying that you've updated the [GET /events API
+documentation](https://zulip.com/api/get-events) to document your new
+event's format for benefit of the developers of Zulip's mobile app,
+terminal app, and other API clients. See the [API documentation
+docs](../documentation/api.md) for details on the OpenAPI
+documentation.
+
+The second is higher-detail check inside `test_events` that this
+specific test generated the expected series of events. Let's look at
+the last line of our example test snippet:
+
+```python
+# ...
+events = self.verify_action(lambda: do_add_default_stream(stream))
+check_default_streams("events[0]", events[0])
+```
+
+We have discussed `verify_action` in some detail, and you will
+note that it returns the actual events generated by the action
+function. It is part of our test discipline in `test_events` to
+verify that the events are formatted in a predictable way.
+
+Ideally, we would test that events match the exact data that we
+expect, but it can be difficult to do this due to unpredictable
+things like database ids. So instead, we just verify the "schema"
+of the event(s). We use a schema checker like `check_default_streams`
+to validate the types of the data.
+
+If you are creating a new event format, then you will have to
+write your own schema checker in `event_schema.py`. Here is
+the example relevant to our example:
+
+```python
+default_streams_event = event_dict_type(
+    required_keys=[
+        ("type", Equals("default_streams")),
+        ("default_streams", ListType(DictType(basic_stream_fields))),
+    ]
+)
+check_default_streams = make_checker(default_streams_event)
+```
+
+Note that `basic_stream_fields` is not shown in these docs. The
+best way to understand how to write schema checkers is to read
+`event_schema.py`. There is a large block comment at the top of
+the file, and then you can skim the rest of the file to see the
+patterns.
+
+When you create a new schema checker for a new event, you not only
+make the `test_events` test more rigorous, you also allow our other
+tools to use the same schema checker to validate event formats in our
+node test fixtures and our OpenAPI documentation.
+
+#### Node testing
+
+Once you've completed backend testing, be sure to add an example event
+in `web/tests/lib/events.cjs`, a test of the
+`server_events_dispatch.js` code for that event in
+`web/tests/dispatch.test.cjs`, and verify your example
+against the two versions of the schema that you declared above using
+`tools/check-schemas`.
+
+#### Code coverage
+
+The final detail we need to ensure that `apply_events` always works
+correctly is to make sure that we have relevant tests for
+every event type that can be generated by Zulip. This can be tested
+manually using `test-backend --coverage BaseAction` and then
+checking that all the calls to `send_event_on_commit` are covered.
+Someday we'll add automation that verifies this directly by inspecting
+the coverage data.
+
+#### page_params
+
+In the Zulip web app, the data returned by the `register` API is
+available via the `page_params` parameter.
+
+### Messages
+
+One exception to the protocol described in the last section is the
+actual messages. Because Zulip clients usually fetch them in a
+separate AJAX call after the rest of the site is loaded, we don't need
+them to be included in the initial state data. To handle those
+correctly, clients are responsible for discarding events related to
+messages that the client has not yet fetched.
+
+Additionally, see
+[the main documentation on sending messages](sending-messages.md).
+
+## Schema changes
+
+When changing the format of events sent into Tornado, it's important
+to make sure we handle backwards-compatibility properly.
+
+- If we're adding a new event type or new fields to an existing event
+  type, we just need to carefully document the changes in the [API
+  documentation](../documentation/api.md), being careful to bump
+  `API_FEATURE_LEVEL` and include a `**Changes**` entry in the updated
+  `GET /events` API documentation. It's also a good idea to and open
+  issues with the mobile and terminal projects to notify them. If the
+  web app should receive the new event type's initial state on browser
+  reload, add it to `FETCH_EVENT_TYPES` in
+  `web/src/server_event_types.ts`, which the web app passes as
+  `fetch_event_types` to `/register`.
+- If we're making changes that could confuse existing client app logic
+  that parses events (e.g., changing the type/meaning of an existing
+  field, or removing a field), we need to be very careful, since Zulip
+  supports old clients connecting to a modern server. See our
+  [release lifecycle](../overview/release-lifecycle.md) documentation
+  for more details on the policy. Our technical solution is to add a
+  `client_capabilities` flag for the new format and have the code
+  continue sending data in the old format for clients that don't
+  declare support for the new capability. `bulk_message_deletion` is
+  a good example to crib from. (A few years later, we'll make the
+  client capability required and remove support for not having it).
+- For most event types, Tornado just passes the event through
+  transparently, and `event_queue.py` requires no changes.
+- However, when changing the format of data used by Tornado code, like
+  renaming the `presence_idle_user_ids` field in `message` events, we
+  need to be careful, because pre-upgrade events may be present in
+  Tornado's queue when we upgrade Tornado. So it's essential to write
+  logic in `event_queue.py` to translate the old format into the new
+  format, or Tornado may crash when upgrading past the relevant
+  commit. We attempt to contain that sort of logic in the `from_dict`
+  function (which is used for changing event queue formats) and
+  `client_capabilities` conditionals (e.g., in
+  `process_deletion_event`). Compatibility code not related to a
+  `client_capabilities` entry should be marked with a
+  `# TODO/compatibility: ...` comment noting when it can be safely deleted;
+  we grep for these comments entries during major releases.
+- Schema changes are a sensitive operation, and like with database
+  schema changes, it's critical to do thoughtful manual testing.
+  E.g., run the mobile app against your test server and verify it
+  handles the new event properly, or arrange for your new Tornado code
+  to actually process a pre-upgrade event and verify via the browser
+  console what came out.

docs/subsystems/full-text-search.md

@@ -0,0 +1,103 @@
+# Full-text search
+
+Zulip supports full-text search, which can be combined arbitrarily
+with Zulip's full suite of narrowing operators. By default, it only
+supports English text, but there is an experimental
+[PGroonga](https://pgroonga.github.io/) integration that provides
+full-text search for all languages.
+
+The user interface and feature set for Zulip's full-text search is
+documented in the in-app "Search filters" reference which can be
+accessed from the Zulip app's gear menu.
+
+## The default full-text search implementation
+
+Zulip uses [PostgreSQL's built-in full-text search
+feature](https://www.postgresql.org/docs/current/textsearch.html),
+with a custom set of English stop words to improve the quality of the
+search results.
+
+In order to optimize the performance of delivering messages, the
+full-text search index is updated for newly sent messages in the
+background, after the message has been delivered. This background
+updating is done by
+`puppet/zulip/files/postgresql/process_fts_updates`, which is usually
+deployed on the database server, but could be deployed on an
+application server instead.
+
+## Multi-language full-text search
+
+Zulip also supports using [PGroonga](https://pgroonga.github.io/) for
+full-text search. While PostgreSQL's built-in full-text search feature
+supports only one language at a time (in Zulip's case, English), the
+PGroonga full-text search engine supports all languages
+simultaneously, including Japanese and Chinese. Once we have tested
+this new backend sufficiently, we expect to switch Zulip deployments
+to always use PGroonga.
+
+### Enabling PGroonga
+
+All steps in this section should be run as the `root` user; on most installs, this can be done by running `sudo -i`.
+
+1. Alter the deployment setting:
+
+   ```bash
+   crudini --set /etc/zulip/zulip.conf machine pgroonga enabled
+   ```
+
+1. Update the deployment to respect that new setting:
+
+   ```bash
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply
+   ```
+
+1. Edit `/etc/zulip/settings.py`, to add:
+
+   ```python
+   USING_PGROONGA = True
+   ```
+
+1. Apply the PGroonga migrations:
+
+   ```bash
+   su zulip -c '/home/zulip/deployments/current/manage.py migrate pgroonga'
+   ```
+
+   Note that the migration may take a long time, and users will be
+   unable to send new messages until the migration finishes.
+
+1. Once the migrations are complete, restart Zulip:
+
+   ```bash
+   su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
+   ```
+
+### Disabling PGroonga
+
+1. Remove the PGroonga migration:
+
+   ```bash
+   su zulip -c '/home/zulip/deployments/current/manage.py migrate pgroonga zero'
+   ```
+
+   If you intend to re-enable PGroonga later, you can skip this step,
+   at the cost of your Message table being slightly larger than it would
+   be otherwise.
+
+1. Edit `/etc/zulip/settings.py`, editing the line containing `USING_PGROONGA` to read:
+
+   ```python
+   USING_PGROONGA = False
+   ```
+
+1. Restart Zulip:
+
+   ```bash
+   su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
+   ```
+
+1. Finally, remove the deployment setting:
+
+   ```bash
+   crudini --del /etc/zulip/zulip.conf machine pgroonga
+   ```

docs/subsystems/hashchange-system.md

@@ -0,0 +1,124 @@
+# URL hashes and deep linking
+
+## Hashchange
+
+The Zulip web application has a nice system of hash (#) URLs that can
+be used to deep-link into the application and allow the browser's
+"back" functionality to let the user navigate between parts of the UI.
+Some examples are:
+
+- `/#settings/your-bots`: Bots section of the settings overlay.
+- `/#channels`: Channels overlay, where the user manages channels
+  (subscription etc.)
+- `/#channels/11/announce`: Channels overlay with channel ID 11 (called
+  "announce") selected.
+- `/#narrow/channel/42-android/topic/fun`: Message feed showing channel
+  "android" and topic "fun". (The `42` represents the id of the
+  channel.)
+
+The main module in the frontend that manages this all is
+`web/src/hashchange.ts` (plus `hash_util.js` for all the parsing
+code), which is unfortunately one of our thorniest modules. Part of
+the reason that it's thorny is that it needs to support a lot of
+different flows:
+
+- The user clicking on an in-app link, which in turn opens an overlay.
+  For example the channels overlay opens when the user clicks the small
+  cog symbol on the left sidebar, which is in fact a link to
+  `/#channels`. This makes it easy to have simple links around the app
+  without custom click handlers for each one.
+- The user uses the "back" button in their browser (basically
+  equivalent to the previous one, as a _link_ out of the browser history
+  will be visited).
+- The user clicking some in-app click handler (e.g., "Channel settings"
+  for an individual channel), that potentially does
+  several UI-manipulating things including, for example, loading the
+  channels overlay, and needs to update the hash without re-triggering
+  the open animation (etc.).
+- Within an overlay like the channels overlay, the user clicks to
+  another part of the overlay, which should update the hash but not
+  re-trigger loading the overlay (which would result in a confusing
+  animation experience).
+- The user is in a part of the web app, and reloads their browser window.
+  Ideally the reloaded browser window should return them to their
+  original state.
+- A server-initiated browser reload (done after a new version is
+  deployed, or when a user comes back after being idle for a while,
+  see [notes below][self-server-reloads]), where we try to preserve
+  extra state (e.g., content of compose box, scroll position within a
+  narrow) using the `/#reload` hash prefix.
+
+When making changes to the hashchange system, it is **essential** to
+test all of these flows, since we don't have great automated tests for
+all of this (would be a good project to add them to the
+[Puppeteer suite][testing-with-puppeteer]) and there's enough complexity
+that it's easy to accidentally break something.
+
+The main external API lives in `web/src/browser_history.js`:
+
+- `browser_history.update` is used to update the browser
+  history, and it should be called when the app code is taking care
+  of updating the UI directly
+- `browser_history.go_to_location` is used when you want the `hashchange`
+  module to actually dispatch building the next page
+
+Internally you have these functions:
+
+- `hashchange.hashchanged` is the function used to handle the hash,
+  whether it's changed by the browser (e.g., by clicking on a link to
+  a hash or using the back button) or triggered internally.
+- `hashchange.do_hashchange_normal` handles most cases, like loading the main
+  page (but maybe with a specific URL if you are narrowed to a
+  channel or topic or direct messages, etc.).
+- `hashchange.do_hashchange_overlay` handles overlay cases. Overlays have
+  some minor complexity related to remembering the page from
+  which the overlay was launched, as well as optimizing in-page
+  transitions (i.e. don't close/re-open the overlay if you can
+  easily avoid it).
+
+## Server-initiated reloads
+
+There are a few circumstances when the Zulip browser window needs to
+reload itself:
+
+- If the browser has been offline for more than 10 minutes, the
+  browser's [event queue][events-system] will have been
+  garbage-collected by the server, meaning the browser can no longer
+  get real-time updates altogether. In this case, the browser
+  auto-reloads immediately in order to reconnect. We have coded an
+  unsuspend callback (based on some clever time logic) that ensures we
+  check immediately when a client unsuspends; grep for `watchdog` to
+  see the code.
+- If a new version of the server has been deployed, we want to reload
+  the browser so that it will start running the latest code. However,
+  we don't want server deploys to be disruptive. So, the backend
+  preserves user-side event queues (etc.) and just pushes a special
+  `restart` event to all clients. That event causes the browser to
+  start looking for a good time to reload, based on when the user is
+  idle (ideally, we'd reload when they're not looking and restore
+  state so that the user never knew it happened!). The logic for
+  doing this is in `web/src/reload.ts`; but regardless we'll reload
+  within 30 minutes unconditionally.
+
+  An important detail in server-initiated reloads is that we
+  desynchronize when browsers start attempting them randomly, in
+  order to avoid a thundering herd situation bringing down the server.
+
+Here are some key functions in the reload system:
+
+- `reload.preserve_state` is called when a server-initiated browser
+  reload happens, and encodes a bunch of data like the current scroll
+  position into the hash.
+- `reload_setup.initialize` handles restoring the preserved state after a
+  reload where the hash starts with `/#reload`.
+
+## All reloads
+
+In addition to saving state as described above when reloading the
+browser, Zulip also does a few bookkeeping things on page reload (like
+cleaning up its event queue, and saving any text in an open compose
+box as a draft).
+
+[testing-with-puppeteer]: ../testing/testing-with-puppeteer.md
+[self-server-reloads]: #server-initiated-reloads
+[events-system]: events-system.md

docs/subsystems/html-css.md

@@ -0,0 +1,308 @@
+# HTML and CSS
+
+## Zulip CSS organization
+
+There are two high-level sections of CSS: the "portico" (logged-out
+pages like `/help/`, `/login/`, etc.), and the app. The Zulip
+application's CSS can be found in the `web/styles/` directory, while
+the portico CSS lives under the `web/styles/portico/` subdirectory.
+
+To generate its CSS files, Zulip uses [PostCSS](https://postcss.org/)
+and a number of PostCSS plugins, including
+[postcss-nesting](https://github.com/csstools/postcss-nesting#readme),
+whose rules are derived from the [CSS Nesting](https://drafts.csswg.org/css-nesting-1/)
+specification.
+
+## Editing Zulip CSS
+
+If you aren't experienced with doing web development and want to make
+CSS changes, we recommend reading the excellent [Chrome developer tools
+guide to the Elements panel and CSS](https://developer.chrome.com/docs/devtools/overview/#elements),
+as well as the [section on viewing and editing CSS](https://developer.chrome.com/docs/devtools/css/)
+to learn about all the great tools that you can use to modify and test
+changes to CSS interactively in-browser (without even having the
+reload the page!).
+
+Our CSS is formatted with [Prettier](https://prettier.io/). You can
+ask Prettier to reformat all code via our [linter
+tool](../testing/linters.md) with `tools/lint --only=prettier --fix`.
+You can also [integrate it with your
+editor](https://prettier.io/docs/en/editors.html).
+
+Zulip's development environment has hot code-reloading configured, so
+changes made in source files will immediately take effect in open
+browser windows, either by live-updating the CSS or reloading the
+browser window (following backend changes).
+
+## CSS style guidelines
+
+### Avoid duplicated code
+
+Without care, it's easy for a web application to end up with thousands
+of lines of duplicated CSS code, which can make it very difficult to
+understand the current styling or modify it. We would very much like
+to avoid such a fate. So please make an effort to reuse existing
+styling, clean up now-unused CSS, etc., to keep things maintainable.
+
+Opt to write CSS in CSS files. Avoid using the `style=` attribute in
+HTML except for styles that are set dynamically. For example, we set
+the colors for specific channels (`{{stream_color}}`) on different
+elements dynamically, in files like `user_stream_list_item.hbs`:
+
+```html
+<span
+  class="stream-privacy-original-color-{{stream_id}} stream-privacy filter-icon"
+  style="color: {{stream_color}}">
+```
+
+But for most other cases, its preferable to define logical classes and
+put your styles in external CSS files such as `zulip.css` or a more
+specific CSS file, if one exists. See the contents of the `web/styles/`
+directory.
+
+### Be consistent with existing similar UI
+
+Ideally, do this by reusing existing CSS declarations, so that any
+improvements we make to the styling can improve all similar UI
+elements.
+
+### Use clear, unique names for classes and object IDs
+
+This makes it much easier to read the code and use `git grep` to find
+where a particular class is used.
+
+Don't use the tag name in a selector unless you have to. In other words,
+use `.foo` instead of `span.foo`. We shouldn't have to care if the tag
+type changes in the future.
+
+Additionally, multi-word class and ID values should be hyphenated,
+also known as _kebab case_. In HTML, opt for `class="my-multiword-class"`,
+with its corresponding CSS selector as `.my-multiword-class`.
+
+## Validating CSS
+
+When changing any part of the Zulip CSS, it's important to check that
+the new CSS looks good at a wide range of screen widths, from very
+wide screen (e.g., 1920px) all the way down to narrow phone screens
+(e.g., 480px).
+
+For complex changes, it's definitely worth testing in a few different
+browsers to make sure things look the same.
+
+## HTML templates
+
+### Behavior
+
+- Templates are automatically recompiled in development when the file
+  is saved; a refresh of the page should be enough to display the latest
+  version. You might need to do a hard refresh, as some browsers cache
+  webpages.
+
+- Variables can be used in templates. The variables available to the
+  template are called the **context**. Passing the context to the HTML
+  template sets the values of those variables to the value they were
+  given in the context. The sections below contain specifics on how the
+  context is defined and where it can be found.
+
+### Backend templates
+
+For text generated in the backend, including logged-out ("portico")
+pages and the web app's base content, we use the [Jinja2][] template
+engine (files in `templates/zerver`).
+
+The syntax for using conditionals and other common structures can be
+found [here][jconditionals].
+
+The context for Jinja2 templates is assembled from a couple places:
+
+- `zulip_default_context` in `zerver/context_processors.py`. This is
+  the default context available to all Jinja2 templates.
+
+- As an argument in the `render` call in the relevant function that
+  renders the template. For example, if you want to find the context
+  passed to `index.html`, you can do:
+
+```console
+$ git grep zerver/app/index.html '*.py'
+zerver/views/home.py:    response = render(request, 'zerver/app/index.html',
+```
+
+The next line in the code being the context definition.
+
+### Frontend templates
+
+For text generated in the frontend, live-rendering HTML from
+JavaScript for things like the main message feed, we use the
+[Handlebars][] template engine (files in `web/templates/`) and
+sometimes work directly from JavaScript code (though as a policy
+matter, we try to avoid generating HTML directly in JavaScript
+wherever possible).
+
+The syntax for using conditionals and other common structures can be
+found [here][hconditionals].
+
+There's no equivalent of `zulip_default_context` for the Handlebars
+templates.
+
+### Toolchain
+
+Handlebars is in our `package.json` and thus ends up in `node_modules`; We use
+handlebars-loader to load and compile templates during the webpack bundling
+stage. In the development environment, webpack will trigger a browser reload
+whenever a template is changed.
+
+### Translation
+
+All user-facing strings (excluding pages only visible to sysadmins or
+developers) should be tagged for [translation][trans].
+
+### Tooltips
+
+Zulip uses [TippyJS](https://atomiks.github.io/tippyjs/) for its tooltips.
+
+## Static asset pipeline
+
+This section documents additional information that may be useful when
+developing new features for Zulip that require front-end changes,
+especially those that involve adding new files. For a more general
+overview, see the [new feature tutorial](../tutorials/new-feature-tutorial.md).
+
+Our [dependencies documentation](dependencies.md) has useful
+relevant background as well.
+
+### Primary build process
+
+Zulip's frontend is primarily JavaScript in the `web/src` directory;
+we are working on migrating these to TypeScript modules. Stylesheets
+are written in CSS extended by various PostCSS plugins; they are
+converted from plain CSS, and we have yet to take full advantage of
+the features PostCSS offers. We use Webpack to transpile and build JS
+and CSS bundles that the browser can understand, one for each entry
+points specified in `web/webpack.*assets.json`; source maps are
+generated in the process for better debugging experience.
+
+In development mode, bundles are built and served on the fly using
+webpack-dev-server with live reloading. In production mode (and when creating a
+release tarball using `tools/build-release-tarball`), the
+`tools/update-prod-static` tool (called by both `tools/build-release-tarball`
+and `tools/upgrade-zulip-from-git`) is responsible for orchestrating the
+webpack build, JS minification and a host of other steps for getting the assets
+ready for deployment.
+
+You can trace which source files are included in which HTML templates
+by comparing the `entrypoint` variables in the HTML templates under
+`templates/` with the bundles declared in `web/webpack.*assets.json`.
+
+### Adding static files
+
+To add a static file to the app (JavaScript, TypeScript, CSS, images, etc),
+first add it to the appropriate place under `static/`.
+
+- Third-party packages from the NPM repository should be added to
+  `package.json` for management by pnpm, this allows them to be upgraded easily
+  and not bloat our codebase. Run `./tools/provision` for pnpm to install the
+  new packages and update its lock file. You should also update
+  `PROVISION_VERSION` in `version.py` in the same commit.
+- Third-party files that we have patched should all go in
+  `web/third/`. Tag the commit with "[third]" when adding or
+  modifying a third-party package. Our goal is to the extent possible
+  to eliminate patched third-party code from the project.
+- Our own JavaScript and TypeScript files live under `web/src`. Ideally,
+  new modules should be written in TypeScript (details on this policy below).
+- CSS files live under `web/styles`.
+- Portico JavaScript ("portico" means for logged-out pages) lives under
+  `web/src/portico`.
+- Custom SVG graphics living under `web/icons` are compiled into
+  custom icon webfonts by webfont-loader according to the
+  `web/icons/template.hbs` template. See
+  [icons](../subsystems/icons.md) for more details on adding icons.
+
+For your asset to be included in a development/production bundle, it
+needs to be accessible from one of the entry points defined either in
+`web/webpack.assets.json` or `web/webpack.dev-assets.json`.
+
+- If you plan to only use the file within the app proper, and not on the login
+  page or other standalone pages, put it in the `app` bundle by importing it
+  in `web/src/bundles/app.ts`.
+- If it needs to be available both in the app and all
+  logged-out/portico pages, import it to
+  `web/src/bundles/common.ts` which itself is imported to the
+  `app` and `common` bundles.
+- If it's just used on a single standalone page which is only used in
+  a development environment (e.g., `/devlogin`) create a new entry
+  point in `web/webpack.dev-assets.json` or it's used in both
+  production and development (e.g., `/stats`) create a new entry point
+  in `web/webpack.assets.json`. Use the `bundle` macro (defined in
+  `templates/zerver/base.html`) in the relevant Jinja2 template to
+  inject the compiled JS and CSS.
+
+If you want to test minified files in development, look for the
+`DEBUG =` line in `zproject/default_settings.py` and set it to `False`.
+
+### How it works in production
+
+A few useful notes are:
+
+- Zulip installs static assets in production in
+  `/home/zulip/prod-static`. When a new version is deployed, before the
+  server is restarted, files are copied into that directory.
+- We use the VFL (versioned file layout) strategy, where each file in
+  the codebase (e.g., `favicon.ico`) gets a new name
+  (e.g., `favicon.c55d45ae8c58.ico`) that contains a hash in it. Each
+  deployment, has a manifest file
+  (e.g., `/home/zulip/deployments/current/staticfiles.json`) that maps
+  codebase filenames to serving filenames for that deployment. The
+  benefit of this VFL approach is that all the static files for past
+  deployments can coexist, which in turn eliminates most classes of
+  race condition bugs where browser windows opened just before a
+  deployment can't find their static assets. It also is necessary for
+  any incremental rollout strategy where different clients get
+  different versions of the site.
+- Some paths for files (e.g., emoji) are stored in the
+  `rendered_content` of past messages, and thus cannot be removed
+  without breaking the rendering of old messages (or doing a
+  mass-rerender of old messages).
+
+### ES6/TypeScript modules
+
+JavaScript modules in the frontend are [ES6
+modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules)
+that are [transpiled by
+webpack](https://webpack.js.org/api/module-methods/#es6-recommended).
+Any variable, function, etc. can be made public by adding the
+[`export`
+keyword](https://developer.mozilla.org/en-US/docs/web/javascript/reference/statements/export),
+and consumed from another module using the [`import`
+statement](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import).
+
+New modules should ideally be written in TypeScript (though in cases
+where one is moving code from an existing JavaScript module, the new
+commit should just move the code, not translate it to TypeScript).
+TypeScript provides more accurate information to development tools,
+allowing for better refactoring, auto-completion and static analysis.
+TypeScript also uses the ES6 module system. See our documentation on
+[TypeScript static types](../testing/typescript.md).
+
+Webpack does not ordinarily allow modules to be accessed directly from
+the browser console, but for debugging convenience, we have a custom
+webpack plugin (`web/debug-require-webpack-plugin.ts`) that exposes
+a version of the `require()` function to the development environment
+browser console for this purpose. For example, you can access our
+`people` module by evaluating
+`let people = require("./src/people")`, or the third-party `lodash`
+module with `let _ = require("lodash")`. This mechanism is **not** a
+stable API and should not be used for any purpose other than
+interactive debugging.
+
+We have one module, `zulip_test`, that’s exposed as a global variable
+using `expose-loader` for direct use in Puppeteer tests and in the
+production browser console. If you need to access a variable or
+function in those scenarios, add it to `zulip_test`. This is also
+**not** a stable API.
+
+[jinja2]: http://jinja.pocoo.org/
+[handlebars]: https://handlebarsjs.com/
+[trans]: https://jinja.palletsprojects.com/en/3.0.x/extensions/#i18n-extension
+[jconditionals]: http://jinja.pocoo.org/docs/2.9/templates/#list-of-control-structures
+[hconditionals]: https://handlebarsjs.com/guide/block-helpers.html#block-helpers
+[translation]: ../translating/translating.md

docs/subsystems/icons.md

@@ -0,0 +1,216 @@
+# Icons
+
+Zulip makes extensive use of icons to decorate elements in the UI as
+well as for compact buttons and other small interactive elements.
+
+## Using icons
+
+- Modern Zulip icons are implemented using a class-based combination
+  like `<i class="zulip-icon zulip-icon-smile"></i>`, which is rendered
+  using generated CSS that maps that class name (`zulip-icon-smile`)
+  to the SVG file located at `web/icons/smile.svg`.
+
+- Older icons use [Font Awesome 4.7](https://fontawesome.com/),
+  declared in our HTML via `<i class="fa fa-paperclip"></i>`. We are
+  migrating away from Font Awesome both for design and licensing
+  reasons (Font Awesome 5.0+ is no longer fully open source).
+
+Always consider [accessibility](../subsystems/accessibility.md) when
+using icons. Typically, this means:
+
+- Icons that are used **purely as a decoration** immediately adjacent a
+  textual label (for example, in our popover menus) should use `aria-hidden`,
+  so that screenreaders ignore them in favor of reading the label.
+
+- Buttons whose **entire label** is an icon should have a
+  [tooltip](../subsystems/html-css.md#tooltips) as well as an
+  `aria-label` declaration, so that screenreaders will explain the
+  button. Generally, the tooltip text should be identical to the
+  `aria-label` text.
+
+## Adding a new icon
+
+A new feature, such as a new menu option, may require a new icon to represent
+it. The issue you're working on may not have an icon specified upfront. In that
+case, you should:
+
+1. Prototype using a [Lucide icon](https://lucide.dev/icons/), which is the
+   preferred source for icons in Zulip. New SVG files must be placed in the
+   `web/icons/` directory to be used (don't forget to `git add` new
+   icons along with your other changes).
+
+1. **When your feature is nearing completion**, post in the [appropriate
+   channel](https://zulip.com/development-community/#where-do-i-send-my-message)
+   in the Zulip development community to discuss what icon should be used (e.g.,
+   #design for web app icons). You can use the discussion thread linked from the
+   issue you're working on if there is one.
+
+1. Once there is general consensus on the icon to be used, be sure to
+   [prepare the icon's SVG file](#preparing-icons-for-use-with-zulip)
+   so that it displays correctly in Zulip.
+
+1. Follow the [attribution guidelines](../contributing/licensing.md)
+   to document the icon's source in `docs/THIRDPARTY/`.
+
+## Preparing icons for use with Zulip
+
+You may discover that your chosen icon looks terribly broken when displayed
+in Zulip. There are two common issues with SVG files that must be manually
+corrected:
+
+1. [Icons that rely on strokes](#correcting-icons-that-include-strokes),
+   rather than paths
+2. [Icons that have an `evenodd` value](#correcting-icons-with-an-evenodd-fill-rule)
+   on the `fill-rule` attribute
+
+Each of those problems requires a different fix, followed by some
+[cleanup of the resulting SVG file](#cleaning-up-the-svg-code).
+
+### Correcting icons that include strokes
+
+For example, the unaltered [Lucide expand icon](https://lucide.dev/icons/expand)
+illustrates what we'd expect to see versus what we actually see after the icon
+goes through Zulip's automated SVG-to-font conversion:
+
+| Expected icon (SVG)                                                               | Broken appearance (SVG to font)                                                      |
+| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
+| ![Expected icon with four arrows](../images/icon-prep-stroke-correct-preview.png) | ![Broken icon with four triangles](../images/icon-prep-stroke-broken-conversion.png) |
+
+What happened there is that the icon uses strokes, which do not properly convert
+to font outlines in Zulip's automated conversion system.
+
+To properly prepare an SVG icon for use with Zulip, you need to make sure that
+all stroke components are converted to paths:
+
+1. Open the SVG file in [Inkscape](https://inkscape.org/), a free and open-source
+   visual editor for vector images, including SVG.
+
+1. Use `Ctrl + A` or otherwise select all components of the icon.
+
+1. Locate Inkscape's Path menu, and choose the Stroke to Path item. You'll
+   immediately see your icon appear to balloon up: this is expected, because
+   the previous stroke in use has now been applied to your icon's converted paths.
+
+1. Examine the Stroke style applied to your paths on the Fill and Stroke tab of
+   Inkscape's object panel. Be sure to reduce the Stroke to 0:
+
+| Stroke incorrectly still applied                                             | Stroke set to zero                                                                       |
+| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
+| ![Ballooned-out icon shapes](../images/icon-prep-stroke-to-path-initial.png) | ![Expected icon shapes without stroke](../images/icon-prep-stroke-to-path-completed.png) |
+
+At this point, you can save your file in the `web/icons/` directory and
+close Inkscape before [cleaning up the SVG file](#cleaning-up-the-svg-code).
+
+### Correcting icons with an `evenodd` fill rule
+
+Certain icons may include a `fill-rule="evenodd"` attribute-value pairing. Those are
+often found on icons where one path appears inside of another. That is the case with
+the custom eye icon here, which turns solid black after going through Zulip's automated
+SVG-to-font conversion:
+
+| Expected icon (SVG)                                                              | Broken appearance (SVG to font)                                                    |
+| -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
+| ![Expected icon with eye shape](../images/icon-prep-evenodd-correct-preview.png) | ![Broken icon with solid shape](../images/icon-prep-evenodd-broken-conversion.png) |
+
+Should you encounter an icon that includes a `fill-rule` (easily spotted in the SVG file's
+code), you'll need to bring it into Inkscape to rework its paths.
+
+1. Open the SVG file in [Inkscape](https://inkscape.org/), a free and open-source
+   visual editor for vector images, including SVG.
+
+1. Use `Ctrl + A` or otherwise select all components of the icon.
+
+1. Locate Inkscape's Path menu, and choose Object to Path. This converts circles
+   and other SVG shapes to paths.
+
+1. Use `Ctrl + A` again to make sure all paths are still selected. From the Path menu,
+   choose Flatten. This typically makes a reasonable combination of the visible areas
+   of multiple paths.
+
+1. Finally, again with `Ctrl + A` to ensure the flattened paths are selected, return to
+   the Path menu and choose Combine. That joins all paths into a single, complex path.
+
+At this point, you can save your file in the `web/icons/` directory and
+close Inkscape before [cleaning up the SVG file](#cleaning-up-the-svg-code).
+
+### Cleaning up the SVG code
+
+While Inkscape does a fine job with altering SVG files for proper SVG-to-font
+conversion, it also leaves a lot of crufty markup behind. So next, open your
+icon's SVG file in your editor of choice.
+
+Once opened in your editor, the file's Inkscape source will look something like this
+(the example here shows only the first 25 lines or so):
+
+```xml
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   width="24"
+   height="24"
+   viewBox="0 0 24 24"
+   fill="none"
+   stroke="currentColor"
+   stroke-width="2"
+   stroke-linecap="round"
+   stroke-linejoin="round"
+   class="lucide lucide-expand-icon lucide-expand"
+   version="1.1"
+   id="svg8"
+   sodipodi:docname="expand.svg"
+   inkscape:version="1.4 (e7c3feb1, 2024-10-09)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg">
+  <defs
+     id="defs8" />
+  <sodipodi:namedview
+     id="namedview8"
+     pagecolor="#ffffff" />
+
+<!-- ...and so on. -->
+```
+
+Your last task is to clean up the SVG's XML so that only its essential components
+remain:
+
+1. Consider using Git to add and temporarily commit the icon as output by Inkscape.
+   If you make a mistake while cleaning up the icon, you can use `git restore` to
+   return to that point.
+
+1. Strip back the opening of the file to include only the `<svg>` tag and its `xmlns`,
+   `width`, `height`, and `viewBox` attributes and their values as set by
+   Inkscape. (There is no need to preserve the `fill` attribute, as it defaults to
+   black for its [initial value](https://www.w3.org/TR/SVG2/painting.html#SpecifyingFillPaint).)
+
+1. Remove all other tags in the file except for the `<path>` tags, which should be
+   cleaned up until they have only a `d` attribute and its value as output by Inkscape.
+
+1. When you are finished, your SVG file's source code should look something like this:
+
+```xml
+<svg width="24" height="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
+  <path d="m 15,14 a 1,1 0 0 0 -0.707031,0.292969 1,1 0 0 0 0,1.414062 l 6,6 a 1,1 0 0 0 1.414062,0 1,1 0 0 0 0,-1.414062 l -6,-6 A 1,1 0 0 0 15,14 Z"/>
+  <path d="m 20.292969,2.2929688 -6,6 a 1,1 0 0 0 0,1.4140625 1,1 0 0 0 1.414062,0 l 6,-6 a 1,1 0 0 0 0,-1.4140625 1,1 0 0 0 -1.414062,0 z"/>
+  <path d="m 21,15.199219 a 1,1 0 0 0 -1,1 V 20 h -3.800781 a 1,1 0 0 0 -1,1 1,1 0 0 0 1,1 H 21 a 1.0001,1.0001 0 0 0 1,-1 v -4.800781 a 1,1 0 0 0 -1,-1 z"/>
+  <path d="m 16.199219,2 a 1,1 0 0 0 -1,1 1,1 0 0 0 1,1 H 20 v 3.8007812 a 1,1 0 0 0 1,1.0000001 1,1 0 0 0 1,-1.0000001 V 3 A 1.0001,1.0001 0 0 0 21,2 Z"/>
+  <path d="m 3,15.199219 a 1,1 0 0 0 -1,1 V 21 a 1.0001,1.0001 0 0 0 1,1 H 7.8007812 A 1,1 0 0 0 8.8007813,21 1,1 0 0 0 7.8007812,20 H 4 v -3.800781 a 1,1 0 0 0 -1,-1 z"/>
+  <path d="m 9,14 a 1,1 0 0 0 -0.7070312,0.292969 l -6,6 a 1,1 0 0 0 0,1.414062 1,1 0 0 0 1.4140625,0 l 6,-6 a 1,1 0 0 0 0,-1.414062 A 1,1 0 0 0 9,14 Z"/>
+  <path d="M 3,2 A 1.0001,1.0001 0 0 0 2,3 V 7.8007812 A 1,1 0 0 0 3,8.8007813 1,1 0 0 0 4,7.8007812 V 4 H 7.8007812 A 1,1 0 0 0 8.8007813,3 1,1 0 0 0 7.8007812,2 Z"/>
+  <path d="m 3,2 a 1,1 0 0 0 -0.7070312,0.2929688 1,1 0 0 0 0,1.4140625 l 6,6 a 1,1 0 0 0 1.4140625,0 1,1 0 0 0 0,-1.4140625 l -6,-6 A 1,1 0 0 0 3,2 Z"/>
+</svg>
+```
+
+With the source code tided up, take a moment to preview your cleaned-up file in your
+browser or operating system. If you're satisfied with how it looks, you can run
+`git add` and `git commit --amend` to commit your cleaned up icon--overwriting the
+output from Inkscape that you committed above.
+
+Your commit should also follow the [attribution guidelines](../contributing/licensing.md)
+and document the icon's original source in `docs/THIRDPARTY/`.
+
+## Updating UI icons in the help center
+
+When changing an icon for an existing feature, be sure to [update the help
+center](../documentation/helpcenter.md#icons) accordingly (`git grep` is your
+friend for finding instances of icons to be replaced).

docs/subsystems/index.md

@@ -0,0 +1,42 @@
+# Subsystems documentation
+
+```{toctree}
+---
+maxdepth: 3
+---
+
+directory-structure
+dependencies
+settings
+html-css
+icons
+accessibility
+events-system
+sending-messages
+notifications
+queuing
+pointer
+markdown
+caching
+performance
+realms
+management-commands
+schema-migrations
+database-concurrency
+hashchange-system
+emoji
+onboarding-steps
+full-text-search
+email
+analytics
+client
+logging
+typing-indicators
+django-upgrades
+input-pills
+unread_messages
+billing
+widgets
+slash-commands
+thumbnailing
+```

docs/subsystems/input-pills.md

@@ -0,0 +1,77 @@
+# UI: input pills
+
+This is a high level and API explanation of the input pill interface in the
+frontend of the Zulip web application.
+
+## Setup
+
+A pill container should have the following markup:
+
+```html
+<div class="pill-container">
+    <div class="input" contenteditable="true"></div>
+</div>
+```
+
+The pills will automatically be inserted in before the ".input" in order.
+
+## Basic usage
+
+```js
+var $pill_container = $("#input_container");
+var pills = input_pill.create({
+    $container: $pill_container,
+    create_item_from_text: user_pill.create_item_from_email,
+    get_text_from_item: user_pill.get_email_from_item,
+    get_display_value_from_item: user_pill.get_display_value_from_item,
+});
+```
+
+You can look at `web/src/user_pill.ts` to see how the above
+methods are implemented. Essentially you just need to convert
+from raw data (like an email) to structured data (like an object
+with display_value, email, and user_id for a user), and vice
+versa.
+
+## Typeahead
+
+Pills almost always work in conjunction with typeahead, and
+you will want to provide a `source` function to typeahead
+that can exclude items from the prior pills. Here is an
+example snippet from our user group settings code.
+
+```js
+source: function () {
+    return user_pill.typeahead_source(pills);
+},
+```
+
+And then in `user_pill.ts`...
+
+```js
+export function typeahead_source(pill_widget) {
+    const persons = people.get_realm_users();
+    return filter_taken_users(persons, pill_widget);
+}
+
+export function filter_taken_users(items, pill_widget) {
+    const taken_user_ids = get_user_ids(pill_widget);
+    items = items.filter((item) => !taken_user_ids.includes(item.user_id));
+    return items;
+}
+```
+
+### `onPillCreate` and `onPillRemove` methods
+
+You can get notifications from the pill code that pills have been
+created/remove.
+
+```js
+pills.onPillCreate(function () {
+    update_save_state();
+});
+
+pills.onPillRemove(function () {
+    update_save_state();
+});
+```

docs/subsystems/logging.md

@@ -0,0 +1,273 @@
+# Logging and error reporting
+
+Having a good system for logging error reporting is essential to
+making a large project like Zulip successful. Without reliable error
+reporting, one has to rely solely on bug reports from users in order
+to produce a working product.
+
+Our goal as a project is to have zero known 500 errors on the backend
+and zero known JavaScript exceptions on the frontend. While there
+will always be new bugs being introduced, that goal is impossible
+without an efficient and effective error reporting framework.
+
+We provide integration with [Sentry][sentry] to make it easier for
+very large installations like zulip.com to manage their exceptions and
+ensure they are all tracked down, but our default email-based system
+is great for small installations.
+
+## Backend error reporting
+
+The [Django][django-errors] framework provides much of the
+infrastructure needed by our error reporting system:
+
+- The ability to send emails to the server's administrators with any
+  500 errors, using `django.utils.log.AdminEmailHandler`.
+- The ability to rate-limit certain errors to avoid sending hundreds
+  of emails in an outage (see `_RateLimitFilter` in
+  `zerver/lib/logging_util.py`)
+- A nice framework for filtering passwords and other important user
+  data from the exception details, which we use in
+  `zerver/filters.py`.
+- Middleware for handling `JsonableError`, which is our standard
+  system for API code to return a JSON-format HTTP error response.
+
+Since 500 errors in any Zulip server are usually a problem the server
+administrator should investigate and/or report upstream, we have this
+email reporting system configured to report errors by default.
+
+[django-errors]: https://docs.djangoproject.com/en/5.0/howto/error-reporting/
+
+### Sentry error logging
+
+Zulip's optional backend [Sentry][sentry] integration will aggregate
+errors to show which users and realms are affected, any logging which
+happened prior to the exception, local variables in each frame of the
+exception, and the full request headers which triggered it.
+
+You can enable it by:
+
+1.  Create a [project][sentry-project] in your Sentry organization
+    with a platform of "Django."
+2.  Copy your [Sentry DSN][sentry-dsn] into `/etc/zulip/settings.py`
+    as `SENTRY_DSN`:
+    ```python3
+    ## Controls the DSN used to report errors to Sentry.io
+    SENTRY_DSN = "https://bbb@bbb.ingest.sentry.io/1234"
+    ```
+3.  As the `zulip` user, restart Zulip by running:
+    ```shell
+    /home/zulip/deployments/current/scripts/restart-server
+    ```
+
+You may also want to enable Zulip's [Sentry deploy
+hook][sentry-deploy-hook].
+
+[sentry-project]: https://docs.sentry.io/product/projects/
+[sentry-dsn]: https://docs.sentry.io/product/sentry-basics/dsn-explainer/
+[sentry-deploy-hook]: ../production/deployment.md#sentry-deploy-hook
+
+### Backend logging
+
+[Django's logging system][django-logging] uses the standard
+[Python logging infrastructure][python-logging]. We have configured
+them so that `logging.exception` and `logging.error` get emailed to
+the server maintainer, while `logging.warning` will just appear in
+`/var/log/zulip/errors.log`. Lower log levels just appear in the main
+server log (as well as in the log for corresponding process, be it
+`django.log` for the main Django processes or the appropriate
+`events_*` log file for a queue worker).
+
+#### Backend logging format
+
+The main Zulip server log contains a line for each backend request.
+It also contains warnings, errors, and the full tracebacks for any
+Python exceptions. In production, it goes to
+`/var/log/zulip/server.log`; in development, it goes to the terminal
+where you run `run-dev`.
+
+In development, it's good to keep an eye on the `run-dev` console
+as you work on backend changes, since it's a great way to notice bugs
+you just introduced.
+
+In production, one usually wants to look at `errors.log` for errors
+since the main server log can be very verbose, but the main server log
+can be extremely valuable for investigating performance problems.
+
+```text
+2016-05-20 14:50:22.056 INFO [zr] 127.0.0.1       GET     302 528ms (db: 1ms/1q) (+start: 123ms) / (unauth@zulip via ?)
+[20/May/2016 14:50:22]"GET / HTTP/1.0" 302 0
+2016-05-20 14:50:22.272 INFO [zr] 127.0.0.1       GET     200 124ms (db: 3ms/2q) /login/ (unauth@zulip via ?)
+2016-05-20 14:50:26.333 INFO [zr] 127.0.0.1       POST    302  37ms (db: 6ms/7q) /accounts/login/local/ (unauth@zulip via ?)
+[20/May/2016 14:50:26]"POST /accounts/login/local/ HTTP/1.0" 302 0
+2016-05-20 14:50:26.538 INFO [zr] 127.0.0.1       POST    200  12ms (db: 1ms/2q) (+start: 53ms) /api/v1/events/internal [1463769771:0/0] (8@zulip via internal)
+2016-05-20 14:50:26.657 INFO [zr] 127.0.0.1       POST    200  10ms (+start: 8ms) /api/v1/events/internal [1463769771:0/0] (8@zulip via internal)
+2016-05-20 14:50:26.959 INFO [zr] 127.0.0.1       GET     200 588ms (db: 26ms/21q) / [1463769771:0] (8@zulip via website)
+```
+
+The format of this output is:
+
+- Timestamp
+- Log level
+- Logger name, abbreviated as "zr" for these Zulip request logs
+- IP address
+- HTTP method
+- HTTP status code
+- Time to process
+- (Optional perf data details, e.g., database time/queries, memcached
+  time/queries, Django process startup time, Markdown processing time,
+  etc.)
+- Endpoint/URL from zproject/urls.py
+- "email via client" showing user account involved (if logged in) and
+  the type of client they used ("web", "Android", etc.).
+
+The performance data details are particularly useful for investigating
+performance problems, since one can see at a glance whether a slow
+request was caused by delays in the database, in the Markdown
+processor, in memcached, or in other Python code.
+
+One useful thing to note, however, is that the database time is only
+the time spent connecting to and receiving a response from the
+database. Especially when response are large, there can often be a
+great deal of Python processing overhead to marshall the data from the
+database into Django objects that is not accounted for in these
+numbers.
+
+#### Searching backend log files
+
+Zulip comes with a tool, `./scripts/log-search`, to quickly search
+through the main `server.log` log file based on a number of different
+axes -- including client IP address, client user-id, request path,
+response code. It can also search the NGINX logs, which provide
+similar information.
+
+Because often the requests to static resources, or to things like user
+avatars, are not as important, they are also filtered out of the
+output by default.
+
+The output shows timestamp, request duration, client IP address,
+response code, and request method, hostname, and path; any property
+which is limited by the tool is not displayed, for conciseness:
+
+```
+zulip@example-prod:~/deployments/current$ ./scripts/log-search realm-name
+22:30:36.593     1ms         2606:2800:220:1:248:1893:25c8:1946       302 GET    /
+22:30:42.508   366ms         2606:2800:220:1:248:1893:25c8:1946       200 GET    /login/
+23:18:30.977     1ms         93.184.216.34                            302 GET    /
+23:18:31.286   132ms         93.184.216.34                            200 GET    /login/
+23:18:48.094    20ms         93.184.216.34                            200 GET    /accounts/password/reset/
+23:18:51.520   149ms         93.184.216.34                            200 GET    /login/
+23:18:59.420    20ms         93.184.216.34                            200 GET    /accounts/password/reset/
+23:19:02.929  1300ms         93.184.216.34                            302 POST   /accounts/password/reset/
+23:19:03.056    93ms         93.184.216.34                            200 GET    /accounts/password/reset/done/
+23:19:08.911    26ms         93.184.216.34                            302 GET    /accounts/password/reset/OA/b56jfp-bd80ee99b98e703456b3bdcd91892be2/
+23:19:09.189   116ms         93.184.216.34                            200 GET    /accounts/password/reset/OA/set-password/
+23:19:18.743   215ms         93.184.216.34                            302 POST   /accounts/password/reset/OA/set-password/
+23:19:18.779    12ms         93.184.216.34                            200 GET    /accounts/password/done/
+23:19:20.796    12ms         93.184.216.34                            200 GET    /accounts/login/
+23:19:29.323   295ms 8@      93.184.216.34                            302 POST   /accounts/login/
+23:19:29.704   362ms 8@      93.184.216.34                            200 GET    /
+23:20:04.980   110ms 8@      93.184.216.34                            200 DELETE /json/users/me/subscriptions
+
+zulip@example-prod:~/deployments/current$ ./scripts/log-search 2606:2800:220:1:248:1893:25c8:1946
+22:30:36.593     1ms          302 GET    https://realm-one.example-prod.example.com/
+22:30:42.508   366ms          200 GET    https://realm-one.example-prod.example.com/login/
+```
+
+See `./scripts/log-search --help` for complete documentation.
+
+## Blueslip frontend error reporting
+
+We have a custom library, called `blueslip` (named after the form used at MIT to
+report problems with the facilities), that logs and reports JavaScript errors in
+the browser. In production, this uses Sentry (if configured) to report and
+aggregate errors. In development, this means displaying a highly visible overlay
+over the message view area, to make exceptions in testing a new feature hard to
+miss.
+
+- Blueslip is implemented in `web/src/blueslip.ts`.
+- In order to capture essentially any error occurring in the browser, in
+  development mode Blueslip listens for the `error` event on `window`.
+- It also has methods for being manually triggered by Zulip JavaScript code for
+  warnings and assertion failures. Explicit `blueslip.error` calls are sent to
+  Sentry, if configured.
+- Blueslip keeps a log of all the notices it has received during a browser
+  session, so that one can see cases where exceptions chained together. You can
+  print this log from the browser console using
+  `blueslip = require("./src/blueslip"); blueslip.get_log()`.
+
+Blueslip supports several error levels:
+
+- `throw new Error(…)`: For fatal errors that cannot be easily recovered
+  from. We try to avoid using it, since it kills the current JS thread, rather
+  than returning execution to the caller.
+- `blueslip.error`: For logging of events that are definitely caused by a bug
+  and thus sufficiently important to be reported, but where we can handle the
+  error without creating major user-facing problems (e.g., an exception when
+  handling a presence update).
+- `blueslip.warn`: For logging of events that are a problem but not important
+  enough to log an error to Sentry in production. They are, however, highlighted
+  in the JS console in development, and appear as breadcrumb logs in Sentry in
+  production.
+- `blueslip.log` (and `blueslip.info`): Logged to the JS console in development
+  and also in the Sentry breadcrumb log in production. Useful for data that
+  might help discern what state the browser was in during an error (e.g., whether
+  the user was in a narrow).
+- `blueslip.debug`: Similar to `blueslip.log`, but are not printed to
+  the JS console in development.
+
+### Sentry JavaScript error logging
+
+Zulip's optional JavaScript [Sentry][sentry] integration will aggregate errors
+to show which users and realms are affected, any logging which happened prior to
+the exception, and any DOM interactions which happened prior to the error.
+
+You can enable it by:
+
+1.  Create a [project][sentry-project] in your Sentry organization
+    with a platform of "JavaScript."
+2.  Copy your [Sentry DSN][sentry-dsn] into `/etc/zulip/settings.py`
+    as `SENTRY_FRONTEND_DSN`:
+    ```python3
+    ## Controls the DSN used to report JavaScript errors to Sentry.io
+    SENTRY_FRONTEND_DSN = "https://bbb@bbb.ingest.sentry.io/1234"
+    ```
+3.  If you wish to [sample][sentry-sample] some fraction of the errors, you
+    should adjust `SENTRY_FRONTEND_SAMPLE_RATE` down from `1.0`.
+4.  As the `zulip` user, restart Zulip by running:
+    ```shell
+    /home/zulip/deployments/current/scripts/restart-server
+    ```
+
+You may also want to enable Zulip's [Sentry deploy
+hook][sentry-deploy-hook].
+
+[sentry-sample]: https://docs.sentry.io/platforms/javascript/configuration/sampling/
+
+## Frontend performance reporting
+
+In order to make it easier to debug potential performance problems in
+the critically latency-sensitive message sending code pathway, we log
+and report to the server the following whenever a message is sent:
+
+- The time the user triggered the message (aka the start time).
+- The time the `send_message` response returned from the server.
+- The time the message was received by the browser from the
+  `get_events` protocol (these last two race with each other).
+- Whether the message was locally echoed.
+- If so, whether there was a disparity between the echoed content and
+  the server-rendered content, which can be used for statistics on how
+  effective our [local echo system](markdown.md) is.
+
+The code is all in `zerver/lib/report.py` and `web/src/sent_messages.ts`.
+
+We have similar reporting for the time it takes to narrow / switch to
+a new view:
+
+- The time the action was initiated
+- The time when the updated message feed was visible to the user
+- The time when the browser was idle again after switching views
+  (intended to catch issues where we generate a lot of deferred work).
+
+[python-logging]: https://docs.python.org/3/library/logging.html
+[django-logging]: https://docs.djangoproject.com/en/5.0/topics/logging/
+[sentry]: https://sentry.io

docs/subsystems/management-commands.md

@@ -0,0 +1,67 @@
+# Management commands
+
+Zulip has a number of [Django management commands][django-docs] that
+live under `{zerver,zilencer,analytics}/management/commands/`.
+
+If you need some Python code to run with a Zulip context (access to
+the database, etc.) in a script, it should probably go in a management
+command. The key thing distinguishing these from production scripts
+(`scripts/`) and development scripts (`tools/`) is that management
+commands can access the database.
+
+While Zulip takes advantage of built-in Django management commands for
+things like managing Django migrations, we also have dozens that we've
+written for a range of purposes:
+
+- Cron jobs to do regular updates, e.g., `update_analytics_counts.py`,
+  `sync_ldap_user_data`, etc.
+- Useful parts of provisioning or upgrading a Zulip development
+  environment or server, e.g., `makemessages`, `compilemessages`,
+  `populate_db`, `fill_memcached_caches`, etc.
+- The actual scripts run by supervisord to run the persistent
+  processes in a Zulip server, e.g., `runtornado` and `process_queue`.
+- For a sysadmin to verify a Zulip server's configuration during
+  installation, e.g., `send_test_email`.
+- As the interface for doing those rare operations that don't have a
+  UI yet, e.g., `deactivate_realm`, `reactivate_realm`,
+  `change_user_email` (for the case where the user doesn't control the
+  old email address).
+- For a sysadmin to easily interact with and script common possible
+  changes they might want to make to the database on a Zulip server.
+  E.g., `send_password_reset_email`, `export`, `purge_queue`.
+
+## Writing management commands
+
+It's generally pretty easy to template off an existing management
+command to write a new one. Some good examples are
+`change_user_email` and `deactivate_realm`. The Django documentation
+is good, but we have a few pieces advice specific to the Zulip
+project.
+
+- Inherit from the `ZulipBaseCommand` class in
+  `zerver/lib/management.py`; this will add some helpful general
+  flags, as well as tools for adding and parsing `--realm` and
+  `--user` flags, so you don't need to write the tedious code of
+  looking those objects up. This is especially important for users,
+  since the library handles the issues around looking up users by
+  email well (if there's a unique user with that email, just modify it
+  without requiring the user to specify the realm as well, but if
+  there's a collision, throw a nice error).
+- Avoid writing a lot of code in management commands; management
+  commands are annoying to unit test, and thus easier to maintain if
+  all the interesting logic is in a nice function that is unit tested
+  (and ideally, also used in Zulip's existing code). Look for code in
+  `zerver/lib/` that already does what you need. For most actions,
+  you can just call a `do_change_foo` type function from
+  `zerver/actions/` to do all the work; this is usually far
+  better than manipulating the database directly, since the library
+  functions used by the UI are maintained to correctly live-update the
+  UI if needed.
+
+Management commands are essentially independent Python scripts with
+access to the Zulip server's database and libraries; so you don't need
+to do anything special like restart the server when iteratively
+testing one, even if testing in a Zulip production environment where
+the server doesn't normally restart whenever a file is edited.
+
+[django-docs]: https://docs.djangoproject.com/en/5.0/howto/custom-management-commands/

docs/subsystems/markdown.md

@@ -0,0 +1,274 @@
+# Markdown implementation
+
+Zulip uses a special flavor of Markdown/CommonMark for its message
+formatting. Our Markdown flavor is unique primarily to add important
+extensions, such as quote blocks and math blocks, and also to do
+previews and correct issues specific to the chat context. Beyond
+that, it has a number of minor historical variations resulting from
+its history predating CommonMark (and thus Zulip choosing different
+solutions to some problems) and based in part on Python-Markdown,
+which is proudly a classic Markdown implementation. We reduce these
+variations with every major Zulip release.
+
+Zulip has two implementations of Markdown. The backend implementation
+at `zerver/lib/markdown/` is based on
+[Python-Markdown](https://pypi.python.org/pypi/Markdown) and is used to
+authoritatively render messages to HTML (and implements
+slow/expensive/complex features like querying the Twitter API to
+render tweets nicely). The frontend implementation is in JavaScript,
+based on [marked.js](https://github.com/chjj/marked)
+(`web/src/echo.ts`), and is used to preview and locally echo
+messages the moment the sender hits Enter, without waiting for round
+trip from the server. Those frontend renderings are only shown to the
+sender of a message, and they are (ideally) identical to the backend
+rendering.
+
+The JavaScript Markdown implementation has a function,
+`markdown.contains_backend_only_syntax`, that is used to check whether a message
+contains any syntax that needs to be rendered to HTML on the backend.
+If `markdown.contains_backend_only_syntax` returns true, the frontend simply won't
+echo the message for the sender until it receives the rendered HTML
+from the backend. If there is a bug where `markdown.contains_backend_only_syntax`
+returns false incorrectly, the frontend will discover this when the
+backend returns the newly sent message, and will update the HTML based
+on the authoritative backend rendering (which would cause a change in
+the rendering that is visible only to the sender shortly after a
+message is sent). As a result, we try to make sure that
+`markdown.contains_backend_only_syntax` is always correct.
+
+## Testing
+
+The Python-Markdown implementation is tested by
+`zerver/tests/test_markdown.py`, and the marked.js implementation and
+`markdown.contains_backend_only_syntax` are tested by
+`web/tests/markdown.test.cjs`.
+
+A shared set of fixed test data ("test fixtures") is present in
+`zerver/tests/fixtures/markdown_test_cases.json`, and is automatically used
+by both test suites; as a result, it is the preferred place to add new
+tests for Zulip's Markdown system. Some important notes on reading
+this file:
+
+- `expected_output` is the expected output for the backend Markdown
+  processor.
+- When the frontend processor doesn't support a feature and it should
+  just be rendered on the backend, we set `backend_only_rendering` to
+  `true` in the fixtures; this will automatically verify that
+  `markdown.contains_backend_only_syntax` rejects the syntax, ensuring
+  it will be rendered only by the backend processor.
+- When the two processors disagree, we set `marked_expected_output` in
+  the fixtures; this will ensure that the syntax stays that way. If
+  the differences are important (i.e. not just whitespace), we should
+  also open an issue on GitHub to track the problem.
+- For mobile push notifications, we need a text version of the
+  rendered content, since the APNS and GCM push notification systems
+  don't support richer markup. Mostly, this involves stripping HTML,
+  but there's some syntax we take special care with. Tests for what
+  this plain-text version of content should be stored in the
+  `text_content` field.
+
+If you're going to manually test some changes in the frontend Markdown
+implementation, the easiest way to do this is as follows:
+
+1. Log in to your development server.
+2. Stop your Zulip server with Ctrl-C, leaving the browser open.
+3. Compose and send the messages you'd like to test. They will be
+   locally echoed using the frontend rendering.
+
+This procedure prevents any server-side rendering. If you don't do
+this, backend will likely render the Markdown you're testing and swap
+it in before you can see the frontend's rendering.
+
+If you are working on a feature that breaks multiple testcases, and want
+to debug the testcases one by one, you can add `"ignore": true` to any
+testcases in `markdown_test_cases.json` that you want to ignore. This
+is a workaround due to lack of comments support in JSON. Revert your
+"ignore" changes before committing. After this, you can run the frontend
+tests with `tools/test-js-with-node markdown` and backend tests with
+`tools/test-backend zerver.tests.test_markdown.MarkdownFixtureTest.test_markdown_fixtures`.
+
+## Changing Zulip's Markdown processor
+
+First, you will likely find these third-party resources helpful:
+
+- **[Python-Markdown](https://pypi.python.org/pypi/Markdown)** is the Markdown
+  library used by Zulip as a base to build our custom Markdown syntax upon.
+- **[Python's XML ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html)**
+  is the part of the Python standard library used by Python Markdown
+  and any custom extensions to generate and modify the output HTML.
+
+When changing Zulip's Markdown syntax, you need to update several
+places:
+
+- The backend Markdown processor (`zerver/lib/markdown/__init__.py`).
+- The frontend Markdown processor (`web/src/markdown.ts` and sometimes
+  `web/third/marked/lib/marked.cjs`), or `markdown.contains_backend_only_syntax` if
+  your changes won't be supported in the frontend processor.
+- If desired, the typeahead logic in `web/src/composebox_typeahead.ts`.
+- The test suite, probably via adding entries to `zerver/tests/fixtures/markdown_test_cases.json`.
+- The in-app Markdown documentation (`markdown_help_rows` in `web/src/info_overlay.ts`).
+- The list of changes to Markdown at the end of this document.
+
+Important considerations for any changes are:
+
+- Security: A bug in the Markdown processor can lead to XSS issues.
+  For example, we should not insert unsanitized HTML from a
+  third-party web application into a Zulip message.
+- Uniqueness: We want to avoid users having a bad experience due to
+  accidentally triggering Markdown syntax or typeahead that isn't
+  related to what they are trying to express.
+- Performance: Zulip can render a lot of messages very quickly, and
+  we'd like to keep it that way. New regular expressions similar to
+  the ones already present are unlikely to be a problem, but we need
+  to be thoughtful about expensive computations or third-party API
+  requests.
+- Database: The backend Markdown processor runs inside a Python thread
+  (as part of how we implement timeouts for third-party API queries),
+  and for that reason we currently should avoid making database
+  queries inside the Markdown processor. This is a technical
+  implementation detail that could be changed with a few days of work,
+  but is an important detail to know about until we do that work.
+- Testing: Every new feature should have both positive and negative
+  tests; they're easy to write and give us the flexibility to refactor
+  frequently.
+
+## Per-realm features
+
+Zulip's Markdown processor's rendering supports a number of features
+that depend on realm-specific or user-specific data. For example, the
+realm could have
+[linkifiers](https://zulip.com/help/add-a-custom-linkifier)
+or [custom emoji](https://zulip.com/help/custom-emoji)
+configured, and Zulip supports mentions for channels, users, and user
+groups (which depend on data like users' names, IDs, etc.).
+
+At a backend code level, these are controlled by the `message_realm`
+object and other arguments passed into `do_convert` (`sent_by_bot`,
+`translate_emoticons`, `mention_data`, etc.). Because
+Python-Markdown doesn't support directly passing arguments into the
+Markdown processor, our logic attaches the data to the Markdown
+processor object via, for example, `_md_engine.zulip_db_data`, and
+then individual Markdown rules can access the data from there.
+
+For non-message contexts (e.g., an organization's profile (aka the
+thing on the right-hand side of the login page), channel descriptions,
+or rendering custom profile fields), one needs to just pass in a
+`message_realm` (see, for example, `zulip_default_context` for the
+organization profile code for this). But for messages, we need to
+pass in attributes like `sent_by_bot` and `translate_emoticons` that
+indicate details about how the user sending the message is configured.
+
+## Zulip's Markdown philosophy
+
+Note that this discussion is based on a comparison with the original
+Markdown, not newer Markdown variants like CommonMark.
+
+Markdown is great for group chat for the same reason it's been
+successful in products ranging from blogs to wikis to bug trackers:
+it's close enough to how people try to express themselves when writing
+plain text (e.g., emails) that it helps more than getting in the way.
+
+The main issue for using Markdown in instant messaging is that the
+Markdown standard syntax used in a lot of wikis/blogs has nontrivial
+error rates, where the author needs to go back and edit the post to
+fix the formatting after typing it the first time. While that's
+basically fine when writing a blog, it gets annoying very fast in a
+chat product; even though you can edit messages to fix formatting
+mistakes, you don't want to be doing that often. There are basically
+2 types of error rates that are important for a product like Zulip:
+
+- What fraction of the time, if you pasted a short technical email
+  that you wrote to your team and passed it through your Markdown
+  implementation, would you need to change the text of your email for it
+  to render in a reasonable way? This is the "accidental Markdown
+  syntax" problem, common with Markdown syntax like the italics syntax
+  interacting with talking about `char *`s.
+
+- What fraction of the time do users attempting to use a particular
+  Markdown syntax actually succeed at doing so correctly? Syntax like
+  required a blank line between text and the start of a bulleted list
+  raise this figure substantially.
+
+Both of these are minor issues for most products using Markdown, but
+they are major problems in the instant messaging context, because one
+can't edit a message that has already been sent before others read it
+and users are generally writing quickly. Zulip's Markdown strategy is
+based on the principles of giving users the power they need to express
+complicated ideas in a chat context while minimizing those two error rates.
+
+## Zulip's changes to Markdown
+
+Below, we document the changes that Zulip has against stock
+Python-Markdown; some of the features we modify / disable may already
+be non-standard.
+
+**Note** This section has not been updated in a few years and is not
+accurate.
+
+### Basic syntax
+
+- Enable `nl2br` extension: this means one newline creates a line
+  break (not paragraph break).
+
+- Allow only `*` syntax for italics, not `_`. This resolves an issue where
+  people were using `_` and hitting it by mistake too often. Asterisks
+  surrounded by spaces won't trigger italics, either (e.g., with stock Markdown
+  `You should use char * instead of void * there` would produce undesired
+  results).
+
+- Allow only `**` syntax for bold, not `__` (easy to hit by mistake if
+  discussing Python `__init__` or something).
+
+- Add `~~` syntax for strikethrough.
+
+- Disable special use of `\` to escape other syntax. Rendering `\\` as
+  `\` was hugely controversial, but having no escape syntax is also
+  controversial. We may revisit this. For now you can always put
+  things in code blocks.
+
+### Lists
+
+- Allow tacking a bulleted list or block quote onto the end of a
+  paragraph, i.e. without a blank line before it.
+
+- Allow only `*` for bulleted lists, not `+` or `-` (previously
+  created confusion with diff-style text sloppily not included in a
+  code block).
+
+- Disable ordered list syntax: stock Markdown automatically renumbers, which
+  can be really confusing when sending a numbered list across multiple
+  messages.
+
+### Links
+
+- Enable auto-linkification, both for `http://...` and guessing at
+  things like `t.co/foo`.
+
+- Force links to be absolute. `[foo](google.com)` will go to
+  `http://google.com`, and not `https://zulip.com/google.com` which
+  is the default behavior.
+
+- Set `title=`(the URL) on every link tag.
+
+- Disable link-by-reference syntax,
+  `[foo][bar]` ... `[bar]: https://google.com`.
+
+- Enable linking to other channels using `#**channelName**`.
+
+### Code
+
+- Enable fenced code block extension, with syntax highlighting.
+
+- Disable line-numbering within fenced code blocks -- the `<table>`
+  output confused our web client code.
+
+### Headings
+
+- Enable headings with syntax `# foo` (syntax `== foo ==` is unsupported).
+
+### Other
+
+- Disabled images with `![]()` (images from links are shown as an inline
+  preview).
+
+- We added the `~~~ quote` block quote syntax.

docs/subsystems/notifications.md

@@ -0,0 +1,166 @@
+# Notifications in Zulip
+
+This is a design document aiming to provide context for developers
+working on Zulip's email notifications and mobile push notifications
+code paths. We recommend first becoming familiar with [sending
+messages](sending-messages.md); this document expands on
+the details of the email/mobile push notifications code path.
+
+## Important corner cases
+
+Here we name a few corner cases worth understanding in designing this
+sort of notifications system:
+
+- The **idle desktop problem**: We don't want the presence of a
+  desktop computer at the office to eat all notifications because the
+  user has an "online" client that they may not have used in 3 days.
+- The **hard disconnect problem**: A client can lose its connection to
+  the Internet (or be suspended, or whatever) at any time, and this
+  happens routinely. We want to ensure that races where a user closes
+  their laptop shortly after a notifiable message is sent does not
+  result in the user never receiving a notification about a message
+  (due to the system thinking that client received it).
+
+## The mobile/email notifications flow
+
+As a reminder, the relevant part of the flow for sending messages is
+as follows:
+
+- `do_send_messages` is the synchronous message-sending code path,
+  and passing the following data in its `send_event_on_commit` call:
+  - Data about the message's content (e.g., mentions, wildcard
+    mentions, and alert words) and encodes it into the `UserMessage`
+    table's `flags` structure, which is in turn passed into
+    `send_event_on_commit` for each user receiving the message.
+  - Data about user configuration relevant to the message, such as
+    `online_push_user_ids` and `stream_notify_user_ids`, are included
+    in the main event dictionary.
+  - The `presence_idle_user_ids` set, containing the subset of
+    recipient users who can potentially receive notifications, but have not
+    interacted with a Zulip client in the last few minutes. (Users who
+    have generally will not receive a notification unless the
+    `enable_online_push_notifications` flag is enabled). This data
+    structure ignores users for whom the message is not notifiable,
+    which is important to avoid this being thousands of `user_ids` for
+    messages to large channels with few currently active users.
+- The Tornado [event queue system](events-system.md)
+  processes that data, as well as data about each user's active event
+  queues, to (1) push an event to each queue needing that message and
+  (2) for notifiable messages, pushing an event onto the
+  `missedmessage_mobile_notifications` and/or `missedmessage_emails`
+  queues. This important message-processing logic has notable extra
+  logic not present when processing normal events, both for details
+  like splicing `flags` to customize event payloads per-user, as well.
+  - The Tornado system determines whether the user is "offline/idle".
+    Zulip's email notifications are designed to not fire when the user
+    is actively using Zulip to avoid spam, and this is where those
+    checks are implemented.
+  - Users in `presence_idle_user_ids` are always considered idle:
+    the variable name means "users who are idle because of
+    presence". This is how we solve the idle desktop problem; users
+    with an idle desktop are treated the same as users who aren't
+    logged in for this check.
+  - However, that check does not handle the hard disconnect problem:
+    if a user was present 1 minute before a message was sent, and then
+    closed their laptop, the user will not be in
+    `presence_idle_user_ids` (because it takes a
+    [few minutes](https://zulip.com/api/update-presence) of being idle for Zulip
+    clients to declare to the server that the user is actually idle),
+    and so without an additional mechanism, messages sent shortly after
+    a user leaves would never trigger a notification (!).
+  - We solve that problem by also notifying if
+    `receiver_is_off_zulip` returns `True`, which checks whether the user has any
+    current events system clients registered to receive `message`
+    events. This check is done immediately (handling soft disconnects,
+    for example, where the user closes their last Zulip tab and we get
+    the `DELETE /events/{queue_id}` request).
+  - The `receiver_is_off_zulip` check is effectively repeated when
+    event queues are garbage-collected (in `missedmessage_hook`) by
+    looking for whether the queue being garbage-collected was the only
+    one; this second check solves the hard disconnect problem, resulting in
+    notifications for these hard-disconnect cases usually coming 10
+    minutes late.
+  - We try to contain the "when to notify" business logic in the
+    `zerver/lib/notification_data.py` class methods. The module has
+    unit tests for all possible situations in
+    `test_notification_data.py`.
+  - The message-edit code path has parallel logic in
+    `maybe_enqueue_notifications_for_message_update` for triggering
+    notifications in cases like a mention added during message
+    editing.
+  - The notification sending logic for message edits
+    inside Tornado has extensive automated test suites; e.g.,
+    `test_message_edit_notifications.py` covers all the cases around
+    editing a message to add/remove a mention.
+  - We may in the future want to add some sort of system for letting
+    users see past notifications, to help with explaining and
+    debugging this system, since it has so much complexity.
+- Desktop notifications are the simplest; they are implemented
+  client-side by the web/desktop app's logic
+  (`web/src/notifications.js`) inspecting the `flags` fields that
+  were spliced into `message` events by the Tornado system, as well as
+  the user's notification settings.
+- The queue processors for those queues make the final determination
+  for whether to send a notification, and do the work to generate an
+  email (`zerver/lib/email_notifications.py`) or mobile
+  (`zerver/lib/push_notifications.py`) notification. We'll detail
+  this process in more detail for each system below, but it's
+  important to know that it's normal for a message to sit in these
+  queues for minutes (and in the future, possibly hours).
+- Both queue processor code paths do additional filtering before
+  sending a notification:
+  - Messages that have already been marked as read by the user before
+    the queue processor runs never trigger a notification.
+  - Messages that were already deleted never trigger a notification.
+  - The user-level settings for whether email/mobile notifications are
+    disabled are rechecked, as the user may have disabled one of these
+    settings during the queuing period.
+  - The **Email notifications queue processor**, `MissedMessageWorker`,
+    takes care to wait for 2 minutes (hopefully in the future this will be a
+    configuration setting) and starts a thread to batch together multiple
+    messages into a single email. These features are unnecessary
+    for mobile push notifications, because we can live-update those
+    details with a future notification, whereas emails cannot be readily
+    updated once sent. Zulip's email notifications are styled similarly
+    to GitHub's email notifications, with a clean, simple design that
+    makes replying from an email client possible (using the [incoming
+    email integration](../production/email-gateway.md)).
+  - The **Push notifications queue processor**,
+    `PushNotificationsWorker`, is a simple wrapper around the
+    `push_notifications.py` code that actually sends the
+    notification. This logic is somewhat complicated by having to track
+    the number of unread push notifications to display on the mobile
+    apps' badges, as well as using the [mobile push notifications
+    service](../production/mobile-push-notifications.md) for self-hosted
+    systems.
+
+The following important constraints are worth understanding about the
+structure of the system, when thinking about changes to it:
+
+- **Bulk database queries** are much more efficient for checking
+  details from the database like "which users receiving this message
+  are online".
+- **Thousands of users**. Zulip supports thousands of users, and we
+  want to avoid `send_event_on_commit()` pushing large amounts of
+  per-user data to Tornado via RabbitMQ for scalability reasons.
+- **Tornado doesn't do database queries**. Because the Tornado system
+  is an asynchronous event-driven framework, and our Django database
+  library is synchronous, database queries are very expensive. So
+  these queries need to be done in either `do_send_messages` or the
+  queue processor logic. (For example, this means `presence` data
+  should be checked in either `do_send_messages` or the queue
+  processors, not in Tornado).
+- **Future configuration**. Notification settings are an area that we
+  expect to only expand with time, with upcoming features like
+  following a topic (to get notifications for messages only within
+  that topic in a channel). There are a lot of different workflows
+  possible with Zulip's threading, and it's important to make it easy
+  for users to set up Zulip's notification to fit as many of those
+  workflows as possible.
+- **Message editing**. Zulip supports editing messages, and that
+  interacts with notifications in ways that require careful handling:
+  Notifications should have
+  the latest edited content (users often fix typos 30 seconds after
+  sending a message), adding a mention when editing a message should
+  send a notification to the newly mentioned user(s), and deleting a
+  message should cancel any unsent notifications for it.

docs/subsystems/onboarding-steps.md

@@ -0,0 +1,40 @@
+# Onboarding Steps
+
+Onboarding steps introduce users to important UI elements. They are an
+effective means of providing context where Zulip's UI may not be self-evident.
+
+Currently, an onboarding step is a one-time notice in the form of a banner or modal.
+Previously, hotspots were another type of onboarding step available.
+
+## Configuring a New Onboarding Step
+
+...is easy! If you think highlighting a certain UI element would improve
+Zulip's user experience, we welcome you to [open an issue](https://github.com/zulip/zulip/issues/new?title=onboarding_step_request:) for discussion.
+
+### Step 1: Add the Onboarding Step Name
+
+In `zerver/lib/onboarding_steps.py`, add the new onboarding step name to the
+`ONE_TIME_NOTICES` list:
+
+```python
+ONE_TIME_NOTICES: List[OneTimeNotice] = [
+    ...
+    OneTimeNotice(
+        name="Provide a concise name",
+    ),
+]
+```
+
+### Step 2: Display the Onboarding Step
+
+When the UI element that is not self-evident appears, use the
+`ONE_TIME_NOTICES_TO_DISPLAY` data structure from `web/src/onboarding_steps.ts`
+and the onboarding step name added in **Step 1** to determine if the
+one-time notice should be displayed. Display the notice.
+
+### Step 3: Mark the Onboarding Step as Read
+
+Once the notice is displayed, use the `post_onboarding_step_as_read` function
+from `web/src/onboarding_steps.ts` to mark the onboarding step as read.
+This will update `ONE_TIME_NOTICES_TO_DISPLAY` so that the user will not see
+the notice again.

docs/subsystems/performance.md

@@ -0,0 +1,367 @@
+# Performance and scalability
+
+This page aims to give some background to help prioritize work on the
+Zulip's server's performance and scalability. By scalability, we mean
+the ability of the Zulip server on given hardware to handle a certain
+workload of usage without performance materially degrading.
+
+First, a few notes on philosophy.
+
+- We consider it an important technical goal for Zulip to be fast,
+  because that's an important part of user experience for a real-time
+  collaboration tool like Zulip. Many UI features in the Zulip web app
+  are designed to load instantly, because all the data required for
+  them is present in the initial HTTP response, and both the Zulip
+  API and web app are architected around that strategy.
+- The Zulip database model and server implementation are carefully
+  designed to ensure that every common operation is efficient, with
+  automated tests designed to prevent the accidental introductions of
+  inefficient or excessive database queries. We much prefer doing
+  design/implementation work to make requests fast over the operational
+  work of running 2-5x as much hardware to handle the same load.
+
+See also [scalability for production users](../production/requirements.md#scalability).
+
+## Load profiles
+
+When thinking about scalability and performance, it's
+important to understand the load profiles for production uses.
+
+Zulip servers typically involve a mixture of two very different types
+of load profiles:
+
+- Open communities like open source projects, online classes,
+  etc. have large numbers of users, many of whom are idle. (Many of
+  the others likely stopped by to ask a question, got it answered, and
+  then didn't need the community again for the next year). Our own
+  [Zulip development community](https://zulip.com/development-community/) is a good
+  example for this, with more than 15K total user accounts, of which
+  only several hundred have logged in during the last few weeks.
+  Zulip has many important optimizations, including [soft
+  deactivation](sending-messages.md#soft-deactivation)
+  to ensure idle users have minimal impact on both server-side
+  scalability and request latency.
+- Fulltime teams, like your typical corporate Zulip installation,
+  have users who are mostly active for multiple hours a day and sending a
+  high volume of messages each. This load profile is most important
+  for self-hosted servers, since many of those are used exclusively by
+  the employees of the organization running the server.
+
+The zulip.com load profile is effectively the sum of thousands of
+organizations from each of those two load profiles.
+
+## Major Zulip endpoints
+
+It's important to understand that Zulip has a handful of endpoints
+that result in the vast majority of all server load, and essentially
+every other endpoint is not important for scalability. We still put
+effort into making sure those other endpoints are fast for latency
+reasons, but were they to be 10x faster (a huge optimization!), it
+wouldn't materially improve Zulip's scalability.
+
+For that reason, we organize this discussion of Zulip's scalability
+around the several specific endpoints that have a combination of
+request volume and cost that makes them important.
+
+That said, it is important to distinguish the load associated with an
+API endpoint from the load associated with a feature. Almost any
+significant new feature is likely to result in its data being sent to
+the client in `page_params` or `GET /messages`, i.e. one of the
+endpoints important to scalability here. As a result, it is important
+to thoughtfully implement the data fetch code path for every feature.
+
+Furthermore, a snappy user interface is one of Zulip's design goals, and
+so we care about the performance of any user-facing code path, even
+though many of them are not material to scalability of the server.
+But when an optimization only saves a few milliseconds that would be
+invisible to the end user, and carries any cost in code readability,
+the optimization is worth considering only if it applies to
+the major endpoints listed below.
+
+In Zulip's documentation, our general rule is to primarily write facts
+that are likely to remain true for a long time. While the numbers
+presented here vary with hardware, usage patterns, and time (there's
+substantial oscillation within a 24 hour period), we expect the rough
+sense of them (as well as the list of important endpoints) is not
+likely to vary dramatically over time.
+
+| Endpoint                | Average time | Request volume | Average impact |
+| ----------------------- | ------------ | -------------- | -------------- |
+| POST /users/me/presence | 25ms         | 36%            | 9000           |
+| GET /messages           | 70ms         | 3%             | 2100           |
+| GET /                   | 300ms        | 0.3%           | 900            |
+| GET /events             | 2ms          | 44%            | 880            |
+| GET /user_uploads/\*    | 12ms         | 5%             | 600            |
+| POST /messages/flags    | 25ms         | 1.5%           | 375            |
+| POST /messages          | 40ms         | 0.5%           | 200            |
+| POST /users/me/\*       | 50ms         | 0.04%          | 20             |
+
+The "Average impact" above is computed by multiplying request volume
+by average time; this tells you roughly that endpoint's **relative**
+contribution to the steady-state total CPU load of the system. It's
+not precise -- waiting for a network request is counted the same as
+active CPU time, but it's extremely useful for providing intuition for
+what code paths are most important to optimize, especially since
+network wait is in practice largely waiting for PostgreSQL or
+memcached to do work.
+
+As one can see, there are two categories of endpoints that are
+important for scalability: those with extremely high request volumes,
+and those with moderately high request volumes that are also
+expensive. It doesn't matter how expensive, for example,
+`POST /users/me/subscriptions` is for scalability, because the volume
+is negligible.
+
+### Tornado
+
+Zulip's Tornado-based [real-time push
+system](events-system.md), and in particular
+`GET /events`, accounts for something like 50% of all HTTP requests to
+a production Zulip server. Despite `GET /events` being extremely
+high-volume, the typical request takes 1-3ms to process, and doesn't
+use the database at all (though it will access `memcached` and
+`redis`), so they aren't a huge contributor to the overall CPU usage
+of the server.
+
+Because these requests are so efficient from a total CPU usage
+perspective, Tornado is significantly less important than other
+services like Presence and fetching message history for overall CPU
+usage of a Zulip installation.
+
+It's worth noting that most (~80%) Tornado requests end the
+longpolling via a `heartbeat` event, which are issued to idle
+connections after about a minute. These `heartbeat` events are
+useless aside from avoiding problems with networks/proxies/NATs that
+are configured poorly and might kill HTTP connections that have been
+idle for a minute. It's likely that with some strategy for detecting
+such situations, we could reduce their volume (and thus overall
+Tornado load) dramatically.
+
+Currently, Tornado is sharded by realm, and optionally by user-id
+within each realm. Sharding by realm is sufficient for arbitrary
+scaling of the number of organizations on a multi-tenant system like
+zulip.com. Sharding by user-id is necessary for very large
+organizations with multiple thousands of active users at once.
+
+### Presence
+
+`POST /users/me/presence` requests, which submit the current user's
+presence information and return the information for all other active
+users in the organization, account for about 36% of all HTTP requests
+on production Zulip servers. See the [presence API
+documentation](https://zulip.com/api/update-presence) for details on
+this system and how it's optimized. For this article, it's important
+to know that presence is one of the most important scalability
+concerns for any chat system, because it cannot be cached long, and is
+structurally a quadratic problem.
+
+Because typical presence requests consume 10-50ms of server-side
+processing time (to fetch and send back live data on all other active
+users in the organization), and are such a high volume, presence is
+the single most important source of steady-state load for a Zulip
+server. This is true for most other chat server implementations as
+well.
+
+There is an ongoing [effort to rewrite the data model for
+presence](https://github.com/zulip/zulip/pull/16381) that we expect to
+result in a substantial improvement in the per-request and thus total
+load resulting from presence requests.
+
+### Fetching page_params
+
+The request to generate the `page_params` portion of `GET /`
+(equivalent to the response from [GET
+/api/v1/register](https://zulip.com/api/register-queue) used by
+mobile/terminal apps) is one of Zulip's most complex and expensive.
+
+Zulip is somewhat unusual among web apps in sending essentially all of the
+data required for the entire Zulip web app in this single request,
+which is part of why the Zulip web app loads very quickly -- one only
+needs a single round trip aside from cacheable assets (avatars, images, JS,
+CSS). Data on other users in the organization, channels, supported
+emoji, custom profile fields, etc., is all included. The nice thing
+about this model is that essentially every UI element in the Zulip
+client can be rendered immediately without paying latency to the
+server; this is critical to Zulip feeling performant even for users
+who have a lot of latency to the server.
+
+There are only a few exceptions where we fetch data in a separate AJAX
+request after page load:
+
+- Message history is managed separately; this is why the Zulip web app will
+  first render the entire site except for the middle panel, and then a
+  moment later render the middle panel (showing the message history).
+- A few very rarely accessed data sets like [message edit
+  history](https://zulip.com/help/view-a-messages-edit-history) are
+  only fetched on demand.
+- A few data sets that are only required for administrative settings
+  pages are fetched only when loading those parts of the UI.
+
+Requests to `GET /` and `/api/v1/register` that fetch `page_params`
+are pretty rare -- something like 0.3% of total requests, but are
+important for scalability because (1) they are the most expensive read
+requests the Zulip API supports and (2) they can come in a thundering
+herd around server restarts (as discussed in [fetching message
+history](#fetching-message-history).
+
+The cost for fetching `page_params` varies dramatically based
+primarily on the organization's size, varying from 90ms-300ms for a
+typical organization but potentially multiple seconds for large open
+organizations with 10,000s of users. There is also smaller
+variability based on a individual user's personal data state,
+primarily in that having 10,000s of unread messages results in a
+somewhat expensive query to find which channels/topics those are in.
+
+We consider any organization having normal `page_params` fetch times
+greater than a second to be a bug, and there is ongoing work to fix that.
+
+It can help when thinking about this to imagine `page_params` as what
+in another web app would have been 25 or so HTTP GET requests, each
+fetching data of a given type (users, channels, custom emoji, etc.); in
+Zulip, we just do all of those in a single API request. In the
+future, we will likely move to a design that does much of the database
+fetching work for different features in parallel to improve latency.
+
+For organizations with 10K+ users and many default channels, the
+majority of time spent constructing `page_params` is spent marshalling
+data on which users are subscribed to which channels, which is an area
+of active optimization work.
+
+### Fetching message history
+
+Bulk requests for message content and metadata
+([`GET /messages`](https://zulip.com/api/get-messages)) account for
+~3% of total HTTP requests. The zulip web app has a few major reasons
+it does a large number of these requests:
+
+- Most of these requests are from users clicking into different views
+  -- to avoid certain subtle bugs, Zulip's web app currently fetches
+  content from the server even when it has the history for the
+  relevant channel/topic cached locally.
+- When a browser opens the Zulip web app, it will eventually fetch and
+  cache in the browser all messages newer than the oldest unread
+  message in a non-muted context. This can be in total extremely
+  expensive for users with 10,000s of unread messages, resulting in a
+  single browser doing 100 of these requests.
+- When a new version of the Zulip server is deployed, every browser
+  will reload within 30 minutes to ensure they are running the latest
+  code. For installations that deploy often like chat.zulip.org and
+  zulip.com, this can result in a thundering herd effect for both `/`
+  and `GET /messages`. A great deal of care has been taking in
+  designing this [auto-reload
+  system](hashchange-system.md#server-initiated-reloads)
+  to spread most of that herd over several minutes.
+
+Typical requests consume 20-100ms to process, much of which is waiting
+to fetch message IDs from the database and then their content from
+memcached. While not large in an absolute sense, these requests are
+expensive relative to most other Zulip endpoints.
+
+Some requests, like full-text search for commonly used words, can be
+more expensive, but they are sufficiently rare in an absolute sense so
+as to be immaterial to the overall scalability of the system.
+
+This server-side code path is already heavily optimized on a
+per-request basis. However, we have technical designs for optimizing
+the overall frequency with which clients need to make these requests
+in two major ways:
+
+- Improving [client-side
+  caching](https://github.com/zulip/zulip/issues/15131) to allow
+  caching of narrows that the user has viewed in the current session,
+  avoiding repeat fetches of message content during a given session.
+- Adjusting the behavior for clients with 10,000s of unread messages
+  to not fetch as much old message history into the cache. See [this
+  issue](https://github.com/zulip/zulip/issues/16697) for relevant
+  design work.
+
+Together, it is likely that these changes will reduce the total
+scalability cost of fetching message history dramatically.
+
+### User uploads
+
+Requests to fetch uploaded files (including user avatars) account for
+about 5% of total HTTP requests. Zulip spends consistently ~10-15ms
+processing one of these requests (mostly authorization logic), before
+handing off delivery of the file to `nginx` (which may itself fetch
+from S3, depending on the configured [upload
+backend](../production/upload-backends.md)).
+
+### Sending and editing messages
+
+[Sending new messages](sending-messages.md) (including
+incoming webhooks) represents less than 0.5% of total request volume.
+That this number is small should not be surprising even though sending
+messages is intuitively the main feature of a chat service: a message
+sent to 50 users triggers ~50 `GET /events` requests.
+
+A typical message-send request takes 20-70ms, with more expensive
+requests typically resulting from [Markdown
+rendering](markdown.md) of more complex syntax. As a
+result, these requests are not material to Zulip's scalability.
+Editing messages and adding emoji reactions are very similar to
+sending them for the purposes of performance and scalability, since
+the same clients need to be notified, and these requests are lower in volume.
+
+That said, we consider the performance of these endpoints to be some
+of the most important for Zulip's user experience, since even with
+local echo, these are some of the places where any request processing
+latency is highly user-visible.
+
+Typing notifications are slightly higher volume than sending messages,
+but are also extremely cheap (~3ms).
+
+### Other endpoints
+
+Other API actions, like subscribing to a channel, editing settings,
+registering an account, etc., are vanishingly rare compared to the
+requests detailed above, fundamentally because almost nobody changes
+these things more than a few dozen times over the lifetime of their
+account, whereas everything above are things that a given user might
+do thousands of times.
+
+As a result, performance work on those requests is generally only
+important for latency reasons, not for optimizing the overall
+scalability of a Zulip server.
+
+## Queue processors and cron jobs
+
+The above doesn't cover all of the work that a production Zulip server
+does; various tasks like sending outgoing emails or recording the data
+that powers [/stats](https://zulip.com/help/analytics) are run by
+[queue processors](queuing.md) and cron jobs, not in
+response to incoming HTTP requests. In practice, all of these have
+been written such that they are immaterial to total load and thus
+architectural scalability, though we do from time to time need to do
+operational work to add additional queue processors for particularly
+high-traffic queues. For all of our queue processors, any
+serialization requirements are at most per-user, and thus it would be
+straightforward to shard by `user_id` or `realm_id` if required.
+
+## Service scalability
+
+In addition to the above, which just focuses on the total amount of
+CPU work, it's also relevant to think about load on infrastructure
+services (memcached, redis, rabbitmq, and most importantly postgres),
+as well as queue processors (which might get backlogged).
+
+In practice, efforts to make an individual endpoint faster will very
+likely reduce the load on these services as well. But it is worth
+considering that database time is a more precious resource than
+Python/CPU time (being harder to scale horizontally).
+
+Most optimizations to make an endpoint cheaper will start with
+optimizing the database queries and/or employing
+[caching](caching.md), and then continue as needed with
+profiling of the Python code and any memcached queries.
+
+For a handful of the critical code paths listed above, we further
+optimize by skipping the Django ORM (which has substantial overhead)
+for narrow sections; typically this is sufficient to result in the
+database query time dominating that spent by the Python application
+server process.
+
+Zulip's [server logs](logging.md) are designed to
+provide insight when a request consumes significant database or
+memcached resources, which is useful both in development and in
+production.

docs/subsystems/pointer.md

@@ -0,0 +1,105 @@
+# Unread counts and the pointer
+
+When you're using Zulip and you reload, or narrow to a channel, how
+does Zulip decide where to place you?
+
+Conceptually, Zulip takes you to the place where you left off
+(e.g., the first unread message), not the most recent messages, to
+facilitate reviewing all the discussions that happened while you were
+away from your computer. The scroll position is then set to keep that
+message in view and away from both the top and bottom of the visible
+section of messages.
+
+But there a lot of details around doing this right, and around
+counting unread messages. Here's how Zulip currently decides which
+message to select, along with some notes on improvements we'd like to
+make to the model.
+
+First a bit of terminology:
+
+- "Narrowing" is the process of filtering to a particular subset of
+  the messages the user has access to.
+
+- The blue cursor box (the "pointer") is around is called the
+  "selected" message. Zulip ensures that the currently selected
+  message is always in-view.
+
+## Pointer logic
+
+### Recipient bar: message you clicked
+
+If you enter a narrow by clicking on a message group's _recipient bar_
+(channel/topic or direct message recipient list at the top of a group
+of messages), Zulip will select the message you clicked on. This
+provides a nice user experience where you get to see the stuff near
+what you clicked on, and in fact the message you clicked on stays at
+exactly the same scroll position in the window after the narrowing as
+it was at before.
+
+### Search, sidebar click, or new tab: unread/recent matching narrow
+
+If you instead narrow by clicking on something in the left sidebar,
+typing some terms into the search box, reloading the browser, or any
+other method that doesn't encode a specific message to visit, Zulip
+will instead select the first unread message matching that narrow, or
+if there are none, the most recent messages matching that narrow.
+
+This provides the nice user experience of taking you to the start of
+the new stuff (with enough messages you've seen before still in view
+at the top to provide you with context), which is usually what you
+want. (When finding the "first unread message", Zulip ignores unread
+messages in muted channels or in muted topics within non-muted
+channels.)
+
+### Unnarrow: previous sequence
+
+When you unnarrow using, for example, the `a` key, you will automatically be
+taken to the same message that was selected in the Combined feed view before
+you narrowed, unless in the narrow you read new messages, in which
+case you will be jumped forward to the first unread and non-muted
+message in the Combined feed view (or the bottom of the feed if there is
+none). This makes for a nice experience reading threads via the Combined feed
+view in sequence.
+
+### Forced reload: state preservation
+
+When the server forces a reload of a browser that's otherwise caught
+up (which happens within 30 minutes when a new version of the server
+is deployed, usually at a type when the user isn't looking at the
+browser), Zulip will preserve the state -- what (if any) narrow the
+user was in, the selected message, and even exact scroll position!
+
+For more on the user experience philosophy guiding these decisions,
+see [the architectural overview](../overview/architecture-overview.md).
+
+## Unread count logic
+
+How does Zulip decide whether a message has been read by the user?
+The algorithm needs to correctly handle a range of ways people might
+use the product. The algorithm is as follows:
+
+- Any message which is selected or above a message which is selected
+  is marked as read. So messages are marked as read as you scroll
+  down the keyboard when the pointer passes over them.
+
+- If the whitespace at the very bottom of the feed is in view, all
+  messages in view are marked as read.
+
+These two simple rules, combined with the pointer logic above, end up
+matching user expectations well for whether the product should treat
+them as having read a set of messages (or not).
+
+One key detail to highlight is that we only mark messages as read
+through these processes in views that contain all messages in a
+thread; search views will never mark messages as read.
+
+## Testing and development
+
+In a Zulip development environment, you can use
+`manage.py mark_all_messages_unread` to set every user's pointer to 0
+and all messages as unread, for convenience in testing unread count
+related logic.
+
+It can be useful to combine this with `manage.py populate_db -n 3000`
+(which rebuilds the database with 3000 initial messages) to ensure a
+large number of messages are present.

docs/subsystems/queuing.md

@@ -0,0 +1,89 @@
+# Queue processors
+
+Zulip uses RabbitMQ to manage a system of internal queues. These are
+used for a variety of purposes:
+
+- Asynchronously doing expensive operations like sending email
+  notifications which can take seconds per email and thus would
+  otherwise time out when 100s are triggered at once (e.g., inviting a
+  lot of new users to a realm).
+
+- Asynchronously doing non-time-critical somewhat expensive operations
+  like updating analytics tables (e.g., UserActivityInternal) which
+  don't have any immediate runtime effect.
+
+- Communicating events to push to clients (browsers, etc.) from the
+  main Zulip Django application process to the Tornado-based events
+  system. Example events might be that a new message was sent, a user
+  has changed their subscriptions, etc.
+
+- Processing mobile push notifications and email mirroring system
+  messages.
+
+- Processing various errors, frontend tracebacks, and slow database
+  queries in a batched fashion.
+
+Needless to say, the RabbitMQ-based queuing system is an important
+part of the overall Zulip architecture, since it's in critical code
+paths for everything from signing up for account, to rendering
+messages, to delivering updates to clients.
+
+We use the `pika` library to interface with RabbitMQ, using a simple
+custom integration defined in `zerver/lib/queue.py`.
+
+### Adding a new queue processor
+
+To add a new queue processor:
+
+- Define the processor in `zerver/worker/` using the `@assign_queue` decorator;
+  it's pretty easy to get the template for an existing similar queue
+  processor. This suffices to test your queue worker in the Zulip development
+  environment (`tools/run-dev` will automatically restart the queue processors
+  and start running your new queue processor code). You can also run a single
+  queue processor manually using, for example,
+  `./manage.py process_queue --queue=user_activity`.
+
+- So that supervisord will know to run the queue processor in
+  production, you will need to add to the `base_queues` variable in
+  `puppet/zulip/manifests/app_frontend_base.pp`; the list there is
+  used to generate `/etc/supervisor/conf.d/zulip.conf`.
+
+The queue will automatically be added to the list of queues tracked by
+`scripts/nagios/check-rabbitmq-consumers`, so Nagios can properly
+check whether a queue processor is running for your queue. You still
+need to update the sample Nagios configuration in `puppet/kandra`
+manually.
+
+### Publishing events into a queue
+
+You can publish events to a RabbitMQ queue using the
+`queue_event_on_commit` function defined in `zerver/lib/queue.py`.
+
+An interesting challenge with queue processors is what should happen
+when queued events in Zulip's backend tests. Our current solution is
+that in the tests, `queue_event_on_commit` will (by default) simple call
+the `consume` method for the relevant queue processor. However,
+`queue_event_on_commit` also supports being passed a function that should
+be called in the tests instead of the queue processor's `consume`
+method. Where possible, we prefer the model of calling `consume` in
+tests since that's more predictable and automatically covers the queue
+processor's code path, but it isn't always possible.
+
+### Clearing a RabbitMQ queue
+
+If you need to clear a queue (delete all the events in it), run
+`./manage.py purge_queue <queue_name>`, for example:
+
+```bash
+./manage.py purge_queue user_activity
+```
+
+You can also use the amqp tools directly. Install `amqp-tools` from
+apt and then run:
+
+```bash
+amqp-delete-queue --username=zulip --password='...' --server=localhost \
+   --queue=user_activity
+```
+
+with the RabbitMQ password from `/etc/zulip/zulip-secrets.conf`.

docs/subsystems/realms.md

@@ -0,0 +1,96 @@
+# Realms in Zulip
+
+Zulip allows multiple _realms_ to be hosted on a single instance.
+Realms are the Zulip codebase's internal name for what we refer to in
+user-facing documentation as an organization (the name "realm" comes
+from [Kerberos](https://web.mit.edu/kerberos/)).
+
+Wherever possible, we avoid using the term `realm` in any user-facing
+string or documentation; "Organization" is the equivalent term used in
+those contexts (and we have linters that attempt to enforce this rule
+in translatable strings). We may in the future modify Zulip's
+internals to use `organization` instead.
+
+The
+[production docs on multiple realms](../production/multiple-organizations.md)
+are also relevant reading.
+
+## Creating realms
+
+There are two main methods for creating realms.
+
+- Using unique link generator
+- Enabling open realm creation
+
+#### Using unique link generator
+
+```bash
+./manage.py generate_realm_creation_link
+```
+
+The above command will output a URL which can be used for creating a
+new realm and an administrator user for that realm. The link expires
+after the creation of the realm. The link also expires if not used
+within 7 days. The expiration period can be changed by modifying
+`CAN_CREATE_REALM_LINK_VALIDITY_DAYS` in settings.py.
+
+## Subdomains
+
+One can host multiple realms in a Zulip server by giving each realm a
+unique subdomain of the main Zulip server's domain. For example, if
+the Zulip instance is hosted at zulip.example.com, and the subdomain
+of your organization is acme you can would acme.zulip.example.com for
+accessing the organization.
+
+For subdomains to work properly, you also have to change your DNS
+records so that the subdomains point to your Zulip installation IP. An
+`A` record with host name value `*` pointing to your IP should do the
+job.
+
+We also recommend upgrading to at least Zulip 1.7, since older Zulip
+releases had much less nice handling for subdomains. See our
+[docs on using subdomains](../production/multiple-organizations.md) for
+user-facing documentation on this.
+
+### Working with subdomains in development environment
+
+Zulip's development environment is designed to make it convenient to
+test the various Zulip configurations for different subdomains:
+
+- Realms are subdomains on `*.zulipdev.com`, just like `*.zulipchat.com`.
+- The root domain (like `zulip.com` itself) is `zulipdev.com` itself.
+- The default realm is hosted on `localhost:9991` rather than
+  `zulip.zulipdev.com`, using the [`REALM_HOSTS`
+  feature](../production/multiple-organizations.md) feature.
+
+Details are below.
+
+By default, Linux does not provide a convenient way to use subdomains
+in your local development environment. To solve this problem, we use
+the **zulipdev.com** domain, which has a wildcard A record pointing to
+127.0.0.1. You can use zulipdev.com to connect to your Zulip
+development server instead of localhost. The default realm with the
+Shakespeare users has the subdomain `zulip` and can be accessed by
+visiting **zulip.zulipdev.com**.
+
+If you are behind a **proxy server**, this method won't work. When you
+make a request to load zulipdev.com in your browser, the proxy server
+will try to get the page on your behalf. Since zulipdev.com points
+to 127.0.0.1 the proxy server is likely to give you a 503 error. The
+workaround is to disable your proxy for `*.zulipdev.com`. The DNS
+lookup should still work even if you disable proxy for
+\*.zulipdev.com. If it doesn't you can add zulipdev.com records in
+`/etc/hosts` file. The file should look something like this.
+
+```text
+127.0.0.1    localhost
+
+127.0.0.1    zulipdev.com
+
+127.0.0.1    zulip.zulipdev.com
+
+127.0.0.1    testsubdomain.zulipdev.com
+```
+
+These records are also useful if you want to, for example, run the
+Puppeteer tests when you are not connected to the Internet.

docs/subsystems/schema-migrations.md

@@ -0,0 +1,716 @@
+# Schema migrations
+
+Zulip uses the [standard Django system for doing schema
+migrations](https://docs.djangoproject.com/en/5.0/topics/migrations/).
+There is some example usage in the [new feature
+tutorial](../tutorials/new-feature-tutorial.md).
+
+This page documents the conventions and constraints that apply to
+writing schema migrations for Zulip. Many of those constraints come
+from how Zulip Cloud deploys, which is described in the next section
+and which most of the rules below trace back to.
+
+## How Zulip Cloud deploys migrations
+
+> **A migration must be safe to apply with the previous release's
+> application code still running, and the previous release's code
+> must continue to work correctly against the new schema until
+> production deploys.**
+
+That rule follows from a few facts about how Zulip Cloud is
+operated. Contributors writing migrations should internalize
+these before writing anything.
+
+- **Staging and production share a single database.** There is one
+  PostgreSQL cluster. Staging is a separate set of application
+  servers (running internal-testing traffic, not live customer
+  traffic), but it points at the same database production uses.
+- **Staging deploys first.** Migrations are applied when staging
+  deploys, against the shared database. Production then deploys
+  some time later — typically about half an hour, occasionally
+  longer — running the new code against the already-migrated
+  database.
+- **During that window, production runs the previous release's code
+  against the new schema.** Old code must continue to read, insert,
+  and update successfully against the migrated database the whole
+  time.
+- **Staging runs migrations against production-scale data.** This
+  is a real safety net: a migration that's too slow, holds locks for
+  too long, or has any other operational problem will manifest at
+  staging-deploy time, on the actual database. But because the
+  database is shared, those problems are also a live production
+  issue while they're happening. The bar is to design the migration
+  correctly up front, not to discover problems on staging.
+- **The deploy tooling can choose which commits go to staging and
+  production independently.** This gives the deploy operator levers:
+  hold a risky migration back from production for a bit, deploy a
+  schema change ahead of the code that uses it, or skip a migration
+  entirely. The cleanest version of this would be reverting a
+  specific commit, but in practice migrations are not always split
+  cleanly enough to make that easy, so we also use a stub-out /
+  `migrate --fake` workflow described in
+  [When a migration can't be online-safe](#when-a-migration-cant-be-online-safe).
+
+This isn't only about the staging/production shared database.
+Cloud's Django processes also restart in a rolling fashion at
+deploy time, so even within a single deploy there's a window
+during which some processes are running the old code and some
+the new — and old and new code must both function against the
+live schema. (Self-hosted deploys typically stop the server, run
+migrations, and start it back up, so neither constraint applies
+there — but Cloud inherits both.)
+
+If you can't satisfy the rule above, the migration needs to be
+deferred until the dependent code is in production; see [When a
+migration can't be online-safe](#when-a-migration-cant-be-online-safe).
+
+## The simple case
+
+If your migration just reflects new fields you've added to
+`zerver/models/*.py`, rebase your branch first, then run
+`./manage.py makemigrations`, rename the generated file if
+Django's auto-picked name is awkward, run `tools/provision` to
+apply it locally, and commit with the migration file added.
+
+For migrations that need custom Python to transform existing
+data, see [Writing RunPython
+migrations](#writing-runpython-migrations); for migrations on
+large tables, see [Making large migrations
+work](#making-large-migrations-work).
+
+## Working with the migration graph
+
+### Numbering conflicts across branches
+
+If you've done your schema change in a branch, and meanwhile another
+schema change has taken place, Django will now have two migrations
+with the same number. There are two easy ways to fix this:
+
+- If your migrations were automatically generated using `manage.py
+makemigrations`, a good option is to just remove your migration and
+  rerun the command after rebasing. Remember to `git rebase` to do
+  this in the commit that changed `models/*.py` if you have a
+  multi-commit branch.
+- If you wrote code as part of preparing your migrations, or prefer
+  this workflow, you can run `./tools/renumber-migrations`, which
+  renumbers your migration(s) and fixes up the "dependencies"
+  entries. It updates the working tree for you (you still need to
+  `git add` the result).
+
+`./tools/test-migrations` checks that the migration graph is
+consistent with the current models and flags auto-named
+migrations that should be renamed. It runs as part of `test-all`
+and is the first thing to run when sorting out a numbering or
+graph problem.
+
+### Release branches
+
+When a release branch needs a migration, but `main` already has new
+migrations, the migration graph must fork. The migration should be
+named and numbered in `main` to follow the usual sequence, but its
+dependency should be set to the last migration that exists on the
+release branch. This should be followed, on `main`, by a migration
+which merges the two resulting tips; you can make such a merge with
+`manage.py makemigrations --merge`. Structuring the fork this way
+makes the migration safe to cherry-pick back into the stable release
+branch. See Django's [Controlling the order of
+migrations](https://docs.djangoproject.com/en/6.0/howto/writing-migrations/#controlling-the-order-of-migrations)
+for the underlying `dependencies` / `run_before` mechanics, or
+`zerver/migrations/0754_merge_20251014_1855.py` for an example of
+the merge migration that joins the forked tips.
+
+## Adding columns safely
+
+The shared-database deploy model means that adding a column requires
+some care: production's old code keeps running `INSERT` and `UPDATE`
+statements against the table that don't list the new column.
+
+### Always set a database default
+
+Set `db_default=` (in addition to Django's `default=`) on any new
+**non-nullable** column introduced via `AddField`. Without
+`db_default`, PostgreSQL has no
+fallback when old code's `INSERT` omits the column, and those
+inserts will fail the `NOT NULL` constraint in production until
+the new code is deployed. (For nullable columns this isn't a
+concern — the column simply takes `NULL` for omitted inserts.)
+
+```python
+migrations.AddField(
+    model_name="realmauditlog",
+    name="scrubbed",
+    field=models.BooleanField(db_default=False, default=False),
+),
+```
+
+`default=` is the Django-level default used by ORM `.create()`,
+`save()`, etc. when the new code runs; `db_default=` is the
+PostgreSQL-level default that protects old code during the deploy
+window. Both are needed.
+
+### Per-row defaults: nullable, backfill, NOT NULL
+
+Some new columns can't take a static default — for example, a
+foreign key to a per-realm `NamedUserGroup`, or a value derived
+from other columns on the same row. The standard shape for these
+is three migrations:
+
+1. **Add the column nullable** (`AddField(..., null=True)`). Old
+   code is unaffected because it doesn't reference the column;
+   any inserts (from old or new code) leave the column `NULL`.
+2. **Backfill the pre-existing rows** in batches with a
+   `RunPython` data migration that has `atomic = False`. See
+   [Making large migrations work](#making-large-migrations-work)
+   for the batching patterns.
+3. **Flip the column to `NOT NULL`** with an `AlterField` that
+   redeclares the field without `null=True` (Django defaults
+   non-`null` field types to `NOT NULL`, so an explicit
+   `null=False` is rarely written).
+
+A canonical recent example is the trio
+`0710_realm_topics_policy.py` →
+`0711_set_default_value_for_realm_topics_policy.py` →
+`0712_alter_realm_topics_policy.py`. Many of the per-stream and
+per-realm group-permission settings follow this pattern as well.
+Django's [Migrations that add unique
+fields](https://docs.djangoproject.com/en/6.0/howto/writing-migrations/#migrations-that-add-unique-fields)
+documents the same structural pattern.
+
+For this sequence to be safe on the shared database, two
+properties of the application code are non-negotiable:
+
+- **The code that populates the column must be in production before
+  the `NOT NULL` migration runs.** It must populate the column on
+  every code path that creates a row. If any path is missed,
+  production will create a `NULL` row and the constraint will reject
+  the insert, failing the request with a 500.
+- **The code that reads the column must tolerate `NULL` until the
+  backfill is complete.** Almost always this means simply not
+  reading the column until step 3 lands.
+
+These are properties of the application code change, not of the
+migrations themselves. Your job as a contributor is to write the
+code so these properties hold, audit every relevant code path,
+and make the backfill idempotent (`filter(column=None)`) so it's
+safe to re-run.
+
+The application code and all three migrations typically ship in
+the same PR. The deploy operator is responsible for sequencing
+them across deploy cycles so that the populating code reaches
+production before the `NOT NULL` migration is applied to the
+shared database — choosing which commits to ship to staging vs.
+production, deferring the `NOT NULL` migration if needed, or
+using the [stub-out / `migrate --fake`
+workflow](#when-a-migration-cant-be-online-safe). Call this
+sequence out in the PR description so the operator knows to
+handle it.
+
+## Indexes
+
+### Adding indexes
+
+Django's regular `AddIndex` operation (corresponding to `CREATE
+INDEX` in SQL) takes a lock that blocks writes to the affected
+table for the duration of the build. On large tables (`Message`,
+`UserMessage`, `Subscription`, `RealmAuditLog`, etc.) that lock
+can last long enough to affect production. Use
+`AddIndexConcurrently` (corresponding to `CREATE INDEX
+CONCURRENTLY`) for any index on those tables, and as a default
+for any index on a table large enough that the lock would be
+noticeable.
+
+`AddIndexConcurrently` requires `atomic = False` on the
+migration class — `CREATE INDEX CONCURRENTLY` cannot run inside
+a transaction.
+
+For an index on plain table columns, `CREATE INDEX` populates
+the index relation's own size/row statistics and the planner
+can already estimate selectivity from existing column
+statistics — no follow-up `ANALYZE` is needed. For **expression
+indexes** (e.g., `Upper(subject)`), PostgreSQL only gathers
+per-expression statistics in `pg_statistic` when `ANALYZE`
+runs, so add a `RunSQL("ANALYZE zerver_<table>")` after the
+`AddIndexConcurrently` so the planner has the distribution info
+it needs to use the new index.
+
+### Renaming indexes
+
+When the index naming scheme changes, prefer `RenameIndex` to
+dropping and rebuilding (see
+`0693_add_conditional_indexes_for_topic.py`, which also adds
+expression indexes; note that it mistakenly omits the `ANALYZE`
+that those would need).
+
+### Dropping indexes
+
+To drop an index `CONCURRENTLY` — for example, the auto-created
+index on a foreign key whose `db_index` you're setting to
+`False` — use
+[SeparateDatabaseAndState](#separating-database-and-state).
+Django's `AlterField(db_index=False)` issues a regular (locking)
+`DROP INDEX`; the separate-state pattern lets you keep Django's
+model tracking in sync while doing the actual drop concurrently.
+
+## Constraints on large tables
+
+Django's `AddConstraint` takes an `ACCESS EXCLUSIVE` lock on the
+table and validates the constraint against every row before
+returning, which can hold writes for a long time on a large
+table. Standard Django doesn't expose the PostgreSQL features
+that avoid this, so the patterns below all reach for `RunSQL` /
+`RunPython` and use [SeparateDatabaseAndState](#separating-database-and-state)
+to keep Django's model state in sync with what we did at the DB
+level.
+
+### `UNIQUE` constraints
+
+Build a unique index `CONCURRENTLY`, then promote it to a table
+constraint via `ALTER TABLE ... ADD CONSTRAINT ... UNIQUE USING
+INDEX` (an instant metadata operation). See
+`0794_alter_directmessagegroup_recipient_and_more.py` for the
+canonical implementation, including the introspection-based
+helper that handles the case where a previous run created the
+index but crashed before promoting it. The state side is
+whatever Django would have generated for the model change —
+e.g., `AlterField` to `OneToOneField` if the uniqueness is on a
+foreign key, or `AddConstraint(UniqueConstraint(...))` for a
+non-foreign-key column.
+
+### `CHECK` constraints
+
+Declare the constraint in the model's `Meta.constraints` first
+(without it, Django's `makemigrations --check` will see drift
+when it compares state to model). Then write a single migration
+that uses `SeparateDatabaseAndState` to do the DB work and
+record the model-state change at the same time:
+
+```python
+migrations.SeparateDatabaseAndState(
+    database_operations=[
+        migrations.RunSQL(
+            "ALTER TABLE zerver_foo ADD CONSTRAINT bar CHECK (...) NOT VALID;",
+            reverse_sql="ALTER TABLE zerver_foo DROP CONSTRAINT bar;",
+        ),
+        migrations.RunSQL(
+            "ALTER TABLE zerver_foo VALIDATE CONSTRAINT bar;",
+            reverse_sql=migrations.RunSQL.noop,
+        ),
+    ],
+    state_operations=[
+        migrations.AddConstraint(
+            model_name="foo",
+            constraint=models.CheckConstraint(...),
+        ),
+    ],
+)
+```
+
+The `NOT VALID` add records the constraint and enforces it for
+new rows without scanning the existing table; `VALIDATE
+CONSTRAINT` then scans without holding `ACCESS EXCLUSIVE`. Both
+can run in a single (atomic) migration.
+
+### Foreign key columns
+
+Adding a foreign key column the default way triggers an `ACCESS
+EXCLUSIVE` lock to validate the constraint. Avoid that with two
+migrations:
+
+1. **Add the column with `db_constraint=False`.** A plain
+   `AddField` — Django emits the column-only DDL, no foreign key
+   constraint, no `SeparateDatabaseAndState` wrapping needed:
+
+   ```python
+   migrations.AddField(
+       model_name="foo",
+       name="bar",
+       field=models.ForeignKey(
+           db_constraint=False,
+           on_delete=models.CASCADE,
+           to="zerver.bar",
+       ),
+   )
+   ```
+
+2. **Wrap the constraint creation in `SeparateDatabaseAndState`.**
+   The DB side adds the constraint `NOT VALID` and validates it;
+   the state side flips `db_constraint` back to its default
+   (`True`) so Django's model tracking matches:
+
+   ```python
+   migrations.SeparateDatabaseAndState(
+       database_operations=[
+           migrations.RunSQL(
+               "ALTER TABLE zerver_foo ADD CONSTRAINT zerver_foo_bar_id_fk "
+               "FOREIGN KEY (bar_id) REFERENCES zerver_bar(id) NOT VALID;",
+               reverse_sql="ALTER TABLE zerver_foo DROP CONSTRAINT zerver_foo_bar_id_fk;",
+           ),
+           migrations.RunSQL(
+               "ALTER TABLE zerver_foo VALIDATE CONSTRAINT zerver_foo_bar_id_fk;",
+               reverse_sql=migrations.RunSQL.noop,
+           ),
+       ],
+       state_operations=[
+           migrations.AlterField(
+               model_name="foo",
+               name="bar",
+               field=models.ForeignKey(
+                   on_delete=models.CASCADE,
+                   to="zerver.bar",
+               ),
+           ),
+       ],
+   )
+   ```
+
+## Writing RunPython migrations
+
+A small data migration looks like this (adapted from
+`0795_rename_old_twitter_profile_fields.py`):
+
+```python
+from django.db import migrations
+from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+from django.db.migrations.state import StateApps
+
+
+def update_twitter_to_x(
+    apps: StateApps, schema_editor: BaseDatabaseSchemaEditor
+) -> None:
+    CustomProfileField = apps.get_model("zerver", "CustomProfileField")
+    # Copied from zerver/models/custom_profile_fields.py
+    EXTERNAL_ACCOUNT = 7
+
+    CustomProfileField.objects.filter(
+        name="Twitter", field_type=EXTERNAL_ACCOUNT, field_data=...
+    ).update(name="X username", ...)
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("zerver", "0794_alter_directmessagegroup_recipient_and_more"),
+    ]
+
+    operations = [
+        migrations.RunPython(
+            update_twitter_to_x,
+            reverse_code=migrations.RunPython.noop,
+            elidable=True,
+        ),
+    ]
+```
+
+The bulk `.filter(...).update(...)` is fine for a small table
+like `CustomProfileField`; for migrations that touch a large
+table, see [Making large migrations
+work](#making-large-migrations-work).
+
+### Hygiene rules
+
+- **Use `apps.get_model("zerver", "Foo")` to access models.** A
+  Zulip server admin runs migrations using whatever code they've
+  checked out, which can be thousands of commits newer than the
+  migration. Importing `Foo` from `zerver.models` would mean the
+  migration sees a model definition that didn't exist when it was
+  written. `apps.get_model` returns the model as the migration
+  graph defines it at that point in time. This works for ORM
+  operations like `Foo.objects.filter(...)`; it doesn't expose
+  Python-only methods or properties, so don't reach for those.
+- **Don't import other code from `zerver` either.** If you need a
+  constant, an enum value, or a small helper, copy it into the
+  migration file with a comment noting where it came from. A linter
+  rule enforces this. The reason is the same as for models: the
+  imported code may have changed beyond recognition by the time the
+  migration runs on a self-hosted server.
+- **Set `reverse_code=migrations.RunPython.noop`** on data migrations
+  unless you have a real inverse to provide. This is the project
+  convention; the alternative (Django's default when `reverse_code` is
+  unset) is to mark the migration irreversible, which is rarely the
+  right semantics for a data migration whose forward effect just
+  rewrites or fills in rows. The rare RunPython migrations which are
+  truly irreversible should leave `reverse_code` unset, with an
+  explicit comment as to why.
+- **Set `elidable=True`** on data migrations whose effect won't be
+  needed once `squashmigrations` collapses old migrations.
+- **Make the data migration idempotent.** A migration may be
+  interrupted partway through (operator action, restart, network
+  blip) and re-run from the beginning. Filter your queryset on the
+  "not yet migrated" condition (`field=None`,
+  `subject__iexact="(no topic)"`, etc.) so re-running is a no-op for
+  rows already done.
+- **Be defensive about settings that may be removed.** If your
+  migration reads a Django setting whose value drives behavior, use
+  `getattr(settings, "FOO", default)` so the migration still runs
+  on a future codebase where the setting has been deleted (see
+  `0743_realm_require_e2ee_push_notifications.py`).
+- **Use `psycopg2.sql.SQL` / `Identifier` for table and column
+  names** in any raw SQL — never string-format them. The
+  migration may end up sharing helpers or being adapted later;
+  building safe-by-construction SQL avoids introducing an
+  injection vector.
+
+### Automated testing
+
+Zulip has support for writing automated tests for your database
+migrations, using the `MigrationsTestCase` test class. This system
+is inspired by [a great blog post][django-migration-test-blog-post]
+on the subject.
+
+We have integrated this system with our test framework so that if
+you use the `use_db_models` decorator, you can use some helper
+methods from `test_classes.py` and friends from inside the tests
+(which is normally not possible in Django's migrations framework).
+
+If you find yourself writing logic in a `RunPython` migration, we
+highly recommend adding a test using this framework. We may end up
+deleting the test later (they can get slow once they are many
+migrations away from current), but it can help prevent disaster
+where an incorrect migration messes up a database in a way that's
+impossible to undo without going to backups.
+
+[django-migration-test-blog-post]: https://www.caktusgroup.com/blog/2016/02/02/writing-unit-tests-django-migrations/
+
+### Pending trigger events when mixing schema and data changes
+
+Mixing data changes (`RunPython`) and `ALTER TABLE` operations in a
+single atomic migration can produce:
+
+```text
+django.db.utils.OperationalError: cannot ALTER TABLE "table_name" because it has pending trigger events
+```
+
+PostgreSQL prohibits schema changes if there are deferred trigger
+events still pending. Django creates foreign-key constraints as
+`DEFERRABLE INITIALLY DEFERRED` in PostgreSQL by default, so foreign
+key constraint checks are queued until commit time. You cannot run
+`ALTER TABLE` while those checks are still queued.
+
+Resolve this by making the migration non-atomic, splitting it into
+two migration files (recommended), or replacing the `RunPython`
+logic with pure SQL.
+
+## Making large migrations work
+
+For migrations that touch many rows or hold locks for non-trivial
+amounts of time, the goal is to keep individual lock durations
+short and total wall-clock time bounded.
+
+### Use `do_batch_update` when it fits
+
+For straightforward bulk updates over an entire table —
+set a column to a value or SQL expression on every row —
+`zerver/lib/migrate.py`'s `do_batch_update` helper handles all
+the iteration for you:
+
+```python
+from zerver.lib.migrate import do_batch_update
+
+def update_is_channel_message(apps, schema_editor):
+    Message = apps.get_model("zerver", "Message")
+    with connection.cursor() as cursor:
+        do_batch_update(
+            cursor,
+            Message._meta.db_table,
+            [SQL("is_channel_message = (SELECT type = 2 ...)")],
+        )
+
+class Migration(migrations.Migration):
+    atomic = False
+    operations = [
+        migrations.RunPython(update_is_channel_message, ...),
+    ]
+```
+
+It computes the table's `MIN(id)` and `MAX(id)`, walks the table
+in batches of 10000 by default, sleeps briefly between batches,
+and extends the upper bound if new rows have arrived during the
+run, so rows inserted concurrently aren't missed. Reach for it first; only write a
+hand-rolled loop when the helper doesn't fit (per-realm
+scoping, multi-step batches, or a non-`UPDATE` operation). See
+`0691_backfill_message_is_channel_message.py` for a complete
+example.
+
+### Push logic into the database
+
+A `RunPython` loop that reads rows in Python, computes a value,
+and writes them back is almost always slower and lock-heavier
+than a single SQL `UPDATE ... FROM (subquery)` per chunk. When
+you can, express the migration as SQL — see
+`0705_stream_subscriber_count_data_migration.py`, which uses one
+`UPDATE ... FROM (subquery)` per realm to backfill stream
+subscriber counts.
+
+### Iterate with `.iterator()`
+
+Django querysets cache their entire result set in memory by
+default. Whenever a migration loops over a queryset that could be
+large, use `.iterator()` to stream rows one at a time:
+
+```python
+for realm in Realm.objects.all().iterator():
+    ...
+```
+
+This applies to any per-row processing — emoji files, audit log
+entries, individual realms — not just to walking large tables.
+
+### Writing a hand-rolled batch loop
+
+When `do_batch_update` doesn't fit, write the batch loop
+yourself. The skeleton:
+
+```python
+class Migration(migrations.Migration):
+    atomic = False
+    ...
+
+def backfill(apps, schema_editor):
+    while lower_bound <= max_id:
+        print(f"Processing {lower_bound}/{max_id}...")
+        with transaction.atomic():
+            Stream.objects.filter(
+                id__range=(lower_bound, lower_bound + BATCH_SIZE - 1),
+                ...
+            ).update(...)
+        lower_bound += BATCH_SIZE
+```
+
+See `0765_set_default_can_create_topic_group.py` for a complete
+example. The considerations:
+
+- **Set [`atomic = False`](https://docs.djangoproject.com/en/6.0/howto/writing-migrations/#non-atomic-migrations)
+  on the migration class** and wrap each batch in its own
+  `transaction.atomic()`. Django's default per-migration
+  transaction would hold row locks for the entire
+  run; per-batch commits release locks between iterations.
+- **Batch by ID range**, not `OFFSET`. `id__range=(lower, upper)`
+  is O(batch size) per page; `OFFSET` makes each subsequent page
+  skip more rows. ID ranges aren't dense (gaps from deletes,
+  scattered filter matches), so the number of rows touched per
+  batch will vary, sometimes considerably — that's not an issue;
+  the goal is to bound how many rows a single transaction holds
+  locks on, not to do equal-sized chunks of work.
+- **For tables with `realm_id`, iterate per realm.** Processing
+  one realm at a time keeps each transaction's lock footprint
+  scoped to that realm and limits the blast radius of any single
+  batch. See `0696_rename_no_topic_to_empty_string_topic.py` and
+  `0782_delete_unused_anonymous_groups.py`.
+- **A few thousand rows is a reasonable batch size**; tune based
+  on per-row cost. If a batch takes more than a few seconds,
+  either the batch is too big or the work belongs in SQL.
+
+### Separating database and state
+
+Occasionally the SQL Django would auto-generate for a model
+change isn't the SQL you want to run — most often because
+Django's standard operations don't have a concurrent variant
+(e.g., `AlterField(db_index=False)` issues a locking `DROP
+INDEX`). `SeparateDatabaseAndState` lets you tell Django to
+update its model-state tracking using one set of operations
+while you actually execute a different set against the database:
+
+```python
+migrations.SeparateDatabaseAndState(
+    database_operations=[migrations.RunPython(...)],
+    state_operations=[migrations.AlterField(...)],
+)
+```
+
+The two sides end up in agreement on the schema — Django's view
+matches the database — but the DB-side change runs as your
+custom SQL or `RunPython` rather than as Django's auto-generated
+DDL. See
+`0791_alter_archivedusermessage_user_profile_and_more.py` for a
+recent example: the state side does `AlterField(db_index=False)`
+on a foreign key, while the database side runs `DROP INDEX
+CONCURRENTLY` on the underlying index.
+
+### Multi-step rollouts: replacing a column
+
+This pattern is only needed because Zulip Cloud migrates online —
+a stop-the-server-and-migrate self-hosted upgrade can rename or
+swap a column in a single migration without coordination. For a
+column replacement on a large table on Cloud, split the
+transition across multiple deploys that are each individually
+correct against the previous deploy's code:
+
+1. Add the new column (with `db_default`) and start writing to it
+   in addition to the old column. Don't read from it yet.
+2. Backfill the new column from the old column.
+3. Confirm that the two columns contain identical data.
+4. Switch readers to the new column. Stop writing to the old
+   column.
+5. Drop the old column.
+
+Each deploy is safe against the previous deploy's running code.
+
+## When a migration can't be online-safe
+
+Sometimes a migration is genuinely incompatible with the old code
+that will be running against the database during the deploy
+window. The common cases are:
+
+- **Dropping or renaming a column, table, or model that the
+  previous release's code still references.** Old code's queries
+  point at a schema element that no longer exists, so they fail
+  the moment the migration applies.
+- **A schema reshape where the intermediate state can't be read
+  correctly by either the old or new code.** For example,
+  restructuring a many-to-many relationship (Django's
+  [worked example](https://docs.djangoproject.com/en/6.0/howto/writing-migrations/#changing-a-manytomanyfield-to-use-a-through-model)
+  for swapping in a `through` model is a useful reference), or
+  splitting one column's data across two new columns where the
+  previous release expects the original column to be authoritative.
+
+Migrations like these can't simply land on `main` and ship: applying
+them to the shared database while production still runs the old
+code would break production. The deploy operator handles them with
+a stub-out / `migrate --fake` workflow:
+
+1. The migration is committed to `main` and runs in dev and CI as
+   normal.
+2. In Cloud's internal deploy branch, the migration is replaced
+   with a stub (a no-op) until the dependent code has reached
+   production.
+3. Deploy the new code to production.
+4. Run `manage.py migrate --fake zerver <previous_migration>` to mark
+   the stub as unapplied.
+5. Restore the real migration.
+6. Run `manage.py migrate zerver <target_migration>` to apply it for
+   real.
+7. Run `manage.py migrate --fake zerver <last_migration>` to move
+   the migration state back to the tip.
+
+If you write a migration that needs this treatment, **call it out
+explicitly in the commit** so the operators are aware when reviewing
+the pending migrations for a deploy.
+
+The stub-out workflow exists because migrations are not always
+split out from the application code that depends on them in
+current practice. If they were each in their own commit — schema
+change, backfill, and `NOT NULL` flip standalone — the operator
+could revert or reorder a single commit instead of editing
+migration files in place. For multi-step migrations where you
+anticipate the operator may need to defer pieces, keeping each
+migration in its own commit is worth the extra discipline.
+
+## Local schema rebuilds
+
+If you follow the processes described above, `tools/provision` and
+`tools/test-backend` should detect any changes to the declared
+migrations and run migrations on (`./manage.py migrate`) or rebuild
+the relevant database automatically as appropriate.
+
+Notably, both `manage.py migrate` (`git grep post_migrate.connect`
+for details) and restarting `tools/run-dev` will flush `memcached`,
+so you shouldn't have to worry about cached objects from a previous
+database schema.
+
+While developing migrations, you may accidentally corrupt your
+databases while debugging your new code. You can always rebuild
+these databases from scratch:
+
+- Use `tools/rebuild-test-database` to rebuild the database used
+  for `test-backend` and other automated tests.
+- Use `tools/rebuild-dev-database` to rebuild the database used in
+  [manual testing](../development/using.md).

docs/subsystems/sending-messages.md

@@ -0,0 +1,395 @@
+# Sending messages
+
+While sending a message in a chat product might seem simple, there's a
+lot of underlying complexity required to make a professional-quality
+experience.
+
+This document aims to explain conceptually what happens when a message
+is sent in Zulip, and why that is correct behavior. It assumes the
+reader is familiar with our
+[real-time sync system](events-system.md) for
+server-to-client communication and
+[new application feature tutorial](../tutorials/new-feature-tutorial.md),
+and we generally don't repeat the content discussed there.
+
+## Message lists
+
+This is just a bit of terminology: A "message list" is what Zulip
+calls the frontend concept of a (potentially narrowed) message feed.
+There are 3 related structures:
+
+- A `message_list_data` just has the sequencing data of which message
+  IDs go in what order.
+- A `message_list` is built on top of `message_list_data` and
+  additionally contains the data for a visible-to-the-user message list
+  (e.g., where trailing bookends should appear, a selected message,
+  etc.).
+- A `message_list_view` is built on top of `message_list` and
+  additionally contains rendering details like a window of up to 400
+  messages that is present in the DOM at the time, scroll position
+  controls, etc.
+
+(This should later be expanded into a full article on message lists
+and narrowing).
+
+## Compose area
+
+The compose box does a lot of fancy things that are out of scope for
+this article. But it also does a decent amount of client-side
+validation before sending a message off to the server, especially
+around mentions (e.g., checking the channel name is a valid channel,
+displaying a warning about the number of recipients before a user can
+use `@**all**` or mention a user who is not subscribed to the current
+channel, etc.).
+
+## Backend implementation
+
+The backend flow for sending messages is similar in many ways to the
+process described in our
+[new application feature tutorial](../tutorials/new-feature-tutorial.md).
+This section details the ways in which it is different:
+
+- There is significant custom code inside the `process_message_event`
+  function in `zerver/tornado/event_queue.py`. This custom code has a
+  number of purposes:
+  - Triggering [email and mobile push
+    notifications](notifications.md) for any users who
+    do not have active clients and have settings of the form "push
+    notifications when offline". In order to avoid doing any real
+    computational work inside the Tornado codebase, this logic aims
+    to just do the check for whether a notification should be
+    generated, and then put an event into an appropriate
+    [queue](queuing.md) to actually send the message.
+    See `maybe_enqueue_notifications` and `zerver/lib/notification_data.py` for
+    this part of the logic.
+  - Splicing user-dependent data (e.g., `flags` such as when the user
+    was `mentioned`) into the events.
+  - Handling the [local echo details](#local-echo).
+  - Handling certain client configuration options that affect
+    messages. E.g., determining whether to send the
+    plaintext/Markdown raw content or the rendered HTML (e.g., the
+    `apply_markdown` and `client_gravatar` features in our
+    [events API docs](https://zulip.com/api/register-queue)).
+- Following our standard naming convention, input validation is done
+  inside the `check_message` function in `zerver/actions/message_send.py`, which is responsible for
+  validating the user can send to the recipient,
+  [rendering the Markdown](markdown.md), etc. --
+  basically everything that can fail due to bad user input.
+- The core `do_send_messages` function (which handles actually sending
+  the message) in `zerver/actions/message_send.py` is one of the most optimized and thus complex parts of
+  the system. But in short, its job is to atomically do a few key
+  things:
+  - Store a `Message` row in the database.
+  - Store one `UserMessage` row in the database for each user who is
+    a recipient of the message (including the sender), with
+    appropriate `flags` for whether the user was mentioned, an alert
+    word appears, etc. See
+    [the section on soft deactivation](#soft-deactivation) for
+    a clever optimization we use here that is important for large
+    open organizations.
+  - Do all the database queries to fetch relevant data for and then
+    send a `message` event to the
+    [events system](events-system.md) containing the
+    data it will need for the calculations described above. This
+    step adds a lot of complexity, because the events system cannot
+    make queries to the database directly.
+  - Trigger any other deferred work caused by the current message,
+    e.g., [outgoing webhooks](https://zulip.com/api/outgoing-webhooks)
+    or embedded bots.
+  - Every query is designed to be a bulk query; we carefully
+    unit-test this system for how many database and memcached queries
+    it makes when sending messages with large numbers of recipients,
+    to ensure its performance.
+
+## Local echo
+
+An essential feature for a good chat experience is local echo
+(i.e. having the message appear in the feed the moment the user hits
+send, before the network round trip to the server). This is essential
+both for freeing up the compose box (for the user to send more
+messages) as well as for the experience to feel snappy.
+
+A sloppy local echo experience (like Google Chat had for over a decade
+for emoji) would just render the raw text the user entered in the
+browser, and then replace it with data from the server when it
+changes.
+
+Zulip aims for a near-perfect local echo experience, which is
+why our [Markdown system](markdown.md) requires both
+an authoritative (backend) Markdown implementation and a secondary
+(frontend) Markdown implementation, the latter used only for the local
+echo feature. Read our Markdown documentation for all the tricky
+details on how that works and is tested.
+
+The rest of this section details how Zulip manages locally echoed
+messages.
+
+- The core function in the frontend codebase
+  `echo.try_deliver_locally`. This checks whether correct local echo
+  is possible (via `markdown.contains_backend_only_syntax`) and useful
+  (whether the message would appear in the current view), and if so,
+  causes Zulip to insert the message into the relevant feed(s).
+- Since the message hasn't been confirmed by the server yet, it
+  doesn't have a message ID. The frontend makes one up, via
+  `local_message.get_next_id_float`, by taking the highest message ID it
+  has seen and adding the decimal `0.01`. The use of a floating point
+  value is critical, because it means the message should sort
+  correctly with other messages (at the bottom) and also won't be
+  duplicated by a real confirmed-by-the-backend message ID. We choose
+  just above the `max_message_id`, because we want any new messages
+  that other users send to the current view to be placed after it in
+  the feed (this decision is somewhat arbitrary; in any case we'll
+  resort it to its proper place once it is confirmed by the server.
+  We do it this way to minimize messages jumping around/reordering
+  visually).
+- The `POST /messages` API request to the server to send the message
+  is passed two special parameters that clients not implementing local
+  echo don't use: `queue_id` and `local_id`. The `queue_id` is the ID
+  of the client's event queue; here, it is used just as a unique
+  identifier for the specific client (e.g., a browser tab) that sent
+  the message. And the `local_id` is, by the construction above, a
+  unique value within that namespace identifying the message.
+- The `do_send_messages` backend code path includes the `queue_id` and
+  `local_id` in the data it passes to the
+  [events system](events-system.md). The events
+  system will extend the `message` event dictionary it delivers to
+  the client containing the `queue_id` with `local_message_id` field,
+  containing the `local_id` that the relevant client used when sending
+  the message. This allows the client to know that the `message`
+  event it is receiving is the same message it itself had sent.
+- Using that information, rather than adding the "new message" to the
+  relevant message feed, it updates the (locally echoed) message's
+  properties (at the very least, message ID and timestamp) and
+  rerenders it in any message lists where it appears. This is
+  primarily done in the `process_from_server` function in
+  `web/src/echo.ts`.
+
+### Local echo in message editing
+
+Zulip also supports local echo in the message editing code path for
+edits to just the content of a message. The approach is analogous
+(using `markdown.contains_backend_only_syntax`, etc.), except we
+don't need any of the `local_id` tracking logic, because the message
+already has a permanent message id; as a result, the whole
+implementation was under 150 lines of code.
+
+## Putting it all together
+
+This section just has a brief review of the sequence of steps all in
+one place:
+
+- User hits send in the compose box.
+- Compose box validation runs; if it passes, the browser locally
+  echoes the message and then sends a request to the `POST /messages`
+  API endpoint.
+- The Django URL routes and middleware run, and eventually call the
+  `send_message_backend` view function in `zerver/views/message_send.py`.
+  (Alternatively, for an API request to send a message via Zulip's
+  REST API, things start here).
+- `send_message_backend` does some validation before triggering the
+  `check_message` + `do_send_messages` backend flow.
+- That backend flow saves the data to the database and triggers a
+  `message` event in the `notify_tornado` queue (part of the events
+  system).
+- The events system processes, and dispatches that event to all
+  clients subscribed to receive notifications for users who should
+  receive the message (including the sender). As a side effect, it
+  adds queue items to the email and push notification queues (which,
+  in turn, may trigger those notifications).
+  - Other clients receive the event and display the new message.
+  - For the client that sent the message, it instead replaces its
+    locally echoed message with the final message it received back
+    from the server (it indicates this to the sender by adding a
+    display timestamp to the message).
+- The `send_message_backend` view function returns
+  a 200 `HTTP` response; the client receives that response and mostly
+  does nothing with it other than update some logging details. (This
+  may happen before or after the client receives the event notifying
+  it about the new message via its event queue.)
+
+## Message editing
+
+Message editing uses a very similar principle to how sending messages
+works. A few details are worth mentioning:
+
+- `maybe_enqueue_notifications_for_message_update` is an analogue of
+  `maybe_enqueue_notifications`, and exists to handle cases like a
+  user was newly mentioned after the message is edited (since that
+  should trigger email/push notifications, even if the original
+  message didn't have one).
+- We use a similar technique to what's described in the local echo
+  section for doing client-side rerendering to update the message feed.
+- In the default configuration, Zulip stores the message edit history
+  (which is useful for forensics but also exposed in the UI), in the
+  `message.edit_history` attribute.
+- We support topic editing, including bulk-updates moving several
+  messages between topics.
+
+### Inline URL previews
+
+Zulip's inline URL previews feature (`zerver/lib/url_preview/`) uses
+variant of the message editing/local echo behavior. The reason is
+that for inline URL previews, the backend needs to fetch the content
+from the target URL, and for slow websites, this could result in a
+significant delay in rendering the message and delivering it to other
+users.
+
+- For this case, Zulip's backend Markdown processor will render the
+  message without including the URL embeds/previews, but it will add a
+  deferred work item into the `embed_links` queue.
+
+- The [queue processor](queuing.md) for the
+  `embed_links` queue will fetch the URLs, and then if they return
+  results, rerun the Markdown processor and notify clients of the
+  updated message `rendered_content`.
+
+- We reuse the `update_message` framework (used for
+  Zulip's message editing feature) in order to avoid needing custom code
+  to implement the notification-and-rerender part of this implementation.
+
+## Soft deactivation
+
+This section details a somewhat subtle issue: How Zulip uses a
+user-invisible technique called "soft deactivation" to handle
+scalability to communities with many thousands of inactive users.
+
+For background, Zulip’s threading model requires tracking which
+individual messages each user has received and read (in other chat
+products, the system either doesn’t track what the user has read at
+all, or just needs to store a pointer for “how far the user has read”
+in each room or channel).
+
+We track these data in the backend in the `UserMessage` table, storing
+rows `(message_id, user_id, flags)`, where `flags` is 32 bits of space
+for boolean data like whether the user has read or starred the
+message. All the key queries needed for accessing message history,
+full-text search, and other key features can be done efficiently with
+the database indexes on this table (with joins to the `Message` table
+containing the actual message content where required).
+
+The downside of this design is that when a new message is sent to a
+channel with `N` recipients, we need to write `N` rows to the
+`UserMessage` table to record those users receiving those messages.
+Each row is just 3 integers in size, but even with modern databases
+and SSDs, writing thousands of rows to a database starts to take a few
+seconds.
+
+This isn’t a problem for most Zulip servers, but is a major problem
+for communities like chat.zulip.org, where there might be 10,000s of
+inactive users who only stopped by briefly to check out the product or
+ask a single question, but are subscribed to whatever the default
+channels in the organization are.
+
+The total amount of work being done here was acceptable (a few seconds
+of total CPU work per message to large public channels), but the
+latency was unacceptable: The server backend was introducing a latency
+of about 1 second per 2000 users subscribed to receive the message.
+While these delays may not be immediately obvious to users (Zulip,
+like many other chat applications,
+[local echoes](markdown.md) messages that a user sends
+as soon as the user hits “Send”), latency beyond a second or two
+significantly impacts the feeling of interactivity in a chat
+experience (i.e. it feels like everyone takes a long time to reply to
+even simple questions).
+
+A key insight for addressing this problem is that there isn’t much of
+a use case for long chat discussions among 1000s of users who are all
+continuously online and actively participating. Channels with a very
+large number of active users are likely to only be used for occasional
+announcements, where some latency before everyone sees the message is
+fine. Even in giant organizations, almost all messages are sent to
+smaller channels with dozens or hundreds of active users, representing
+some organizational unit within the community or company.
+
+However, large, active channels are common in open source projects,
+standards bodies, professional development groups, and other large
+communities with the rough structure of the Zulip development
+community. These communities usually have thousands of user accounts
+subscribed to all the default channels, even if they only have dozens
+or hundreds of those users active in any given month. Many of the
+other accounts may be from people who signed up just to check the
+community out, or who signed up to ask a few questions and may never
+be seen again.
+
+The key technical insight is that if we can make the latency scale
+with the number of users who actually participate in the community,
+not the total size of the community, then our database write limited
+send latency of 1 second per 2000 users is totally fine. But we need
+to do this in a way that doesn’t create problems if any of the
+thousands of “inactive” users come back (or one of the active users
+sends a direct message to one of the inactive users), since it’s
+impossible for the software to know which users are eventually coming
+back or will eventually be interacted with by an existing user.
+
+We solved this problem with a solution we call “soft deactivation”;
+users that are soft-deactivated consume less resources from Zulip in a
+way that is designed to be invisible both to other users and to the
+user themself. If a user hasn’t logged into a given Zulip
+organization for a few weeks, they are tagged as soft-deactivated.
+
+The way this works internally is:
+
+- We (usually) skip creating UserMessage rows for soft-deactivated
+  users when a message is sent to a channel where they are subscribed.
+
+- If/when the user ever returns to Zulip, we can at that time
+  reconstruct the UserMessage rows that they missed, and create the rows
+  at that time (or, to avoid a latency spike if/when the user returns to
+  Zulip, this work can be done in a nightly cron job). We can construct
+  those rows later because we already have the data for when the user
+  might have been subscribed or unsubscribed from channels by other
+  users, and, importantly, we also know that the user didn’t interact
+  with the UI since the message was sent (and thus we can safely assume
+  that the messages have not been marked as read by the user). This is
+  done in the `add_missing_messages` function, which is the core of the
+  soft-deactivation implementation.
+
+- The “usually” above is because there are a few flags that result
+  from content in the message (e.g., a message that mentions a user
+  results in a “mentioned” flag in the UserMessage row), that we need to
+  keep track of. Since parsing a message can be expensive (>10ms of
+  work, depending on message content), it would be too inefficient to
+  need to re-parse every message when a soft-deactivated user comes back
+  to Zulip. Conveniently, those messages are rare, and so we can just
+  create UserMessage rows which would have “interesting” flags at the
+  time they were sent without any material performance impact. And then
+  `add_missing_messages` skips any messages that already have a
+  `UserMessage` row for that user when doing its backfill.
+
+The end result is the best of both worlds:
+
+- Nobody's view of the world is different because the user was
+  soft-deactivated (resulting in no visible user-experience impact), at
+  least if one is running the cron job. If one does not run the cron
+  job, then users returning after being away for a very long time will
+  potentially have a (very) slow loading experience as potentially
+  100,000s of UserMessage rows might need to be reconstructed at once.
+- On the latency-sensitive message sending and fanout code path, the
+  server only needs to do work for users who are currently interacting
+  with Zulip.
+
+Empirically, we've found this technique completely resolved the "send
+latency" scaling problem. The latency of sending a message to a channel
+now scales only with the number of active subscribers, so one can send
+a message to a channel with 5K subscribers of which 500 are active, and
+it’ll arrive in the couple hundred milliseconds one would expect if
+the extra 4500 inactive subscribers didn’t exist.
+
+There are a few details that require special care with this system:
+
+- [Email and mobile push
+  notifications](notifications.md). We need to make
+  sure these are still correctly delivered to soft-deactivated users;
+  making this work required careful work for those code paths that
+  assumed a `UserMessage` row would always exist for a message that
+  triggers a notification to a given user.
+- Digest emails, which use the `UserMessage` table extensively to
+  determine what has happened in channels the user can see. We can use
+  the user's subscriptions to construct what messages they should have
+  access to for this feature.
+- Soft-deactivated users experience high loading latency when
+  returning after being idle for months. We optimize this by
+  triggering a soft reactivation for users who receive email or push
+  notification for direct messages or personal mentions, or who
+  request a password reset, since these are good leading indicators
+  that a user is likely to return to Zulip.

docs/subsystems/settings.md

@@ -0,0 +1,190 @@
+# Settings system
+
+The page documents the Zulip settings system, and hopefully should
+help you decide how to correctly implement new settings you're adding
+to Zulip.
+
+We have two types of administrative settings in Zulip:
+
+- **Server settings** are set via configuration files, and apply to
+  the whole Zulip installation.
+- **Realm settings** (or **organization settings**) are usually
+  set via the /#organization page in the Zulip web application, and
+  apply to a single Zulip realm/organization. (Which, for most Zulip
+  servers, is the only realm on the server).
+
+Philosophically, the goals of the settings system are to make it
+convenient for:
+
+- Zulip server administrators to configure
+  Zulip's feature set for their server without needing to patch Zulip
+- Realm administrators to configure settings for their organization
+  independently without needing to talk with the server administrator.
+- Secrets (passwords, API keys, etc.) to be stored in a separate place
+  from shareable configuration.
+
+## Server settings
+
+Zulip uses the [Django settings
+system](https://docs.djangoproject.com/en/5.0/topics/settings/), which
+means that the settings files are Python programs that set a lot of
+variables with all-capital names like `EMAIL_GATEWAY_PATTERN`. You can
+access these anywhere in the Zulip Django code using, for example:
+
+```python
+from django.conf import settings
+print(settings.EMAIL_GATEWAY_PATTERN)
+```
+
+Additionally, if you need to access a Django setting in a shell
+script (or just on the command line for debugging), you can use, for
+example:
+
+```console
+$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN
+%s@localhost:9991
+```
+
+The `DJANGO_SETTINGS_MODULE` environment variable is set to
+`zproject.settings`, which is what tells Django to effectively import
+`zproject.settings.*` into `django.conf.settings` to make them
+accessible there.
+
+However, Zulip has separated those settings that we expect a system
+administrator to change (with nice documentation) from the ~1000 lines
+of settings needed by the Zulip Django app. As a result, there are a
+few files involved in the Zulip settings for server administrators.
+In a production environment, we have:
+
+- `/etc/zulip/settings.py` (the template is in the Zulip repo at
+  `zproject/prod_settings_template.py`) is the main system
+  administrator-facing settings file for Zulip. It contains all the
+  server-specific settings, such as how to send outgoing email, the
+  hostname of the PostgreSQL database, etc., but does not contain any
+  secrets (e.g., passwords, secret API keys, cryptographic keys, etc.).
+  The way we generally do settings that can be controlled with shell
+  access to a Zulip server is to put a default in
+  `zproject/default_settings.py`, and then override it here. As this
+  is the main documentation for Zulip settings, we recommend that
+  production installations [carefully update `/etc/zulip/settings.py`
+  every major
+  release](../production/upgrade.md#updating-settingspy-inline-documentation)
+  to pick up new inline documentation.
+
+- `/etc/zulip/zulip-secrets.conf` (generated by
+  `scripts/setup/generate_secrets.py` as part of installation)
+  contains secrets used by the Zulip installation. These are read
+  using the [standard Python
+  `RawConfigParser`](https://docs.python.org/3/library/configparser.html#configparser.RawConfigParser),
+  and accessed in `zproject/computed_settings.py` by the `get_secret`
+  or `get_mandatory_secret` function defined in `zproject/config.py`.
+  All secrets/API keys/etc. used by the Zulip Django application should
+  be stored here. The secrets mandatory to start the Zulip Django app
+  are read using `get_mandatory_secret` and the others are read with
+  `get_secret`. If any of the mandatory secrets is missing, a
+  `ZulipSettingsError` is raised.
+
+- `zproject/settings.py` is the main Django settings file for Zulip.
+  It imports everything from `zproject/configured_settings.py` and
+  `zproject/computed_settings.py`.
+
+- `zproject/configured_settings.py` imports everything from
+  `zproject/default_settings.py`, then in a prod environment imports
+  `/etc/zulip/settings.py` via a symlink.
+
+- `zproject/default_settings.py` has the default values for the settings the
+  user would set in `/etc/zulip/settings.py`.
+
+- `zproject/computed_settings.py` contains all the settings that are
+  constant for all Zulip installations or computed as a function of
+  `zproject/configured_settings.py` (e.g., configuration for logging,
+  static assets, middleware, etc.).
+
+In a development environment, we have `zproject/settings.py`, and
+additionally:
+
+- `zproject/dev_settings.py` has the custom settings for the Zulip development
+  environment; these are set after importing `prod_settings_template.py`.
+
+- `zproject/dev-secrets.conf` replaces
+  `/etc/zulip/zulip-secrets.conf`, and is not tracked by Git. This
+  allows you to configure your development environment to support
+  features like [authentication
+  options](../development/authentication.md) that require secrets to
+  work. It is also used to set certain settings that in production
+  belong in `/etc/zulip/settings.py`, e.g., `SOCIAL_AUTH_GITHUB_KEY`.
+  You can see a full list with `git grep development_only=True`, or
+  add additional settings of this form if needed.
+
+- If you need to override a setting in your development environment,
+  you can do so by creating a `zproject/custom_dev_settings.py`
+  setting the values you'd like to override (the test suites ignore
+  this file). This optional file is processed just after
+  `dev_settings.py` in `configured_settings.py`, so
+  `zproject/computed_settings.py` will correctly use your custom
+  settings when calculating any computed settings that depend on them.
+
+- `zproject/test_settings.py` imports everything from
+  `zproject/settings.py` and `zproject/test_extra_settings.py`.
+
+- `zproject/test_extra_settings.py` has the (default) settings used
+  for the Zulip tests (both backend and Puppeteer), which are applied on
+  top of the development environment settings.
+
+When adding a new server setting to Zulip, you will typically add it
+in two or three places:
+
+- `zproject/default_settings.py`, with a default value
+  for production environments.
+
+- If the settings has a secret key,
+  you'll add a `get_secret` or `get_mandatory_secret` call in
+  `zproject/computed_settings.py` (and the
+  user will add the value when they configure the feature).
+
+- In an appropriate section of `zproject/prod_settings_template.py`,
+  with documentation in the comments explaining the setting's
+  purpose and effect.
+
+- Possibly also `zproject/dev_settings.py` and/or
+  `zproject/test_settings.py`, if the desired value of the setting for
+  Zulip development and/or test environments is different from the
+  default for production.
+
+Most settings should be enabled in the development environment, to
+maximize convenience of testing all of Zulip's features; they should
+be enabled by default in production if we expect most Zulip sites to
+want those settings.
+
+### Testing non-default settings
+
+You can write tests for settings using, for example,
+`with self.settings(TERMS_OF_SERVICE=None)`. However, this only works
+for settings which are checked at runtime, not settings which are only
+accessed in initialization of Django (or Zulip) internals
+(e.g., `DATABASES`). See the [Django docs on overriding settings in
+tests][django-test-settings] for more details.
+
+[django-test-settings]: https://docs.djangoproject.com/en/5.0/topics/testing/tools/#overriding-settings
+
+## Realm settings
+
+Realm settings are preferred for any configuration that is a matter of
+organizational policy (as opposed to technical capabilities of the
+server). As a result, configuration options for user-facing
+functionality is almost always added as a new realm setting, not a
+server setting. The [new feature tutorial][doc-newfeat] documents the
+process for adding a new realm setting to Zulip.
+
+So for example, the following server settings will eventually be
+replaced with realm settings:
+
+- `NAME_CHANGES_DISABLED`
+- `INLINE_IMAGE_PREVIEW`
+- `ENABLE_GRAVATAR`
+- Which authentication methods are allowed should probably appear in
+  both places; in server settings indicating the capabilities of the
+  server, and in the realm settings indicating which methods the realm
+  administrator wants to allow users to log in with.
+
+[doc-newfeat]: ../tutorials/new-feature-tutorial.md

docs/subsystems/slash-commands.md

@@ -0,0 +1,55 @@
+# Slash commands
+
+Slash commands are commands (mainly for power users) to quickly do
+some stuff from the compose box. The codebase refers to these as "zcommand"s,
+and both these terms are often user interchangeably.
+
+Currently supported slash commands are:
+
+- `/light` and `/dark` to change the UI theme
+- `/ping` to ping to server and get back the time for the round
+  trip. Mainly for testing.
+- `/fluid-width` and `/fixed-width` to toggle that setting
+
+It is important to distinguish slash commands from the
+[widget system](/subsystems/widgets.md). Slash commands essentially
+**do not send** messages (and could have very well had their
+own "command prompt" (but don't), since they have nothing to do with
+message sending), while widgets are special kinds of messages.
+
+### Data flow
+
+These commands have client-side support in `zcommands.js`.
+They send commands to the server using the `/json/command`
+endpoint.
+
+In the case of "/ping", the server code in `zcommand.py`
+basically just acks the client. The client then computes
+the round trip time and shows a little message above
+the compose box that the user can see and then dismiss.
+
+For commands like "/light" and "/dark", the server does
+a little bit of logic to toggle the user's dark theme
+setting, and this is largely done inside `zcommand.py`.
+The server sends a very basic response, and then
+the client actually changes the display colors. The
+client also shows the user a little message above
+the compose box instructing them how to reverse the
+change.
+
+(It's possible that we don't really need a general
+`/json/zcommand` endpoint for these, and we
+may decide later to just use custom
+API endpoints for each command. There's some logic
+in having a central API for these, though, since they
+are typically things that only UI-based clients will
+invoke, and they may share validation code.)
+
+It is the client's responsibility to correctly detect and
+process when a user uses a slash command, and not instead
+send a message with the raw content.
+
+## Typeahead
+
+Typeahead for both slash commands (and widgets) is implemented
+via the `slash_commands` object in `web/src/composebox_typeahead.ts`.

docs/subsystems/thumbnailing.md

@@ -0,0 +1,152 @@
+# Thumbnailing
+
+## libvips
+
+Zulip uses the [`libvips`](https://www.libvips.org/) image processing toolkit
+for thumbnailing, as a low-memory and high-performance image processing
+library. Some smaller images are thumbnailed synchronously inside the Django
+process, but the majority of the work is offloaded to one or more `thumbnail`
+worker processes.
+
+Thumbnailing is a notoriously high-risk surface from a security standpoint,
+since it parses arbitrary binary user input with often complex grammars. On
+versions of `libvips` which support it (>= 8.13, on or after Ubuntu 24.04 or
+Debian 12), Zulip limits `libvips` to only the image parsers and libraries whose
+image formats we expect to parse, all of which are fuzz-tested by
+[`oss-fuzz`](https://google.github.io/oss-fuzz/).
+
+## Avatars
+
+Avatar images are served at two of potential resolutions (100x100 and 500x500,
+the latter of which is called "medium"), and always as PNGs. These are served
+from a "dumb" endpoint -- that is, if S3 is used, we provide a direct link to
+the content in the S3 bucket (or a Cloudfront distribution in front of it), and
+the request does not pass through the Zulip server. This is because avatars are
+referenced in emails, and thus their URLs need to be permanent and
+publicly-accessible. This also means that any choice of resolution and file
+format needs to be entirely done by the client.
+
+Avatars are thumbnailed synchronously upon upload into 100x100 and 500x500 PNGs;
+the originals are not preserved. The smallest dimension is scaled to fit, and
+the largest dimension is cropped centered; the image may be scaled _up_ to fit
+the 100x100 or 500x500 dimensions. To generate the filename, the server hashes
+the avatar salt (a server-side secret), the user-id, and a per-user sequence
+(the "version") to produce a filename which is not enumerable, and can only be
+determined by the server. Hashing the version means that avatars can be served
+with long-lasting caching headers.
+
+The original avatar image is stored adjacent to the thumbnailed versions,
+enabling later re-thumbnailing to other dimensions or formats without requiring
+users to re-upload it.
+
+## Emoji
+
+Emoji URLs are hard-coded into emails, and as such their URLs need to be
+permanent and publicly-accessible. They are served at a consistent 1:1 aspect
+ratio, and while they may be rendered at different scales based on the
+line-height of the client, we only need to store them at one resolution.
+
+Emoji are thumbnailed synchronously into 64x64 images, and they are saved in
+the same file format that they were uploaded in. Transparent pixels are added
+to the smaller dimension to make the image square after resizing. The filename
+of the emoji is based on a hash of the avatar salt (a server-side secret) and
+the emoji's id -- but because the filename is stored in the database, it can be
+anything with sufficient entropy to not be enumerable or have collisions.
+
+For animated emoji, a separate "still" version of the emoji is generated from
+the first frame, as a 64x64 PNG image. This is currently mostly unused, but is
+intended to be part of a user preference to disable emoji animations (see
+[#13434](https://github.com/zulip/zulip/issues/13434)). Current use is limited
+to [user status](https://zulip.com/help/status-and-availability) display in
+the buddy list. When a user uses an animated emoji as their status, the
+"still" version is used.
+
+The original emoji is stored adjacent to the thumbnailed version, enabling later
+re-thumbnailing to other dimensions or formats without requiring users to
+re-upload it.
+
+There is no technical reason that we preserve the uploader's choice of file
+format, or that we use PNGs as the file format for the "still" version. Both of
+these would plausibly benefit from being WebP images.
+
+## Realm logos
+
+Realm logos are converted to PNGs, thumbnailed down to fit within 800x100; a
+1000x10 pixel image will end up as 800x8, and a 10x20 will end up 10x20. The
+original is stored adjacent to the converted thumbnail.
+
+## Realm icons
+
+Realm icons are converted to PNGs, and treated identical to avatars, albeit only
+producing the 100x100 size.
+
+## File uploads
+
+### Images
+
+When an image file (as determined by the browser-supplied content-type) is
+uploaded, we immediately upload the original content into S3 or onto disk. Its
+headers are then examined, and used to create an ImageAttachment row, with
+properties determined from the image; `thumbnail_metadata` is left empty. A
+task is dispatched to the `thumbnail` worker to generate thumbnails in all of
+the format/size combinations that the server currently has configured.
+
+Because we parse the image headers enough to extract size information at upload
+time, this also serves as a check that the upload is indeed a valid image. If
+the image is determined to be invalid at this stage, the file upload returns
+200, but the message content is left with a link to the uploaded content, not an
+inline image.
+
+When a message is sent, it checks the ImageAttachment rows for each referenced
+image; if they have a non-empty `thumbnail_metadata`, then it writes out an
+`img` tag pointing to one of them (see below); otherwise, it writes out a
+specially-tagged "spinner" image, which indicates the server is still processing
+the upload. The image tag encodes the original dimensions and if the image is
+animated into the rendered content so clients can reserve the appropriate space
+in the viewport.
+
+If a message is rendered with a spinner, it also inserts the image into the
+`thumbnail` worker's queue. This is generally redundant -- the image was
+inserted into the queue when the image was uploaded. The exception is if the
+image was uploaded prior to the existence of thumbnailing support, in which case
+the additional queue insertion is required to have the spinner ever resolve.
+Since the worker takes no action if all necessary thumbnails already exist,
+this has little cost in general.
+
+The `thumbnail` worker generates the thumbnails, uploads them to S3 or disk, and
+then updates the `thumbnail_metadata` of the ImageAttachment row to contain a
+list of formats/sizes which thumbnails were generated in. At the time of commit,
+if there are already messages which reference the attachment row, then we do a
+"silent" update of all of them to remove the "spinner" and insert an image.
+
+In either case, the image which is inserted into the message body is at a
+"reasonable" scale and format, as decided by the server. The paths to all the
+generated thumbnails are not specified in the message content -- instead, the
+client is told at registration time the set of formats/sizes which the server
+supports, and knows how to transform any single thumbnailed path into any of the
+other supported thumbnail variants. The client is responsible for choosing the
+most appropriate format/size based on viewport size and format support, and
+rewriting the URL accordingly.
+
+All requests for images go through `/user_uploads`, which is processed by
+Django. Any request for an ImageAttachment URL is first determined to be a valid
+format/size for the server's current configuration; if is not valid, the server
+may return any other thumbnail of its choosing (preferring similar sizes, and
+accepted formats based on the client's `Accepts` header).
+
+If the request is for a thumbnail format/size which is supported by the server,
+but not in the ImageAttachment's `thumbnail_metadata` (as would happen if the
+server's supported set is added to over time) then the server should generate,
+store, and return the requested format/size on-demand.
+
+### Migrations
+
+Historical image uploads have ImageAttachment rows generated for them, but not
+thumbnails. If the message content is re-rendered (for instance, due to being
+edited) then it will trigger the image to be thumbnailed.
+
+### Videos and PDFs
+
+The thumbnailing system only processes images; it does not transcode videos or produce
+image renderings of documents (e.g., PDFs), though those are natural potential
+extensions.

docs/subsystems/typing-indicators.md

@@ -0,0 +1,181 @@
+# Typing indicators
+
+Zulip supports a feature called "typing indicators."
+
+Typing indicators are status messages (or visual indicators) that
+tell you when another user is writing a message to you. Zulip's
+typing UI is similar to what you see in other chat/text systems.
+
+This document describes how we have implemented the feature in
+the Zulip web app, and our main audience is developers who want to
+understand the system and possibly improve it. Any client should
+be able follow the protocol documented here.
+
+Typing indicators are implemented for both direct message conversations
+and channel conversations in the web app.
+
+There are two major roles for users in this system:
+
+- The "writing user" is composing a message.
+- The "receiving user" is waiting to receive a message (or possibly
+  ready to shift their attention elsewhere).
+
+Any Zulip user can play either one of these roles, and sometimes
+they can be playing both roles at once. Having said that, you
+can generally understand the system in terms of a single message
+being composed by the "writing user."
+
+On a high level the typing indicators system works like this:
+
+- The client for the "writing user" sends requests to the server.
+- The server broadcasts events to other users.
+- The clients for "receiving users" receive events and conditionally
+  show typing indicators, depending on where the clients are narrowed.
+
+## Privacy settings
+
+Note that there is a user-level privacy setting to disable sending
+typing notifications that a client should check when implementing
+the "writing user" protocol below. See `send_private_typing_notifications`
+in the `UserBaseSettings` model in `zerver/models/users.py` and in the
+`user_settings` object in the `POST /register` response.
+
+## Writing user
+
+When a "writing user" starts to compose a message, the client
+sends a request to `POST /typing` with an `op` of `start` and
+a list of potential message recipients. The web app function
+that facilitates this is called `send_typing_notification_ajax`.
+
+If the "writing user" is composing a long message, we want to send
+repeated updates to the server so that downstream clients know the
+user is still typing. Zulip messages tend to be longer than
+messages in other chat/text clients, so this detail is important.
+
+We have a small state machine in `web/src/typing_status.ts`
+that makes sure subsequent "start" requests get sent out. The
+frequency of these requests is determined by
+`server_typing_started_wait_period_milliseconds` in the
+`POST /register` response.
+
+If the "writing user" goes for a while without any text input,
+then we send a request to `POST /typing` with an `op` of `stop`.
+The time period a client should wait before sending the request
+is determined by `server_typing_stopped_wait_period_milliseconds`
+in the `POST /register` response. We also immediately send "stop"
+notification when the user explicitly aborts composing a message
+by closing the compose box (or other actions).
+
+A common scenario is that a user is just pausing to think for a few
+seconds, but they still intend to finish the message. Of course,
+that's hard to distinguish from the scenario of the user got pulled
+away from their desk. For the former case, where the "writing user"
+completes the message with lots of pauses for thinking, a series of
+"start" and "stop" messages may be sent over time. Timeout values
+reflect tradeoffs, where we have to guess how quickly people type,
+how long they pause to think, and how frequently they get interrupted.
+
+## Server
+
+The server piece of typing notifications is currently pretty
+straightforward, since we take advantage of Zulip's
+[events system](events-system.md).
+
+We deliberately designed the server piece to be stateless,
+which minimizes the possibility of backend bugs and gives clients
+more control over the user experience.
+
+As such, the server piece here is basically a single Django view
+function with a small bit of library support to send out events
+to clients.
+
+Requests come in to `send_notification_backend`, which is in
+`zerver/views/typing.py`. For direct message typing notifications,
+the call to `check_send_typing_notification` does the heavy lifting.
+
+One of the main things that the server does is to validate that
+the user IDs in the `to` parameter are for valid, active users in
+the realm.
+
+Once the request has been validated, the server sends events to
+potential recipients of the message. The event type for that
+payload is `typing`. See the function `do_send_typing_notification`
+in `zerver/actions/typing.py` for more details.
+
+For channel typing notifications, the server also handles the logic
+for determining which users should receive the typing events based
+on channel subscribers.
+
+## Receiving user
+
+When a user plays the role of a "receiving user," the client handles
+incoming "typing" events from the server, and the client will
+display a typing indicator only when both of these conditions are
+true:
+
+- The "writing user" is still likely typing.
+- The "receiving user" is in a view where they'd see the eventual
+  message.
+
+The client code starts by processing events, and it maintains data
+structures, and then it eventually shows or hides status messages.
+
+We'll describe the flow of data through the web app
+as a concrete example.
+
+The events will come in to `web/src/server_events_dispatch.js`.
+The `stop` and `start` operations get further handled by
+`web/src/typing_events.ts`.
+
+The main goal is then to triage which events should lead to
+display changes.
+
+The web app client maintains a list of incoming "typists" using
+code in `web/src/typing_data.ts`. The API here has functions
+like the following:
+
+- `add_typist`
+- `remove_typist`
+- `get_group_typists`
+- `get_all_direct_message_typists`
+
+One subtle thing that the client has to do here is to maintain
+timers for typing notifications. The value of
+`server_typing_started_expiry_period_milliseconds` in the
+`POST /register` response is used to determine when the
+"writing user" has abandoned the message. Of course, the
+"writing user" will also explicitly send us `stop` notifications
+at certain times.
+
+When it finally comes to displaying the notification, the web
+app eventually calls `render_notifications_for_narrow`.
+
+## Ecosystem
+
+Even though the server is stateless, any developer working on
+a client needs to be mindful of timing/network considerations
+that affect the overall system.
+
+In general, client developers should agree on timeout parameters
+for how frequently we "kickstart" typing notifications for users
+sending long messages. This means standardizing the "writing
+user" piece of the system. It's possible that certain clients
+will have slightly different mechanisms for detecting that users
+have abandoned messages, but the re-transmit frequency should be
+similar.
+
+When implementing the "receiving user" piece, it's important to
+realize how clients behave on the other end of the protocol. It's
+possible, for example, to never receive a "stop" notification
+from a client that was shut down abruptly. You should allow
+reasonable amounts of time for the other side to send notifications,
+allowing for network delays and server delays, but you should
+not let the notifications become too "sticky" either.
+
+## Roadmap
+
+An area for refinement is to tune the timing values a bit.
+Right now, we are possibly too aggressive about sending `stop`
+messages when users are just pausing to think. It's possible
+to better account for typing speed or other heuristic things
+like how much of the message has already been typed.

docs/subsystems/unread_messages.md

@@ -0,0 +1,52 @@
+# Unread message synchronization
+
+In general displaying unread counts for all channels and topics may require
+downloading an unbounded number of messages. Consider a user who has a muted
+channel or topic and has not read the backlog in a month; to have an accurate
+unread count we would need to load all messages this user has received in the
+past month. This is inefficient for web clients and even more for mobile
+devices.
+
+We work around this by including a list of unread message ids in the initial
+state grouped by relevant conversation keys. This data is included in the
+`unread_msgs` key if both `update_message_flags` and `message` are required
+in the register call.
+
+```json
+{
+    "count": 4,
+    "huddles": [
+        {
+            "user_ids_string": "3,4,6",
+            "unread_message_ids": [
+                34
+            ]
+        }
+    ],
+    "streams": [
+        {
+            "stream_id": 1,
+            "topic": "test",
+            "unread_message_ids": [
+                33
+            ]
+        }
+    ],
+    "pms": [
+        {
+            "sender_id": 3,
+            "unread_message_ids": [
+                31,
+                32
+            ]
+        }
+    ],
+    "mentions": [31, 34]
+}
+```
+
+Three event types are required to correctly maintain the `unread_msgs`. New
+messages can be created without the unread flag by the `message` event type.
+The unread flag can be added and removed by the `update_message_flags` event,
+and the topic of unread messages can be updated by the `update_message` event
+type.

docs/subsystems/widgets.md

@@ -0,0 +1,332 @@
+# Widgets
+
+## What is a widget?
+
+Widgets are special kinds of messages. These include:
+
+- polls
+- TODO lists
+- `/me` messages
+- Trivia bot
+
+Some widgets are used with a leading `/` (like `/poll Tea or coffee?`), similar
+to [slash commands](/subsystems/slash-commands.md), but these two concepts
+are very different. Slash commands have nothing to do with message sending.
+
+The trivia_quiz_bot does not use `/`'s. Instead, it sends "extra_data"
+in messages to invoke **zforms** (which enable button-based UIs in the
+messages).
+
+## `/me` messages
+
+These are the least complex. We use our markdown processors to
+detect if a message is a `/me` message, plumb the flag through
+the message object (as `is_me_message`) and have the clients
+format it correctly. Related code (for the web app) lies in
+`message_list_view.ts` in `_maybe_get_me_message`.
+
+## Polls, todo lists, and games
+
+The most interactive widgets that we built during
+2018 are for polls, todo lists, and games. You
+launch widgets by sending one of the following messages:
+
+- /poll
+- /todo
+
+The web app client provides the "widget experience" by
+default. Other clients just show raw messages like
+"/poll", and should be adding support
+for widgets soon.
+
+Our customers have long requested a poll/survey widget.
+See [this issue](https://github.com/zulip/zulip/issues/9736).
+There are workaround ways to do polls using things like
+emoji reactions, but our poll widget provides a more
+interactive experience.
+
+### Data flow
+
+Some important code entities for the widget implementation are:
+
+- `SubMessage` database table
+- `/json/submessage` API endpoint
+- `web/src/submessage.js`
+- `web/src/poll_widget.js`
+- `web/src/widgetize.ts`
+- `web/src/zform.ts`
+- `web/templates/widgets/`
+- `zerver/lib/widget.py`
+- `zerver/views/submessage.py`
+
+Both **poll** and **todo** widgets use the "submessage" architecture.
+We'll use the poll widget as an example.
+
+The `SubMessage` table, as the name indicates, allows
+you to associate multiple submessages to any given
+`Message` row. When a message gets sent, there's a
+hook inside of `widget.py` that will detect widgets
+like "/poll". If a message needs to be
+widgetized, an initial `SubMessage` row will be
+created with an appropriate `msg_type` (and persisted
+to the database). This data will also be included
+in the normal Zulip message event payload. Clients
+can choose to ignore the submessage-related data, in
+which case they'll gracefully degrade to seeing "/poll".
+Of course, the web app client actually recognizes the
+appropriate widgets.
+
+The web app client will next collect poll options and votes
+from users. The web app client has
+code in `submessage.js` that dispatches events
+to `widgetize.ts`, which in turn sends events to
+individual widgets. The widgets know how to render
+themselves and set up click/input handlers to collect
+data. They can then post back to `/json/submessage`
+to attach more data to the message (and the
+details are encapsulated with a callback). The server
+will continue to persist `SubMessage` rows in the
+database. These rows are encoded as JSON, and the
+schema of the messages is driven by the individual widgets.
+Most of the logic is in the client; things are fairly opaque
+to the server at this point.
+
+If a client joins Zulip after a message has accumulated
+several submessage events, it will see all of those
+events the first time it sees the parent message. Clients
+need to know how to build/rebuild their state as each
+submessage comes in. They also need to tolerate
+misformatted data, ideally just dropping data on the floor.
+If a widget throws an exception, it's caught before the
+rest of the message feed is affected.
+
+As far as rendering is concerned, each widget module
+is given a parent `elem` when its `activate` function
+is called. This is just a `<div>` inside of the parent
+message in the message pane. The widget has access to
+jQuery and template.render, and the developer can create
+new templates in `web/templates/widgets/`.
+
+A good way to learn the system is to read the code
+in `web/src/poll_widget.js`. It is worth noting that
+writing a new widget requires only minor backend
+changes in the current architecture. This could change
+in the future, but for now, a frontend developer mostly
+needs to know JS, CSS, and HTML.
+
+It may be useful to think of widgets in terms of a
+bunch of clients exchanging peer-to-peer messages. The
+server's only real role is to decide who gets delivered
+which submessages. It's a lot like a "subchat" system.
+
+### Backward compatibility
+
+Our "submessage" widgets are still evolving, and we want
+to have a plan for allowing future progress without
+breaking old messages.
+
+Widget developers can revise code to improve a
+widget's visual polish without too much concern
+for breaking how old messages get widgetized. They will need to
+be more cautious if they change the actual data
+structures passed around in the submessage payloads.
+
+For significant schema changes, it would be worthwhile to add
+some kind of versioning scheme inside of `SubMessages`, either
+at the DB level or more at the JSON level within fields.
+This has yet to be designed. One thing to consider is that
+most widgets are somewhat ephemeral in nature, so it's not
+the end of the world if upgrades cause some older messages
+to be obsolete, as long as the code degrades gracefully.
+
+Mission-critical widgets should have a deprecation strategy.
+For example, you could add optional features for one version
+bump and then only make them mandatory for the next version,
+as long as you don't radically change the data model. And
+if you're truly making radical changes, you can always
+write a Django migration for the `SubMessage` data.
+
+### Adding widgets
+
+Right now we don't have a plugin model for the above widgets;
+they are served up by the core Zulip server implementation.
+Of course, anybody who wishes to build their own widget
+has the option of forking the server code and self-hosting,
+but we want to encourage folks to submit widget
+code to our codebase in PRs. If we get to a critical mass
+of contributed widgets, we will want to explore a more
+dynamic mechanism for "plugging in" code from outside sources,
+but that is not in our immediate roadmap.
+
+This is sort of a segue to the next section of this document.
+Suppose you want to write your own custom bot, and you
+want to allow users to click buttons to respond to options,
+but you don't want to have to modify the Zulip server codebase
+to turn on those features. This is where our "zform"
+architecture comes to the rescue.
+
+## zform (trivia quiz bot)
+
+This section will describe our "zform" architecture.
+
+For context, imagine a naive trivia bot. The trivia bot
+sends a question with the answers labeled as A, B, C,
+and D. Folks who want to answer the bot send back an
+answer have to send an actual Zulip message with something
+like `@trivia_bot answer A to Q01`, which is kind of
+tedious to type. Wouldn't it be nice if the bot could
+serve up some kind of buttons with canned replies, so
+that the user just hits a button?
+
+That is where zforms come in. Zulip's trivia bot sends
+the Zulip server a JSON representation of a form it
+wants rendered, and then the client renders a generic
+"**zform**" with buttons corresponding to `short_name` fields
+inside a `choices` list inside of the JSON payload.
+
+Here is what an example payload looks like:
+
+```json
+{
+    "extra_data": {
+        "type": "choices",
+        "heading": "05: What color is a blueberry?",
+        "choices": [
+            {
+                "type": "multiple_choice",
+                "reply": "answer 05 A",
+                "long_name": "red",
+                "short_name": "A"
+            },
+            {
+                "type": "multiple_choice",
+                "reply": "answer 05 B",
+                "long_name": "blue",
+                "short_name": "B"
+            },
+            {
+                "type": "multiple_choice",
+                "reply": "answer 05 C",
+                "long_name": "yellow",
+                "short_name": "C"
+            },
+            {
+                "type": "multiple_choice",
+                "reply": "answer 05 D",
+                "long_name": "orange",
+                "short_name": "D"
+            }
+        ]
+    },
+    "widget_type": "zform"
+}
+```
+
+When users click on the buttons, **generic** click
+handlers automatically simulate a client reply using
+a field called `reply` (in `choices`) as the content
+of the message reply. Then the bot sees the reply
+and grades the answer using ordinary chat-bot coding.
+
+The beautiful thing is that any third party developer
+can enhance bots that are similar to the **trivia_quiz**
+bot without touching any Zulip code, because **zforms**
+are completely generic.
+
+## Data flow
+
+We can walk through the steps from the bot generating
+the **zform** to the client rendering it.
+
+First,
+[here](https://github.com/zulip/python-zulip-api/blob/main/zulip_bots/zulip_bots/bots/trivia_quiz/trivia_quiz.py)
+is the code that produces the JSON.
+
+```py
+def format_quiz_for_widget(quiz_id: str, quiz: Dict[str, Any]) -> str:
+    widget_type = 'zform'
+    question = quiz['question']
+    answers = quiz['answers']
+
+    heading = quiz_id + ': ' + question
+
+    def get_choice(letter: str) -> Dict[str, str]:
+        answer = answers[letter]
+        reply = 'answer ' + quiz_id + ' ' + letter
+
+        return dict(
+            type='multiple_choice',
+            short_name=letter,
+            long_name=answer,
+            reply=reply,
+        )
+
+    choices = [get_choice(letter) for letter in 'ABCD']
+
+    extra_data = dict(
+        type='choices',
+        heading=heading,
+        choices=choices,
+    )
+
+    widget_content = dict(
+        widget_type=widget_type,
+        extra_data=extra_data,
+    )
+    payload = json.dumps(widget_content)
+    return payload
+```
+
+The above code processes data that is specific to a trivia
+quiz, but it follows a generic schema.
+
+The bot sends the JSON payload to the server using the
+`send_reply` callback.
+
+The bot framework looks for the optional `widget_content`
+parameter in `send_reply` and includes that in the
+message payload it sends to the server.
+
+The server validates the schema of `widget_content` using
+`check_widget_content`.
+
+Then code inside of `zerver/lib/widget.py` builds a single
+`SubMessage` row to contain the **zform** payload, and the
+server also sends this payload to all clients who are
+recipients of the parent message.
+
+When the message gets to the client, the codepath for **zform**
+is actually quite similar to what happens with a more
+customized widget like **poll**. (In fact, **zform** is a
+sibling of **poll** and **zform** just has a somewhat more
+generic job to do.) In `web/src/widgetize.ts` you will see
+where this code converges, with snippets like this:
+
+```js
+widgets.poll = poll_widget;
+widgets.todo = todo_widget;
+widgets.zform = zform;
+```
+
+The code in `web/src/zform.ts` renders the form (not
+shown here) and then sets up a click handler like below:
+
+```js
+    $elem.find('button').on('click', function (e) {
+        e.stopPropagation();
+
+        // Grab our index from the markup.
+        var idx = $(e.target).attr('data-idx');
+
+        // Use the index from the markup to dereference our
+        // data structure.
+        var reply_content = data.choices[idx].reply;
+
+        transmit.reply_message(
+            opts.message,
+            reply_content,
+        );
+    });
+```
+
+And then we are basically done!