snapshot of https://github.com/zulip/zulip.git @ 48c69911c7dd4e03b20552eb3bd77ad8151c1357 (2026-09-15)

docs/testing/continuous-integration.md

@@ -0,0 +1,129 @@
+# Continuous integration (CI)
+
+The Zulip server uses [GitHub Actions](https://docs.github.com/en/actions) for continuous
+integration. GitHub Actions runs frontend, backend and end-to-end production
+installer tests. This page documents useful tools and tips when using
+GitHub Actions and debugging issues with it.
+
+## Goals
+
+The overall goal of our CI is to avoid regressions and minimize the
+total time spent debugging Zulip. We do that by trying to catch as
+many possible future bugs as possible, while minimizing both latency
+and false positives, both of which can waste a lot of developer time.
+There are a few implications of this overall goal:
+
+- If a test is failing nondeterministically in CI, we consider that to
+  be an urgent problem.
+- If the tests become a lot slower, that is also an urgent problem.
+- Everything we do in CI should also have a way to run it quickly
+  (under 1 minute, preferably under 3 seconds), in order to iterate fast
+  in development. Except when working on the CI configuration itself, a
+  developer should never have to repeatedly wait 10 minutes for a full CI
+  run to iteratively debug something.
+
+## GitHub Actions
+
+### Useful debugging tips and tools
+
+- GitHub Actions stores timestamps for every line in the logs. They
+  are hidden by default; you can see them by toggling the
+  `Show timestamps` option in the menu on any job's log page. (You can
+  get this sort of timestamp in a development environment by piping
+  output to `ts`).
+
+- GitHub Actions runs on every branch you push on your Zulip fork.
+  This is helpful when debugging something complicated.
+
+- You can also ssh into a container to debug failures. SSHing into
+  the containers can be helpful, especially in rare cases where the
+  tests are passing in your computer but failing in the CI. There are
+  various
+  [Actions](https://github.com/marketplace?type=actions&query=debug+ssh)
+  available on GitHub Marketplace to help you SSH into a container. Use
+  whichever you find easiest to set up.
+
+### Suites
+
+We run multiple jobs during a GitHub Actions build to efficiently run
+Zulip's various test suites, some of them multiple times because we
+support multiple versions of the base OS. See the [Actions
+tabs](https://github.com/zulip/zulip/actions) for full list of Actions
+that we run.
+
+Files which define GitHub workflows live in `.github/workflows` directory.
+`zulip-ci.yml` is the main file where most of the tests are run.
+`production-suite.yml` builds a Zulip release tarball, which is
+then installed in a fresh container. Various Nagios and other
+checks are run to confirm the installation worked.
+
+`zulip-ci.yml` is designed to run our main test suites on all of our
+supported platforms. Out of them, only one of them runs the frontend
+tests, since `puppeteer` is slow and unlikely to catch issues that
+depend on the version of the base OS and/or Python. Similarly, only a
+(different) one runs the documentation tests.
+
+Our code for running the tests in CI lives under `tools/ci`; but that
+logic is mostly thin wrappers around [Zulip's test
+suites](testing.md) or production installer.
+
+The `Legacy OS` tests are designed to ensure we give good error
+messages when trying to upgrade Zulip servers running on very old base
+OS versions with EOL Python versions that Zulip no longer supports.
+
+### Configuration
+
+The remaining details in this section are primarily relevant for doing
+development on our CI system and/or provisioning process.
+
+The first key of the job section is `docker`. The docker key specifies
+the image GitHub Actions should get from [Docker Hub][docker-hub] for running
+the job. Once GitHub Actions fetches the image from Docker Hub, it will spin
+up a docker container. See [images](#images) section to know more about
+the images we use in GitHub Actions for testing.
+
+[docker-hub]: https://hub.docker.com/r/zulip/ci
+
+After booting the container from the configured image, GitHub Actions will
+create the directory mentioned in `working_directory` and all the
+steps will be run from here.
+
+The `steps` section describes everything: fetching the Zulip
+code, provisioning, fetching caught data, running tests and uploading
+coverage reports. The steps with prefix `*` reference aliases, which
+are defined in the `aliases` section at the top of the file.
+
+### Images
+
+GitHub Actions tests are run in containers that are spun off from the
+images maintained by Zulip team. The Docker images can be generated by
+running `tools/ci/build-docker-images`; see instructions at the top of
+`tools/ci/Dockerfile` for more information.
+
+### Performance optimizations
+
+#### Caching
+
+An important element of making GitHub Actions perform effectively is caching
+between jobs the various caches that live under `/srv/` in a Zulip
+development or production environment. In particular, we cache the
+following:
+
+- Python virtualenvs
+- node_modules directories
+
+This has a huge impact on the performance of running tests in GitHub Actions
+CI; without these caches, the average test time would be several times
+longer.
+
+We have designed these caches carefully (they are also used in
+production and the Zulip development environment) to ensure that each
+is named by a hash of its dependencies and ubuntu distribution name,
+so Zulip should always be using the same version of dependencies it
+would have used had the cache not existed. In practice, bugs are
+always possible, so be mindful of this possibility.
+
+A consequence of this caching is that test jobs for branches which
+modify `package.json`, `pyproject.toml`, and other key dependencies
+will be significantly slower than normal, because they won't get to
+benefit from the cache.

docs/testing/index.md

@@ -0,0 +1,18 @@
+# Code testing
+
+```{toctree}
+---
+maxdepth: 3
+---
+
+testing
+linters
+testing-with-django
+testing-with-node
+testing-with-puppeteer
+mypy
+typescript
+continuous-integration
+manual-testing
+philosophy
+```

docs/testing/linters.md

@@ -0,0 +1,262 @@
+# Linters
+
+A linter is a program that runs on the source code of software and
+reports potentially problematic code to the user. Linters help enforce
+coding guidelines, from minor details like whitespace formatting or
+capitalization patterns for strings to preventing problematic coding
+patterns that can lead to security bugs.
+
+## Overview
+
+Zulip does extensive linting of much of its source code, including
+Python/JavaScript/TypeScript files, HTML templates (Django/Handlebars), CSS files,
+JSON fixtures, Markdown documents, puppet manifests, and shell scripts.
+
+For some files we simply check for small things like trailing whitespace,
+but for other files, we are quite thorough about checking semantic
+correctness.
+
+Obviously, a large reason for linting code is to enforce the [Zulip
+coding standards](../contributing/code-style.md). But we also use the linters to
+prevent common coding errors.
+
+We borrow some open source tools for much of our linting, and the links
+below will direct you to the official documentation for these projects.
+
+- [ESLint](https://eslint.org)
+- [mypy](http://mypy-lang.org/)
+- [Prettier](https://prettier.io/)
+- [Puppet](https://puppet.com/) (puppet provides its own mechanism for
+  validating manifests)
+- [Ruff](https://docs.astral.sh/ruff/)
+- [stylelint](https://github.com/stylelint/stylelint)
+
+Zulip also uses some home-grown code to perform tasks like validating
+indentation in template files, enforcing coding standards that are unique
+to Zulip, allowing certain errors from third party linters to pass through,
+and exempting legacy files from lint checks.
+
+## Running the linters
+
+If you run `./tools/test-all`, it will automatically run the linters.
+You can also run them individually or pass specific files:
+
+```bash
+./tools/lint
+./tools/lint web/src/compose.ts
+./tools/lint web/src/
+```
+
+`./tools/lint` has many useful options; you can read about them in its
+internal documentation using `./tools/lint --help`. Of particular
+note are:
+
+- `--fix`: Several of our linters support automatically fixing basic
+  issues; this option will ask `tools/lint` to run those.
+- `--verbose`: Provides detailed information on how to fix many common
+  linter errors not covered by `--fix`.
+- `--skip` and `--only`: Only run certain linters.
+- `-m`: Only check modified files.
+
+Finally, you can rely on our continuous integration setup to run linters for you,
+but it is good practice to run lint checks locally.
+
+:::{important}
+We provide a
+[Git pre-commit hook](../git/zulip-tools.md#set-up-git-repo-script)
+that can automatically run `tools/lint` on just the files that
+changed (in a few 100ms) whenever you make a commit. This can save
+you a lot of time, by automatically detecting linter errors as you
+make them.
+:::
+
+**Note:** The linters only check files that Git tracks. Remember to `git add`
+new files before running lint checks.
+
+Our linting tools generally support the ability to lint files
+individually--with some caveats--and those options will be described
+later in this document.
+
+## General considerations
+
+Once you have read the [Zulip coding guidelines](../contributing/code-style.md), you can
+be pretty confident that 99% of the code that you write will pass through
+the linters fine, as long as you are thorough about keeping your code clean.
+And, of course, for minor oversights, `lint` is your friend, not your foe.
+
+Occasionally, our linters will complain about things that are more of
+an artifact of the linter limitations than any actual problem with your
+code. There is usually a mechanism where you can bypass the linter in
+extreme cases, but often it can be a simple matter of writing your code
+in a slightly different style to appease the linter. If you have
+problems getting something to lint, you can submit an unfinished PR
+and ask the reviewer to help you work through the lint problem, or you
+can find other people in the [Zulip Community](https://zulip.com/development-community/)
+to help you.
+
+Also, bear in mind that 100% of the lint code is open source, so if you
+find limitations in either the Zulip home-grown stuff or our third party
+tools, feedback will be highly appreciated.
+
+Finally, one way to clean up your code is to thoroughly exercise it
+with tests. The [Zulip test documentation](testing.md)
+describes our test system in detail.
+
+## Lint checks
+
+Most of our lint checks get performed by `./tools/lint`. These include the
+following checks:
+
+- Check Python code with Ruff.
+- Check Python formatting with Ruff.
+- Check JavaScript and TypeScript code with ESLint.
+- Check CSS, JavaScript, TypeScript, and YAML formatting with Prettier.
+- Check Python code for custom Zulip rules.
+- Check non-Python code for custom Zulip rules.
+- Check Puppet manifests with the Puppet validator.
+- Check HTML templates for matching tags and indentations.
+- Check CSS for parsability and formatting.
+- Check JavaScript code for addClass calls.
+- Running `mypy` to check static types in Python code. Our
+  [documentation on using mypy](mypy.md) covers mypy in
+  more detail.
+- Running `tsc` to compile TypeScript code. Our [documentation on
+  TypeScript](typescript.md) covers TypeScript in more detail.
+
+The rest of this document pertains to the checks that occur in `./tools/lint`.
+
+## lint
+
+Zulip has a script called `lint` that lives in our "tools" directory.
+It is the workhorse of our linting system, although in some cases it
+dispatches the heavy lifting to other components such as Ruff,
+eslint, and other home grown tools.
+
+You can find the source code [here](https://github.com/zulip/zulip/blob/main/tools/lint).
+
+In order for our entire lint suite to run in a timely fashion, the `lint`
+script performs several lint checks in parallel by forking out subprocesses.
+
+Note that our project does custom regex-based checks on the code. The code for these
+types of checks mostly lives [here](https://github.com/zulip/zulip/tree/main/tools/linter_lib).
+
+### Special options
+
+You can use the `-h` option for `lint` to see its usage. One particular
+flag to take note of is the `--modified` flag, which enables you to only run
+lint checks against files that are modified in your Git repo. Most of the
+"sub-linters" respect this flag, but some will continue to process all the files.
+Generally, a good workflow is to run with `--modified` when you are iterating on
+the code, and then run without that option right before committing new code.
+
+If you need to troubleshoot the linters, there is a `--verbose-timing`
+option that can give you clues about which linters may be running
+slow, for example.
+
+### Lint checks
+
+The next part of this document describes the lint checks that we apply to
+various file types.
+
+#### Generic source code checks
+
+We check almost our entire codebase for trailing whitespace. Also, we
+disallow tab (\t) characters in all but two files.
+
+We also have custom regex-based checks that apply to specific file types.
+For relatively minor files like Markdown files and JSON fixtures, this
+is the extent of our checking.
+
+Finally, we're checking line length in Python code (and hope to extend
+this to other parts of the codebase soon). You can use
+`#ignorelinelength` for special cases where a very long line makes
+sense (e.g., a link in a comment to an extremely long URL).
+
+#### Python code
+
+Our Python code is formatted using
+[Ruff](https://docs.astral.sh/ruff/) (using the options in the
+`[tool.ruff]` section of `pyproject.toml`). The `lint` script enforces
+this by running Ruff in check mode, or in write mode with `--fix`.
+
+The bulk of our Python linting also gets outsourced to Ruff.
+
+Zulip also has custom regex-based rules that it applies to Python code.
+Look for `python_rules` in the source code for `lint`. Note that we
+provide a mechanism to exclude certain lines of codes from these checks.
+Often, it is simply the case that our regex approach is too crude to
+correctly exonerate certain valid constructs. In other cases, the code
+that we exempt may be deemed not worthwhile to fix.
+
+#### JavaScript code
+
+We check our JavaScript code in a few different ways:
+
+- We run eslint.
+- We check code formatting with Prettier.
+- We perform custom Zulip regex checks on the code.
+
+#### Puppet manifests
+
+We use Puppet as our tool to manage configuration files, using
+Puppet "manifests." To lint Puppet manifests, we use the "parser validate"
+option of Puppet.
+
+#### HTML templates
+
+Zulip uses two HTML templating systems:
+
+- [Django templates](https://docs.djangoproject.com/en/5.0/topics/templates/)
+- [handlebars](https://handlebarsjs.com/)
+
+Zulip has an internal tool that validates both types of templates for
+correct indentation and matching tags. You can find the code here:
+
+- driver: [check-templates](https://github.com/zulip/zulip/blob/main/tools/check-templates)
+- engine: [lib/template_parser.py](https://github.com/zulip/zulip/blob/main/tools/lib/template_parser.py)
+
+We exempt some legacy files from indentation checks, but we are hoping to
+clean those files up eventually.
+
+#### CSS
+
+Zulip uses [stylelint](https://github.com/stylelint/stylelint) to lint
+its CSS; see our
+[configuration](https://github.com/zulip/zulip/blob/main/stylelint.config.js)
+for the rules we currently enforce.
+
+#### Shell scripts
+
+Zulip uses [shellcheck](https://github.com/koalaman/shellcheck) to
+lint our shell scripts. We recommend the
+[shellcheck gallery of bad code][shellcheck-bad-code] as a resource on
+how to not write bad shell.
+
+[shellcheck-bad-code]: https://github.com/koalaman/shellcheck/blob/master/README.md#user-content-gallery-of-bad-code
+
+#### Markdown, shell scripts, JSON fixtures
+
+We mostly validate miscellaneous source files like `.json`, and `.md`
+files for whitespace issues.
+
+## Philosophy
+
+If you want to help improve Zulip's system for linting, here are some
+considerations.
+
+#### Speed
+
+We want our linters to be fast enough that most developers
+will feel comfortable running them in a pre-commit hook, so we run
+our linters in parallel and support incremental checks.
+
+#### Accuracy
+
+We try to catch as many common mistakes as possible, either via a
+linter or an automated test.
+
+#### Completeness
+
+Our goal is to have most common style issues by caught by the linters, so new
+contributors to the codebase can efficiently fix produce code with correct
+style without needing to go back-and-forth with a reviewer.

docs/testing/manual-testing.md

@@ -0,0 +1,563 @@
+# Manual testing
+
+As a general rule, we like to have automated tests for everything that
+can be practically tested. However, there are certain types of bugs
+that are best caught with old fashioned manual testing (also called
+manual QA). Manual testing not only catches bugs, but it also helps
+developers learn more about the system and think about the existing
+semantics of a feature they're working on.
+
+This doc assumes you know how to set up a local development server
+and open the Zulip app in the browser. It also assumes a basic
+knowledge of how to use Zulip.
+
+## Basic stuff
+
+When testing Zulip manually, here are things to focus on:
+
+- The best bugs to catch are security/permissions bugs.
+- Don't rush manual testing. Look for small details like
+  display glitches.
+- Always test with multiple users (you can use incognito windows
+  to facilitate this).
+- Always keep the inspector console open and watch for warnings
+  or errors.
+- Be methodical about collecting information on bugs. (You will
+  eventually want to create tickets, but you may want to consolidate
+  your own notes before filing tickets.)
+
+You generally want to test with Cordelia as the primary user,
+and use Hamlet as her primary conversation partner. Use Iago
+when you need to test administrative functions. Send messages
+to Othello or Prospero if you want to verify things such as
+Cordelia not being able to receive messages not intended for her.
+
+The rest of this document groups tasks into basic areas of
+functionality of the system. If you have multiple people testing
+at once, you can divvy up QA tasks by these sections in the doc.
+
+### Message view
+
+We mostly test the message view as part of testing everything
+else, but there are few things to specially test here.
+
+Try using all the navigation hotkeys:
+
+- Up/k
+- Down/j
+- PgUp/K
+- PgDn/J/Spacebar
+- End (or fn-right-arrow on OSX)
+- also try scrolling aggressively with the mouse
+
+Try narrowing from the message view:
+
+- Hotkeys
+  - use a to go to Combined feed
+  - use s to narrow to a channel (select message first
+    and verify in sidebar)
+  - use S to narrow to the topic (and verify in sidebar)
+  - use v to navigate to direct messages
+- Click on the recipient bar
+  - narrow to a channel
+  - narrow to a topic
+  - narrow to direct messages with one user
+  - narrow to a group direct message
+- Click on the Zulip logo
+  - narrow to a topic
+  - click on the Zulip logo (and verify you're in the Recent conversations view)
+
+### Messagebox
+
+With messagebox we want to test mainly the functioning of all
+all the keyboard shortcuts and click handlers.
+
+Apart from that there are three views of a message box, we want
+to test their appearance too:
+
+- Message that includes sender
+- Message without the sender
+- "/me" message
+
+Here's how we're going to test the message appearances:
+
+- narrow to a new topic and send a message (this message will include sender)
+  - edit the message ("EDITED" label should appear beside sender name)
+- send another message (will not include sender)
+  - edit the message ("EDITED" label should appear in the left column, where the avatar is)
+- send a "/me" message (`/me test message`)
+  - message should appear alongside sender name
+  - edit the message ("EDITED" label should appear beside the message)
+
+For all the three cases, we need to test the click handlers and
+the hotkeys too:
+
+- Sender popover:
+  - click on the avatar and sender name for messages which include sender
+  - press 'u' to open the sender popover for all messages
+- Message reply:
+  - click on message to reply
+  - use 'r' or Return hotkey to reply
+  - use '>' to quote message
+  - use '@' to mention and reply
+- Reactions:
+  - click on the reactions button to open menu
+  - use ':' to open the reactions menu
+  - react to a message
+- Chevron
+  - click on chevron to open menu
+  - use 'i' to open chevron menu
+- Message edit
+  - click on message edit/view source button
+  - use 'i' + Return to edit/view source message
+  - click on the 'copy and close' option in view source and verify positioning of 'Copied!' label.
+- Star a message:
+  - click on the star button in the right column
+  - use 'Ctrl + S' to star a message
+- Message length
+  - send a long message and see if 'Show more' button appears
+  - click on the 'more' or 'collapse' link
+  - use i to collapse/expand a message irrespective of message length
+- use 'v' to show all images in the thread
+- use 'M' to mute the thread
+
+Play with the screen size to check if the messagebox appears
+fine in different screens too.
+
+### Message editing
+
+With message editing we mainly want to exercise topic changes.
+
+Here are some tasks:
+
+- Do lots of editing
+
+  - send a message to the topic "original"
+  - edit the message content
+  - send two messages to the "original" channel
+  - start to edit a message but then cancel
+  - change the topic for the first message to "change1" (just
+    this message)
+  - narrow back to "original"
+  - send one more message to the channel
+  - change the topic for the last two messages to "change2"
+  - narrow back to "original"
+  - send two more messages to the channel
+  - edit the 2nd message on topic and change all messages to
+    "change3"
+
+- Test UI entry points
+  - hit "i" then down arrow to edit with the popup
+  - use the popup using the mouse
+  - enter edit mode using the pencil icon
+
+### Narrowing
+
+Zulip uses the term "narrowing" to refer to opening different views
+of your messages, whether by clicking on sidebar options, recipient
+bars, or by using search. The main focus of these tasks should
+be watching unread counts. Of course, you also want to see messages
+show up in the message pane. And, finally, you should make sure
+that no messages outside the narrow show up in Cordelia's view.
+
+:::{important}
+Make sure that Cordelia is subscribed to Verona but not
+subscribed to Denmark; if not, you should use different channels
+for your testing.
+:::
+
+When testing narrows, you want to have Hamlet send the same message
+several times in a row, while cycling Cordelia through various narrows.
+
+Here are the main tasks for Hamlet (and each message gets sent several
+times):
+
+- Send Cordelia/Othello a direct message.
+- Send Cordelia a direct message.
+- Send Othello a direct message.
+- Post to Verona/foo.
+- Post to Verona/bar.
+- Post to Denmark/foo.
+- Post to Denmark/foo and mention Cordelia.
+
+For each of the above types of messages, you will want to cycle
+through the following views for Cordelia (and have Hamlet send new
+messages after each narrow):
+
+- Go to Combined feed view.
+- Go to Direct message feed view.
+- Go to Direct messages w/Hamlet.
+- Go to Direct messages w/Hamlet and Othello.
+- Go to Verona view.
+- Go to Verona/bar view.
+- Go to Verona/foo view.
+- Go to Denmark view.
+- Go to Denmark/foo view.
+
+There are 56 things to test here. If you can get into a rhythm
+where you can test each case in about 30 seconds, then the whole
+exercise is about 30 minutes, assuming no bugs.
+
+### Composing messages
+
+We have pretty good automated tests for our Markdown processor, so
+manual testing is targeted more to other interactions. For composing
+a message, pay attention to details like what is automatically
+populated and where the focus is placed.
+
+- Hotkeys
+
+  - use r to reply to a channel message
+  - use r to reply to a direct message
+  - use R to reply to the author of a direct message
+  - use R to reply to the author of a direct message channel
+  - use c to compose a channel message
+  - use x to compose a new direct message
+
+- Buttons
+
+  - Narrow to a channel and click on "New topic"
+  - Narrow to "Direct message feed" and click on "New topic"
+  - Narrow to a channel and click on "New direct message"
+  - Narrow to "Direct message feed" and click on "New direct message"
+
+- Topics
+
+  - Compose/send a message to a channel with no topic.
+  - Compose/send a message to a channel with a new topic.
+  - Compose/send a message to a channel with autocomplete.
+  - Compose/send a message to a channel manually typing an
+    existing topic.
+
+- Formatting stuff
+
+  - Use the "A" icon to get Markdown help.
+  - Use the eyeball icon to show a preview and send from preview mode.
+  - Toggle in and out of preview before sending a message.
+  - Use @-mention to mention Hamlet (and send him a message).
+  - Use `#**devel**` syntax and send to Hamlet, then follow the link.
+  - Create a bulleted list.
+  - Use the emoji icon to find an emoji in the picker.
+
+- Attachments
+
+  - Send a message with an attachment using the paperclip icon.
+  - Send a message with multiple attachments.
+  - Copy an image from the clipboard.
+  - Use drag/drop from the desktop to upload an image.
+
+- Drafts
+
+  - Start composing a message then click outside the compose box.
+  - Use "restore drafts" to restore the draft.
+  - Start composing then use "Esc" to abort the message.
+  - Use "restore drafts" to restore the draft.
+  - Start composing a channel message and then abort using
+    the little "x" icon in the compose box.
+  - Click on "New direct message" and restore the draft. (You
+    should now be sending to a channel.)
+
+- Click to send
+  - Turn off Enter-to-send.
+    - Send a two-paragraph message using Tab and Enter.
+    - Send a two-paragraph message using Ctrl-Enter or Cmd-Enter.
+  - Turn on Enter-to-send.
+    - Hit Enter to send.
+
+### Popover menus
+
+For this task you just want to go through all of our popover menus
+and exercise them. The main nuance here is that you occasionally want
+to click somewhere on the UI outside of an existing popover to see if
+the popover menu is "too sticky." Also, occasionally actions will be
+somewhat jarring; for example, if you mute a message in the current view,
+then the message will disappear from the view.
+
+Here are the things to test:
+
+- Channel sidebar menus (click ellipsis when hovering over channel filters)
+
+  - Channel settings (just make sure it goes there)
+  - Narrow (and then have Hamlet send a message)
+  - Pin/unpin (do both)
+  - Compose (send a message to the channel)
+  - Mark as read (scroll back and then have Hamlet send you a message)
+  - Mute/unmute (do both)
+  - Unsubscribe (and then go to Channel settings in the gear menu to resubscribe)
+  - Choose custom color (play around with this)
+
+- Topic sidebar menus (click ellipsis when hovering over topics)
+
+  - Narrow (and then have Hamlet send a message)
+  - Mute/unmute (try both)
+  - Mark as read (scroll back and then have Hamlet send you a message)
+
+- Left-message-pane menus (click on person's name)
+
+  - Verify email
+  - Verify date message sent
+  - Send a direct message (make sure compose box is filled out ok)
+  - Narrow to direct messages with
+  - Narrow to direct messages sent by
+
+- Right-pane-pane menus (click on chevron when hovering)
+
+  - use "i" hotkey to open the menu
+  - Edit a message you sent (using the down-arrow key to navigate the popup)
+  - View Source for somebody else's message (make sure
+    it's not editable)
+  - Reply (send a message)
+  - Collapse/uncollapse (try both)
+  - Mute/unmute (try both, watch left sidebar)
+  - Link to this conversation
+
+- Buddy list menus (click ellipsis when hovering over users)
+  - Narrow to direct messages with
+  - Narrow to message sent by
+  - Compose a message to
+
+### Sidebar filtering
+
+This is a fairly quick task where we test the search filters on the left sidebar
+and the buddy list. If Cordelia is not subscribed to Denmark, subscribe her to
+that channel.
+
+- Channels filtering
+
+  - Use "w" hotkey to open the search.
+  - Filter on "d".
+  - Pin/unpin Denmark.
+  - Clear filter.
+  - Use "A" and "D" hotkeys to cycle through the channels.
+  - Filter again and then click somewhere else.
+
+- Buddy list filtering
+  - Use "q" hotkey to open the search.
+  - Filter for Hamlet, Prospero, Othello, etc.
+  - Log on Hamlet and log off Hamlet while filtering for Hamlet.
+  - Log on/log off Hamlet while filtering for Othello.
+  - Log on/log off Hamlet while not filtering at all.
+  - Filter again and then click somewhere else.
+
+### Channel permissions
+
+This is an important category to test, because we obviously do not
+want to have bugs where people can read messages on channels they
+should not have access to.
+
+The general flow here is for Hamlet to create the channels and verify
+that Cordelia has the correct visibility to them.
+
+First, we start off with "positive" tests.
+
+- Positive tests
+  - Have Hamlet create a public channel w/Cordelia subscribed and
+    have him post a message to the channel.
+  - Have Hamlet create a public channel without Cordelia and then...
+    - Have Hamlet post to the channel.
+    - Have Cordelia subscribe to the channel.
+    - Verify Cordelia can see the previous message.
+    - Have Cordelia post a message to the channel.
+  - Have Hamlet create a private channel with Cordelia
+    invited and test a two-way conversation between the two
+    users.
+
+For negative tests, we want to dig a little deeper to find back
+doors for Cordelia to access the channel. Here are some techniques
+to try:
+
+- Try to have her compose a message to the channel by
+  circumventing autocomplete.
+- Try to have her narrow to the channel using channel:foo
+  in search.
+- Go to channel settings and see if the channel shows up.
+
+For public channels, it's ok for Cordelia to know the channel exists,
+and she can subsequently subscribe. For private channels, she should
+not even know they exist (until she's invited, of course).
+
+- Negative tests
+  - Have Hamlet create a public channel without inviting Cordelia.
+    - Verify Cordelia can see the channel in her settings.
+    - Verify Cordelia can't compose a message to the channel.
+    - Verify that Cordelia sees nothing when Hamlet posts to
+      the channel.
+  - Have Hamlet create a public channel with Cordelia, but then
+    have Iago revoke her subscription using the admin page.
+    - Verify that the channel appears in Cordelia's left sidebar
+      and then goes away.
+    - Try to have Cordelia view the channel using a sneaky
+      search along the lines of `channel:foo`.
+  - Have Hamlet create a private channel without inviting Cordelia.
+    - Verify Cordelia can't compose a message to the channel.
+
+### Search
+
+The main task for testing search is to play around with search
+suggestions (autocomplete). Once you select an option, verify the
+message view is consistent with the search and that the left sidebar
+reflects the current narrow. If a search comes up legitimately
+empty, have Hamlet send a message that matches the search.
+
+Here are searches you should be able to do with autocomplete:
+
+- channel:design
+- channel:Verona topic:Verona1
+- channel:Verona keyword
+- sent by me
+- @-mentions
+- starred messages
+- messages sent by Hamlet
+- direct messages with Hamlet
+- direct messages with Hamlet matching keyword "foo"
+
+There are some things you can try that don't come up in autocomplete:
+
+- -channel:Verona (exclude Verona)
+- channel:Verona channel:devel (should return no results)
+
+Miscellaneous:
+
+- Use the "/" hotkey to start a search.
+- Use the "x" icon to clear a search.
+- Use the "Esc" hotkey to clear a search.
+
+### Channel settings
+
+Test various UI entry points into channel settings:
+
+- Use small gear menu in left sidebar, then filter to "devel".
+- Use popover menu in left sidebar next to "devel".
+- Use gear menu above buddy list and filter to "devel".
+- Use gear menu and click on "devel."
+- Use gear menu and then click on chevron menu next to "devel."
+  (I'm not sure why we still have the chevron at this writing.)
+
+Create new public channel "public1" and add Hamlet:
+
+- Type "public1" in the text box and then click "Create new channel."
+- Select "People must be invited" and then verify you can't
+  select "Announce new channel in #[announcement channel]".
+- Select "Anyone can join" again to make it be public.
+- Check the checkbox for Hamlet.
+- Hit the "Create" button.
+
+Test subscribe/unsubscribe:
+
+- Log in as Hamlet and go to his channel settings.
+- As Cordelia, unsubscribe from "public1" using the checkmark in the
+  channel settings page.
+- Verify that Hamlet sees that Cordelia has unsubscribed (and the
+  subscriber count should decrement).
+- As Cordelia, resubscribe to "public1."
+- Verify Hamlet sees that change.
+
+As Cordelia, exercise different options in **Create Channel**
+dialog by creating channels s1, s2, s3, etc.:
+
+- s1: anyone can join, announce it, and add Hamlet using filter feature
+- s2: people must be invited
+- s3: anyone can join, don't announce
+- s4: check all, then uncheck all, then invite only Hamlet
+- s5: invite everybody but Hamlet
+- s6:
+  - create the channel as public, but don't subscribe anybody initially
+  - then click on channel options to add Hamlet using "Add" button
+
+Test per-channel options:
+
+- Use "devel" channel and send a message to it
+- Do mute and unmute, have Hamlet send messages
+- Test notifications on/off, have Hamlet send messages
+- Test pin and unpin, view left sidebar
+- Change channel color, and then view the left sidebar and the All
+  messages view
+- Verify channel subscriber counts in the main channel view
+
+### User settings
+
+You can modify per-user settings by choosing "Settings" in the gear menu.
+Do these tasks as Cordelia.
+
+- Your account
+  - Change full name (Hamlet should see the name change)
+  - Customize profile picture
+  - Deactivate account (and then log in as Iago to re-activate Cordelia)
+- Preferences
+  - Right now, these unfortunately require reloads to take effect.
+  - Default language (change to Spanish)
+  - 24-hour time (and then test going back to AM/PM)
+- Notifications
+  - Channel message
+    - turn off notifications at user level
+      - create a new channel
+      - have Hamlet send a message
+    - turn on notifications at user level
+      - create a new channel
+      - have Hamlet send a message
+      - then turn off notifications for that channel
+      - have Hamlet send another message
+  - Direct messages and @-mentions
+    - Test Desktop/Audible options
+    - You can ignore other stuff for now
+- Bots/API key
+  - Create a bot with a generic avatar and send it a direct message
+  - Create a bot with a custom avatar and send it a direct message
+  - Change your API key
+- Alert words
+  - Create an alert word
+  - Have Hamlet send you a message that includes the alert word
+- Zulip labs
+  - Turn on auto-scroll to new messages (and have Hamlet send you one)
+  - Turn on/off "Enable desktop notifications for new channels" and test.
+    (We may eliminate this option soon.)
+
+### Keyboard shortcuts
+
+We mostly test keyboard shortcuts as part of other tasks.
+
+Here are the tasks for this section:
+
+- Use the "?" hotkey to open the keyboard help
+- Proofread the dialog for typos.
+- Close the dialog.
+- Re-open the keyboard help using the gear menu.
+- Find a hotkey that you don't frequently use and experiment with its
+  usage.
+
+### Miscellaneous menu options
+
+Make sure that these options launch appropriate help screens:
+
+- Proofread and try a couple random options:
+  - Message formatting
+  - Search filters
+- Make sure help launches in a separate browser tab:
+  - Desktop and mobile apps
+  - Integrations
+  - API documentation
+
+### Inviting users/tutorial
+
+Here are the tasks:
+
+- Invite ignore@zulip.com using the link beneath the buddy list but
+  then don't take further action.
+- Fully invite foo@zulip.com using the gear menu.
+- Go to the development console to get the login link for foo@zulip.com.
+- Go through the signup flow.
+- Follow the tutorial.
+- Use the gear menu to log out.
+- Log back in as Cordelia (admittedly, this step doesn't really QA
+  much of our production code, since the login flow is customized for
+  the development environment).
+
+### To be continued...
+
+This document does not cover settings/admin options yet. The main
+things to do when testing the settings system are:
+
+- Verify that changes are synced to other users.
+- Verify error messages appear if you do something wrong and look right.
+- For organization settings, verify that they look right in read-only
+  mode (i.e. when not logged into an administrator account).

docs/testing/mypy.md

@@ -0,0 +1,561 @@
+# Python static type checker (mypy)
+
+[mypy](http://mypy-lang.org/) is a compile-time static type checker
+for Python, allowing optional, gradual typing of Python code. Zulip
+was fully annotated with mypy's Python 2 syntax in 2016, before our
+migration to Python 3 in late 2017. In 2018 and 2020, we migrated
+essentially the entire codebase to the nice PEP 484 (Python 3 only)
+and PEP 526 (Python 3.6) syntax for static types:
+
+```python
+user_dict: Dict[str, UserProfile] = {}
+
+def get_user(email: str, realm: Realm) -> UserProfile:
+    ... # Actual code of the function here
+```
+
+You can learn more about it at:
+
+- The
+  [mypy cheat sheet for Python 3](https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html)
+  is the best resource for quickly understanding how to write the PEP
+  484 type annotations used by mypy correctly.
+
+- The
+  [Python type annotation spec in PEP 484](https://www.python.org/dev/peps/pep-0484/).
+
+- Our [blog post on being an early adopter of mypy][mypy-blog-post] from 2016.
+
+- Our [best practices](#best-practices) section below.
+
+The mypy type checker is run automatically as part of Zulip's Travis
+CI testing process in the `backend` build.
+
+[mypy-blog-post]: https://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/
+
+## Installing mypy
+
+mypy is installed by default in the Zulip development environment.
+
+## Running mypy on Zulip's code locally
+
+To run mypy on Zulip's python code, you can run the command:
+
+```bash
+tools/run-mypy
+```
+
+Mypy outputs errors in the same style as a compiler would. For
+example, if your code has a type error like this:
+
+```python
+foo = 1
+foo = '1'
+```
+
+you'll get an error like this:
+
+```console
+test.py: note: In function "test":
+test.py:200: error: Incompatible types in assignment (expression has type "str", variable has type "int")
+```
+
+## Mypy is there to find bugs in Zulip before they impact users
+
+For the purposes of Zulip development, you can treat `mypy` like a
+much more powerful linter that can catch a wide range of bugs. If,
+after running `tools/run-mypy` on your Zulip branch, you get mypy
+errors, it's important to get to the bottom of the issue, not just do
+something quick to silence the warnings, before we merge the changes.
+Possible explanations include:
+
+- A bug in any new type annotations you added.
+- A bug in the existing type annotations.
+- A bug in Zulip!
+- Some Zulip code is correct but confusingly reuses variables with
+  different types.
+- A bug in mypy (though this is increasingly rare as mypy is now
+  fairly mature as a project).
+
+Each explanation has its own solution, but in every case the result
+should be solving the mypy warning in a way that makes the Zulip
+codebase better. If you're having trouble, silence the warning with
+an `Any` or `# type: ignore[code]` so you're not blocked waiting for help,
+add a `# TODO: ` comment so it doesn't get forgotten in code review,
+and ask for help in chat.zulip.org.
+
+## Mypy stubs for third-party modules
+
+For the Python standard library and some popular third-party modules,
+the [typeshed project](https://github.com/python/typeshed) has
+[stubs](https://github.com/python/mypy/wiki/Creating-Stubs-For-Python-Modules),
+basically the equivalent of C header files defining the types used in
+these Python APIs.
+
+For other third-party modules that we call from Zulip, one either
+needs to add an `ignore_missing_imports` entry in `pyproject.toml` in the
+root of the project, letting `mypy` know that it's third-party code,
+or add type stubs to the `stubs/` directory, which has type stubs that
+mypy can use to type-check calls into that third-party module.
+
+It's easy to add new stubs! Just read the docs, look at some of
+existing examples to see how they work, and remember to remove the
+`ignore_missing_imports` entry in `pyproject.toml` when you add them.
+
+For any third-party modules that don't have stubs, `mypy` treats
+everything in the third-party module as an `Any`, which is the right
+model (one certainly wouldn't want to need stubs for everything just
+to use `mypy`!), but means the code can't be fully type-checked.
+
+## Working with types from django-stubs
+
+For features that are difficult to be expressed with static type
+annotations, type analysis is supplemented with mypy plugins. Zulip's
+Python codebases uses the Django web framework, and such a plugin is
+required in order for `mypy` to correctly infer the types of most code
+interacting with Django model classes (i.e. code that accesses the
+database).
+
+We use the `mypy_django_plugin` plugin from the
+[django-stubs](https://github.com/typeddjango/django-stubs) project,
+which supports accurate type inference for classes like
+`QuerySet`. For example, `Stream.objects.filter(realm=realm)` is
+simple Django code to fetch all the channels in a realm. With this
+plugin, mypy will correctly determine its type is `QuerySet[Stream]`,
+aka a standard, lazily evaluated Django query object that can be
+iterated through to access `Stream` objects, without the developer
+needing to do an explicit annotation.
+
+When declaring the types for functions that accept a `QuerySet`
+object, you should always supply the model type that it accepts as the
+first type parameter.
+
+```python
+def foo(user: QuerySet[UserProfile]) -> None:
+    ...
+```
+
+For `.values_list`, supply the type of the column as the second type
+parameter.
+
+```python
+def get_book_page_counts() -> QuerySet[Book, int]:
+    return Book.objects.filter().values_list("page_count", flat=True)
+```
+
+For `.values`, we prefer to define a `TypedDict` containing the
+key-value pairs for the columns.
+
+```python
+class BookMetadata(TypedDict):
+    id: int
+    name: str
+
+def get_book_meta_data(
+    book_ids: List[int],
+) -> QuerySet[Book, BookMetadata]:
+    return Book.objects.filter(id__in=book_ids).values("name", "id")
+```
+
+When writing a helper function that returns the response from a test
+client, it should be typed as `TestHttpResponse` instead of
+`HttpResponse`. This type is only defined in the Django stubs, so it
+has to be conditionally imported only when type
+checking. Conventionally, we alias it as `TestHttpResponse`, which is
+internally named `_MonkeyPatchedWSGIResponse` within django-stubs.
+
+```python
+from typing import TYPE_CHECKING
+from zerver.lib.test_classes import ZulipTestCase
+
+if TYPE_CHECKING:
+    from django.test.client import _MonkeyPatchedWSGIResponse as TestHttpResponse
+
+class FooTestCase(ZulipTestCase):
+    def helper(self) -> "TestHttpResponse":
+        return self.client_get("/bar")
+```
+
+We sometimes encounter inaccurate type annotations in the Django
+stubs project. We prefer to address these by [submitting a pull
+request](https://github.com/typeddjango/django-stubs/pulls) to fix the
+issue in the upstream project, just like we do with `typeshed` bugs.
+
+## Using @overload to accurately describe variations
+
+Sometimes, a function's type is most precisely expressed as a few
+possibilities, and which possibility can be determined by looking at
+the arguments. You can express that idea in a way mypy understands
+using `@overload`. For example, `check_list` returns a `Validator`
+function that verifies that an object is a list, raising an exception
+if it isn't.
+
+It supports being passed a `sub_validator`, which will verify that
+each element in the list has a given type as well. One can express
+the idea "If `sub_validator` validates that something is a `ResultT`,
+`check_list(sub_validator)` validates that something is a
+`List[ResultT]` as follows:
+
+```python
+@overload
+def check_list(sub_validator: None, length: Optional[int]=None) -> Validator[List[object]]:
+    ...
+@overload
+def check_list(sub_validator: Validator[ResultT],
+               length: Optional[int]=None) -> Validator[List[ResultT]]:
+    ...
+def check_list(sub_validator: Optional[Validator[ResultT]]=None,
+               length: Optional[int]=None) -> Validator[List[ResultT]]:
+```
+
+The first overload expresses the types for the case where no
+`sub_validator` is passed, in which case all we know is that it
+returns a `Validator[List[object]]`; whereas the second defines the
+type logic for the case where we are passed a `sub_validator`.
+
+**Warning:** Mypy only checks the body of an overloaded function
+against the final signature and not against the more restrictive
+`@overload` signatures. This allows some type errors to evade
+detection by mypy:
+
+```python
+@overload
+def f(x: int) -> int: ...
+@overload
+def f(x: str) -> int: ...  # oops
+def f(x: Union[int, str]) -> Union[int, str]:
+    return x
+
+x: int = f("three!!")
+```
+
+Due to this potential for unsafety, we discourage overloading unless
+it's absolutely necessary. Consider writing multiple functions with
+different names instead.
+
+See the [mypy overloading documentation][mypy-overloads] for more details.
+
+[mypy-overloads]: https://mypy.readthedocs.io/en/stable/more_types.html#function-overloading
+
+## Best practices
+
+### When is a type annotation justified?
+
+Usually in fully typed code, mypy will protect you from writing a type
+annotation that isn't justified by the surrounding code. But when you
+need to write annotations at the border between untyped and typed
+code, keep in mind that **a type annotation should always represent a
+guarantee,** not an aspiration. If you have validated that some value
+is an `int`, it can go in an `int` annotated variable. If you are
+going to validate it later, it should not. When in doubt, an `object`
+annotation is always safe.
+
+Mypy understands many Python constructs like `assert`, `if`,
+`isinstance`, and logical operators, and uses them to automatically
+narrow the type of validated objects in many cases.
+
+```python
+def f(x: object, y: Optional[str]) -> None:
+    if isinstance(x, int):
+        # Within this if block, mypy infers that x: int
+        print(x + 1)
+    assert y is not None
+    # After that assert statement, mypy infers that y: str
+    print(y.strip())
+```
+
+It won't be able do this narrowing if the validation is hidden behind
+a function call, so sometimes it's helpful for a validation function
+to return the type-narrowed value back to the caller even though the
+caller already has it. (The validators in `zerver/lib/validator.py`
+are examples of this pattern.)
+
+### Avoid the `Any` type
+
+Mypy provides the [`Any`
+type](https://mypy.readthedocs.io/en/stable/dynamic_typing.html) for
+interoperability with untyped code, but it is completely unchecked.
+You can put an value of an arbitrary type into an expression of type
+`Any`, and get an value of an arbitrary type out, and mypy will make
+no effort to check that the input and output types match. So using
+`Any` defeats the type safety that mypy would otherwise provide.
+
+```python
+x: Any = 5
+y: str = x  # oops
+print(y.strip())  # runtime error
+```
+
+If you think you need to use `Any`, consider the following safer
+alternatives first:
+
+- To annotate a dictionary where different keys correspond to values
+  of different types, instead of writing `Dict[str, Any]`, try
+  declaring a
+  [**`dataclass`**](https://mypy.readthedocs.io/en/stable/additional_features.html#dataclasses)
+  or a
+  [**`TypedDict`**](https://mypy.readthedocs.io/en/stable/more_types.html#typeddict).
+
+- If you're annotating a class or function that might be used with
+  different data types at different call sites, similar to the builtin
+  `List` type or the `sorted` function, [**generic
+  types**](https://mypy.readthedocs.io/en/stable/generics.html) with
+  `TypeVar` might be what you need.
+
+- If you need to accept data of several specific possible types at a
+  single site, you may want a [**`Union`
+  type**](https://mypy.readthedocs.io/en/stable/kinds_of_types.html#union-types).
+  `Union` is checked: before using `value: Union[str, int]` as a
+  `str`, mypy requires that you validate it with an
+  `instance(value, str)` test.
+
+- If you really have no information about the type of a value, use the
+  **`object` type**. Since every type is a subtype of `object`, you
+  can correctly annotate any value as `object`. The [difference
+  between `Any` and
+  `object`](https://mypy.readthedocs.io/en/stable/dynamic_typing.html#any-vs-object)
+  is that mypy will check that you safely validate an `object` with
+  `isinstance` before using it in a way that expects a more specific
+  type.
+
+- A common way for `Any` annotations to sneak into your code is the
+  interaction with untyped third-party libraries. Mypy treats any
+  value imported from an untyped library as annotated with `Any`, and
+  treats any type imported from an untyped library as equivalent to
+  `Any`. Consider providing real type annotations for the library by
+  [**writing a stub file**](#mypy-stubs-for-third-party-modules).
+
+### Avoid `cast()`
+
+The [`cast`
+function](https://mypy.readthedocs.io/en/stable/type_narrowing.html#casts) lets you
+provide an annotation that Mypy will not verify. Obviously, this is
+completely unsafe in general.
+
+```python
+x = cast(str, 5)  # oops
+print(x.strip())  # runtime error
+```
+
+Instead of using `cast`:
+
+- You can use a [variable
+  annotation](https://mypy.readthedocs.io/en/stable/type_inference_and_annotations.html#explicit-types-for-variables)
+  to be explicit or to disambiguate types that mypy can check but
+  cannot infer.
+
+  ```python
+  l: List[int] = []
+  ```
+
+- You can use an [`isinstance`
+  test](https://mypy.readthedocs.io/en/stable/common_issues.html#complex-type-tests)
+  to safely verify that a value really has the type you expect.
+
+### Avoid `# type: ignore` comments
+
+Mypy allows you to ignore any type checking error with a
+[`# type: ignore`
+comment](https://mypy.readthedocs.io/en/stable/common_issues.html#spurious-errors-and-locally-silencing-the-checker),
+but you should avoid this in the absence of a very good reason, such
+as a bug in mypy itself. If there are no safe options for dealing
+with the error, prefer an unchecked `cast`, since its unsafety is
+somewhat more localized.
+
+Our linter requires all `# type: ignore` comments to be [scoped to the
+specific error
+code](https://mypy.readthedocs.io/en/stable/error_codes.html) being
+ignored, and followed by an explanation such as a link to a GitHub
+issue.
+
+### Avoid other unchecked constructs
+
+- As mentioned
+  [above](#using-overload-to-accurately-describe-variations), we
+  **discourage writing overloaded functions** because their bodies are
+  not checked against the `@overload` signatures.
+
+- **Avoid `Callable[..., T]`** (with literal ellipsis `...`), since
+  mypy cannot check the types of arguments passed to it. Provide the
+  specific argument types (`Callable[[int, str], T]`) in simple cases,
+  or use [callback
+  protocols](https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols)
+  in more complex cases.
+
+### Use `Optional` and `None` correctly
+
+The [`Optional`
+type](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html#built-in-types)
+is for optional values, which are values that could be `None`. For
+example, `Optional[int]` is equivalent to `Union[int, None]`.
+
+The `Optional` type is **not for optional parameters** (unless they
+are also optional values as above). This signature does not use the
+`Optional` type:
+
+```python
+def func(flag: bool = False) -> str:
+    ...
+```
+
+A collection such as `List` should only be `Optional` if `None` would
+have a different meaning than the natural meaning of an empty
+collection. For example:
+
+- An include list where the default is to include everything should be
+  `Optional` with default `None`.
+- An exclude list where the default is to exclude nothing should be
+  non-`Optional` with default `[]`.
+
+Don't test an `Optional` value using truthiness (`if value:`,
+`not value`, `value or default_value`), especially when the type might
+have falsy values other than `None`.
+
+```python
+s: Optional[str]
+if not s:  # bad: are we checking for None or ""?
+    ...
+if s is None:  # good
+    ...
+```
+
+### Read-only types
+
+The basic Python collections
+[`List`](https://docs.python.org/3/library/typing.html#typing.List),
+[`Dict`](https://docs.python.org/3/library/typing.html#typing.Dict),
+and [`Set`](https://docs.python.org/3/library/typing.html#typing.Set)
+are mutable, but it's confusing for a function to mutate a collection
+that was passed to it as an argument, especially by accident. To
+avoid this, prefer annotating function parameters with read-only
+types:
+
+- [`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)
+  instead of `List`,
+- [`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping)
+  instead of `Dict`,
+- [`AbstractSet`](https://docs.python.org/3/library/typing.html#typing.AbstractSet)
+  instead of `Set`.
+
+This is especially important for parameters with default arguments,
+since a mutable default argument is confusingly shared between all
+calls to the function.
+
+```python
+def f(items: Sequence[int] = []) -> int:
+    items.append(1)  # mypy catches this mistake
+    return sum(items)
+```
+
+In some cases the more general
+[`Collection`](https://docs.python.org/3/library/typing.html#typing.Collection)
+or
+[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)
+types might be appropriate. (But don’t use `Iterable` for a value
+that might be iterated multiple times, since a one-use iterator is
+`Iterable` too.)
+
+For example, if a function gets called with either a `list` or a `QuerySet`,
+and it only iterates the object once, the parameter can be typed as `Iterable`.
+
+```python
+def f(items: Iterable[Realm]) -> None:
+    for item in items:
+        ...
+
+realms_list: List[Realm] = [zulip, analytics]
+realms_queryset: QuerySet[Realm] = Realm.objects.all()
+
+f(realms_list)      # OK
+f(realms_queryset)  # Also OK
+```
+
+A function's return type can be mutable if the return value is always
+a freshly created collection, since the caller ends up with the only
+reference to the value and can freely mutate it without risk of
+confusion. But a read-only return type might be more appropriate for
+a function that returns a reference to an existing collection.
+
+Read-only types have the additional advantage of being [covariant
+rather than
+invariant](https://mypy.readthedocs.io/en/latest/common_issues.html#invariance-vs-covariance):
+if `B` is a subtype of `A`, then `List[B]` may not be converted to
+`List[A]`, but `Sequence[B]` may be converted to `Sequence[A]`.
+
+### Typing decorators
+
+A simple decorator that operates on functions of a fixed signature
+works with no issues:
+
+```python
+def fancy(func: Callable[[str], str]) -> Callable[[int], str]:
+    def wrapped_func(n: int) -> str:
+        print("so fancy")
+        return func(str(n))
+    return wrapped_func
+
+@fancy
+def f(s: str) -> str:
+    return s
+```
+
+A decorator with an argument also works:
+
+```python
+def fancy(message: str) -> Callable[[Callable[[str], str]], Callable[[int], str]]:
+    def wrapper(func: Callable[[str], str]) -> Callable[[int], str]:
+        def wrapped_func(n: int) -> str:
+            print(message)
+            return func(str(n))
+        return wrapped_func
+    return wrapper
+
+@fancy("so fancy")
+def f(s: str) -> str:
+    return s
+```
+
+And a [generic
+decorator](https://mypy.readthedocs.io/en/stable/generics.html#declaring-decorators)
+that operates on functions of arbitrary signatures can be written
+[with a `cast`](https://github.com/python/mypy/issues/1927) if the
+output signature is always the same as the input signature:
+
+```python
+FuncT = TypeVar("FuncT", bound=Callable[..., object])
+
+def fancy(func: FuncT) -> FuncT:
+    def wrapped_func(*args: object, **kwargs: object) -> object:
+        print("so fancy")
+        return func(*args, **kwargs)
+    return cast(FuncT, wrapped_func)  # https://github.com/python/mypy/issues/1927
+
+@fancy
+def f(s: str) -> str:
+    return s
+```
+
+(A generic decorator with an argument would return
+`Callable[[FuncT], FuncT]`.)
+
+But Mypy doesn't yet support the advanced type annotations that would
+be needed to correctly type generic signature-changing decorators,
+such as `zerver.decorator.authenticated_json_view`, which passes an
+extra argument to the inner function. For these decorators we must
+unfortunately give up some type safety by falling back to
+`Callable[..., T]`.
+
+## Troubleshooting advice
+
+All of our linters, including mypy, are designed to only check files
+that have been added in Git (this is by design, since it means you
+have untracked files in your Zulip checkout safely). So if you get a
+`mypy` error like this after adding a new file that is referenced by
+the existing codebase:
+
+```console
+mypy | zerver/models.py:1234: note: Import of 'zerver.lib.markdown_wrappers' ignored
+mypy | zerver/models.py:1234: note: (Using --follow-imports=error, module not passed on command line)
+```
+
+The problem is that you need to `git add` the new file.

docs/testing/philosophy.md

@@ -0,0 +1,251 @@
+# Testing philosophy
+
+Zulip's automated tests are a huge part of what makes the project able
+to make progress. This page records some of the key principles behind
+how we have designed our automated test suites.
+
+Greg's talk at FlutterCon 2025,
+["Writing better tests with less work"](https://www.youtube.com/watch?v=nf_XYpfsZNg),
+is another fantastic resource, in video format.
+
+## Effective testing allows us to move quickly
+
+Zulip's engineering strategy can be summarized as "move quickly
+without breaking things". Despite reviewing many code submissions
+from new contributors without deep expertise in the code they are
+changing, Zulip's maintainers spend most of the time they spend
+integrating changes on product decisions and code
+structure/readability questions, not on correctness, style, or
+lower-level issues.
+
+This is possible because we have spent years systematically investing
+in testing, tooling, code structure, documentation, and development
+practices to help ensure that our contributors write code that needs
+relatively few changes before it can be merged. The testing element
+of this is to have reliable, extensive, easily extended test suites
+that cover most classes of bugs. Our testing systems have been
+designed to minimize the time spent manually testing or otherwise
+investigating whether changes are correct.
+
+For example, our [infrastructure for testing
+authentication](../development/authentication.md) allows using a mock
+LDAP database in both automated tests and the development environment,
+making it much easier now to refactor and improve this important part of
+the product than it was when you needed to set up an LDAP server and
+populate it with some test data in order to test LDAP authentication.
+
+While not every part of Zulip has a great test suite, many components
+do, and for those components, the tests mean that new contributors can
+often make substantive changes and have them be
+more or less correct by the time they share the
+changes for code review. More importantly, it means that maintainers
+save most of the time that would otherwise be spent verifying that the
+changes are simply correct, and instead focus on making sure that the
+codebase remains readable, well-structured, and well-tested.
+
+## Test suite performance and reliability are critical
+
+When automated test suites are slow or unreliable, developers will
+avoid running them, and furthermore, avoid working on improving them
+(both the system and individual tests). Because changes that make
+tests slow or unreliable are often unintentional side effects of other
+development, problems in this area tend to accumulate as a codebase
+grows. As a result, barring focused effort to prevent this outcome,
+any large software project will eventually have its test suite rot
+into one that is slow, unreliable, untrustworthy, and hated. We aim
+for Zulip to avoid that fate.
+
+So we consider it essential to maintaining every automated test suite
+setup in a way where it is fast and reliable (tests pass both in CI
+and locally if there are no problems with the developer's changes).
+
+Concretely, our performance goals are for the full backend suite
+(`test-backend`) to complete in about a minute, and our full frontend
+suite (`test-js-with-node`) to run in under 10 seconds.
+
+It'd be a long blog post to summarize everything we do to help achieve
+these goals, but a few techniques are worth highlighting:
+
+- Our test suites are designed to not access the Internet, since the
+  Internet might be down or unreliable in the test environment. Where
+  outgoing HTTP requests are required to test something, we mock the
+  responses with libraries like `responses`.
+- We carefully avoid the potential for contamination of data inside
+  services like PostgreSQL, Redis, and memcached from different tests.
+  - Every test case prepends a unique random prefix to all keys it
+    uses when accessing Redis and memcached.
+  - Every test case runs inside a database transaction, which is
+    aborted after the test completes. Each test process interacts
+    only with a fresh copy of a special template database used for
+    server tests that is destroyed after the process completes.
+- We rigorously investigate non-deterministically failing tests as though
+  they were priority bugs in the product.
+
+## Integration testing or unit testing?
+
+Developers frequently ask whether they should write "integration
+tests" or "unit tests". Our view is that tests should be written
+against interfaces that you're already counting on keeping stable, or
+already promising people you'll keep stable. In other words,
+interfaces that you or other people are already counting on mostly not
+changing except in compatible ways.
+
+So writing tests for the Zulip server against Zulip's end-to-end API
+is a great example of that: the API is something that people have
+written lots of code against, which means all that code is counting on
+the API generally continuing to work for the ways they're using it.
+
+The same would be true even if the only users of the API were our own
+project's clients like the mobile apps -- because there are a bunch of
+already-installed copies of our mobile apps out there, and they're
+counting on the API not suddenly changing incompatibly.
+
+One big reason for this principle is that when you write tests against
+an interface, those tests become a cost you pay any time you change
+that interface -- you have to go update a bunch of tests.
+
+So in a big codebase if you have a lot of "unit tests" that are for
+tiny internal functions, then any time you refactor something and
+change the internal interfaces -- even though you just made them up,
+and they're completely internal to that codebase so there's nothing
+that will break if you change them at will -- you have to go deal with
+editing a bunch of tests to match the new interfaces. That's
+especially a lot of work if you try to take the tests seriously,
+because you have to think through whether the tests breaking are
+telling you something you should actually listen to.
+
+In some big codebases, this can lead to tests feeling a lot like
+busywork... and it's because a lot of those tests really are
+busywork. And that leads to developers not being committed to
+maintaining and expanding the test suite in a thoughtful way.
+
+But if your tests are written against an external API, and you make
+some refactoring change and a bunch of tests break... now that's
+telling you something very real! You can always edit the tests... but
+the tests are stand-ins for real users and real code out there beyond
+your reach, which will break the same way.
+
+So you can still make the change... but you have to deal with figuring
+out an appropriate migration or backwards-compatibility strategy for
+all those real users out there. Updating the tests is one of the easy
+parts. And those changes to the tests are a nice reminder to code
+reviewers that you've changed an interface, and the reviewer should
+think carefully about whether those interface changes will be a
+problem for any existing clients and whether they're properly reflected
+in any documentation for that interface.
+
+Some examples of this philosophy:
+
+- If you have a web service that's mainly an API, you want to write
+  your tests for that API.
+- If you have a CLI program, you want to write your tests against the
+  CLI.
+- If you have a compiler, an interpreter, etc., you want essentially
+  all your tests to be example programs, with a bit of metadata for
+  things like "should give an error at this line" or "should build and
+  run, and produce this output".
+
+In the Zulip context:
+
+- Zulip uses the same API for our web app as for our mobile clients and
+  third-party API clients, and most of our server tests are written
+  against the Zulip API.
+- The tests for Zulip's incoming webhooks work by sending actual
+  payloads captured from the real third-party service to the webhook
+  endpoints, and verifies that the webhook produces the expected Zulip
+  message as output, to test the actual interface.
+
+So, to summarize our approach to integration vs. unit testing:
+
+- While we aim to achieve test coverage of every significant code path
+  in the Zulip server, which is commonly associated with unit testing,
+  most of our tests are integration tests in the sense of sending a
+  complete HTTP API query to the Zulip server and checking that the
+  HTTP response and the internal state of the server following the request
+  are both correct.
+- Following the end-to-end principle in system design, where possible
+  we write tests that execute a complete flow (e.g., registering a new
+  Zulip account) rather than testing the implementations of individual
+  functions.
+- We invest in the performance of Zulip in part to give users a great
+  experience, but just as much to make our test suite fast enough
+  that we can write our tests this way.
+
+## Avoid duplicating code with security impact
+
+Developing secure software with few security bugs is extremely
+difficult. An important part of our strategy for avoiding security
+logic bugs is to design patterns for how all of our code that
+processes untrusted user input can be well tested without either
+writing (and reviewing!) endless tests or requiring every developer to
+be good at thinking about security corner cases.
+
+Our strategy for this is to write a small number of carefully-designed
+functions like `access_stream_by_id` that we test carefully, and then
+use linting and other coding conventions to require that all access to
+data from code paths that might share that data with users be mediated
+through those functions. So rather than having each view function do
+it own security checks for whether the user can access a given channel,
+and needing to test each of those copies of the logic, we only need to
+do that work once for each major type of data structure and level of
+access.
+
+These `access_*_by_*` functions are written in a special style, with each
+conditional on its own line (so our test coverage tooling helps verify
+that every case is tested), detailed comments, and carefully
+considered error-handling to avoid leaking information such as whether
+the channel ID requested exists or not.
+
+We will typically also write tests for a given view verifying that it
+provides the appropriate errors when improper access is attempted, but
+these tests are defense in depth; the main way we prevent invalid
+access to channels is not offering developers a way to get a `Stream`
+object in server code except as mediated through these security check
+functions.
+
+## Share test setup code
+
+It's very common to need to write tests for permission checking or
+error handling code. When doing this, it's best to share the test
+setup code between success and failure tests.
+
+For example, when testing a function that returns a boolean (as
+opposed to an exception with a specific error messages), it's often
+better to write a single test function, `test_foo`, that calls the
+function several times and verifies its output for each value of the
+test conditions.
+
+The benefit of this strategy is that you guarantee that the test setup
+only differs as intended: Done well, it helps avoid the otherwise
+extremely common failure mode where a `test_foo_failure` test passes
+for the wrong reason. (e.g., the action fails not because of the
+permission check, but because a required HTTP parameter was only added
+to an adjacent `test_foo_success`).
+
+## What isn't tested probably doesn't work
+
+Even the very best programmers make mistakes constantly. Further, it's
+impossible to do large codebase refactors (which are important to
+having a readable, happy, correct codebase) if doing so has a high
+risk of creating subtle bugs.
+
+As a result, it's important to test every change. For business logic,
+the best option is usually a high-quality automated test, that is
+designed to be robust to future refactoring.
+
+But for some things, like documentation and CSS, the only way to test
+is to view the element in a browser and try things that might not
+work. What to test will vary with what is likely to break. For
+example, after a significant change to Zulip's Markdown documentation,
+if you haven't verified every special bit of formatting visually and
+clicked every new link, there's a good chance that you've introduced a
+bug.
+
+Manual testing not only catches bugs, but it also helps developers
+learn more about the system and think about the existing semantics of
+a feature they're working on.
+
+When submitting a pull request that affects UI, it's extremely helpful
+to show a screencast of your feature working, because that allows a
+reviewer to save time that would otherwise be spent manually testing
+your changes.

docs/testing/testing-with-django.md

@@ -0,0 +1,499 @@
+# Backend Django tests
+
+## Overview
+
+Zulip uses the Django framework for its Python backend. We
+use the testing framework from
+[django.test](https://docs.djangoproject.com/en/5.0/topics/testing/)
+to test our code. We have thousands of automated tests that verify that
+our backend works as expected.
+
+All changes to the Zulip backend code should be supported by tests. We
+enforce our testing culture during code review, and we also use
+coverage tools to measure how well we test our code. We mostly use
+tests to prevent regressions in our code, but the tests can have
+ancillary benefits such as documenting interfaces and influencing
+the design of our software.
+
+If you have worked on other Django projects that use unit testing, you
+will probably find familiar patterns in Zulip's code. This document
+describes how to write tests for the Zulip backend, with a particular
+emphasis on areas where we have either wrapped Django's test framework
+or just done things that are kind of unique in Zulip.
+
+## Running tests
+
+Our tests live in `zerver/tests/`. You can run them with
+`./tools/test-backend`. The tests run in parallel using multiple
+threads in your development environment, and can finish in under 30s
+on a fast machine. When you are in iterative mode, you can run
+individual tests or individual modules, following the dotted.test.name
+convention below:
+
+```bash
+cd /srv/zulip
+./tools/test-backend zerver.tests.test_queue_worker.WorkerTest
+```
+
+There are many command line options for running Zulip tests, such
+as a `--verbose` option. The
+best way to learn the options is to use the online help:
+
+```bash
+./tools/test-backend --help
+```
+
+We also have ways to instrument our tests for finding code coverage,
+URL coverage, and slow tests. Use the `-h` option to discover these
+features. We also have a `--profile` option to facilitate profiling
+tests.
+
+By default, `test-backend` will run all requested tests, and report
+all failures at the end. You can configure it to stop after the first
+error with the `--stop` option (or `-x`).
+
+Another useful option is `--rerun`, which will rerun just the tests
+that failed in the last test run.
+
+**Webhook integrations**. For performance, `test-backend` with no
+arguments will not run webhook integration tests (`zerver/webhooks/`),
+which would otherwise account for about 25% of the total runtime.
+When working on webhooks, we recommend instead running
+`test-backend zerver/webhooks` manually (or better, the direction for
+the specific webhooks you're working on). And of course our CI is
+configured to always use `test-backend --include-webhooks` and run all
+of the tests.
+
+## Writing tests
+
+Before you write your first tests of Zulip, it is worthwhile to read
+the rest of this document.
+
+To get a hang of commonly used testing techniques, read
+[zerver/tests/test_example.py](https://github.com/zulip/zulip/blob/main/zerver/tests/test_example.py).
+You can also read some of the existing tests in `zerver/tests`
+to get a feel for other patterns we use.
+
+A good practice is to get a "failing test" before you start to implement
+your feature. First, it is a useful exercise to understand what needs to happen
+in your tests before you write the code, as it can help drive out simple
+design or help you make incremental progress on a large feature. Second,
+you want to avoid introducing tests that give false positives. Ensuring
+that a test fails before you implement the feature ensures that if somebody
+accidentally regresses the feature in the future, the test will catch
+the regression.
+
+Another important files to skim are
+[zerver/lib/test_helpers.py](https://github.com/zulip/zulip/blob/main/zerver/lib/test_helpers.py),
+which contains test helpers.
+[zerver/lib/test_classes.py](https://github.com/zulip/zulip/blob/main/zerver/lib/test_classes.py),
+which contains our `ZulipTestCase` and `WebhookTestCase` classes.
+
+### Setting up data for tests
+
+All tests start with the same fixture data. (The tests themselves
+update the database, but they do so inside a transaction that gets
+rolled back after each of the tests complete. For more details on how the
+fixture data gets set up, refer to `tools/setup/generate-fixtures`.)
+
+The fixture data includes a few users that are named after
+Shakesepeare characters, and they are part of the "zulip.com" realm.
+
+Generally, you will also do some explicit data setup of your own. Here
+are a couple useful methods in ZulipTestCase:
+
+- make_stream
+- subscribe
+- unsubscribe
+- send_stream_message
+- send_personal_message
+
+More typically, you will use methods directly from the backend code.
+(This ensures more end-to-end testing, and avoids false positives from
+tests that might not consider ancillary parts of data setup that could
+influence tests results.)
+
+Here are some example action methods that tests may use for data setup:
+
+- check_send_message
+- do_change_user_role
+- do_create_user
+- do_make_stream_private
+
+### Testing code that accesses the filesystem
+
+Some tests need to access the filesystem (e.g., `test_upload.py` tests
+for `LocalUploadBackend` and the data import tests). Doing
+this correctly requires care to avoid problems like:
+
+- Leaking files after every test (which are clutter and can eventually
+  run the development environment out of disk) or
+- Interacting with other parallel processes of this `test-backend` run
+  (or another `test-backend` run), or with later tests run by this
+  process.
+
+To avoid these problems, you can do the following:
+
+- Use a subdirectory of `settings.TEST_WORKER_DIR`; this is a
+  subdirectory of `/var/<uuid>/test-backend` that is unique to the
+  test worker thread and will be automatically deleted when the
+  relevant `test-backend` process finishes.
+- Delete any files created by the test in the test class's `tearDown`
+  method (which runs even if the test fails); this is valuable to
+  avoid conflicts with other tests run later by the same test process.
+
+Our common testing infrastructure handles some of this for you,
+e.g., it replaces `settings.LOCAL_UPLOADS_DIR` for each test process
+with a unique path under `/var/<uuid>/test-backend`. And
+`UploadSerializeMixin` manages some of the cleanup work for
+`test_upload.py`.
+
+### Testing with mocks
+
+This section is a beginner's guide to mocking with Python's
+`unittest.mock` library. It will give you answers to the most common
+questions around mocking, and a selection of commonly used mocking
+techniques.
+
+#### What is mocking?
+
+When writing tests, _mocks allow you to replace methods or objects with fake entities
+suiting your testing requirements_. Once an object is mocked, **its original code does not
+get executed anymore**.
+
+Rather, you can think of a mocked object as an initially empty shell:
+Calling it won't do anything, but you can fill your shell with custom code, return values, etc.
+Additionally, you can observe any calls made to your mocked object.
+
+#### Why is mocking useful?
+
+When writing tests, it often occurs that you make calls to functions
+taking complex arguments. Creating a real instance of such an argument
+would require the use of various different libraries, a lot of
+boilerplate code, etc. Another scenario is that the tested code
+accesses files or objects that don't exist at testing time. Finally,
+it is good practice to keep tests independent from others. Mocks help
+you to isolate test cases by simulating objects and methods irrelevant
+to a test's goal.
+
+In all of these cases, you can "mock out" the function calls / objects
+and replace them with fake instances that only implement a limited
+interface. On top of that, these fake instances can be easily
+analyzed.
+
+Say you have a module `greetings` defining the following functions:
+
+```python
+def fetch_database(key: str) -> str:
+    # ...
+    # Do some look-ups in a database
+    return data
+
+def greet(name_key: str) -> str:
+    name = fetch_database(name_key)
+    return "Hello" + name
+```
+
+- You want to test `greet()`.
+
+- In your test, you want to call `greet("Mario")` and verify that it returns the correct greeting:
+
+  ```python
+  from greetings import greet
+
+  def test_greet() -> str:
+      greeting = greet("Mario")
+      assert greeting == "Hello Mr. Mario Mario"
+  ```
+
+-> **You have a problem**: `greet()` calls `fetch_database()`. `fetch_database()` does some look-ups in
+a database. _You haven't created that database for your tests, so your test would fail, even though
+the code is correct._
+
+- Luckily, you know that `fetch_database("Mario")` should return "Mr. Mario Mario".
+
+  - _Hint_: Sometimes, you might not know the exact return value, but one that is equally valid and works
+    with the rest of the code. In that case, just use this one.
+
+-> **Solution**: You mock `fetch_database()`. This is also referred to as "mocking out" `fetch_database()`.
+
+```python
+from unittest.mock import patch
+
+def test_greet() -> None:
+    # Mock `fetch_database()` with an object that acts like a shell: It still accepts calls like `fetch_database()`,
+    # but doesn't do any database lookup. We "fill" the shell with a return value; This value will be returned on every
+    # call to `fetch_database()`.
+    with patch("greetings.fetch_database", return_value="Mr. Mario Mario"):
+        greeting = greetings.greet("Mario")
+        assert greeting == "Hello Mr. Mario Mario"
+```
+
+That's all. Note that **this mock is suitable for testing `greet()`, but not for testing `fetch_database()`**.
+More generally, you should only mock those functions you explicitly don't want to test.
+
+#### How does mocking work under the hood?
+
+Since Python 3.3, the standard mocking library is `unittest.mock`. `unittest.mock` implements the basic mocking class `Mock`.
+It also implements `MagicMock`, which is the same as `Mock`, but contains many default magic methods (in Python,
+those are the ones starting with a dunder `__`). From the docs:
+
+> In most of these examples the Mock and MagicMock classes are interchangeable. As the MagicMock is the more capable class
+> it makes a sensible one to use by default.
+
+`Mock` itself is a class that principally accepts and records any and all calls. A piece of code like
+
+```python
+from unittest import mock
+
+foo = mock.Mock()
+foo.bar('quux')
+foo.baz
+foo.qux = 42
+```
+
+is _not_ going to throw any errors. Our mock silently accepts all these calls and records them.
+`Mock` also implements methods for us to access and assert its records, e.g.,
+
+```python
+foo.bar.assert_called_with('quux')
+```
+
+Finally, `unittest.mock` also provides a method to mock objects only within a scope: `patch()`. We can use `patch()` either
+as a decorator or as a context manager. In both cases, the mock created by `patch()` will apply for the scope of the decorator /
+context manager. `patch()` takes only one required argument `target`. `target` is a string in dot notation that _refers to
+the name of the object you want to mock_. It will then assign a `MagicMock()` to that object.
+As an example, look at the following code:
+
+```python
+from unittest import mock
+from os import urandom
+
+with mock.patch('__main__.urandom', return_value=42):
+    print(urandom(1))
+    print(urandom(1)) # No matter what value we plug in for urandom, it will always return 42.
+print(urandom(1)) # We exited the context manager, so the mock doesn't apply anymore. Will return a random byte.
+```
+
+_Note that calling `mock.patch('os.urandom', return_value=42)` wouldn't work here_: `os.urandom` would be the name of our patched
+object. However, we imported `urandom` with `from os import urandom`; hence, we bound the `urandom` name to our current module
+`__main__`.
+
+On the other hand, if we had used `import os.urandom`, we would need to call `mock.patch('os.urandom', return_value=42)` instead.
+
+#### Boilerplate code
+
+- Including the Python mocking library:
+
+  ```python
+  from unittest import mock
+  ```
+
+- Mocking a class with a context manager:
+
+  ```python
+  with mock.patch('module.ClassName', foo=42, return_value='I am a mock') as my_mock:
+    # In here, 'module.ClassName' is mocked with a MagicMock() object my_mock.
+    # my_mock has an attribute named foo with the value 42.
+    # var = module.ClassName() will assign 'I am a mock' to var.
+  ```
+
+- Mocking a class with a decorator:
+
+  ```python
+  @mock.patch('module.ClassName', foo=42, return_value='I am a mock')
+  def my_function(my_mock):
+      # ...
+      # In here, 'module.ClassName' will behave as in the previous example.
+  ```
+
+- Mocking a class attribute:
+
+  ```python
+  with mock.patch.object(module.ClassName, 'class_method', return_value=42)
+    # In here, 'module.ClassName' has the same properties as before, except for 'class_method'
+    # Calling module.ClassName.class_method() will now return 42.
+  ```
+
+  Note the missing quotes around module.ClassName in the patch.object() call.
+
+#### Zulip mocking practices
+
+For mocking we generally use the "mock" library and use `mock.patch` as
+a context manager or decorator. We also take advantage of some context managers
+from Django as well as our own custom helpers. Here is an example:
+
+```python
+with self.settings(RATE_LIMITING=True):
+    with mock.patch('zerver.decorator.rate_limit_user') as rate_limit_mock:
+        api_result = my_webhook(request)
+
+self.assertTrue(rate_limit_mock.called)
+```
+
+Follow [this link](../subsystems/settings.md#testing-non-default-settings) for more
+information on the "settings" context manager.
+
+Zulip has several features, like outgoing webhooks or social
+authentication, that made outgoing HTTP requests to external
+servers. We test those features using the excellent
+[responses](https://pypi.org/project/responses/) library, which has a
+nice interface for mocking `requests` calls to return whatever HTTP
+response from the external server we need for the test. you can find
+examples with `git grep responses.add`. Zulip's own `HostRequestMock`
+class should be used only for low-level tests for code that expects to
+receive Django HttpRequest object.
+
+## Zulip testing philosophy
+
+If there is one word to describe Zulip's philosophy for writing tests,
+it is probably "flexible." (Hopefully "thorough" goes without saying.)
+
+When in doubt, unless speed concerns are prohibitive,
+you usually want your tests to be somewhat end-to-end, particularly
+for testing endpoints.
+
+These are some of the testing strategies that you will see in the Zulip
+test suite...
+
+### Endpoint tests
+
+We strive to test all of our URL endpoints. The vast majority of Zulip
+endpoints support a JSON interface. Regardless of the interface, an
+endpoint test generally follows this pattern:
+
+- Set up the data.
+- Log in with `self.login()` or set up an API key.
+- Use a Zulip test helper to hit the endpoint.
+- Assert that the result was either a success or failure.
+- Check the data that comes back from the endpoint.
+
+Generally, if you are doing endpoint tests, you will want to create a
+test class that is a subclass of `ZulipTestCase`, which will provide
+you helper methods like the following:
+
+- api_auth
+- assert_json_error
+- assert_json_success
+- client_get
+- client_post
+- get_api_key
+- get_streams
+- login
+- send_message
+
+### Library tests
+
+For certain Zulip library functions, especially the ones that are
+not intrinsically tied to Django, we use a classic unit testing
+approach of calling the function and inspecting the results.
+
+For these types of tests, you will often use methods like
+`self.assertEqual()`, `self.assertTrue()`, etc., which come with
+[unittest](https://docs.python.org/3/library/unittest.html#unittest.TestCase)
+via Django.
+
+### Fixture-driven tests
+
+Particularly for testing Zulip's integrations with third party systems,
+we strive to have a highly data-driven approach to testing. To give a
+specific example, when we test our GitHub integration, the test code
+reads a bunch of sample inputs from a JSON fixture file, feeds them
+to our GitHub integration code, and then verifies the output against
+expected values from the same JSON fixture file.
+
+Our fixtures live in `zerver/tests/fixtures`.
+
+### Mocks and stubs
+
+We use mocks and stubs for all the typical reasons:
+
+- to more precisely test the target code
+- to stub out calls to third-party services
+- to make it so that you can [run the Zulip tests on the airplane without wifi][no-internet]
+
+[no-internet]: testing.md#internet-access-inside-test-suites
+
+A detailed description of mocks, along with useful coded snippets, can be found in the section
+[Testing with mocks](#testing-with-mocks).
+
+### Template tests
+
+In [zerver/tests/test_templates.py](https://github.com/zulip/zulip/blob/main/zerver/tests/test_templates.py)
+we have a test that renders all of our backend templates with
+a "dummy" context, to make sure the templates don't have obvious
+errors. (These tests won't catch all types of errors; they are
+just a first line of defense.)
+
+### SQL performance tests
+
+A common class of bug with Django systems is to handle bulk data in
+an inefficient way, where the backend populates objects for join tables
+with a series of individual queries that give O(N) latency. (The
+remedy is often just to call `select_related()`, but sometimes it
+requires a more subtle restructuring of the code.)
+
+We try to prevent these bugs in our tests by using a context manager
+called `queries_captured()` that captures the SQL queries used by
+the backend during a particular operation. We make assertions about
+those queries, often simply by using the `assert_database_query_count`
+that checks the number of queries.
+
+### Event-based tests
+
+The Zulip backend has a mechanism where it will fetch initial data
+for a client from the database, and then it will subsequently apply
+some queued up events to that data to the data structure before notifying
+the client. The `BaseAction.do_test()` helper helps tests
+verify that the application of those events via apply_events() produces
+the same data structure as performing an action that generates said event.
+
+This is a bit esoteric, but if you read the tests, you will see some of
+the patterns. You can also learn more about our event system in the
+[new feature tutorial](../tutorials/new-feature-tutorial.md#handle-database-interactions).
+
+### Negative tests
+
+It is important to verify error handling paths for endpoints, particularly
+situations where we need to ensure that we don't return results to clients
+with improper authentication or with limited authorization. A typical test
+will call the endpoint with either a non-logged in client, an invalid API
+key, or missing input fields. Then the test will call `assert_json_error()`
+to verify that the endpoint is properly failing.
+
+## Testing considerations
+
+Here are some things to consider when writing new tests:
+
+- **Duplication** We try to avoid excessive duplication in tests.
+  If you have several tests repeating the same type of test setup,
+  consider making a setUp() method or a test helper.
+
+- **Network independence** Our tests should still work if you don't
+  have an internet connection. For third party clients, you can simulate
+  their behavior using fixture data. For third party servers, you can
+  typically simulate their behavior using mocks.
+
+- **Coverage** We have 100% line coverage on several of our backend
+  modules. You can use the `--coverage` option to generate coverage
+  reports, and new code should have 100% coverage, which generally
+  requires testing not only the "happy path" but also error handling
+  code and edge cases. It will generate a nice HTML report that you can
+  view right from your browser (the tool prints the URL where the report
+  is exposed in your development environment).
+
+  The HTML report also displays which tests executed each line, which
+  can be handy for finding existing tests for a code path you're
+  working on.
+
+- **Console output** A properly written test should print nothing to
+  the console; use `with self.assertLogs` to capture and verify any
+  logging output. Note that we reconfigure various loggers in
+  `zproject/test_extra_settings.py` where the output is unlikely to be
+  interesting when running our test suite.
+  `test-backend --ban-console-output` checks for stray print statements.
+
+Note that `test-backend --coverage` will assert that
+various specific files in the project have 100% test coverage and
+throw an error if their coverage has fallen. One of our project goals
+is to expand that checking to ever-larger parts of the codebase.

docs/testing/testing-with-node.md

@@ -0,0 +1,298 @@
+# JavaScript/TypeScript unit tests
+
+Our node-based unit tests system is the preferred way to test
+JavaScript/TypeScript code in Zulip. We prefer it over the [Puppeteer
+black-box whole-app testing](testing-with-puppeteer.md),
+system since it is much (>100x) faster and also easier to do correctly
+than the Puppeteer system.
+
+You can run this test suite as follows:
+
+```bash
+tools/test-js-with-node
+```
+
+See `test-js-with-node --help` for useful options; even though the
+whole suite is quite fast, it still saves time to run a single test by
+name when debugging something.
+
+The JS unit tests are written to work with node. You can find them
+in `web/tests`. Here is an example test from
+`web/tests/stream_data.test.cjs`:
+
+```js
+(function test_get_by_id() {
+    stream_data.clear_subscriptions();
+    var id = 42;
+    var sub = {
+        name: 'Denmark',
+        subscribed: true,
+        color: 'red',
+        stream_id: id
+    };
+    stream_data.add_sub_for_tests('Denmark', sub);
+    sub = stream_data.get_sub('Denmark');
+    assert.equal(sub.color, 'red');
+    sub = sub_store.get(id);
+    assert.equal(sub.color, 'red');
+}());
+```
+
+The names of the node tests generally align with the names of the
+modules they test. If you modify a JS module in `web/src` you should
+see if there are corresponding test in `web/tests`. If
+there are, you should strive to follow the patterns of the existing tests
+and add your own tests.
+
+A good first test to read is
+[example1.test.cjs](https://github.com/zulip/zulip/blob/main/web/tests/example1.test.cjs).
+(And then there are several other example files.)
+
+## How the node tests work
+
+Unlike the [Puppeteer unit tests](testing-with-puppeteer.md),
+which use a headless Chromium browser connected to a running Zulip
+development server, our node unit tests don't have a browser, don't
+talk to a server, and generally don't use a complete virtual DOM (a
+handful of tests use the `jsdom` library for this purpose) because
+those slow down the tests a lot, and often don't add much value.
+
+Instead, the preferred model for our unit tests is to mock DOM
+manipulations (which in Zulip are almost exclusively done via
+`jQuery`) using a custom library
+[zjquery](https://github.com/zulip/zulip/blob/main/web/tests/lib/zjquery.cjs).
+
+The
+[unit test file](https://github.com/zulip/zulip/blob/main/web/tests/zjquery.test.cjs)
+for `zjquery` is designed to be also serve as nice documentation for
+how to use `zjquery`, and is **highly recommended reading** for anyone
+working on or debugging the Zulip node tests.
+
+Conceptually, the `zjquery` library provides minimal versions of most
+`jQuery` DOM manipulation functions, and has a convenient system for
+letting you set up return values for more complex functions. For
+example, if the code you'd like to test calls `$obj.find()`, you can
+use `$obj.set_find_results(selector, $value)` to set up `zjquery` so
+that calls to `$obj.find(selector)` will return `$value`. See the unit
+test file for details.
+
+This process of substituting `jQuery` functions with our own code for
+testing purposes is known as "stubbing". `zjquery` does not stub all
+possible interactions with the dom, as such, you may need to write out
+the stub for a function you're calling in your patch. Typically the stub
+is just placed in the test file, to prevent bloating of `zjquery`
+with functions that are only used in a single test.
+
+If you need to stub, you will see an error of this form:
+`Error: You must create a stub for $("#foo").bar`
+
+The `zjquery` library itself is only about 500 lines of code, and can
+also be a useful resource if you're having trouble debugging DOM
+access in the unit tests.
+
+It is typically a good idea to figure out how to stub a given function
+based on how other functions have been stubbed in the same file.
+
+## Handling dependencies in unit tests
+
+The other big challenge with doing unit tests for a JavaScript project
+is that often one wants to limit the scope the production code being
+run, just to avoid doing extra setup work that isn't relevant to the
+code you're trying to test. For that reason, each unit test file
+explicitly declares all of the modules it depends on, with a few
+different types of declarations depending on whether we want to:
+
+- Exercise the module's real code for deeper, more realistic testing?
+- Stub out the module's interface for more control, speed, and
+  isolation?
+- Do some combination of the above?
+
+For all the modules where you want to run actual code, add statements
+like the following toward the top of your test file:
+
+```js
+zrequire('util');
+zrequire('stream_data');
+zrequire('Filter', 'js/filter');
+```
+
+For modules that you want to completely stub out, use a pattern like
+this:
+
+```js
+const reminder = mock_esm("../../web/src/reminder", {
+    is_deferred_delivery: noop,
+});
+
+// then maybe further down
+reminder.is_deferred_delivery = () => true;
+```
+
+One can similarly stub out functions in a module's exported interface
+with either `noop` functions or actual code.
+
+Finally, there's the hybrid situation, where you want to borrow some
+of a module's real functionality but stub out other pieces. Obviously,
+this is a pretty strong code smell that the other module might be
+lacking in cohesion, but sometimes it's not worth going down the
+rabbit hole of trying to improve that. The pattern here is this:
+
+```js
+// Import real code.
+zrequire('narrow_state');
+
+// And later...
+narrow_state.stream = function () {
+    return 'office';
+};
+```
+
+## Creating new test modules
+
+The test runner (`index.cjs`) automatically runs all .test.cjs files in the
+`web/tests` directory, so you can simply start editing a file
+in that directory to create a new test.
+
+## Verifying HTML templates with `mock_template`
+
+As a project, we prefer [end-to-end
+testing][testing-philosophy-end-to-end] where possible, since those
+tests are usually both more effective at catching bugs and cheaper to
+maintain than tests that make heavy use of mocks.
+
+One place where mocks can often be useful is when testing logic for
+rendering an HTML template in Zulip. The interesting logic that one
+wants to verify can be split between two places:
+
+- Computing the **context data** object passed into the HTML template.
+- Conditional logic in the HTML template itself.
+
+It can work well to write tests that verify properties of the computed
+HTML template, for example, is a given CSS class present in the
+result.
+
+But often, one can write a more readable test by instead verifying
+the values of parameters in the context passed into the template
+rendering. The `mock_template` function in Zulip's testing library
+is designed to support this.
+
+We use `mock_template` in our unit tests to verify that the JS code is
+calling the template with the expected context data. And then we use
+the results of mock_template to supply the JS code with either the
+actual HTML from the template or some kind of zjquery stub.
+
+The `mock_template` function accepts 3 parameters:
+
+- The path within `web/templates` to the [Handlebars
+  template](../subsystems/html-css.md) that you'd like to mock.
+- Whether to call the actual template rendering function so that you
+  can verify the HTML generated by this specific template. Since
+  Handlebars rendering in tests takes time and rarely catches bugs, we
+  recommend using `false` if you're only planning to check the context
+  data.
+- A callback function that you can use to include assertions about
+  what parameters were passed into the template. This function
+  receives a `data` parameter, with the context data, and an `html`
+  parameter if the real template was rendered.
+
+The following illustrates the two common patterns for using this method.
+
+```js
+run_test("test something calling template", ({mock_template}) => {
+    mock_template("path/to/template.hbs", false, (data) => {
+        assert.deepEqual(data, {...};
+        // or assert.deepEqual(data.foo, {...});
+        return "stub-for-zjquery";
+    });
+
+    mock_template("path/to/template.hbs", true, (data, html) => {
+        assert.deepEqual(data, {...};
+        assert.ok(html.startWith(...));
+        return html;
+    });
+});
+```
+
+[testing-philosophy-end-to-end]: https://zulip.readthedocs.io/en/stable/testing/philosophy.html#integration-testing-or-unit-testing
+
+## Coverage reports
+
+You can automatically generate coverage reports for the JavaScript unit
+tests like this:
+
+```bash
+tools/test-js-with-node --coverage
+```
+
+If tests pass, you will get instructions to view coverage reports
+in your browser.
+
+Note that modules that we don't test _at all_ aren't listed in the
+report, so this tends to overstate how good our overall coverage is,
+but it's accurate for individual files. You can also click a filename
+to see the specific statements and branches not tested. 100% branch
+coverage isn't necessarily possible, but getting to at least 80%
+branch coverage is a good goal.
+
+The overall project goal is to get to 100% node test coverage on all
+data/logic modules (UI modules are lower priority for unit testing).
+
+## Editor debugger integration
+
+Our node test system is pretty simple, and it's possible to configure
+the native debugger features of popular editors to allow stepping
+through the code. Below we document the editors where someone has put
+together detailed instructions for how to do so. Contributions of
+notes for other editors are welcome!
+
+## Webstorm integration setup
+
+These instructions assume you're using the Vagrant development environment.
+
+1. Set up [Vagrant in WebStorm][vagrant-webstorm].
+
+2. In WebStorm, navigate to `Preferences -> Tools -> Vagrant` and
+   configure the following:
+
+   - `Instance folder` should be the root of the `zulip` repository on
+     your host (where the Vagrantfile is located).
+   - `Provider` should be `virtualbox` on macOS and Docker on Linux
+   - In `Boxes`, choose the one used for Zulip (unless you use
+     Virtualbox for other things, there should only be one option).
+
+   You shouldn't need to set these additional settings:
+
+   - `Vagrant executable` should already be correctly `vagrant`.
+   - `Environment Variables` is not needed.
+
+3. You'll now need to set up a WebStorm "Debug Configuration". Open
+   the `Run/Debug Configuration` menu and create a new `Node.js` config:
+   1. Under `Node interpreter:` click the 3 dots to the right side and
+      click on the little plus in the bottom left of the
+      `Node.js Interpreters` window.
+   1. Select `Add Remote...`.
+      1. In the `Configure Node.js Remote Interpreter`, window select `Vagrant`
+      1. Wait for WebStorm to connect to Vagrant. This will be displayed
+         by the `Vagrant Host URL` section updating to contain the Vagrant
+         SSH URL, e.g., `ssh://vagrant@127.0.0.1:2222`.
+      1. **Set the `Node.js interpreter path` to `/usr/local/bin/node`**
+      1. Hit `OK` 2 times to get back to the `Run/Debug Configurations` window.
+   1. Under `Working Directory` select the root `zulip` directory.
+   1. Under `JavaScript file`, enter `web/tests/lib/index.cjs`
+      -- this is the root script for Zulip's node unit tests.
+
+Congratulations! You've now set up the integration.
+
+## Running tests with the debugger
+
+To use Webstorm to debug a given node test file, do the following:
+
+1. Under `Application parameters` choose the node test file that you
+   are trying to test (e.g., `web/tests/message_store.test.cjs`).
+1. Under `Path Mappings`, set `Project Root` to `/srv/zulip`
+   (i.e. where the `zulip` Git repository is mounted in the Vagrant guest).
+1. Use the WebStorm debugger; see [this overview][webstorm-debugging]
+   for details on how to use it.
+
+[webstorm-debugging]: https://blog.jetbrains.com/webstorm/2018/01/how-to-debug-with-webstorm/
+[vagrant-webstorm]: https://www.jetbrains.com/help/webstorm/vagrant-support.html?section=Windows%20or%20Linux

docs/testing/testing-with-puppeteer.md

@@ -0,0 +1,167 @@
+# Web frontend black-box Puppeteer tests
+
+While our [node test suite](testing-with-node.md) is the
+preferred way to test most frontend code because they are easy to
+write and maintain, some code is best tested in a real browser, either
+because of navigation (e.g., login) or because we want to verify the
+interaction between Zulip logic and browser behavior (e.g., copy/paste,
+keyboard shortcuts, etc.).
+
+## Running tests
+
+You can run this test suite as follows:
+
+```bash
+tools/test-js-with-puppeteer
+```
+
+See `tools/test-js-with-puppeteer --help` for useful options,
+especially running specific subsets of the tests to save time when
+debugging.
+
+The test files live in `web/e2e-tests` and make use
+of various useful helper functions defined in
+`web/e2e-tests/lib/common.ts`.
+
+## How Puppeteer tests work
+
+The Puppeteer tests use a real Chromium browser (powered by
+[puppeteer](https://github.com/puppeteer/puppeteer)), connected to a
+real Zulip development server. These are black-box tests: Steps in a
+Puppeteer test are largely things one might do as a user of the Zulip
+web app, like "Type this key", "Wait until this HTML element
+appears/disappears", or "Click on this HTML element".
+
+For example, this function might test the `x` keyboard shortcut to
+open the compose box for a new direct message:
+
+```js
+async function test_private_message_compose_shortcut(page) {
+    await page.keyboard.press("KeyX");
+    await page.waitForSelector("#private_message_recipient", {visible: true});
+    await common.pm_recipient.expect(page, "");
+    await close_compose_box(page);
+}
+```
+
+The test function presses the `x` key, waits for the
+`#private_message_recipient` input element to appear, verifies its
+content is empty, and then closes the compose box. The
+`waitForSelector` step here (and in most tests) is critical; tests
+that don't wait properly often fail nonderministically, because the
+test will work or not depending on whether the browser updates the UI
+before or after executing the next step in the test.
+
+Black-box tests are fantastic for ensuring the overall health of the
+project, but are also slow, costly to maintain, and require care to
+avoid nondeterministic failures, so we usually prefer to write a Node
+test instead when both are options.
+
+They also can be a bit tricky to understand for contributors not
+familiar with [async/await][learn-async-await].
+
+## Debugging Puppeteer tests
+
+The following questions are useful when debugging Puppeteer test
+failures you might see in [continuous
+integration](continuous-integration.md):
+
+- Does the flow being tested work properly in the Zulip browser UI?
+  Test failures can reflect real bugs, and often it's easier and more
+  interactive to debug an issue in the normal Zulip development
+  environment than in the Puppeteer test suite.
+- Does the change being tested adjust the HTML structure in a way that
+  affects any of the selectors used in the tests? If so, the test may
+  just need to be updated for your changes.
+- Does the test fail deterministically when you run it locally using
+  e.g., `./tools/test-js-with-puppeteer compose.ts`? If so, you can
+  iteratively debug to see the failure.
+- Does the test fail nondeterministically? If so, the problem is
+  likely that a `waitForSelector` statement is either missing or not
+  waiting for the right thing. Tests fail nondeterministically much
+  more often on very slow systems like those used for Continuous
+  Integration (CI) services because small races are amplified in those
+  environments; this often explains failures in CI that cannot be
+  easily reproduced locally.
+- Does the test fail when you are typing (filling the form) on a modal
+  or other just-opened UI widget? Puppeteer starts typing after focusing on
+  the text field, sending keystrokes one after another. So, if
+  application code explicitly focuses the modal after it
+  appears/animates, this could cause the text field that Puppeteer is
+  trying to type into to lose focus, resulting in partially typed data.
+  The recommended fix for this is to wait until the modal is focused before
+  starting typing, like this:
+  ```JavaScript
+  await page.waitForFunction(":focus").attr("id") === modal_id);
+  ```
+
+These tools/features are often useful when debugging:
+
+- You can use `console.log` statements both in Puppeteer tests and the
+  code being tested to print-debug.
+- Zulip's Puppeteer tests are configured to generate screenshots of
+  the state of the test browser when an assert statement fails; these
+  are stored under `var/puppeteer/*.png` and are extremely helpful for
+  debugging test failures.
+- TODO: Mention how to access Puppeteer screenshots in CI.
+- TODO: Add an option for using the `headless: false` debugging mode
+  of Puppeteer so you can watch what's happening, and document how to
+  make that work with Vagrant.
+- TODO: Document `--interactive`.
+- TODO: Document how to run 100x in CI to check for nondeterministic
+  failures.
+- TODO: Document any other techniques/ideas that were helpful when porting
+  the Casper suite.
+- The Zulip server powering these tests is just `run-dev` with some
+  extra [Django settings](../subsystems/settings.md) from
+  `zproject/test_extra_settings.py` to configure an isolated database
+  so that the tests will not interfere/interact with a normal
+  development environment. The console output while running the tests
+  includes the console output for the server; any Python exceptions
+  are likely actual bugs in the changes being tested.
+
+See also [Puppeteer upstream's debugging
+tips](https://github.com/puppeteer/puppeteer#debugging-tips); some
+tips may require temporary patches to functions like `run_test` or
+`ensure_browser` in `web/e2e-tests/lib/common.ts`.
+
+## Writing Puppeteer tests
+
+Probably the easiest way to learn how to write Puppeteer tests is to
+study some of the existing test files. There are a few tips that can
+be useful for writing Puppeteer tests in addition to the debugging
+notes above:
+
+- Run just the file containing your new tests as described above to
+  have a fast debugging cycle.
+- When you're done writing a test, run it 100 times in a loop to
+  verify it does not fail nondeterministically (see above for notes on
+  how to get CI to do it for you); this is important to avoid
+  introducing extremely annoying nondeterministic failures into
+  `main`.
+- With black-box browser tests like these, it's very important to write your code
+  to wait for browser's UI to update before taking any action that
+  assumes the last step was processed by the browser (e.g., after you
+  click on a user's avatar, you need an explicit wait for the profile
+  popover to appear before you can try to click on a menu item in that
+  popover). This means that before essentially every action in your
+  Puppeteer tests, you'll want to use `waitForSelector` or similar
+  wait function to make sure the page or element is ready before you
+  interact with it. The [puppeteer docs site](https://pptr.dev/) is a
+  useful reference for the available wait functions.
+- When using `waitForSelector`, you always want to use the
+  `{visible: true}` option; otherwise the test will stop waiting as
+  soon as the target selector is present in the DOM even if it's
+  hidden. For the common UI pattern of having an element always be
+  present in the DOM whose presence is managed via show/hide rather
+  than adding/removing it from the DOM, `waitForSelector` without
+  `visible: true` won't wait at all.
+- The test suite uses a smaller set of default user accounts and other
+  data initialized in the database than the normal development
+  environment; specifically, it uses the same setup as the [backend
+  tests](testing-with-django.md). To see what differs from
+  the development environment, check out the conditions on
+  `options["test_suite"]` in
+  `zilencer/management/commands/populate_db.py`.
+
+[learn-async-await]: https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous/Async_await

docs/testing/testing.md

@@ -0,0 +1,154 @@
+# Testing overview
+
+Zulip takes pride in its extensive, carefully designed test suites.
+For example, `test-backend` runs a complete test suite (~98% test
+coverage; 100% on core code) for the Zulip server in under a minute on
+a fast laptop; very few web apps of similar scope can say something
+similar.
+
+This page focused on the mechanics of running automated tests in a
+[development environment](../development/overview.md); you may also
+want to read about our [testing philosophy](philosophy.md)
+and [continuous integration
+setup](continuous-integration.md).
+
+Manual testing with a web browser is primarily discussed in the docs
+on [using the development environment](../development/using.md).
+
+## Running tests
+
+Zulip tests must be run inside a Zulip development environment; if
+you're using Vagrant, you may need to enter it with `vagrant ssh`.
+
+You can run all of the test suites (similar to our continuous integration)
+as follows:
+
+```bash
+./tools/test-all
+```
+
+However, you will rarely want to do this while actively developing,
+because it takes a long time. Instead, your edit/refresh cycle will
+typically involve running subsets of the tests with commands like these:
+
+```bash
+./tools/lint zerver/models/__init__.py # Lint the file you just changed
+./tools/test-backend zerver.tests.test_markdown.MarkdownEmbedsTest.test_inline_youtube
+./tools/test-backend MarkdownEmbedsTest # Run `test-backend --help` for more options
+./tools/test-js-with-node util
+# etc.
+```
+
+The commands above will all run in just a few seconds. Many more
+useful options are discussed in each tool's documentation (e.g.,
+`./tools/test-backend --help`).
+
+## Major test suites
+
+Zulip has a handful of major tests suite that every developer will
+eventually work with, each with its own page detailing how it works:
+
+- [Linters](linters.md): Our dozen or so linters run in parallel.
+- [Django](testing-with-django.md): Server/backend Python tests.
+- [Node](testing-with-node.md): JavaScript tests for the
+  frontend run via node.js.
+- [Puppeteer](testing-with-puppeteer.md): End-to-end
+  UI tests run via a Chromium browser.
+
+## Other test suites
+
+Additionally, Zulip also has about a dozen smaller tests suites:
+
+- `tools/test-migrations`: Checks whether the `zerver/migrations`
+  migration content the models defined in `zerver/models/*.py`. See our
+  [schema migration documentation](../subsystems/schema-migrations.md)
+  for details on how to do database migrations correctly.
+- `tools/test-documentation`: Checks for broken links in this
+  ReadTheDocs documentation site.
+- `tools/test-help-documentation`: Checks for broken links in the
+  `/help/` help center documentation, and related pages.
+- `tools/test-api`: Tests that the API documentation at `/api`
+  actually works; the actual code for this is defined in
+  `zerver/openapi/python_examples.py`.
+- `tools/check-capitalization`: Checks whether translated strings (aka
+  user-facing strings) correctly follow Zulip's capitalization
+  conventions. This requires some maintenance of an exclude list
+  (`tools.lib.capitalization.IGNORED_PHRASES`) of proper nouns
+  mentioned in the Zulip project, but helps a lot in avoiding new
+  strings being added that don't match our style.
+- `tools/check-frontend-i18n`: Checks for a common bug in Handlebars
+  templates, of using the wrong syntax for translating blocks
+  containing variables.
+- `./tools/test-run-dev`: Checks that `run-dev` starts properly;
+  this helps prevent bugs that break the development environment.
+- `./tools/test-queue-worker-reload`: Verifies that Zulip's queue
+  processors properly reload themselves after code changes.
+- `./tools/setup/optimize-svg`: Checks whether all integration logo SVG
+  graphics are optimized.
+  logos are properly optimized for size (since we're not going to edit
+  third-party logos, this helps keep the Zulip codebase from getting huge).
+- `./tools/test-tools`: Automated tests for various parts of our
+  development tooling (mostly various linters) that are not used in
+  production.
+
+Each of these has a reason (usually, performance or a need to do messy
+things to the environment) why they are not part of the handful of
+major test suites like `test-backend`, but they all contribute
+something valuable to helping keep Zulip bug-free.
+
+## Internet access inside test suites
+
+As a policy matter, the Zulip test suites should never make outgoing
+HTTP or other network requests. This is important for 2 major
+reasons:
+
+- Tests that make outgoing Internet requests will fail when the user
+  isn't on the Internet.
+- Tests that make outgoing Internet requests often have a hidden
+  dependency on the uptime of a third-party service, and will fail
+  nondeterministically if that service has a temporary outage.
+  Nondeterministically failing tests can be a big waste of
+  developer time, and we try to avoid them wherever possible.
+
+As a result, Zulip's major test suites should never access the
+Internet directly. Since code in Zulip does need to access the
+Internet (e.g., to access various third-party APIs), this means that
+the Zulip tests use mocking to basically hardcode (for the purposes of
+the test) what responses should be used for any outgoing Internet
+requests that Zulip would make in the code path being tested.
+
+This is easy to do using test fixtures (a fancy word for fixed data
+used in tests) and the `mock.patch` function to specify what HTTP
+response should be used by the tests for every outgoing HTTP (or other
+network) request. Consult
+[our guide on mocking](testing-with-django.md#zulip-mocking-practices) to
+learn how to mock network requests easily; there are also a number of
+examples throughout the codebase.
+
+We partially enforce this policy in the main Django/backend test suite
+by overriding certain library functions that are used in outgoing HTTP
+code paths (`httplib2.Http().request`, `requests.request`, etc.) to
+throw an exception in the backend tests. While this is enforcement is
+not complete (there a lot of other ways to use the Internet from
+Python), it is easy to do and catches most common cases of new code
+depending on Internet access.
+
+This enforcement code results in the following exception:
+
+```pytb
+File "tools/test-backend", line 120, in internet_guard
+  raise Exception("Outgoing network requests are not allowed in the Zulip tests."
+Exception: Outgoing network requests are not allowed in the Zulip tests.
+...
+```
+
+#### Documentation tests
+
+The one exception to this policy is our documentation tests, which
+will attempt to verify that the links included in our documentation
+aren't broken. Those tests end up failing nondeterministically fairly
+often, which is unfortunate, but there's simply no other correct way
+to verify links other than attempting to access them. The compromise
+we've implemented is that in CI, these tests only verify links to
+websites controlled by the Zulip project (zulip.com, our GitHub,
+our ReadTheDocs), and not links to third-party websites.

docs/testing/typescript.md

@@ -0,0 +1,101 @@
+# TypeScript static types
+
+Zulip is early in the process of migrating our codebase to use
+[TypeScript](https://www.typescriptlang.org/), the leading static type
+system for JavaScript. It works as an extension of the ES6 JavaScript
+standard, and provides similar benefits to our use of
+[the mypy static type system for Python](mypy.md).
+
+We expect to eventually migrate the entire JavaScript codebase to
+TypeScript, though our current focus is on getting the tooling and
+migration process right, not on actually migrating the codebase.
+
+As a result, the details in this document are preliminary ideas for
+discussion and very much subject to change.
+
+A typical piece of TypeScript code looks like this:
+
+```ts
+setdefault(key: K, value: V): V {
+    const mapping = this._items[this._munge(key)];
+    if (mapping === undefined) {
+        return this.set(key, value);
+    }
+    return mapping.v;
+}
+```
+
+The following resources are valuable for learning TypeScript:
+
+- The main documentation on [TypeScript syntax][typescript-handbook].
+
+## Type checking
+
+TypeScript types are checked by the TypeScript compiler, `tsc`, which
+is run as part of our [lint checks](linters.md). You can run the
+compiler yourself with `tools/run-tsc`, which will check all the
+TypeScript files once, or `tools/run-tsc --watch`, which will
+continually recheck the files as you edit them.
+
+## Linting and style
+
+We use the ESLint plugin for TypeScript to lint TypeScript code, just
+like we do for JavaScript. Our long-term goal is to use an idiomatic
+TypeScript style for our TypeScript codebase.
+
+However, because we are migrating an established JavaScript codebase,
+we plan to start with a style that is closer to the existing
+JavaScript code, so that we can easily migrate individual modules
+without too much code churn. A few examples:
+
+- TypeScript generally prefers explicit `return undefined;`, whereas
+  our existing JavaScript style uses just `return;`.
+- With TypeScript, we expect to make heavy use of `let` and `const`
+  rather than `var`.
+- With TypeScript/ES6, we may no longer need to use `_.each()` as our
+  recommended way to do loop iteration.
+
+For each of the details, we will either want to bulk-migrate the
+existing JavaScript codebase before the migration or plan to do it
+after JS->TS migration for a given file, so that we don't need to
+modify these details as part of converting a file from JavaScript to
+TypeScript.
+
+A possibly useful technique for this will be setting some eslint
+override rules at the top of individual files in the first commit that
+converts them from JS to TS.
+
+## Migration strategy
+
+Our plan is to order which modules we migrate carefully, starting with
+those that:
+
+- Appear frequently as reverse dependencies of other modules
+  (e.g., `people.js`). These are most valuable to do first because
+  then we have types on the data being interacted with by other
+  modules when we migrate those.
+- Don't have large open pull requests (to avoid merge conflicts); one
+  can scan for these using [TinglingGit](https://github.com/zulip/TinglingGit).
+- Have good unit test coverage, which limits the risk of breaking
+  correctness through refactoring. Use
+  `tools/test-js-with-node --coverage` to get a coverage report.
+
+When migrating a module, we want to be especially thoughtful about
+putting together a commit structure that makes mistakes unlikely and
+the changes easy to verify. For example:
+
+- First a commit that just converts the language to TypeScript adding
+  types. The result may potentially have some violations of the
+  long-term style we want (e.g., not using `const`). Depending on how
+  we're handling linting, we set some override eslint rules at the top
+  of the module at this stage so CI still passes.
+- Then a commit just migrating use of `var` to `const/let` without
+  other changes (other than removing any relevant linter overrides).
+- Etc.
+
+With this approach, we should be able to produce a bunch of really
+simple commits that can be merged the same day they're written without
+significant risk of introducing regressions from typos, refactors that
+don't quite work how they were expected to, etc.
+
+[typescript-handbook]: https://www.typescriptlang.org/docs/handbook/basic-types.html