snapshot of https://github.com/zulip/zulip.git @ 48c69911c7dd4e03b20552eb3bd77ad8151c1357 (2026-09-15)

docs/tutorials/index.md

@@ -0,0 +1,13 @@
+# Developer tutorials
+
+```{toctree}
+---
+maxdepth: 3
+---
+
+new-feature-tutorial
+writing-views
+life-of-a-request
+reading-list
+shell-tips
+```

docs/tutorials/life-of-a-request.md

@@ -0,0 +1,217 @@
+# Life of a request
+
+It can sometimes be confusing to figure out how to write a new feature,
+or debug an existing one. Let us try to follow a request through the
+Zulip codebase, and dive deep into how each part works.
+
+We will use as our example the creation of users through the API, but we
+will also highlight how alternative requests are handled.
+
+## A request is sent to the server, and handled by [nginx](https://nginx.org/en/docs/)
+
+When Zulip is deployed in production, all requests go through nginx.
+For the most part we don't need to know how this works, except for when
+it isn't working. nginx does the first level of routing--deciding which
+application will serve the request (or deciding to serve the request
+itself for static content).
+
+In development, `tools/run-dev` fills the role of nginx. Static files
+are in your Git checkout under `static`, and are served unminified.
+
+## Static files are [served directly][served-directly] by nginx
+
+[served-directly]: https://github.com/zulip/zulip/blob/main/puppet/zulip/files/nginx/zulip-include-frontend/app
+
+Static files include JavaScript, CSS, and static assets (like emoji, avatars).
+
+File not found errors (404) are served using a Django URL, so that we
+can use configuration variables (like whether the user is logged in)
+in the 404 error page.
+
+```nginx
+location /static/ {
+    alias /home/zulip/prod-static/;
+    # Set a nonexistent path, so we just serve the nice Django 404 page.
+    error_page 404 /django_static_404.html;
+}
+```
+
+## nginx routes other requests [between Django and Tornado][tornado-django]
+
+[tornado-django]: ../overview/architecture-overview.md#django-and-tornado
+
+All our connected clients hold open long-polling connections so that
+they can receive events (messages, presence notifications, and so on) in
+real-time. Events are served by Zulip's `tornado` application.
+
+Nearly every other kind of request is served by the `zerver` Django
+application.
+
+[Here is the relevant nginx routing configuration.][nginx-config-link]
+
+[nginx-config-link]: https://github.com/zulip/zulip/blob/main/puppet/zulip/files/nginx/zulip-include-frontend/app
+
+## Django routes the request to a view in urls.py files
+
+There are various
+[urls.py](https://docs.djangoproject.com/en/5.0/topics/http/urls/)
+files throughout the server codebase, which are covered in more detail
+in
+[the directory structure doc](../subsystems/directory-structure.md).
+
+The main Zulip Django app is `zerver`. The routes are found in
+`zproject/urls.py`.
+
+There are HTML-serving, REST API, and webhook url patterns. We
+will look at how each of these types of requests are handled, and focus
+on how the REST API handles our user creation example.
+
+## Views serving HTML are internationalized by server path
+
+If we look in
+[zproject/urls.py](https://github.com/zulip/zulip/blob/main/zproject/urls.py),
+we can see something called `i18n_urls`. These urls show up in the
+address bar of the browser, and serve HTML.
+
+For example, the `/features` page (preview
+[here](https://zulip.com/features/)) gets translated in Chinese at
+`zh-hans/features/` (preview
+[here](https://zulip.com/zh-hans/features/)).
+
+Note the `zh-hans` prefix--that url pattern gets added by `i18n_patterns`.
+
+## API endpoints use [REST](https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm)
+
+Our example is a REST API endpoint. It's a PUT to `/users`.
+
+With the exception of incoming webhooks (which we do not usually control the
+format of) and logged-out endpoints, Zulip uses REST
+for its API. This means that we use:
+
+- POST for creating something new where we don't have a unique
+  ID. Also used as a catch-all if no other verb is appropriate.
+- PUT for creating something for which we have a unique ID.
+- DELETE for deleting something
+- PATCH for updating or editing attributes of something.
+- GET to get something (read-only)
+- HEAD to check the existence of something to GET, without getting it;
+  useful to check a link without downloading a potentially large link
+- OPTIONS (handled automatically, see more below)
+
+Of these, PUT, DELETE, HEAD, OPTIONS, and GET are _idempotent_, which
+means that we can send the request multiple times and get the same
+state on the server. You might get a different response after the first
+request, as we like to give our clients an error so they know that no
+new change was made by the extra requests.
+
+POST is not idempotent--if I send a message multiple times, Zulip will
+show my message multiple times. PATCH is special--it can be
+idempotent, and we like to write API endpoints in an idempotent fashion,
+as much as possible.
+
+This [cookbook](http://restcookbook.com/) and
+[tutorial](https://www.restapitutorial.com/) can be helpful if you are
+new to REST web applications.
+
+### PUT is only for creating new things
+
+If you're used to using PUT to update or modify resources, you might
+find our convention a little strange.
+
+We use PUT to create resources with unique identifiers, POST to create
+resources without unique identifiers (like sending a message with the
+same content multiple times), and PATCH to modify resources.
+
+In our example, `create_user_backend` uses PUT, because there's a unique
+identifier, the user's email.
+
+### OPTIONS
+
+The OPTIONS method will yield the allowed methods.
+
+This request:
+`OPTIONS https://chat.zulip.org/api/v1/users`
+yields a response with this HTTP header:
+`Allow: GET, HEAD, POST`
+
+We can see this reflected in [zproject/urls.py](https://github.com/zulip/zulip/blob/main/zproject/urls.py):
+
+```python
+rest_path('users',
+          GET=get_members_backend,
+          POST=create_user_backend),
+```
+
+In this way, the API is partially self-documenting.
+
+### Incoming webhook integrations may not be RESTful
+
+Zulip endpoints that are called by other services for integrations have
+to conform to the service's request format. They are likely to use
+only POST.
+
+## Django calls rest_dispatch for REST endpoints, and authenticates
+
+For requests that correspond to a REST url pattern, Zulip configures
+its url patterns (see
+[zerver/lib/rest.py](https://github.com/zulip/zulip/blob/main/zerver/lib/rest.py))
+so that the action called is `rest_dispatch`. This method will
+authenticate the user, either through a session token from a cookie,
+or from an `email:api-key` string given via HTTP basic auth for API
+clients.
+
+It will then look up what HTTP verb was used (GET, POST, etc) to make
+the request, and then figure out which view to show from that.
+
+In our example,
+
+```python
+GET=get_members_backend,
+PUT=create_user_backend
+```
+
+are supplied as arguments to `rest_path`, along with the
+[HTTPRequest](https://docs.djangoproject.com/en/5.0/ref/request-response/).
+The request has the HTTP verb `PUT`, which `rest_dispatch` can use to
+find the correct view to show:
+`zerver.views.users.create_user_backend`.
+
+## The view will authorize the user, extract request variables, and validate them
+
+This is covered in good detail in the [writing views doc](writing-views.md).
+
+## Results are given as JSON
+
+Our API works on JSON requests and responses. Every API endpoint should
+`raise JsonableError` in the case of an error, which gives a JSON string:
+
+```json
+{"result": "error", "msg": "<some error message>", "code": "BAD_REQUEST"}
+```
+
+in a [Django HttpResponse
+object](https://docs.djangoproject.com/en/5.0/ref/request-response/)
+with a `Content-Type` of 'application/json'.
+
+To pass back data from the server to the calling client, in the event of
+a successfully handled request, we use `json_success(request, data)`.
+
+The `request` argument is a [Django HttpRequest
+object](https://docs.djangoproject.com/en/5.0/ref/request-response/).
+The `data` argument is a Python object which can be converted to a JSON
+string and has a default value of an empty Python dictionary.
+
+Zulip stores additional metadata it has associated with that HTTP
+request in a `RequestNotes` object, which is primarily accessed in
+common code used in all requests (middleware, logging, rate limiting,
+etc.).
+
+This will result in a JSON string:
+
+```json
+{"result": "success", "msg": "", "data": {"var_name1": "var_value1", "var_name2": "var_value2"}}
+```
+
+with an HTTP 200 status and a `Content-Type` of 'application/json'.
+
+That's it!

docs/tutorials/new-feature-tutorial.md

@@ -0,0 +1,760 @@
+# Writing a new application feature
+
+The changes needed to add a new feature will vary, of course, but this
+document provides a general outline of what you may need to do, as well
+as an example of the specific steps needed to add a new feature: adding
+a new option to the application that is dynamically synced through the
+data system in real-time to all browsers the user may have open.
+
+As you read this, you may find you need to learn about Zulip's
+real-time push system; the
+[real-time push and events](../subsystems/events-system.md)
+documentation has a detailed explanation of how everything works. You
+may also find it beneficial to read Zulip's
+[architectural overview](../overview/architecture-overview.md).
+Zulip is a web application built using the
+[Django framework](https://www.djangoproject.com/), and some of the
+processes listed in this tutorial, such as database migrations and
+tests, use Django's tooling.
+
+Zulip's [directory structure](../subsystems/directory-structure.md)
+will also be helpful to review when creating a new feature. Many
+aspects of the structure will be familiar to Django developers. Visit
+[Django's documentation](https://docs.djangoproject.com/en/5.0/#index-first-steps)
+for more information about how Django projects are typically
+organized. And finally, the
+[message sending](../subsystems/sending-messages.md) documentation on
+the additional complexity involved in sending messages.
+
+## General process
+
+### Files impacted
+
+This tutorial will walk through adding a new feature to a Realm (an
+organization in Zulip). The following files are involved in the process:
+
+**Backend**
+
+- `zerver/models/realms.py`: Defines the database model.
+- `zerver/views/realm.py`: Contains the view function that implements the
+  API endpoint for editing Realm objects.
+- `zerver/actions/realm_settings.py`: Contains code for updating and
+  interacting with the database.
+- `zerver/lib/events.py`: Ensures that the state Zulip sends to clients
+  is always consistent and correct.
+
+**Frontend**
+
+- `web/src/state_data.ts`: Defines the expected state of the data that
+  is received from the Zulip server.
+- `web/templates/settings/organization_settings_admin.hbs`: Defines
+  the structure of the organization settings admin modal.
+- `web/src/admin.ts`: Handles building the organization settings admin
+  modal.
+- `web/src/settings_org.ts`: Handles form submission for changes to an
+  organization's settings.
+- `web/src/server_events_dispatch.js`: Handles events coming from the
+  server.
+
+**Backend testing**
+
+- `zerver/tests/test_realm.py`: Contains end-to-end API tests for
+  updating realm settings.
+- `zerver/tests/test_events.py`: Tests for possible race bugs in the
+  implementation of events.
+
+**Frontend testing**
+
+- `web/e2e-tests/admin.test.ts`: Contains end-to-end tests for the
+  organization admin settings pages.
+- `web/tests/dispatch.test.cjs`: Unit tests for handling of events.
+
+**Documentation**
+
+- `zerver/openapi/zulip.yaml`: Contains OpenAPI definitions for the
+  Zulip REST API.
+- `api_docs/changelog.md`: Documentation listing all changes to the
+  Zulip Server API.
+- `starlight_help/...`: The user-facing documentation (help center)
+  for the application.
+
+### Adding a field to the database
+
+**Update the model:** The server accesses the underlying database in
+`zerver/models/realms.py`. Add a new field in the appropriate class.
+
+**Create and run the migration:** To create and apply a migration, run the
+following commands:
+
+```bash
+./manage.py makemigrations
+./manage.py migrate
+```
+
+It's highly recommended to read our
+[database migration documentation](../subsystems/schema-migrations.md)
+to learn more about creating and applying database migrations.
+
+**Test your changes:** Once you've run the migration, [restart the
+development
+server](../development/remote.md#running-the-development-server).
+
+### Backend changes
+
+We have a framework that automatically handles many of the steps for the
+most common types of UserProfile and Realm settings. We refer to this as the
+`property_types` framework. However, it is valuable to understand
+the flow of events even if the `property_types` framework means you don't
+have to write much code for a new setting.
+
+**Database interaction:** Add any necessary code for updating and
+interacting with the database in `zerver/actions/realm_settings.py`.
+It should update the database and send an event announcing the change.
+
+**Application state:** Modify the `fetch_initial_state_data` and
+`apply_event` functions in `zerver/lib/events.py` to update the state
+based on the field you just created.
+
+**Backend implementation:** Make any other modifications to the backend
+required for your feature to do what it's supposed to do (this will
+be unique to the feature you're implementing).
+
+**New views:** Add any new application views to `zproject/urls.py`, or
+update the appropriate existing view in `zerver/views/...`. This
+includes both views that serve HTML (new pages on Zulip) as well as new
+API endpoints that serve JSON-formatted data.
+
+**Testing:** At the very least, add a test of your event data flowing
+through the system in `test_events.py` and an API test (e.g., for a
+Realm setting, in `test_realm.py`).
+
+### Frontend changes
+
+**JavaScript/TypeScript:** Zulip's JavaScript and TypeScript sources are
+located in the directory `web/src/`. The exact files you may need to change
+depend on your feature. If you've added a new event that is sent to clients,
+be sure to add a handler for it in `web/src/server_events_dispatch.js`.
+
+**CSS:** The primary CSS file is `web/styles/zulip.css`. If your new
+feature requires UI changes, you may need to add additional CSS to this
+file.
+
+**Templates:** The initial page structure is rendered via Jinja2
+templates located in `templates/zerver/app`. For JavaScript, Zulip uses
+Handlebars templates located in `web/templates`. Templates are
+precompiled as part of the build/deploy process.
+
+Zulip is fully internationalized, so when writing both HTML templates
+or JavaScript/TypeScript/Python code that generates user-facing strings,
+be sure to [tag those strings for translation](../translating/translating.md).
+
+**Testing:** There are two types of frontend tests: node-based unit
+tests and blackbox end-to-end tests. The blackbox tests are run in a
+headless Chromium browser using Puppeteer and are located in
+`web/e2e-tests/`. The unit tests use Node's `assert` module are located
+in `web/tests/`. For more information on writing and running tests, see
+the [testing documentation](../testing/testing.md).
+
+### Documentation changes
+
+After implementing the new feature, you should document it and update
+any existing documentation that might be relevant to the new feature.
+For detailed information on the kinds of documentation Zulip has, see
+[Documentation](../documentation/overview.md).
+
+**Help center documentation:** You will likely need to at least update,
+extend and link to articles in the `starlight_help/` directory that are
+related to your new feature. [Writing help center articles](../documentation/helpcenter.md)
+provides more detailed information about writing and editing feature
+`starlight_help/` directory articles.
+
+**API documentation:** A new feature will probably impact the REST API
+documentation as well, which will mean updating `zerver/openapi/zulip.yaml`
+and modifying `api_docs/changelog.md` for a new feature
+level. [Documenting REST API endpoints](../documentation/api.md)
+explains Zulip's API documentation system and provides a step by step
+guide to adding or updating documentation for an API endpoint.
+
+## Example feature
+
+This example describes the process of adding a new realm setting to Zulip.
+For the purposes of this tutorial, this new setting will be called
+`my_fantastic_feature`.
+
+:::{tip}
+
+A useful tool that we'll highlight throughout this tutorial is
+`git grep`. If you're unfamiliar with using `git grep`, then check out
+[this blog post](https://laurynmm.github.io/2021/12/22/git-grep.html)
+for an introduction to it, as well as the [official `git grep`
+documentation](https://git-scm.com/docs/git-grep.)
+
+:::
+
+### Update the model
+
+First, update the database and model to store the new setting. Add a new
+boolean field, `my_fantastic_feature`, to the Realm model in
+`zerver/models/realms.py`.
+
+```diff
+ # zerver/models/realms.py
+
+ class Realm(models.Model):
+     # ...
+     require_unique_names = models.BooleanField(default=False)
+     name_changes_disabled = models.BooleanField(default=False)
+     email_changes_disabled = models.BooleanField(default=False)
+     avatar_changes_disabled = models.BooleanField(default=False)
+
++    # My fantastic feature for the new feature tutorial.
++    my_fantastic_feature = models.BooleanField(default=False)
+```
+
+The Realm model also contains an attribute, `property_types`, which
+other backend functions use to handle most realm settings without any
+custom code for the setting (more on this process below). The attribute
+is a dictionary, where the key is the name of the realm field and the
+value is the field's type. Add the new field to the `property_types`
+dictionary.
+
+```diff
+ # zerver/models/realms.py
+
+ class Realm(models.Model)
+     # ...
+     # Define the types of the various automatically managed properties
+     property_types: dict[str, type | UnionType] = dict(
+         allow_message_editing=bool,
+         avatar_changes_disabled=bool,
+         # ...
++        my_fantastic_feature=bool,
+         # ...
+```
+
+**The majority of realm settings can be included in `property_types`.**
+However, there are some properties that need custom logic and thus cannot
+use this framework. Any properties that define a relationship to another
+model (e.g., a `ForeignKey` field) do not use the `property_types`
+framework, for example `moderation_request_channel` property and the
+`can_add_custom_emoji_group` property.
+
+When creating a realm property that is not a boolean, character or
+integer field, or when adding a field that is dependent on other fields,
+do not add the field to the `property_types` dictionary. The steps below
+will point out where to write additional code for these cases.
+
+### Create the migration
+
+Create the migration file using the Django `makemigrations` command:
+`./manage.py makemigrations`. Make sure to commit the generated file to
+git: `git add zerver/migrations/NNNN_my_fantastic_feature.py` (NNNN is a
+number that is equal to the current number of migrations.)
+
+If you run into problems, the
+[Django migration documentation](https://docs.djangoproject.com/en/5.0/topics/migrations/)
+is helpful.
+
+### Test your migration changes
+
+Apply the migration using Django's `migrate` command, `./manage.py migrate`:
+
+```console
+shell $ ./manage.py migrate
+Operations to perform:
+  Synchronize unmigrated apps: staticfiles, analytics, pipeline
+  Apply all migrations: zilencer, confirmation, sessions, guardian, zerver, sites, auth, contenttypes
+Synchronizing apps without migrations:
+  Creating tables...
+    Running deferred SQL...
+  Installing custom SQL...
+Running migrations:
+  Rendering model states... DONE
+  Applying zerver.NNNN_my_fantastic_feature... OK
+```
+
+Once you've run the migration, [restart the development
+server](../development/remote.md#running-the-development-server).
+
+### Handle database interactions
+
+Next, we will implement the backend part of this feature. Like typical
+apps, we will need our backend to update the database and send some
+response to the client that made the request.
+
+Beyond that, we need to orchestrate notifications about the setting
+change to _other_ clients (or other users, if you will). Clients find out
+about settings through two closely related code paths. When a client
+first contacts the server, the server sends the client its initial state.
+Subsequently, clients subscribe to "events," which can (among other
+things) indicate that settings have changed.
+
+For the backend piece, we will need our action to make a call to
+`send_event_on_commit` to send the event to clients that are active
+(The event is only sent after the current database transaction
+commits, hence the name). We will also need to modify
+`fetch_initial_state_data` so that the new field is passed to
+clients. See [our event system docs](../subsystems/events-system.md)
+for all the gory details.
+
+Getting back to implementation details...
+
+If you are working on a feature that is in the realm `property_types`
+dictionary, you will not need to add code to
+`zerver/actions/realm_settings.py`, but we will describe what the
+process in that file does:
+
+In `zerver/actions/realm_settings.py`, the function `do_set_realm_property`
+takes in the name of a realm property to update and the value it should
+have. This function updates the database and triggers an event to
+notify clients about the change. It uses the field's type, specified
+in the `Realm.property_types` dictionary, to validate the type of the
+value before updating the property; this is primarily an assertion to
+help catch coding mistakes, not to check for bad user input.
+
+After updating the given realm field, `do_set_realm_property` creates
+an 'update' event with the name of the property and the new value. It
+then calls `send_event_on_commit`, passing the event and the list of
+users whose browser sessions should be notified as the second argument.
+The latter argument can be a single user (if the setting is a personal
+one, like time display format), members in a particular channel only or
+all active users in an realm.
+
+:::{tip}
+
+Use `git grep` to find and read the `do_set_realm_property`
+function. Are there `property_types` that are handled differently in
+that function? Which ones? Does the function indicate why?
+
+:::
+
+If the new realm property being added does not fit into the
+`property_types` framework (such as the `moderation_request_channel`
+field), you'll need to create a new function to explicitly update this
+field and send an event. For example:
+
+```python
+# zerver/actions/realm_settings.py
+
+def do_set_realm_stream(
+    realm: Realm,
+    field: Literal[
+        "moderation_request_channel",
+        "new_stream_announcements_stream",
+        "signup_announcements_stream",
+        "zulip_update_announcements_stream",
+    ],
+    stream: Stream | None,
+    stream_id: int,
+    *,
+    acting_user: UserProfile | None,
+) -> None:
+    # We could calculate more of these variables from `field`, but
+    # it's probably more readable to not do so.
+    if field == "moderation_request_channel":
+        old_value = realm.moderation_request_channel_id
+        realm.moderation_request_channel = stream
+        property = "moderation_request_channel_id"
+```
+
+### Update application state
+
+`zerver/lib/events.py` contains code to ensure that your new setting is
+included in the data sent down to clients: both when a new client is
+loaded and when changes happen. This file also automatically handles
+realm settings in the `property_types` dictionary, so you would not need
+to change this file if your setting fits that framework.
+
+The `fetch_initial_state_data` function is responsible for sending data
+when a client is loaded (data added to the `state` here will be available
+both in `page_params` in the browser, as well as to API clients like the
+mobile app). The `apply_event` function in `zerver/lib/events.py` is
+important for making sure the `state` is always correct, even in the
+event of rare race conditions.
+
+```python
+# zerver/lib/events.py
+
+def fetch_initial_state_data(
+    user_profile: UserProfile | None,
+    # ...
+    if want("realm"):
+        # The realm bundle includes both realm properties and server
+        # properties, since it's rare that one would want one and not
+        # the other. We expect most clients to want it.
+        # ...
+        for property_name in Realm.property_types:
+            state["realm_" + property_name] = getattr(realm, property_name)
+
+        for setting_name in Realm.REALM_PERMISSION_GROUP_SETTINGS:
+            setting_group_id = getattr(realm, setting_name + "_id")
+            state["realm_" + setting_name] = get_group_setting_value_for_register_api(
+                setting_group_id, anonymous_group_membership_data_dict
+            )
+        # ...
+
+def apply_event(
+    user_profile: UserProfile,
+    # ...
+) -> None:
+    # ...
+    elif event["type"] == "realm":
+        if event["op"] == "update":
+            field = "realm_" + event["property"]
+            state[field] = event["value"]
+           # ...
+```
+
+If your new realm property fits the `property_types` framework, you don't
+need to change `fetch_initial_state_data` or `apply_event`. However, if
+you are adding a property that is handled separately, you will need to
+explicitly add the property to the `state` dictionary in the
+`fetch_initial_state_data` function.
+
+For our example, one won't need to change `apply_event` since its default
+code for `realm` event types handles this case correctly, but for a
+totally new type of feature, a few lines in that function may be needed.
+
+### Add a new view
+
+You will need to add a view for clients to access that will call the
+`actions.py` code to update the database. This example feature
+adds a new parameter that will be sent to clients when the
+application loads and should be accessible via JavaScript. There is
+already a view that does this for related flags: `update_realm` in
+`zerver/views/realm.py`. So in this case, we can add our code to the
+existing view instead of creating a new one.
+
+You'll need to add a parameter for the new field to the `update_realm`
+function in `zerver/views/realm.py` (and add the appropriate mypy type
+annotation).
+
+```diff
+ # zerver/views/realm.py
+
+@require_realm_admin
+@typed_endpoint
+def update_realm(
+    request: HttpRequest,
+    user_profile: UserProfile,
+     *,
+     allow_message_editing: Json[bool] | None = None,
+     authentication_methods: Json[dict[str, Any]] | None = None,
+     # ...
++    my_fantastic_feature: Json[bool] | None = None,
+     # ...
+ ):
+     # ...
+```
+
+If this feature fits the `property_types` framework and does not
+require additional validation, this is the only change to make to
+`zerver/views/realm.py`.
+
+Text fields or other realm properties that need additional validation
+can be handled at the beginning of `update_realm`.
+
+```python
+# zerver/views/realm.py
+
+    # Additional validation/error checking beyond types go here, so
+    # the entire request can succeed or fail atomically.
+    if default_language is not None and default_language not in get_available_language_codes():
+        raise JsonableError(_("Invalid language '{language}'").format(language=default_language))
+# ...
+```
+
+The code in `update_realm` loops through the `property_types` dictionary
+and calls `do_set_realm_property` on any property to be updated from
+the request.
+
+If the new feature is not in `property_types`, you will need to write code
+to call the function you wrote in `actions.py` that updates the database
+with the new value. E.g., for `moderation_request_channel`, we created
+`do_set_realm_moderation_request_channel`, which we will call here:
+
+```python
+# zerver/views/realm.py
+
+# import do_set_realm_authentication_methods from actions.py
+from zerver.actions.realm_settings import (
+    # ...
+    do_set_realm_moderation_request_channel,
+    # ...
+)
+# ...
+# ...
+    # Channel-valued settings are not yet fully supported by the
+    # property_types framework, and thus have explicit blocks here.
+    if moderation_request_channel_id is not None and (
+        realm.moderation_request_channel is None
+        or realm.moderation_request_channel.id != moderation_request_channel_id
+    ):
+        new_moderation_request_channel_id = None
+        if moderation_request_channel_id >= 0:
+            (new_moderation_request_channel_id, sub) = access_stream_by_id(
+                user_profile, moderation_request_channel_id, require_content_access=False
+            )
+        do_set_realm_moderation_request_channel(
+            realm,
+            new_moderation_request_channel_id,
+            moderation_request_channel_id,
+            acting_user=user_profile,
+        )
+        data["moderation_request_channel_id"] = moderation_request_channel_id
+# ...
+```
+
+This completes the backend implementation. A great next step is to
+write automated backend tests for your new feature.
+
+### Backend tests
+
+To test the new setting syncs correctly with the `property_types`
+framework, one usually just needs to add a line in each of
+`zerver/tests/test_events.py` and `zerver/tests/test_realm.py` with a
+list of values to switch between in the test. In the case of a boolean
+field, no action is required, because those tests will correctly assume
+that the only values to test are `True` and `False`.
+
+In `test_events.py`, the function that runs tests for the `property_types`
+framework is `do_set_realm_property_test`, and in `test_realm.py`, it is
+`do_test_realm_update_api`.
+
+One would still need to add tests for whether the setting actually
+controls the feature it is supposed to control. At this time, our example,
+`my_fantastic_feature`, does not change any behavior on the backend, so
+we don't need to add any additional tests. If you're using this tutorial
+as a guide for creating an actual new feature, then at this point you
+should add any necessary backend tests.
+
+Visit Zulip's [Django testing](../testing/testing-with-django.md)
+documentation to learn more about the backend testing framework.
+
+:::{tip}
+
+Pick one of the other `Realm.property_types` settings and
+use `git grep` to find examples of backend tests that were written for
+that setting: `git grep REALM_SETTING zerver/tests/`.
+
+:::
+
+Also note that you may already need to update the API documentation for
+your new feature to pass new or existing backend tests at this point.
+The tutorial for [writing REST API endpoints](../documentation/api.md)
+can be a helpful resource, especially the section on [debugging schema
+validation errors](../documentation//api.md#debugging-schema-validation-errors).
+
+### What is my fantastic feature?
+
+Since you've now set up the backend for your new fantastic feature, let's
+take a moment to discuss what that feature actually will be. Here's where
+you get to be creative!
+
+Think of some fun, visual change that you could make to the Zulip web app
+in your development environment. Maybe toggling `my_fantastic_feature` will
+change the app font to [Dingbats](https://en.wikipedia.org/wiki/Dingbat).
+Maybe it will change the color of the font in the message feed. Maybe it
+will make all user status emoji render upside down 🙃. That's up to you!
+
+The next part of this tutorial will go through the process of setting up
+the toggle for turning your feature on and off via the organization
+settings admin modal in the web-app. After that, your final challenge will
+be to make the changes to the codebase to make your fantastic idea into a
+reality in your development environment.
+
+### Update the frontend
+
+After completing the process of adding a new realm setting on the backend,
+you'll want make the required frontend changes.
+
+Since we've updated the data sent for a realm to include a new property,
+we need to update the `realm_schema` on the frontend to match our changes
+in `zerver/lib/events.py`. This can be done in `web/src/state_data.ts`:
+
+```diff
+ // web/src/state_data.ts
+
+ // Sync this with zerver.lib.events.do_events_register.
+ export const realm_schema = z.object({
+     // ...
+     realm_move_messages_within_stream_limit_seconds: z.nullable(z.number()),
++    realm_my_fantastic_feature: z.boolean(),
+     realm_name_changes_disabled: z.boolean(),
+     // ...
+```
+
+In order for an admin user to enable and disable our new realm setting, a
+checkbox needs to be added to the organization settings page (and its
+value added to the data sent back to server when a realm is updated) and
+the change event needs to be handled on the client.
+
+To add the checkbox to the organization settings admin modal, you'll need
+to modify one of the templates for the organization settings modal:
+`web/templates/settings/organization...`. If you're adding a non-checkbox
+field, you'll need to specify the type of the field via the
+`data-setting-widget-type` attribute in the Handlebars template.
+
+For simplicity, let's add the checkbox for `realm_my_fantastic_feature` to
+the "Message feed" section in `web/templates/settings/organization_settings_admin.hbs`.
+Since there are a number of realm settings controlled via checkbox, there
+is a `settings_checkbox` partial template already set up for you to reuse.
+
+:::{tip}
+
+Use `git grep` to see where the other realm settings in the
+"Message feed" section of that template are in the `web/` directory.
+
+Also, if you're not familiar with the organization settings modal, log in
+as Iago in your development environment and follow the Zulip help center
+instructions for [customizing organization settings](https://zulip.com/help/customize-organization-settings).
+
+:::
+
+Then add the new form control in `web/src/admin.ts`.
+
+```diff
+ // web/src/admin.ts
+
+ export function build_page() {
+     const options = {
+        custom_profile_field_types: realm.custom_profile_field_types,
+        full_name: current_user.full_name,
+         // ...
++        realm_my_fantastic_feature: realm.realm_my_fantastic_feature,
+         // ...
+```
+
+You'll also want to add a label to the `admin_settings_label` object
+in that same file. Note that these labels are marked for
+[translation](../translating/internationalization.md).
+
+Finally, update `server_events_dispatch.js` to handle related events
+coming from the server. There is an object, `realm_settings`, in the
+function `dispatch_normal_event`. The keys in this object are setting
+names and the values are the UI updating functions to run when an event
+has occurred.
+
+If there is no relevant UI change to make other than in settings page
+itself, the value should be `noop`. However, if you are writing a
+function to update the UI after a given setting has changed, your
+function should be referenced in the `realm_settings` of
+`server_events_dispatch.js`. See for example
+`settings_emoji.update_custom_emoji_ui`.
+
+```diff
+ // web/src/server_events_dispatch.js
+
+ function dispatch_normal_event(event) {
+     switch (event.type) {
+     // ...
+     case 'realm':
+         var realm_settings = {
+             add_custom_emoji_policy: settings_emoji.update_custom_emoji_ui,
+             allow_edit_history: noop,
+             // ...
++            my_fantastic_feature: ???,
+             // ...
+         };
+```
+
+Checkboxes and other common input elements handle the UI updates
+automatically through the logic in `settings_org.sync_realm_settings`.
+
+The rest of the `dispatch_normal_events` function updates the state of the
+application if an update event has occurred on a realm property and runs
+the associated function to update the application's UI, if necessary.
+
+Now that you've got the admin checkbox set up to toggle your new feature
+on and off, you should work on implementing your fantastic idea! Remember
+to use `git grep` to help as you become familiar with the codebase.
+
+### Manually test
+
+Here are few important cases you should consider when manually testing
+your changes:
+
+- For organization settings where we have a "save/discard" model, make
+  sure both the "Save" and "Discard changes" buttons are working
+  properly when toggling the realm setting.
+
+- If your setting is dependent on another setting, carefully check
+  that both are properly synchronized. For example, the input element
+  for `realm_waiting_period_threshold_custom_input` is shown only when
+  we have selected the custom time limit option in the
+  `realm_waiting_period_threshold` dropdown.
+
+- Do some manual testing for the real-time synchronization of input
+  elements across the browsers and just like "Discard changes" button,
+  check whether dependent settings are synchronized properly (this is
+  easy to do by opening two browser windows to the settings page, and
+  making changes in one while watching the other).
+
+- Each subsection has independent "Save" and "Discard changes"
+  buttons, so changes and saving in one subsection shouldn't affect
+  the others.
+
+### Frontend tests
+
+A great next step is to write frontend tests. There are two types of
+frontend tests: [node-based unit tests](../testing/testing-with-node.md) and
+[Puppeteer end-to-end tests](../testing/testing-with-puppeteer.md).
+
+At the minimum, you'll want to update `web/tests/lib/example_realm.cjs`
+for the new realm setting that you've added.
+
+Beyond that, you should add any applicable tests that verify the
+behavior of the setting you just created.
+
+:::{tip}
+
+If you've identified a similar feature on the frontend to the
+one that you're creating, use your `git grep` skills to find and read
+through the frontend tests for that feature.
+
+:::
+
+### Update documentation
+
+If you're using this tutorial as a guide to add an actual new feature to
+Zulip, it's really important to make sure that your new feature is well
+documented.
+
+A recommended way to document a new feature is to update and/or augment
+Zulip's existing [help center documentation](https://zulip.com/help/)
+to reflect your changes and additions.
+
+At the very least, this will involve modifying (or adding) a Markdown
+file documenting the feature in the `starlight_help/` directory of the
+main Zulip server repository, where the source for Zulip's end user
+documentation is stored. Details about writing, editing and testing these
+files can be found in:
+[Writing help center articles](../documentation/helpcenter.md).
+
+Also, new features will often impact Zulip's REST API documentation,
+which is found in `zerver/openapi/zulip.yaml`. You may have noticed
+this during the testing process as the Zulip test suite should fail if
+there is a change to the API without a corresponding update to the
+documentation.
+
+The best way to understand writing and updating Zulip's API
+documentation is to read more about Zulip's
+[REST API documentation process](../documentation/api.md)
+and [OpenAPI configuration](../documentation/openapi.md).
+
+In particular, if there is an API change, you should use the
+`create-api-changelog` tool to create a file for the API changelog
+entry for your new feature. The API feature level allows the
+developers of mobile clients and other tools using the Zulip API to
+programmatically determine whether the Zulip server they are
+interacting with supports a given feature; see the
+[Zulip release lifecycle](../overview/release-lifecycle.md).
+
+### Share your fantastic feature
+
+If you're using this tutorial to learn about the codebase and have
+created a fun, fantastic feature, share a screenshot and/or a GIF
+from your development environment of your unique changes to the
+web-app UI in the Zulip development community's
+[new members](https://chat.zulip.org/#topics/channel/95-new-members)
+channel!

docs/tutorials/reading-list.md

@@ -0,0 +1,138 @@
+# Reading list
+
+Here you can find a list of relevant resources that you may find useful for
+learning new skills, or polishing the ones you already have.
+
+The list covers a wide variety of topics, from basic Python coding to
+general development guidelines.
+
+Feel free to create a pull request in Zulip's [GitHub repository](https://github.com/zulip/zulip/) with any interesting books, articles or
+videos you would like to see in this list.
+
+Some titles have been shortened for organizational purposes.
+
+## General programming/IT
+
+_Book_ - [Clean Code: A Handbook of Agile Software Craftsmanship](https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882) _(Not free!)_
+
+_Books_ - [Free programming books list](https://github.com/vhf/free-programming-books)
+
+_Blog_ - [Free Code Camp blog](https://medium.freecodecamp.com)
+
+_Blog_ - [Idle Words talks transcripts](https://idlewords.com/talks)
+
+_Tutorial_ - [HTTP Can Do That?!, by Sumana Harihareswara (PyCon 2016)](https://youtu.be/HsLrXt2l-kg)
+
+_Video_ - [Minimum Viable Documentation, by Matthew Lyon (WriteTheDocs 2014)](https://youtu.be/bEZcodengwk)
+
+_Video_ - [NoOps, by Kelsey Hightower (DepOpsDays 2016)](https://youtu.be/ajT90pC3ris)
+
+_Video_ - [The mind behind Linux (TED interview)](https://youtu.be/o8NPllzkFhE)
+
+_Tutorial_ - [Learn code the hard way](https://learncodethehardway.org)
+
+_Tutorial_ - [What happens when...](https://github.com/alex/what-happens-when)
+
+_Article_ - [An Interview With Linus Torvalds](https://techcrunch.com/2012/04/19/an-interview-with-millenium-technology-prize-finalist-linus-torvalds)
+
+_Article_ - [Effective Learning Strategies for Programmers](https://akaptur.com/blog/2015/10/10/effective-learning-strategies-for-programmers/)
+
+_Article_ - [Readme Driven Development](https://tom.preston-werner.com/2010/08/23/readme-driven-development.html)
+
+_Article_ - [Systematic Debugging](https://akaptur.com/blog/2013/07/24/systematic-debugging)
+
+_Paper_ - [Floating-Point Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html)
+
+## Python
+
+_Video_ - [Intro to Python for beginners, by Jessica McKellar (PyCon 2013)](https://youtu.be/rkx5_MRAV3A)
+
+_Video_ - [Breaking the rules, by Jessica McKellar (PyCon Sweden)](https://youtu.be/C0fnHhY9UOc)
+
+_Video_ - [Build & break a Python sandbox, by Jessica McKellar (PyCon 2014)](https://pyvideo.org/pycon-us-2014/building-and-breaking-a-python-sandbox.html)
+
+_Video_ - [Cache me if you can, by Guillaume Ardaud (PyCon 2014)](https://pyvideo.org/pycon-us-2014/cache-me-if-you-can-memcached-caching-patterns.html)
+
+_Video_ - [Loop like a native, by Ned Batchelder (PyCon 2013)](https://youtu.be/EnSu9hHGq5o)
+
+_Video_ - [Modern Dictionaries, by Raymond Hettinger (SF Python)](https://youtu.be/p33CVV29OG8)
+
+_Video_ - [Python Language, by Guido van Rossum (PyCon 2016)](https://youtu.be/YgtL4S7Hrwo)
+
+_Video_ - [The Mighty Dictionary, by Brandon Rhodes (PyCon 2010)](https://pyvideo.org/pycon-us-2010/the-mighty-dictionary-55.html)
+
+_Article_ - [Static types in Python, oh my(py)!](https://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy)
+
+_Guide_ - [The Hitchhiker’s Guide to Python!](https://docs.python-guide.org/)
+
+## Java/Android
+
+_Course_ - [Android Development for Beginners](https://www.udacity.com/course/android-development-for-beginners--ud837)
+
+_Blog_ - [Java Tutorials for Beginners](https://www.geeksforgeeks.org/java/)
+
+## JavaScript/ECMAScript
+
+_Tutorial_ - [clean-code-javascript Software engineering principles](https://github.com/ryanmcdermott/clean-code-javascript)
+
+_Course_ - [React native and redux course](https://www.udemy.com/course/the-complete-react-native-and-redux-course/) (_Not free!_)
+
+_Video_ - [TypeScript vs. CoffeeScript vs. ES6](https://www.youtube.com/watch?v=Ae4h9GC9cCg)
+
+## TypeScript
+
+_Tutorial_ - [TypeScript handbook section on base types](https://www.typescriptlang.org/docs/handbook/basic-types.html)
+
+_Book_ - [TypeScript Deep Dive](https://basarat.gitbooks.io/typescript/)
+
+_Guide_ - [TypeScript Declaration Files Introduction](https://www.typescriptlang.org/docs/handbook/declaration-files/introduction.html)
+
+## Git/version control systems (VCS)
+
+You may want to take a look first at our [Git and GitHub guide](../git/index.md).
+
+## Computer science/algorithms
+
+_Blog_ - [GeeksforGeeks](https://www.geeksforgeeks.org)
+
+_Book_ [Introduction to Algorithms](https://mitpress.mit.edu/books/introduction-algorithms) (_Not free!_)
+
+_Blog_ - [Setosa data visualization and visual explanations](https://setosa.io)
+
+_Course_ - [Algorithms, Part I](https://www.coursera.org/learn/algorithms-part1)
+
+_Course_ - [Open Source Society University](https://ossu.firebaseapp.com)
+
+_Course_ - [MIT CSAIL 6.828: Operative Systems Engineering](https://pdos.csail.mit.edu/6.828/2016)
+
+## Community experience
+
+_Book_ - [Producing Open Source Software](https://producingoss.com/en/)
+
+_Article_ - [Advice on Starting And Running A New Open Source Project](https://www.harihareswara.net/sumana/2016/08/04/1)
+
+_Article_ - [How to ask good questions](https://jvns.ca/blog/good-questions)
+
+_Article_ - [Notes for New FLOSS Contributors](https://www.harihareswara.net/sumana/2016/10/12/0)
+
+_Article_ - [To be mentored](https://trueskawka.github.io/zulip/outreachy/blog/2017/01/02/to-be-mentored.html)
+
+_Article_ - [To mentor](https://trueskawka.github.io/zulip/outreachy/gci/blog/2017/01/03/to-mentor.html)
+
+[List of good projects for new contributors](https://github.com/MunGell/awesome-for-beginners)
+
+## Competitions/camps
+
+[CodeForces](https://codeforces.com)
+
+[Free Code Camp](https://www.freecodecamp.com)
+
+## Massive open online courses (MOOC) platforms
+
+[Coursera](https://www.coursera.org)
+
+[edX](https://www.edx.org)
+
+[MIT OpenCourseWare](https://ocw.mit.edu)
+
+[Udacity](https://www.udacity.com)

docs/tutorials/shell-tips.md

@@ -0,0 +1,350 @@
+# Shell tips
+
+The _shell_ is a **command line interpreter**. To use it you can open a
+_terminal_ (sometimes called a _console_). This is how most terminal windows
+look like:
+
+![An example shell window](../images/shell-screenshot.png)
+
+If you haven't used it before, you should probably take a look at
+[this tutorial](http://linuxcommand.org/lc3_learning_the_shell.php).
+
+If you're using Windows,
+[these videos](https://www.youtube.com/playlist?list=PL6gx4Cwl9DGDV6SnbINlVUd0o2xT4JbMu)
+may be useful too, but keep in mind that the following tips only apply to
+Linux/macOS environments (Unix shells). You can also use a tool, for example
+[Cygwin](https://www.cygwin.com/), to have a Unix-like shell on Windows.
+
+## The prompt (`$`)
+
+When searching Google, or Zulip's docs, you'll find commands that begin
+with a dollar sign `$` or a dollar sign preceded by some text
+(e.g., `(venv)john@laptop:~$`).
+
+This is called the **prompt**, and it's only an indicator that the shell is
+awaiting new orders. The prompt can contain useful information, let's look
+at `(venv)john@laptop:~$`:
+
+- `(venv)` informs the user that they're currently in a virtual environment
+  (more on [Python virtual
+  environments](https://docs.python-guide.org/dev/virtualenvs/))
+- the `john` before `@` is the username
+- the `laptop` is the host machine name
+- the `~` after the colon informs the user they're currently in the home
+  folder of the user `john`
+
+You shouldn't type the prompt or the text preceding it, since it isn't a
+part of the commands.
+
+## Tilde character (`~`)
+
+It's very frequent to see the tilde (`~`) in paths. The tilde is an
+abbreviation for your home directory (`/home/YOUR_USERNAME` most of the times).
+
+That's why the following is exactly the same, if the user running it is
+`john`:
+
+```console
+$ cd ~
+$ cd /home/john
+```
+
+## Change directory (`cd`)
+
+When you're using the shell, you work inside a directory (the one specified in
+the prompt). This way you can point to files relative to your current
+directory, instead of writing the whole path.
+
+Imagine you have a file called `ideas.txt` inside `/home/john/notes/`, and
+you want to edit it using `nano`. You could use:
+
+```console
+$ nano /home/john/notes/ideas.txt
+```
+
+However, that isn't very practical, especially if you are working with
+longer paths.
+
+That's why it's very useful to change the path where you are currently
+located (usually known as **working directory**). To do that, you use `cd`
+(**c**hange **d**irectory):
+
+```console
+$ cd /home/john/notes/
+~/notes$ nano ideas.txt
+```
+
+Or, if you're the user `john`: `cd ~/notes`.
+
+You can now access to all the files inside `/home/john/notes` directly, without
+needing to type the whole path.
+
+Relative paths make it much easier to move through files and
+directories, too.
+
+## Running commands as root (`sudo`)
+
+You may have noticed that many commands begin with `sudo`. This informs the
+shell that the following command must be run as the root - a user that by
+default has access to all commands and files on a Unix operating system (i.e.
+a user with administrator privileges). That's why you may be asked for a
+password in those cases: the system verifies you have permission to act as
+the _root_ user.
+
+In case you were wondering, the name `sudo` comes from **s**uper **u**ser
+**do**.
+
+## Escaping characters
+
+Some characters cannot be used directly in the shell, because they have a
+special meaning. Consider the following example:
+
+```console
+$ echo "He said hello"
+He said hello
+```
+
+What if you wanted to display double quotes? You can't use
+`echo "He said "hello""`, because in that case you're using the
+double quotes for two different purposes:
+
+- Delimiting the string you want to use, from `He` to `"hello"`.
+- Quoting something, by literally printing `"`.
+
+You have to specify which double quotes are used in each case. When you want
+one of those "special characters" to be literally printed, that's called
+**character escaping**. To escape a character, simply add a backslash (`\`)
+before it.
+
+Returning to our example:
+
+```console
+$ echo "He said \"hello\""
+He said "hello"
+```
+
+As you can see, the double quotes with the backslash are shown, but the ones
+without it are used as string delimiters.
+
+Double quotes aren't the only case of special characters. Some others are `$`,
+`#`, `{` or `}`, but there are many more. The backslash itself can be escaped
+as well, using the same procedure: `\\`.
+
+## Sequencing commands
+
+It's also possible to run multiple commands in a single line. For that purpose,
+the shell provides two different separators:
+
+- **Semicolon `;`**: runs a command, and once it has finished, runs the next
+  one:
+
+  ```console
+  $ echo "Hello"; echo "World!"
+  Hello
+  World!
+  ```
+
+- **Double ampersand `&&`**: runs a command, and **only if** it finished
+  without errors, it proceeds with the next one:
+
+  ```console
+  $ qwfvijwe && echo "Hello"
+  qwfvijwe: command not found
+  ```
+
+  Notice that it doesn't print `Hello` at the end, because the previous
+  command (`qwfvijwe`) returned an error.
+
+  When using an incorrect command with a semicolon, the `Hello` will still
+  be printed:
+
+  ```console
+  $ qwfvijwe; echo "Hello"
+  qwfvijwe: command not found
+  Hello
+  ```
+
+## Splitting commands into multiple lines
+
+Sometimes you end up with a very long command, that is hard to read and may
+be unclear. This is a problem, especially if you want to share that command,
+e.g., in a documentation file.
+
+In those cases, you can use a backslash at the end of each line, to inform the
+shell "wait, there's more on the next line".
+
+This is an example, taken from the docs on how to install the Zulip development
+environment:
+
+```bash
+sudo apt-get -y purge vagrant && \
+curl -fLO https://releases.hashicorp.com/vagrant/2.0.2/vagrant_2.0.2_x86_64.deb && \
+sudo dpkg -i vagrant*.deb && \
+sudo apt-get -y install build-essential git ruby lxc lxc-templates cgroup-lite redir && \
+vagrant plugin install vagrant-lxc && \
+vagrant lxc sudoers
+```
+
+It's all a single command, joined using the double ampersand explained in
+[Sequencing commands](#sequencing-commands). If you're typing it manually,
+you don't need to include the backslashes, just write it all on the same line,
+and hit <kbd>ENTER</kbd>/<kbd>RETURN</kbd> at the end.
+
+If you think about it, what is happening here is actually another case of
+character escaping. The newline character (the one that appears when you hit
+<kbd>ENTER</kbd>) usually means "read this command". However, here we want to
+literally have the newline character, and thus the `\<newline>`.
+
+The newline character is invisible (we only see a line break), but it's still
+there!
+
+## Arguments
+
+Most commands need additional data to work, like a path or a file. That extra
+information is called an **argument**, and it's specified after the name of the
+command, like this:
+
+```console
+$ cd /home/john/notes
+```
+
+Here, the command is `cd`, and the first (and only) argument is
+`/home/john/notes`:
+
+- `cd` - _command_: changes your current directory.
+
+- `/home/john/notes` - _argument_: the directory where you want to go.
+
+In each command the arguments are specified in different ways, and have
+different meanings.
+
+Sometimes, a command can accept arguments indicated with dashes. Here's another
+example of arguments usage:
+
+```console
+$ nano -C /home/john/backups --mouse todo.txt
+```
+
+As you can see, some arguments imply that more information has to be specified,
+while others don't.
+
+In this case, we're saying: "Bash, use the app `nano` to open the file
+`todo.txt`, enabling mouse support, and saving the backup files to
+`/home/john/backups`". The different parts are:
+
+- `nano` - _command_: program that allows editing text easily.
+
+- `-C` - _argument_: needs you to indicate where the backups should be stored,
+  and thus you have to add an additional argument after it, to specify the
+  directory (`/home/john/backups` in the example).
+
+- `--mouse` - _argument_: is just an option you set, `nano` doesn't need
+  anything else to make it work. Thus, there isn't any extra argument for that.
+
+Note that the `todo.txt` is the file we want to open! It has nothing to do with
+the previous argument. This will probably clarify it (taken from `nano`'s
+help):
+
+```console
+Usage: nano [OPTIONS] [FILE]...
+```
+
+So, in the options you indicate the arguments, and `FILE` is... well, the file.
+
+Don't worry, you don't have to memorize the meaning of
+all the arguments for every single command. There are
+[tools](#understanding-commands) that help you with that :wink:.
+
+## Shebang
+
+You can run some files directly, without specifying a program to interpret
+them.
+
+That's why you may have seen cases, in the Zulip codebase or
+elsewhere, when some Python scripts are called with `python`:
+
+```console
+$ python my_program.py
+```
+
+While other times, `python` isn't used:
+
+```console
+$ ./my_program.py
+```
+
+In the latter case, the operating system will look at the beginning of
+the script to identify what interpreter (like `python` in this
+example) to run it with.
+
+The note telling the OS how to interpret the file goes on the file's
+very first line, and it's called a **shebang**. In our Python scripts,
+it looks like this:
+
+```python
+#!/usr/bin/env python3
+```
+
+With this, you're telling the operating system: "if I tell you to run
+this, ask `/usr/bin/env python3` how to understand it".
+
+The result is that the operating system is asked to run the script, it
+runs the command specified in the shebang, with the script filename
+added as a command-line argument. So, returning to our example with
+`my_program.py`, when you run `./my_program.py`, what happens under
+the hood is equivalent to:
+
+```console
+$ /usr/bin/env python3 ./my_program.py
+```
+
+The purpose of `/usr/bin/env` in our shebangs is as a way to locate
+the `python3` program in your current environment, the same one the
+shell would use if you ran `python3 my_program.py`. You may see
+Python scripts outside of Zulip with a shebang like
+`#!/usr/bin/python3`; but because of the way Python virtualenvs work,
+this has the effect of running the script outside of any currently
+activated virtualenv. We use `/usr/bin/env` to keep our scripts
+running inside the virtualenv where we've installed all our
+dependencies.
+
+Another example of a shebang is the one used in Bash scripts. In those
+cases, `#!/bin/bash` or `#!/bin/sh` is used.
+
+## Understanding commands
+
+Frequently, you may find commands that you don't understand, or don't
+know what they do. You can use `man <command>` to see the **man**ual page for
+that specific command. Also, you may find useful
+[explainshell](https://explainshell.com/), a webpage that explains what most
+commands do, part by part.
+Finally, [docopt](http://docopt.org/) can help you understand the syntax used
+by command-line tools to describe their interface, and make sense of strings like
+`cd: usage: cd [-L|[-P [-e]] [-@]] [dir]`).
+
+## Cheatsheet
+
+There are many more commands in the shell, besides the ones explained in this
+file.
+[Here](https://www.git-tower.com/blog/command-line-cheat-sheet/) you can find
+a simple yet useful cheatsheet, created by Tower, that could help you
+understand and remember what other common commands do (e.g., `ls`).
+
+## Git
+
+Probably at this point you've heard about Git. It's basically a tool that most
+developers use to manage all the changes in their code.
+
+At first it seems like magic, but once you get the basic concepts you find it
+extremely useful and even easy to use (at least the 99% of the time).
+
+To learn more about how to use it, read
+[our docs](../git/index.md) on Git and
+GitHub.
+
+[This cheatsheet][git-cheat-detailed]
+will be useful in your journey, as well.
+
+![Git - XKCD 1597](https://imgs.xkcd.com/comics/git.png)
+
+[git-cheat-detailed]: ../git/cheat-sheet.md#detailed-cheat-sheet

docs/tutorials/writing-views.md

@@ -0,0 +1,415 @@
+# Writing views in Zulip
+
+## What this covers
+
+This page documents how views work in Zulip. You may want to read the
+[new feature tutorial](new-feature-tutorial.md)
+and treat this as a reference.
+
+If you have experience with Django, much of this will be familiar, but
+you may want to read about how REST requests are dispatched, and how
+request authentication works.
+
+This document supplements the [new feature tutorial](new-feature-tutorial.md)
+and the [testing](../testing/testing.md)
+documentation.
+
+## What is a view?
+
+A view in Zulip is everything that helps implement a server endpoint.
+Every path that the Zulip server supports (doesn't show a 404 page
+for) is a view. The obvious ones are those you can visit in your
+browser, for example
+[/integrations](https://zulip.com/integrations/), which shows the
+integration documentation. These paths show up in the address bar of
+the browser. There are other views that are only seen by software,
+namely the API views. They are used to build the various clients that
+Zulip has, namely the web client (which is also used by the desktop
+client) and the mobile clients.
+
+## Modifying urls.py
+
+A view is anything with an entry in the appropriate urls.py, usually
+`zproject/urls.py`. Zulip views either serve HTML (pages for browsers)
+or JSON (data for Zulip clients on all platforms, custom bots, and
+integrations).
+
+The format of the URL patterns in Django is [documented
+here](https://docs.djangoproject.com/en/5.0/topics/http/urls/), and
+the Zulip specific details for these are discussed in detail in the
+[life of a request doc](life-of-a-request.md#options).
+
+We have two Zulip-specific conventions we use for internationalization and for
+our REST API, respectively.
+
+## Writing human-readable views
+
+If you're writing a new page for the website, make sure to add it
+to `i18n_urls` in `zproject/urls.py`
+
+```diff
+     i18n_urls = [
+     ...
++    path('quote-of-the-day', TemplateView.as_view(template_name='zerver/qotd.html')),
++    path('postcards', 'zerver.views.postcards'),
+]
+```
+
+As an example, if a request comes in for Spanish, language code `es`,
+the server path will be something like: `es/features/`.
+
+### Decorators used for webpage views
+
+This section documents a few simple decorators that we use for webpage
+views, as an introduction to view decorators.
+
+`require_post`:
+
+```py
+
+@require_post
+def accounts_register(request: HttpRequest) -> HttpResponse:
+```
+
+This decorator ensures that the request was a POST--here, we're
+checking that the registration submission page is requested with a
+post, and inside the function, we'll check the form data. If you
+request this page with GET, you'll get an HTTP 405 METHOD NOT ALLOWED
+error.
+
+`zulip_login_required`:
+
+This decorator verifies that the browser is logged in (i.e. has a
+valid session cookie) before providing the view for this route, or
+redirects the browser to a login page. This is used in the root path
+(`/`) of the website for the web client. If a request comes from a
+browser without a valid session cookie, they are redirected to a login
+page. It is a small fork of Django's
+[login_required][login-required-link], adding a few extra checks
+specific to Zulip.
+
+```py
+@zulip_login_required
+def home(request: HttpRequest) -> HttpResponse:
+```
+
+[login-required-link]: https://docs.djangoproject.com/en/5.0/topics/auth/default/#django.contrib.auth.decorators.login_required
+
+### Writing a template
+
+Templates for the main website are found in
+[templates/zerver/app](https://github.com/zulip/zulip/tree/main/templates/zerver/app).
+
+## Writing API REST endpoints
+
+These are code-parsable views that take x-www-form-urlencoded or JSON
+request bodies, and return JSON-string responses. Almost all Zulip
+view code is in the implementations of API REST endpoints.
+
+The REST API does authentication of the user through `rest_dispatch`,
+which is documented in detail at
+[zerver/lib/rest.py](https://github.com/zulip/zulip/blob/main/zerver/lib/rest.py).
+This method will authenticate the user either through a session token
+from a cookie on the browser, or from a base64 encoded `email:api-key`
+string given via HTTP basic auth for API clients.
+
+```py
+>>> import requests
+>>> r = requests.get('https://api.github.com/user', auth=('hello@example.com', '0123456789abcdeFGHIJKLmnopQRSTUV'))
+>>> r.status_code
+-> 200
+```
+
+### Request variables
+
+Most API views will have some arguments that are passed as part of the
+request to control the behavior of the view. In any well-engineered
+view, you need to write code to parse and validate that the arguments
+exist and have the correct form. For many applications, this leads to
+one of several bad outcomes:
+
+- The code isn't written, so arguments aren't validated, leading to
+  bugs and confusing error messages for users of the API.
+- Every function starts with a long list of semi-redundant validation
+  code, usually with highly inconsistent error messages.
+- Every view function comes with another function that does the
+  validation that has the problems from the last bullet point.
+
+In Zulip, we solve this problem with a special decorator called
+`typed_endpoint` which allows a developer to declare the
+arguments a view function takes and validate their types all within
+the `def` line of the function. This framework uses
+[Pydantic V2](https://docs.pydantic.dev/dev/) to perform data validation
+and parsing for the view arguments. We like this framework because we
+have found it makes the validation code compact, readable, and
+conveniently located in the same place as the method it is validating
+arguments for.
+
+Here's an example:
+
+```py
+from zerver.decorator import require_realm_admin
+from zerver.lib.typed_endpoint import typed_endpoint
+
+@require_realm_admin
+@typed_endpoint
+def create_user_backend(
+        request: HttpRequest,
+        user_profile: UserProfile,
+        *,
+        email: str,
+        password: str,
+        full_name: str,
+    ):
+    # ... code here
+```
+
+The `typed_endpoint` decorator parses the declared
+[keyword-only arguments](https://docs.python.org/3/glossary.html#term-parameter)
+of the decorated function, and for each argument that has been declared,
+it extracts the HTTP parameter with that name from the request,
+parses it according to the type annotation, and then passes it to
+the function. It will return an nicely JSON formatted HTTP 400 error
+in the event that an argument is missing, doesn't parse as JSON, or
+otherwise is invalid.
+
+`require_realm_admin` is another decorator which checks the
+authorization of the given `user_profile` to make sure it belongs to a
+realm administrator (and thus has permission to create a user); we
+show it here primarily to show how `typed_endpoint` should be
+the inner decorator.
+
+The implementation of `typed_endpoint` is documented in detail
+in
+[zerver/lib/typed_endpoint.py](https://github.com/zulip/zulip/blob/main/zerver/lib/typed_endpoint.py)
+
+Pydantic also helps us with request variable validation. For example:
+
+- `msg_ids: Json[list[int]]` will check that the `msg_ids`
+  HTTP parameter is a list of integers, marshalled as JSON,
+  and pass it into the function as the `msg_ids` Python
+  keyword argument.
+
+- `streams_raw: Annotated[Json[list[str]], ApiParamConfig("subscriptions")]`
+  will check that the "subscriptions" HTTP parameter is a list of
+  strings, marshalled as JSON, and pass it into the function with the
+  Python keyword argument `streams_raw`.
+
+- `message_id: Json[NonNegativeInt]` will check that the `message_id`
+  HTTP parameter is a string containing a JSON encoded non-negative
+  integer.
+
+[Annotated](https://docs.python.org/3/library/typing.html#typing.Annotated)
+can be used in combination with
+[Pydantic's validators](https://docs.pydantic.dev/latest/api/functional_validators/)
+to provide additional validation for the arguments.
+
+- `name: Annotated[str, StringConstraints(max_length=60)]` will check that the
+  `name` HTTP parameter is a string containing up to 60 characters.
+
+- Since there is no need to JSON-encode strings
+  (lists, integers, bools and complex objects require JSON encoding), usually simply
+  `my_string: str` is correct. One can pass, for example,
+  `Annotated[str, check_string_in_validator(...)]` where one wants to run a
+  validator on the value of a string.
+
+Default values can be specified for optional arguments similar to how we would specify
+default values in regular python function.
+
+- `is_default_stream: Json[bool] = False` will assign False to the `is_default_stream` argument
+  if no value is specified when making a request to the endpoint.
+
+- We can use `None` as the default value for optional arguments when we don't
+  want to specify any specific default value, for example,
+  `narrow: Json[list[NarrowParameter]] | None = None`. This does not allow the
+  caller to pass `None` as the value, the only way `narrow` can be set to `None` is
+  by using the default value.
+
+[Pydantic models](https://docs.pydantic.dev/latest/concepts/models/) can be used to
+define the schema of complex objects that can be passed to the endpoint.
+
+Here's an example:
+
+```py
+from typing import Annotated
+
+from pydantic import BaseModel, StringConstraints, model_validator
+
+class AddSubscriptionData(BaseModel):
+    name: str
+    color: str | None = None
+    description: (
+        Annotated[str, StringConstraints(max_length=Stream.MAX_DESCRIPTION_LENGTH)] | None
+    ) = None
+
+    @model_validator(mode="after")
+    def validate_terms(self) -> "AddSubscriptionData":
+        # ... Validation logic here
+        return self
+```
+
+- `add: Json[list[AddSubscriptionData]]` will require the `add` argument to be a list of objects
+  having the keys that are specified in the `AddSubscriptionData` model.
+
+- `@model_validator` can be used to specify additional validation logic for the model.
+
+See
+[zerver/lib/typed_endpoint_validators.py](https://github.com/zulip/zulip/blob/main/zerver/lib/typed_endpoint_validators.py)
+for more validators and their documentation.
+
+### Deciding which HTTP verb to use
+
+When writing a new API view, you should writing a view to do just one
+type of thing. Usually that's either a read or write operation.
+
+If you're reading data, GET is the best option. Other read-only verbs
+are HEAD, which should be used for testing if a resource is available to
+be read with GET, without the expense of the full GET. OPTIONS is also
+read-only, and used by clients to determine which HTTP verbs are
+available for a given path. This isn't something you need to write, as
+it happens automatically in the implementation of `rest_dispatch`--see
+[zerver/lib/rest.py](https://github.com/zulip/zulip/blob/main/zerver/lib/rest.py)
+for more.
+
+If you're creating new data, try to figure out if the thing you are
+creating is uniquely identifiable. For example, if you're creating a
+user, there's only one user per email. If you can find a unique ID,
+you should use PUT for the view. If you want to create the data multiple
+times for multiple requests (for example, requesting the send_message
+view multiple times with the same content should send multiple
+messages), you should use POST.
+
+If you're updating existing data, use PATCH.
+
+If you're removing data, use DELETE.
+
+### Idempotency
+
+When writing a new API endpoint, with the exception of things like
+sending messages, requests should be safe to repeat, without impacting
+the state of the server. This is _idempotency_.
+
+You will often want to return an error if a request to change
+something would do nothing because the state is already as desired, to
+make debugging Zulip clients easier. This means that the response for
+repeated requests may not be the same, but the repeated requests won't
+change the server more than once or cause unwanted side effects.
+
+### Making changes to the database
+
+If the view does any modification to the database, that change is done
+in a helper function in `zerver/actions/*.py`. Those functions are
+responsible for doing a complete update to the state of the server,
+which often entails both updating the database and sending any events
+to notify clients about the state change. When possible, we prefer to
+design a clean boundary between the view function and the actions
+function is such that all user input validation happens in the view
+code (i.e. all 400 type errors are thrown there), and the actions code
+is responsible for atomically executing the change (this is usually
+signalled by having the actions function have a name starting with
+`do_`). So in most cases, errors in an actions function will be the
+result of an operational problem (e.g., lost connection to the
+database) and lead to a 500 error. If an actions function is
+responsible for validation as well, it should have a name starting
+with `check_`.
+
+For example, in [zerver/views/realm.py](https://github.com/zulip/zulip/blob/main/zerver/views/realm.py):
+
+```py
+@require_realm_admin
+@typed_endpoint
+def update_realm(
+    request: HttpRequest,
+    user_profile: UserProfile,
+    *,
+    name: Annotated[str | None, StringConstraints(max_length=Realm.MAX_REALM_NAME_LENGTH)] = None,
+    # ...
+):
+    realm = user_profile.realm
+    # ...
+            do_set_realm_property(realm, k, v, acting_user=user_profile)
+    # ...
+```
+
+`realm.save()` actually saves the changes to the realm to the
+database, and `send_event_on_commit` sends the event to active clients
+belonging to the provided list of users (in this case, all active
+users in the Zulip realm), once the current transaction completes.
+
+### Calling from the web application
+
+You should always use `channel.<method>` to make an `HTTP <method>` call
+to the Zulip JSON API. As an example, in
+[web/src/admin.ts](https://github.com/zulip/zulip/blob/main/web/src/admin.ts)
+
+```js
+var url = "/json/realm";
+var data = {
+    name: JSON.stringify(new_name),
+}
+channel.patch({
+    url: url,
+    data: data,
+    success: function (response_data) {
+        if (response_data.name !== undefined) {
+            ui_report.success($t({defaultMessage: "Name changed!"}), name_status);
+        }
+        ...
+```
+
+### Calling from an API client
+
+Here's how you might manually make a call from python:
+
+```py
+payload = {'name': new_name}
+
+# email and API key
+api_auth = ('hello@example.com', '0123456789abcdeFGHIJKLmnopQRSTUV')
+
+r = requests.patch(SERVER_URL + 'api/v1/realm',
+                   data=json.dumps(payload),
+                   auth=api_auth,
+                  )
+```
+
+This is simply an illustration; we recommend making use of the [Zulip
+Python API bindings](https://zulip.com/api/) since they provide
+a nice interface for accessing the API.
+
+## Legacy endpoints used by the web client
+
+New features should conform the REST API style. The legacy, web-only
+endpoints can't effectively enforce usage of a browser, so they aren't
+preferable from a security perspective, and it is generally a good idea
+to make your feature available to other clients, especially the mobile
+clients.
+
+These endpoints make use the older authentication decorator
+`authenticated_json_view`, so you may see it in the code.
+
+## Incoming webhook integrations
+
+Incoming webhooks are called by other services, often to send a message as part
+of those services' integrations. They are most often POST requests, and
+often there is very little you can customize about them. Usually you can
+expect that the webhook for a service will allow specification for the
+target server for the webhook, and an API key.
+
+If the webhook does not have an option to provide a bot email, use the
+`webhook_view` decorator, to fill in the `user_profile` and
+`request.client` fields of a request:
+
+```py
+@webhook_view("PagerDuty", all_event_types=ALL_EVENT_TYPES)
+@typed_endpoint
+def api_pagerduty_webhook(
+    request: HttpRequest,
+    user_profile: UserProfile,
+    *,
+    payload: JsonBodyPayload[WildValue],
+```
+
+`request.client` will be the result of `get_client("ZulipPagerDutyWebhook")`
+in this example and it will be passed to `check_send_stream_message`. For
+more information, see [Clients in Zulip](../subsystems/client.md).