upstream/upstream-zulip
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

snapshot of https://github.com/zulip/zulip.git @ 48c69911c7dd4e03b20552eb3bd77ad8151c1357 (2026-09-15)
Some checks failed
Code scanning / CodeQL (push) Failing after 15s
Details
Zulip production suite / Ubuntu 22.04 production build (push) Failing after 26s
Details
Zulip production suite / Debian 12 production install with custom db name and user (push) Has been skipped
Details
Zulip production suite / Ubuntu 22.04 production install and PostgreSQL upgrade with pgroonga (push) Has been skipped
Details
Zulip production suite / Ubuntu 24.04 production install (push) Has been skipped
Details
Zulip production suite / Ubuntu 26.04 production install (push) Has been skipped
Details
Zulip production suite / Debian 13 production install (push) Has been skipped
Details
Zulip production suite / 7.0 Version Upgrade (push) Has been skipped
Details
Zulip production suite / 8.0 Version Upgrade (push) Has been skipped
Details
Zulip production suite / 6.0 Version Upgrade (push) Has been skipped
Details
Zulip production suite / 10.0 Version Upgrade (push) Has been skipped
Details
Zulip production suite / 9.0 Version Upgrade (push) Has been skipped
Details
Zulip production suite / 12.0 Version Upgrade (push) Has been skipped
Details
Zulip production suite / 11.0 Version Upgrade (push) Has been skipped
Details
GitHub Actions Security Analysis with zizmor / zizmor (push) Failing after 3s
Details
Zulip CI / Debian 12 (Python 3.11, backend + documentation) (push) Failing after 26s
Details
Zulip CI / Ubuntu 22.04 (Python 3.10, backend + frontend) (push) Failing after 3s
Details
Zulip CI / Ubuntu 24.04 (Python 3.12, backend) (push) Failing after 25s
Details
Zulip CI / Ubuntu 26.04 (Python 3.14, backend) (push) Failing after 25s
Details
Zulip CI / Debian 13 (Python 3.13, backend) (push) Failing after 28s
Details
Zulip production suite / Required jobs (push) Has been cancelled
Details
Zulip CI / Required jobs (push) Has been cancelled
Details
API Documentation Update Check / check-feature-level-updated (push) Has been cancelled
Details
API Documentation Update Check / notify-if-api-docs-changed (push) Has been cancelled
Details

Browse Source
This commit is contained in:
upstream-mirror
2026-09-15 15:05:30 +05:00
commit 64898b2c79
8999 changed files with 1757358 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

249
docs/development/authentication.md Normal file
View File

@@ -0,0 +1,249 @@
# Authentication in the development environment
This page documents special notes that are useful for configuring
Zulip's various [authentication
methods](../production/authentication-methods.md) for testing in a
development environment.
Many of these authentication methods involve a complex interaction
between Zulip, an external service, and the user's browser. Because
browsers can (rightly!) be picky about the identity of sites you
interact with, the preferred way to set up authentication methods in a
development environment is provide secret keys so that you can go
through the real flow.
The steps to do this are a variation of the steps discussed in the
production documentation, including the comments in
`zproject/prod_settings_template.py`. The differences here are driven
by the fact that `dev_settings.py` is in Git, so it is inconvenient
for local [settings configuration](../subsystems/settings.md). As a
result, in the development environment, we allow setting certain
settings in the untracked file `zproject/dev-secrets.conf` (which is
also serves as `/etc/zulip/zulip-secrets.conf`).
Below, we document the procedure for each of the major authentication
methods supported by Zulip.
### Email and password
Zulip's default EmailAuthBackend authenticates users by verifying
control over their email address, and then allowing them to set a
password for their account. There are two development environment
details worth understanding:
- All of our authentication flows in the development environment have
special links to the `/emails` page (advertised in `/devtools`),
which shows all emails that the Zulip server has "sent" (emails are
not actually sent by the development environment), to make it
convenient to click through the UI of signup, password reset, etc.
- There's a management command,
`manage.py print_initial_password username@example.com`, that prints
out **default** passwords for the development environment users.
Note that if you change a user's password in the development
environment, those passwords will no longer work. It also prints
out the user's **current** API key.
### Google
- Visit [the Google developer
console](https://console.developers.google.com) and navigate to "APIs
& services" > "Credentials". Create a "Project", which will correspond
to your dev environment.
- Navigate to "APIs & services" > "Library", and find the "Identity
Toolkit API". Choose "Enable".
- Return to "Credentials", and select "Create credentials". Choose
"OAuth client ID", and follow prompts to create a consent screen, etc.
For "Authorized redirect URIs", fill in
`http://auth.zulipdev.com:9991/complete/google/` .
- You should get a client ID and a client secret. Copy them. In
`dev-secrets.conf`, set `social_auth_google_key` to the client ID
and `social_auth_google_secret` to the client secret.
### GitHub
- Register an OAuth2 application with GitHub at one of
<https://github.com/settings/developers> or
`https://github.com/organizations/<your-org>/settings/applications`.
Specify `http://auth.zulipdev.com:9991/complete/github/` as the callback URL.
- You should get a page with settings for your new application,
showing a client ID and a client secret. In `dev-secrets.conf`, set
`social_auth_github_key` to the client ID and `social_auth_github_secret`
to the client secret.
### GitLab
- Register an OAuth application with GitLab at
<https://gitlab.com/oauth/applications>.
Specify `http://auth.zulipdev.com:9991/complete/gitlab/` as the callback URL.
- You should get a page containing the Application ID and Secret for
your new application. In `dev-secrets.conf`, enter the Application
ID as `social_auth_gitlab_key` and the Secret as
`social_auth_gitlab_secret`.
### Apple
- Visit <https://developer.apple.com/account/resources/>,
Enable App ID and Create a Services ID with the instructions in
<https://help.apple.com/developer-account/?lang=en#/dev1c0e25352> .
When prompted for a "Return URL", enter
`http://auth.zulipdev.com:9991/complete/apple/` .
- [Create a Sign in with Apple private key](https://help.apple.com/developer-account/?lang=en#/dev77c875b7e)
- In `dev-secrets.conf`, set
- `social_auth_apple_services_id` to your
"Services ID" (eg. com.application.your).
- `social_auth_apple_app_id` to "App ID" or "Bundle ID".
This is only required if you are testing Apple auth on iOS.
- `social_auth_apple_key` to your "Key ID".
- `social_auth_apple_team` to your "Team ID".
- Put the private key file you got from apple at the path
`zproject/dev_apple.key`.
### Discord
- Register an OAuth application with Discord at
<https://discord.com/developers/applications>.
Specify `http://auth.zulipdev.com:9991/complete/discord/` as the callback URL.
- You should get a page containing the Client ID and Secret for
your new application. In `dev-secrets.conf`, enter the Client
ID as `social_auth_discord_key` and the Secret as
`social_auth_discord_secret`.
### SAML
- Sign up for a [developer Okta account](https://developer.okta.com/).
- Set up SAML authentication by following
[Okta's documentation](https://developer.okta.com/docs/guides/saml-application-setup/overview/).
Specify:
- `http://localhost:9991/complete/saml/` for the "Single sign on URL"`. If you are using a dev droplet, this will
be `http://auth.{your_droplet_name}.zulipdev.org:9991/complete/saml/`
- `http://localhost:9991` for the "Audience URI (SP Entity ID)". If you are using a dev droplet, this will be
`http://zulip.{your_droplet_name}.zulipdev.org:9991/`
- Skip "Default RelayState".
- Skip "Name ID format".
- Set 'Email` for "Application username format".
- Provide "Attribute statements" of `email` to `user.email`,
`first_name` to `user.firstName`, and `last_name` to `user.lastName`.
- Assign at least one account in the "Assignments" tab. You'll use it for
signing up / logging in to Zulip.
- Visit the big "Setup instructions" button on the "Sign on" tab.
- Edit `zproject/dev-secrets.conf` to add the two values provided:
- Set `saml_url = http...` from "Identity Provider Single Sign-On
URL".
- Set `saml_entity_id = http://...` from "Identity Provider Issuer".
- Download the certificate and put it at the path `zproject/dev_saml.cert`.
- Now you should have working SAML authentication!
- You can sign up to the target realm with the account that you've "assigned"
in the previous steps (if the account's email address is allowed in the realm,
so you may have to change the realm settings to allow the appropriate email domain)
and then you'll be able to log in freely. Alternatively, you can create an account
with the email in any other way, and then just use SAML to log in.
### When SSL is required
Some OAuth providers (such as Facebook) require HTTPS on the callback
URL they post back to, which isn't supported directly by the Zulip
development environment. If you run a
[remote Zulip development server](remote.md), we have
instructions for
[an nginx reverse proxy with SSL](remote.md#using-an-nginx-reverse-proxy)
that you can use for your development efforts.
## Testing LDAP in development
Before Zulip 2.0, one of the more common classes of bug reports with
Zulip's authentication was users having trouble getting [LDAP
authentication](../production/authentication-methods.md#ldap-including-active-directory)
working. The root cause was because setting up a local LDAP server
for development was difficult, which meant most developers were unable
to work on fixing even simple issues with it.
We solved this problem for our unit tests long ago by using the
popular [fakeldap](https://github.com/zulip/fakeldap) library. And in
2018, we added convenient support for using fakeldap in the Zulip
development environment as well, so that you can go through all the
actual flows for LDAP configuration.
- To enable fakeldap, set `FAKE_LDAP_MODE` in
`zproject/dev_settings.py` to one of the following options. For more
information on these modes, refer to
[our production docs](../production/authentication-methods.md#ldap-including-active-directory):
- `a`: If users' email addresses are in LDAP and used as username.
- `b`: If LDAP only has usernames but email addresses are of the form
username@example.com
- `c`: If LDAP usernames are completely unrelated to email addresses.
- To disable fakeldap, set `FAKE_LDAP_MODE` back to `None`.
- In all fakeldap configurations, users' fake LDAP passwords are equal
to their usernames (e.g., for `ldapuser1@zulip.com`, the password is
`ldapuser1`).
- `FAKE_LDAP_NUM_USERS` in `zproject/dev_settings.py` can be used to
specify the number of LDAP users to be added. The default value for
the number of LDAP users is 8.
### Testing avatar and custom profile field synchronization
The fakeldap LDAP directories we use in the development environment
are generated by the code in `zerver/lib/dev_ldap_directory.py`, and
contain data one might want to sync, including avatars and custom
profile fields.
We also have configured `AUTH_LDAP_USER_ATTR_MAP` in
`zproject/dev_settings.py` to sync several of those fields. For
example:
- Modes `a` and `b` will set the user's avatar on account creation and
update it when `manage.py sync_ldap_user_data` is run.
- Mode `b` is configured to automatically have the `birthday` and
`Phone number` custom profile fields populated/synced.
- Mode `a` is configured to deactivate/reactivate users whose accounts
are disabled in LDAP when `manage.py sync_ldap_user_data` is run.
(Note that you'll likely need to edit
`zerver/lib/dev_ldap_directory.py` to ensure there are some accounts
configured to be disabled).
### Automated testing
For our automated tests, we generally configure custom LDAP data for
each individual test, because that generally means one can understand
exactly what data is being used in the test without looking at other
resources. It also gives us more freedom to edit the development
environment directory without worrying about tests.
## Two factor authentication
Zulip uses [django-two-factor-auth][0] as a beta 2FA integration.
To enable 2FA, set `TWO_FACTOR_AUTHENTICATION_ENABLED` in settings to
`True`, then log in to Zulip and add an OTP device from the settings
page. Once the device is added, password based authentication will ask
for a one-time-password. In the development environment, this
one-time-password will be printed to the console when you try to
log in. Just copy-paste it into the form field to continue.
Direct development logins don't prompt for 2FA one-time-passwords, so
to test 2FA in development, make sure that you log in using a
password. You can get the passwords for the default test users using
`./manage.py print_initial_password`.
## Password form implementation
By default, Zulip uses `autocomplete=off` for password fields where we
enter the current password, and `autocomplete="new-password"` for
password fields where we create a new account or change the existing
password. This prevents the browser from auto-filling the existing
password.
Visit <https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/autocomplete> for more details.
[0]: https://github.com/Bouke/django-two-factor-auth

15
docs/development/index.md Normal file
View File

@@ -0,0 +1,15 @@
# Development environment
```{toctree}
---
maxdepth: 3
---
Development environment installation <overview>
Recommended setup <setup-recommended>
Advanced setup <setup-advanced>
Using the development environment <using>
Developing remotely <remote>
Authentication in the development environment <authentication>
Testing the installer <test-install>
```

87
docs/development/overview.md Normal file
View File

@@ -0,0 +1,87 @@
# Development environment installation
## Requirements
The Zulip development environment can be installed on **macOS,
Windows, and Linux** (Debian or Ubuntu recommended). You'll need at least **2GB
of available RAM**.
Installing the Zulip development environment requires downloading several hundred
megabytes of dependencies, so you will need an **active, reasonably fast,
internet connection throughout the entire installation process.** You can
[configure a proxy][configure-proxy] if you need one.
## Recommended setup
**For first-time contributors, we recommend using the
[Vagrant development environment][install-vagrant]** on
macOS and Linux based OS and [WSL 2 setup][install-via-wsl] on Windows.
## Vagrant setup
[Vagrant setup][install-vagrant] creates a virtual machine (for Windows and macOS) or a
Linux container (otherwise) inside which the Zulip server and all
related services will run. Vagrant adds a bit of overhead to using the
Zulip development server, but provides an isolated environment that is
easy to install, update, and uninstall. It has been well-tested and
performs well.
## Advanced setup
Zulip also supports a wide range of ways to install the Zulip
development environment:
- On Linux platforms, you can **[install directly][install-direct]**.
- On Windows, you can **[install directly][install-via-wsl]** via WSL 2.
## Slow internet connections
If you have a very slow network connection, however, you may want to
avoid using Vagrant (which involves downloading an Ubuntu virtual
machine or Linux container) and [install directly][install-direct] on
a Linux system.
An alternative option if you have poor network connectivity is to rent
a cloud server and install the Zulip development environment for
remote use. See the [next section][self-install-remote] for details.
## Installing remotely
The Zulip development environment works well on remote virtual
machines. This can be a good alternative for those with poor network
connectivity or who have limited storage/memory on their local
machines.
We recommend giving the Zulip development environment its **own
virtual machine**, running one of
[the supported platforms for direct installation][install-direct],
with at least **2GB of memory**.
If the Zulip development environment will be the only thing running on
the remote virtual machine, we recommend installing
[directly][install-direct]. Otherwise, we recommend the
[Vagrant][install-vagrant] method so you can easily uninstall if you
need to.
## Next steps
Once you've installed the Zulip development environment, you'll want
to read these documents to learn how to use it:
- [Using the development environment][using-dev-env]
- [Testing][testing] (and [Configuring CI][ci])
And if you've set up the Zulip development environment on a remote
machine, take a look at our tips for
[developing remotely][dev-remote].
[dev-remote]: remote.md
[install-direct]: setup-advanced.md#installing-directly-on-ubuntu-debian-centos-or-fedora
[install-vagrant]: setup-recommended.md
[self-install-remote]: #installing-remotely
[self-slow-internet]: #slow-internet-connections
[configure-proxy]: setup-recommended.md#specifying-a-proxy
[using-dev-env]: using.md
[testing]: ../testing/testing.md
[ci]: ../git/cloning.md#step-3-configure-continuous-integration-for-your-fork
[install-via-wsl]: setup-recommended.md

320
docs/development/remote.md Normal file
View File

@@ -0,0 +1,320 @@
# Developing on a remote machine
The Zulip developer environment works well on remote virtual machines. This can
be a good alternative for those with poor network connectivity or who have
limited storage/memory on their local machines.
We recommend giving the Zulip development environment its own virtual
machine with at least 2GB of memory. If the Zulip development
environment will be the only thing running on the remote virtual
machine, we recommend installing
[directly][install-direct]. Otherwise, we recommend the
[Vagrant][install-vagrant] method so you can easily uninstall if you
need to.
## Connecting to the remote environment
The best way to connect to your server is using the command line tool `ssh`.
- On macOS and Linux/UNIX, `ssh` is a part of Terminal.
- On Windows, `ssh` comes with [Bash for Git][git-bash].
Open _Terminal_ or _Bash for Git_, and connect with the following:
```console
$ ssh username@host
```
If you have poor internet connectivity, we recommend using
[Mosh](https://mosh.org/) as it is more reliable over slow or unreliable
networks.
## Setting up user accounts
You will need a non-root user account with sudo privileges to set up
the Zulip development environment. If you have one already, continue
to the next section.
You can create a new user with sudo privileges by running the
following commands as root:
- You can create a `zulipdev` user by running the command
`adduser zulipdev`. Run through the prompts to assign a password and
user information. (You can pick any username you like for this user
account.)
- You can add the user to the sudo group by running the command
`usermod -aG sudo zulipdev`.
- Finally, you can switch to the user by running the command
`su - zulipdev` (or just log in to that user using `ssh`).
## Setting up the development environment
After you have connected to your remote server, you need to install the
development environment.
If the Zulip development environment will be the only thing running on
the remote virtual machine, we recommend installing
[directly][install-direct]. Otherwise, we recommend the
[Vagrant][install-vagrant] method so you can easily uninstall if you
need to.
The main difference from the standard instructions is that for a
remote development environment, and you're not using our Digital Ocean
Droplet infrastructure (which handles `EXTERNAL_HOST` for you), you'll
need to run `export EXTERNAL_HOST=<REMOTE_IP>:9991` in a shell before
running `run-dev` (and see also the `--interface=''` option
documented below).
If your server has a static IP address, we recommend putting this
command in `~/.bashrc`, so you don't need to remember to run it every
time. This allows you to access Zulip running in your development
environment using a browser on another host.
## Running the development server
Once you have set up the development environment, you can start up the
development server with the following command in the directory where
you cloned Zulip:
```bash
./tools/run-dev --interface=''
```
This will start up the Zulip server on port 9991. You can then
navigate to `http://<REMOTE_IP>:9991/devlogin` and you should see something like
this screenshot of the Zulip development environment:
![Image of Zulip development environment](../images/zulip-devlogin.png)
The `--interface=''` option makes the Zulip development environment
accessible from any IP address (in contrast with the much more secure
default of only being accessible from localhost, which is great for
developing on your laptop).
To properly secure your remote development environment, you can
[port forward](https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding)
using ssh instead of running the development environment on an exposed
interface. For example, if you're running Zulip on a remote server
such as a DigitalOcean Droplet or an AWS EC2 instance, you can set up
port-forwarding to access Zulip by running the following command in
your terminal:
```bash
ssh -L 3000:127.0.0.1:9991 <username>@<remote_server_ip> -N
```
Now you can access Zulip by navigating to `http://127.0.0.1:3000` in
your local computer's browser.
For more information, see [Using the development
environment][rtd-using-dev-env].
## Making changes to code on your remote development server
To see changes on your remote development server, you need to do one of the following:
- [Edit locally](#editing-locally): Clone Zulip code to your computer and
then use your favorite editor to make changes. When you want to see changes
on your remote Zulip development instance, sync with Git.
- [Edit remotely](#editing-remotely): Edit code directly on your remote
Zulip development instance using a [Web-based IDE](#web-based-ide) (recommended for
beginners) or a [command line editor](#command-line-editors), or a
[desktop IDE](#desktop-gui-editors) using a plugin to sync your
changes to the server when you save.
#### Editing locally
If you want to edit code locally install your favorite text editor. If you
don't have a favorite, here are some suggestions:
- [atom](https://atom.io/)
- [emacs](https://www.gnu.org/software/emacs/)
- [vim](https://www.vim.org/)
- [spacemacs](https://github.com/syl20bnr/spacemacs)
- [sublime](https://www.sublimetext.com/)
- [PyCharm](https://www.jetbrains.com/pycharm/)
Next, follow our [Git and GitHub guide](../git/index.md) to clone and configure
your fork of zulip on your local computer.
Once you have cloned your code locally, you can get to work.
##### Syncing changes
The easiest way to see your changes on your remote development server
is to **push them to GitHub** and then **fetch and merge** them from
the remote server.
For more detailed instructions about how to do this, see our [Git & GitHub
guide][rtd-git-guide]. In brief, the steps are as follows.
On your **local computer**:
1. Open _Terminal_ (macOS/Linux) or _Git for BASH_.
2. Change directory to where you cloned Zulip (e.g., `cd zulip`).
3. Use `git add` and `git commit` to stage and commit your changes (if you
haven't already).
4. Push your commits to GitHub with `git push origin branchname`.
Be sure to replace `branchname` with the name of your actual feature branch.
Once `git push` has completed successfully, you are ready to fetch the commits
from your remote development instance:
1. In _Terminal_ or _Git BASH_, connect to your remote development
instance with `ssh user@host`.
2. Change to the zulip directory (e.g., `cd zulip`).
3. Fetch new commits from GitHub with `git fetch origin`.
4. Change to the branch you want to work on with `git checkout branchname`.
5. Merge the new commits into your branch with `git merge origin/branchname`.
#### Editing remotely
There are a few good ways to edit code in your remote development
environment:
- With a command-line editor like vim or emacs run over SSH.
- With a desktop GUI editor like VS Code or Atom and a plugin for
syncing your changes to the remote server.
- With a web-based IDE like CodeAnywhere.
We document these options below; we recommend using whatever editor
you prefer for development in general.
##### Desktop GUI editors
If you use [TextMate](https://macromates.com), Atom, VS Code, or a
similar GUI editor, tools like
[Visual Studio Code Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) and
[rmate](https://github.com/textmate/rmate) that are designed to
integrate that editor with remote development over SSH allow you to
develop remotely from the comfort of your local machine.
Similar packages/extensions exist for other popular code editors as
well; contributions of precise documentation for them are welcome!
- [VSCode Remote - SSH][vscode-remote-ssh]: Lets you use Visual Studio
Code against a remote repository with a similar user experience to
developing locally.
[vscode-remote-ssh]: https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh
- [rmate](https://github.com/textmate/rmate) for TextMate + VS Code:
1. Install the extension
[Remote VSCode](https://marketplace.visualstudio.com/items?itemName=rafaelmaiolla.remote-vscode).
2. On your remote machine, run:
```console
$ mkdir -p ~/bin
$ curl -fL -o ~/bin/rmate https://raw.githubusercontent.com/textmate/rmate/master/bin/rmate
$ chmod a+x ~/bin/rmate
```
3. Make sure the remote server is running in VS Code (you can
force-start through the Command Palette).
4. SSH to your remote machine using:
```console
$ ssh -R 52698:localhost:52698 user@example.org
```
5. On your remote machine, run:
```console
$ rmate [options] file
```
and the file should open up in VS Code. Any changes you make now will be saved remotely.
##### Command line editors
Another way to edit directly on the remote development server is with
a command line text editor on the remote machine.
Two editors often available by default on Linux systems are:
- **Nano**: A very simple, beginner-friendly editor. However, it lacks a lot of
features useful for programming, such as syntax highlighting, so we only
recommended it for quick edits to things like configuration files. Launch by
running command `nano <filename>`. Exit by pressing _Ctrl-X_.
- **[Vim](https://www.vim.org/)**: A very powerful editor that can take a while
to learn. Launch by running `vim <filename>`. Quit Vim by pressing _Esc_,
typing `:q`, and then pressing _Enter_. Vim comes with a program to learn it
called `vimtutor` (just run that command to start it).
Other options include:
- [emacs](https://www.gnu.org/software/emacs/)
- [spacemacs](https://github.com/syl20bnr/spacemacs)
##### Web-based IDE
If you are relatively new to working on the command line, or just want to get
started working quickly, we recommend web-based IDE
[Codeanywhere][codeanywhere].
To set up Codeanywhere for Zulip:
1. Create a [Codeanywhere][codeanywhere] account and log in.
2. Create a new **SFTP-SSH** project. Use _Public key_ for authentication.
3. Click **GET YOUR PUBLIC KEY** to get the new public key that
Codeanywhere generates when you create a new project. Add this public key to
`~/.ssh/authorized_keys` on your remote development instance.
4. Once you've added the new public key to your remote development instance, click
_CONNECT_.
Now your workspace should look similar this:
![Codeanywhere workspace][img-ca-workspace]
#### Next steps
Next, read the following to learn more about developing for Zulip:
- [Git & GitHub guide][rtd-git-guide]
- [Using the development environment][rtd-using-dev-env]
- [Testing][rtd-testing]
[install-direct]: setup-advanced.md#installing-directly-on-ubuntu-debian-centos-or-fedora
[install-vagrant]: setup-recommended.md
[rtd-git-guide]: ../git/index.md
[rtd-using-dev-env]: using.md
[rtd-testing]: ../testing/testing.md
[git-bash]: https://git-for-windows.github.io/
[codeanywhere]: https://codeanywhere.com/
[img-ca-settings]: ../images/codeanywhere-settings.png
[img-ca-workspace]: ../images/codeanywhere-workspace.png
## Using an nginx reverse proxy
For some applications (e.g., developing an OAuth2 integration for
Facebook), you may need your Zulip development to have a valid SSL
certificate. While `run-dev` doesn't support that, you can do this
with an `nginx` reverse proxy sitting in front of `run-dev`.
The following instructions assume you have a Zulip Droplet working and
that the user is `zulipdev`; edit accordingly if the situation is
different.
1. First, get an SSL certificate; you can use
[our certbot wrapper script used for production](../production/ssl-certificates.md#certbot-recommended)
by running the following commands as root:
```bash
# apt install -y crudini
mkdir -p /var/lib/zulip/certbot-webroot/
# if nginx running this will fail and you need to run `service nginx stop`
/home/zulipdev/zulip/scripts/setup/setup-certbot \
hostname.example.com \
--email=username@example.com --method=standalone
```
1. Install nginx configuration:
```bash
apt install -y nginx-full
cp -a /home/zulipdev/zulip/tools/droplets/zulipdev /etc/nginx/sites-available/
ln -nsf /etc/nginx/sites-available/zulipdev /etc/nginx/sites-enabled/
nginx -t # Verifies your nginx configuration
service nginx reload # Actually enabled your nginx configuration
```
1. Start the Zulip development environment in HTTPS mode with the following command:
```bash
env EXTERNAL_HOST="hostname.example.com" ./tools/run-dev --behind-https-proxy --interface=''
```

86
docs/development/request-remote.md Normal file
View File

@@ -0,0 +1,86 @@
---
orphan: true
---
# How to request a remote Zulip development instance
Under specific circumstances, typically during sprints, hackathons, and
Google Code-in, Zulip can provide you with a virtual machine with the
development environment already set up.
The machines (droplets) are being generously provided by
[DigitalOcean](https://www.digitalocean.com/). Thank you DigitalOcean!
## Step 1: Join GitHub and create SSH keys
To contribute to Zulip and to use a remote Zulip developer instance, you'll
need a GitHub account. If you don't already have one, sign up
[here][github-join].
You'll also need to [create SSH keys and add them to your GitHub
account][github-help-add-ssh-key].
## Step 2: Create a fork of zulip/zulip
Zulip uses a **forked-repo** and **[rebase][gitbook-rebase]-oriented
workflow**. This means that all contributors create a fork of the [Zulip
repository][github-zulip-zulip] they want to contribute to and then submit pull
requests to the upstream repository to have their contributions reviewed and
accepted.
When we create your Zulip dev instance, we'll connect it to your fork of Zulip,
so that needs to exist before you make your request.
While you're logged in to GitHub, navigate to [zulip/zulip][github-zulip-zulip]
and click the **Fork** button. (See [GitHub's help article][github-help-fork]
for further details).
## Step 3: Make request via chat.zulip.org
Now that you have a GitHub account, have added your SSH keys, and forked
zulip/zulip, you are ready to request your Zulip developer instance.
If you haven't already, create an account on https://chat.zulip.org/.
Next, join the [development
help](https://chat.zulip.org/#narrow/channel/49-development-help) channel. Create a
new **channel message** with your GitHub username as the **topic** and request
your remote dev instance. **Please make sure you have completed steps 1 and 2
before doing so**. A core developer should reply letting you know they're
working on creating it as soon as they are available to help.
Once requested, it will only take a few minutes to create your instance. You
will be contacted when it is complete and available.
## Next steps
Once your remote dev instance is ready:
- Connect to your server by running
`ssh zulipdev@<username>.zulipdev.org` on the command line
(Terminal for macOS and Linux, Bash for Git on Windows).
- There is no password; your account is configured to use your SSH keys.
- Once you log in, you should see `(zulip-server) ~$`.
- To start the dev server, `cd zulip` and then run `./tools/run-dev`.
- While the dev server is running, you can see the Zulip server in your browser
at http://zulip.username.zulipdev.org:9991.
- The development server actually runs on all subdomains of
`username.zulipdev.org`; this is important for testing Zulip's
support for multiple organizations in your development server.
Once you've confirmed you can connect to your remote server, take a look at:
- [developing remotely](remote.md) for tips on using the remote dev
instance, and
- our [Git & GitHub guide](../git/index.md) to learn how to use Git with Zulip.
Next, read the following to learn more about developing for Zulip:
- [Using the development environment](using.md)
- [Testing](../testing/testing.md)
[github-join]: https://github.com/join
[github-help-add-ssh-key]: https://help.github.com/en/articles/adding-a-new-ssh-key-to-your-github-account
[github-zulip-zulip]: https://github.com/zulip/zulip/
[github-help-fork]: https://help.github.com/en/articles/fork-a-repo
[gitbook-rebase]: https://git-scm.com/book/en/v2/Git-Branching-Rebasing

295
docs/development/setup-advanced.md Normal file
View File

@@ -0,0 +1,295 @@
# Advanced setup
Contents:
- [Installing directly on Ubuntu, Debian, CentOS, or Fedora](#installing-directly-on-ubuntu-debian-centos-or-fedora)
- [Installing using Vagrant with VirtualBox on Windows 10](#installing-using-vagrant-with-virtualbox-on-windows-10)
- [Using the Vagrant Hyper-V provider on Windows](#using-the-vagrant-hyper-v-provider-on-windows-beta)
- [Newer versions of supported platforms](#newer-versions-of-supported-platforms)
## Installing directly on Ubuntu, Debian, CentOS, or Fedora
:::{warning}
There is no supported uninstallation process with the direct-install
method. If you want that, use [the Vagrant environment](setup-recommended.md),
where you can just do `vagrant destroy` to clean up the development environment.
:::
One can install the Zulip development environment directly on a Linux
host by following these instructions. Currently supported platforms
are:
- Ubuntu 22.04, 24.04, 26.04
- Debian 12, 13
- CentOS 7 (beta)
- Fedora 43, 44 (beta)
- RHEL 7 (beta)
**Note**: You should not use the `root` user to run the installation.
If you are using a [remote server](remote.md), see
the
[section on creating appropriate user accounts](remote.md#setting-up-user-accounts).
Start by [cloning your fork of the Zulip repository][zulip-rtd-git-cloning]
and [connecting the Zulip upstream repository][zulip-rtd-git-connect]:
```bash
git clone --config pull.rebase git@github.com:YOURUSERNAME/zulip.git
cd zulip
git remote add -f upstream https://github.com/zulip/zulip.git
```
CentOS, Fedora, and RHEL users should ensure that python3 is installed on their
systems (Debian and Ubuntu distributions already include it):
```bash
# On CentOS/Fedora/RHEL, you must first install python3.
# For example, this command installs python3 with yum:
yum install python
```
With python3 installed, change into the directory where you have cloned
Zulip and run the following commands:
```bash
# From inside a clone of zulip.git:
./tools/provision
source .venv/bin/activate
./tools/run-dev # starts the development server
```
Once you've done the above setup, you can pick up the [documentation
on using the Zulip development
environment](setup-recommended.md#step-4-developing),
ignoring the parts about `vagrant` (since you're not using it).
## Installing using Vagrant with VirtualBox on Windows 10
:::{note}
We recommend using [WSL 2 for Windows development](setup-recommended.md)
because it is easier to set up and provides a substantially better experience.
:::
1. Install [Git for Windows][git-bash], which installs _Git BASH_.
2. Install [VirtualBox][vbox-dl] (latest).
3. Install [Vagrant][vagrant-dl] (latest).
(Note: While _Git BASH_ is recommended, you may also use [Cygwin][cygwin-dl].
If you do, make sure to **install default required packages** along with
**git**, **curl**, **openssh**, and **rsync** binaries.)
Also, you must have hardware virtualization enabled (VT-x or AMD-V) in your
computer's BIOS.
#### Running Git BASH as an administrator
It is important that you **always run Git BASH with administrator
privileges** when working on Zulip code, as not doing so will cause
errors in the development environment (such as symlink creation). You
might wish to configure your Git BASH shortcut to always run with
these privileges enabled (see this [guide][bash-admin-setup] for how
to set this up).
##### Enable native symlinks
The Zulip code includes symbolic links (symlinks). By default, native Windows
symlinks are not enabled in either Git BASH or Cygwin, so you need to do a bit
of configuration. **You must do this before you clone the Zulip code.**
In **Git for BASH**:
Open **Git BASH as an administrator** and run:
```console
$ git config --global core.symlinks true
```
Now confirm the setting:
```console
$ git config core.symlinks
true
```
If you see `true`, you are ready for [Step 2: Get Zulip code](setup-recommended.md#step-2-get-zulip-code).
Otherwise, if the above command prints `false` or nothing at all, then symlinks
have not been enabled.
In **Cygwin**:
Open a Cygwin window **as an administrator** and do this:
```console
christie@win10 ~
$ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile
```
Next, close that Cygwin window and open another. If you `echo` $CYGWIN you
should see:
```console
christie@win10 ~
$ echo $CYGWIN
winsymlinks:native
```
Now you are ready for [Step 2: Get Zulip code](setup-recommended.md#step-2-get-zulip-code).
(Note: The **GitHub Desktop client** for Windows has a bug where it
will automatically set `git config core.symlink false` on a repository
if you use it to clone a repository, which will break the Zulip
development environment, because we use symbolic links. For that
reason, we recommend avoiding using GitHub Desktop client to clone
projects and to instead follow these instructions exactly.)
[cygwin-dl]: https://cygwin.com
[git-bash]: https://git-for-windows.github.io
[vbox-dl]: https://www.virtualbox.org/wiki/Downloads
[vagrant-dl]: https://www.vagrantup.com/downloads.html
[bash-admin-setup]: https://superuser.com/questions/1002262/run-applications-as-administrator-by-default-in-windows-10
## Using the Vagrant Hyper-V provider on Windows (beta)
You should have [Vagrant](https://www.vagrantup.com/downloads) and
[Hyper-V][hyper-v] installed on your system. Ensure they both work as
expected.
[hyper-v]: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v
**NOTE**: Hyper-V is available only on Windows Enterprise, Pro, or Education.
1. Start by [cloning your fork of the Zulip repository][zulip-rtd-git-cloning]
and [connecting the Zulip upstream repository][zulip-rtd-git-connect]:
```bash
git clone --config pull.rebase git@github.com:YOURUSERNAME/zulip.git
cd zulip
git remote add -f upstream https://github.com/zulip/zulip.git
```
1. You will have to open up powershell with administrator rights in
order to use Hyper-V. Then provision the development environment:
```bash
vagrant up --provider=hyperv
```
You should get output like this:
```console
Bringing machine 'default' up with 'hyperv' provider...
==> default: Verifying Hyper-V is enabled...
==> default: Verifying Hyper-V is accessible...
<other stuff>...
==> default: Waiting for the machine to report its IP address...
default: Timeout: 120 seconds
default: IP: 172.28.119.70
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 172.28.122.156
==> default: Machine booted and ready!
==> default: Preparing SMB shared folders...
Vagrant requires administrator access for pruning SMB shares and
may request access to complete removal of stale shares.
==> default: Starting the machine...
<other stuff>...
default: Username (user[@domain]): <your-machine-username>
default: Password (will be hidden):
```
At this point, you will be prompted for your Windows administrator
username and password (not your Microsoft account credentials).
1. SSH into your newly created virtual machine
```bash
vagrant ssh
```
This will ssh you into the bash shell of the Zulip development environment
where you can execute bash commands.
1. Set the `EXTERNAL_HOST` environment variable.
```console
(zulip-server) vagrant@ubuntu-18:/srv/zulip$ export EXTERNAL_HOST="$(hostname -I | xargs):9991"
(zulip-server) vagrant@ubuntu-18:/srv/zulip$ echo $EXTERNAL_HOST
```
The output will be like:
```console
172.28.122.156:9991
```
Make sure you note down this down. This is where your zulip development web
server can be accessed.
:::{important}
The output of the above command changes every time you restart the Vagrant
development machine. Thus, it will have to be run every time you bring one up.
This quirk is one reason this method is marked experimental.
:::
1. You should now be able to start the Zulip development server.
```console
(zulip-server) vagrant@ubuntu-18:/srv/zulip$ ./tools/run-dev
```
The output will look like:
```console
Starting Zulip on:
http://172.30.24.235:9991/
Internal ports:
9991: Development server proxy (connect here)
9992: Django
9993: Tornado
9994: webpack
```
Visit the indicated URL in your web browser.
1. You can stop the development environment using `vagrant halt`, and restart it
using `vagrant up` and then going through steps **3** and **4** again.
### Problems you may encounter
1. If you get the error `Hyper-V could not initialize memory`, this is
likely because your system has insufficient free memory to start
the virtual machine. You can generally work around this error by
closing all other running programs and running
`vagrant up --provider=hyperv` again. You can reopen the other
programs after the provisioning is completed. If it still isn't
enough, try restarting your system and running the command again.
2. Be patient the first time you run `./tools/run-dev`.
As with other installation methods, please visit [#provision
help][provision-help] in the [Zulip development community
server](https://zulip.com/development-community/) if you need help.
[provision-help]: https://chat.zulip.org/#narrow/channel/21-provision-help
## Newer versions of supported platforms
You can use
[our provisioning tool](#installing-directly-on-ubuntu-debian-centos-or-fedora)
to set up the Zulip development environment on current versions of
these platforms reliably and easily, so we no longer maintain manual
installation instructions for these platforms.
If `tools/provision` doesn't yet support a newer release of Debian or
Ubuntu that you're using, we'd love to add support for it. It's
likely only a few lines of changes to `tools/lib/provision.py` and
`scripts/lib/setup-apt-repo` if you'd like to do it yourself and
submit a pull request, or you can ask for help in
[#development help](https://chat.zulip.org/#narrow/channel/49-development-help)
in [the Zulip development community](https://zulip.com/development-community/),
and a core team member can help guide you through adding support for the platform.
[zulip-rtd-git-cloning]: ../git/cloning.md#step-1b-clone-to-your-machine
[zulip-rtd-git-connect]: ../git/cloning.md#step-1c-connect-your-fork-to-zulip-upstream

1056
docs/development/setup-recommended.md Normal file
View File

File diff suppressed because it is too large Load Diff

36
docs/development/setup/install-docker.md Normal file
View File

@@ -0,0 +1,36 @@
##### 2. Create the `docker` group and add yourself to it:
```console
$ sudo groupadd docker
$ sudo usermod -aG docker $USER
```
You will need to log out and log back in (or reboot) for this change to take effect.
If it worked, you will see `docker` in your list of groups:
```console
$ groups | grep docker
YOURUSERNAME adm cdrom sudo dip plugdev lpadmin sambashare docker
```
##### 3. Make sure the Docker daemon is running
Docker runs as a background service (daemon), which must be running for Docker commands to work.
You can check using the following:
```console
$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-07-15 23:20:46 IST; 18min ago
```
If the service is not running, you'll see `Active: inactive (dead)` on
the second line, and will need to enable and start the Docker service
using the following:
```console
$ sudo systemctl unmask docker
$ sudo systemctl enable docker
$ sudo systemctl start docker
```

121
docs/development/setup/shared-vagrant-errors.md Normal file
View File

@@ -0,0 +1,121 @@
#### Vagrant guest doesn't show (zulip-server) at start of prompt
This is caused by provisioning failing to complete successfully. You
can see the errors in `var/log/provision.log`; it should end with
something like this:
```text
ESC[94mZulip development environment setup succeeded!ESC[0m
```
The `ESC` stuff are the terminal color codes that make it show as a nice
blue in the terminal, which unfortunately looks ugly in the logs.
If you encounter an incomplete `/var/log/provision.log file`, you need to
update your environment. Re-provision your Vagrant machine; if the problem
persists, please come chat with us (see instructions above) for help.
After you provision successfully, you'll need to exit your `vagrant ssh`
shell and run `vagrant ssh` again to get the virtualenv setup properly.
#### ssl read error
If you receive the following error while running `vagrant up`:
```console
SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
```
It means that either your network connection is unstable and/or very
slow. To resolve it, run `vagrant up` until it works (possibly on a
better network connection).
#### ssh connection closed by remote host
On running `vagrant ssh`, if you see the following error:
```console
ssh_exchange_identification: Connection closed by remote host
```
It usually means the Vagrant guest is not running, which is usually
solved by rebooting the Vagrant guest via `vagrant halt; vagrant up`. See
[Vagrant was unable to communicate with the guest machine](/development/setup-recommended.md#vagrant-was-unable-to-communicate-with-the-guest-machine)
for more details.
#### Vagrant was unable to communicate with the guest machine
If you see the following error when you run `vagrant up`:
```console
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.
If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.
If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.
```
This has a range of possible causes, that usually amount to a bug in
Virtualbox or Vagrant. If you see this error, you usually can fix it
by rebooting the guest via `vagrant halt; vagrant up`.
#### Vagrant up fails with subprocess.CalledProcessError
The `vagrant up` command basically does the following:
- Downloads an Ubuntu image and starts it using a Vagrant provider.
- Uses `vagrant ssh` to connect to that Ubuntu guest, and then runs
`tools/provision`, which has a lot of subcommands that are
executed via Python's `subprocess` module. These errors mean that
one of those subcommands failed.
To debug such errors, you can log in to the Vagrant guest machine by
running `vagrant ssh`, which should present you with a standard shell
prompt. You can debug interactively by using, for example,
`cd zulip && ./tools/provision`, and then running the individual
subcommands that failed. Once you've resolved the problem, you can
rerun `tools/provision` to proceed; the provisioning system is
designed to recover well from failures.
The Zulip provisioning system is generally highly reliable; the most common
cause of issues here is a poor network connection (or one where you need a
proxy to access the Internet and haven't [configured the development
environment to use it](/development/setup-recommended.md#specifying-a-proxy)).
Once you've provisioned successfully, you'll get output like this:
```console
Zulip development environment setup succeeded!
(zulip-server) vagrant@vagrant:/srv/zulip$
```
If the `(zulip-server)` part is missing, this is because your
installation failed the first time before the Zulip virtualenv was
created. You can fix this by just closing the shell and running
`vagrant ssh` again, or using `source .venv/bin/activate`.
Finally, if you encounter any issues that weren't caused by your
Internet connection, please report them! We try hard to keep Zulip
development environment provisioning free of bugs.
##### `pip install` fails during `vagrant up` on Linux
Likely causes are:
1. Networking issues
2. Insufficient RAM. Check whether you've allotted at least two
gigabytes of RAM, which is the minimum Zulip
[requires](/development/setup-recommended.md#requirements). If
not, go to your VM settings and increase the RAM, then restart
the VM.

14
docs/development/setup/unix-troubleshoot.md Normal file
View File

@@ -0,0 +1,14 @@
#### Unmet dependencies error
When running `vagrant up` or `provision`, if you see the following error:
```console
==> default: E:unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
```
It means that your local apt repository has been corrupted, which can
usually be resolved by executing the command:
```console
$ apt-get -f install
```

36
docs/development/setup/vagrant-halt.md Normal file
View File

@@ -0,0 +1,36 @@
To shut down but preserve the development environment so you can use
it again later use `vagrant halt` or `vagrant suspend`.
You can do this from the same Terminal/Git BASH window that is running
run-dev by pressing ^C to halt the server and then typing `exit`. Or you
can halt Vagrant from another Terminal/Git BASH window.
From the window where run-dev is running:
```console
2016-05-04 18:33:13,330 INFO 127.0.0.1 GET 200 92ms /register/ (unauth@zulip via ?)
^C
KeyboardInterrupt
(zulip-server) vagrant@vagrant:/srv/zulip$ exit
logout
Connection to 127.0.0.1 closed.
$
```
Now you can suspend the development environment:
```console
$ vagrant suspend
==> default: Saving VM state and suspending execution...
```
If `vagrant suspend` doesn't work, try `vagrant halt`:
```console
$ vagrant halt
==> default: Attempting graceful shutdown of VM...
```
Check out the Vagrant documentation to learn more about
[suspend](https://www.vagrantup.com/docs/cli/suspend.html) and
[halt](https://www.vagrantup.com/docs/cli/halt.html).

15
docs/development/setup/vagrant-rebuild.md Normal file
View File

@@ -0,0 +1,15 @@
If you ever want to recreate your development environment again from
scratch (e.g., to test a change you've made to the provisioning
process, or because you think something is broken), you can do so
using `vagrant destroy` and then `vagrant up`. This will usually be
much faster than the original `vagrant up` since the base image is
already cached on your machine (it takes about 5 minutes to run with a
fast Internet connection).
Any additional programs (e.g., Zsh, emacs, etc.) or configuration that
you may have installed in the development environment will be lost
when you recreate it. To address this, you can create a script called
`tools/custom_provision` in your Zulip Git checkout; and place any
extra setup commands there. Vagrant will run `tools/custom_provision`
every time you run `vagrant provision` (or create a Vagrant guest via
`vagrant up`).

11
docs/development/setup/vagrant-resume.md Normal file
View File

@@ -0,0 +1,11 @@
When you're ready to work on Zulip again, run `vagrant up` (no need to
pass the `--provider` option required above). You will also need to
connect to the virtual machine with `vagrant ssh` and re-start the
Zulip server:
```console
$ vagrant up
$ vagrant ssh
(zulip-server) vagrant@vagrant:/srv/zulip$ ./tools/run-dev
```

64
docs/development/setup/vagrant-ssh.md Normal file
View File

@@ -0,0 +1,64 @@
Once `vagrant up` has completed, connect to the development environment
with `vagrant ssh`:
```console
$ vagrant ssh
```
You should see output that starts like this:
```console
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-92-generic x86_64)
```
Congrats, you're now inside the Zulip development environment!
You can confirm this by looking at the command prompt, which starts
with `(zulip-server) vagrant@`. If it just starts with `vagrant@`, your
provisioning failed and you should look at the
[troubleshooting section](/development/setup-recommended.md#troubleshooting-and-common-errors).
Next, start the Zulip server:
```console
(zulip-server) vagrant@vagrant:/srv/zulip$ ./tools/run-dev
```
You will see something like:
```console
Starting Zulip on:
http://localhost:9991/
Internal ports:
9991: Development server proxy (connect here)
9992: Django
9993: Tornado
9994: webpack
Tornado server (re)started on port 9993
2023-12-15 20:57:14.206 INFO [process_queue] 13 queue worker threads were launched
frontend:
frontend (webpack 5.89.0) compiled successfully in 8054 ms
```
Now the Zulip server should be running and accessible. Verify this by
navigating to <http://localhost:9991/devlogin> in the browser on your main machine.
You should see something like this:
![Image of Zulip devlogin](/images/zulip-devlogin.png)
The Zulip server will continue to run and send output to the terminal window.
When you navigate to Zulip in your browser, check your terminal and you
should see something like:
```console
2016-05-04 18:21:57,547 INFO 127.0.0.1 GET 302 582ms (+start: 417ms) / (unauth@zulip via ?)
[04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0
2016-05-04 18:21:57,568 INFO 127.0.0.1 GET 301 4ms /login (unauth@zulip via ?)
[04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0
2016-05-04 18:21:57,819 INFO 127.0.0.1 GET 200 209ms (db: 7ms/2q) /login/ (unauth@zulip via ?)
```

25
docs/development/setup/vagrant-up-details.md Normal file
View File

@@ -0,0 +1,25 @@
The first time you run this command it will take some time because Vagrant
does the following:
- downloads the base Ubuntu 22.04 virtual machine/Docker image
- configures this virtual machine/container for use with Zulip,
- creates a shared directory mapping your clone of the Zulip code inside the
virtual machine/container at `~/zulip`
- runs the `./tools/provision` script inside the virtual machine/container, which
downloads all required dependencies, sets up the Python environment for
the Zulip development server, and initializes a default test
database. We call this process "provisioning", and it is documented
in some detail in our [dependencies documentation](/subsystems/dependencies.md).
You will need an active internet connection during the entire
process. (See [Specifying a proxy](/development/setup-recommended.md#specifying-a-proxy) if you need a
proxy to access the internet.) `vagrant up` can fail while
provisioning if your Internet connection is unreliable. To retry, you
can use `vagrant provision` (`vagrant up` will just boot the guest
without provisioning after the first time). Other common issues are
documented in the
[Troubleshooting and common errors](/development/setup-recommended.md#troubleshooting-and-common-errors)
section. If that doesn't help, please visit
[#provision help](https://chat.zulip.org/#narrow/channel/21-provision-help)
in the [Zulip development community server](https://zulip.com/development-community/) for
real-time help.

7
docs/development/setup/vagrant-up.md Normal file
View File

@@ -0,0 +1,7 @@
Change into the zulip directory and tell Vagrant to start the Zulip
development environment with `vagrant up`:
```console
$ cd zulip
$ vagrant up --provider=docker
```

18
docs/development/setup/vagrant-update.md Normal file
View File

@@ -0,0 +1,18 @@
If after rebasing onto a new version of the Zulip server, you receive
new errors while starting the Zulip server or running tests, this is
probably not because Zulip's `main` branch is broken. Instead, this
is likely because we've recently merged changes to the development
environment provisioning process that you need to apply to your
development environment. To update your environment, you'll need to
re-provision your Vagrant machine using `vagrant provision` (this just
runs `tools/provision` from your Zulip checkout inside the Vagrant
guest); this should complete in about a minute.
After provisioning, you'll want to
[(re)start the Zulip development server](/development/setup-recommended.md#step-3-start-the-development-environment).
If you run into any trouble, [#provision
help](https://chat.zulip.org/#narrow/channel/21-provision-help) in the
[Zulip development community
server](https://zulip.com/development-community/) is a great place to ask for
help.

34
docs/development/setup/vscode-vagrant.md Normal file
View File

@@ -0,0 +1,34 @@
If your preferred editor is Visual Studio Code, the [Visual Studio
Code Remote - SSH](https://code.visualstudio.com/docs/remote/ssh)
extension is recommended for editing files when developing with
Vagrant. When you have it installed, you can run:
```console
$ code .
```
to open VS Code connected to your Vagrant environment. See the
[Remote development over SSH](https://code.visualstudio.com/docs/remote/ssh-tutorial) tutorial for more information.
When using this plugin with Vagrant, you will want to run the command
`vagrant ssh-config` from your `zulip` folder:
```console
$ vagrant ssh-config
Host default
HostName 127.0.0.1
User vagrant
Port 2222
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile /PATH/TO/zulip/.vagrant/machines/default/docker/private_key
IdentitiesOnly yes
LogLevel FATAL
PubkeyAcceptedKeyTypes +ssh-rsa
HostKeyAlgorithms +ssh-rsa
```
Then copy that config into your `~/.ssh/config` file. You may want to change
the host name from `default` to something more descriptive, like `zulip`.
Finally, refresh the known remotes in Visual Studio Code's Remote Explorer.

196
docs/development/setup/winvm-troubleshoot.md Normal file
View File

@@ -0,0 +1,196 @@
#### Vagrant was unable to mount VirtualBox shared folders
For the following error:
```console
Vagrant was unable to mount VirtualBox shared folders. This is usually
because the filesystem "vboxsf" is not available. This filesystem is
made available via the VirtualBox Guest Additions and kernel
module. Please verify that these guest additions are properly
installed in the guest. This is not a bug in Vagrant and is usually
caused by a faulty Vagrant box. For context, the command attempted
was:
mount -t vboxsf -o uid=1000,gid=1000 keys /keys
```
If this error starts happening unexpectedly, then just run:
```console
$ vagrant halt
$ vagrant up
```
to reboot the guest. After this, you can do `vagrant provision` and
`vagrant ssh`.
#### os.symlink error
If you receive the following error while running `vagrant up`:
```console
==> default: Traceback (most recent call last):
==> default: File "./emoji_dump.py", line 75, in <module>
==> default:
==> default: os.symlink('unicode/{}.png'.format(code_point), 'out/{}.png'.format(name))
==> default: OSError
==> default: :
==> default: [Errno 71] Protocol error
```
Then Vagrant was not able to create a symbolic link.
First, if you are using Windows, **make sure you have run Git BASH (or
Cygwin) as an administrator**. By default, only administrators can
create symbolic links on Windows. Additionally [UAC][windows-uac], a
Windows feature intended to limit the impact of malware, can prevent
even administrator accounts from creating symlinks. [Turning off
UAC][disable-uac] will allow you to create symlinks. You can also try
some of the solutions mentioned
[here](https://superuser.com/questions/124679/how-do-i-create-a-link-in-windows-7-home-premium-as-a-regular-user).
[windows-uac]: https://docs.microsoft.com/en-us/windows/security/identity-protection/user-account-control/how-user-account-control-works
[disable-uac]: https://stackoverflow.com/questions/15320550/why-is-secreatesymboliclinkprivilege-ignored-on-windows-8
If you ran Git BASH as administrator but you already had VirtualBox
running, you might still get this error because VirtualBox is not
running as administrator. In that case: close the Zulip VM with
`vagrant halt`; close any other VirtualBox VMs that may be running;
exit VirtualBox; and try again with `vagrant up --provision` from a
Git BASH running as administrator.
Second, VirtualBox does not enable symbolic links by default. Vagrant
starting with version 1.6.0 enables symbolic links for VirtualBox shared
folder.
You can check to see that this is enabled for your virtual machine with
`vboxmanage` command.
Get the name of your virtual machine by running `vboxmanage list vms` and
then print out the custom settings for this virtual machine with
`vboxmanage getextradata YOURVMNAME enumerate`:
```console
$ vboxmanage list vms
"zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d}
$ vboxmanage getextradata zulip_default_1462498139595_55484 enumerate
Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1
Key: supported, Value: false
```
If you see "command not found" when you try to run VBoxManage, you need to
add the VirtualBox directory to your path. On Windows this is mostly likely
`C:\Program Files\Oracle\VirtualBox\`.
If `vboxmanage enumerate` prints nothing, or shows a value of 0 for
VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable
symbolic links by running this command in Terminal/Git BASH/Cygwin:
```console
$ vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip 1
```
The virtual machine needs to be shut down when you run this command.
#### Hyper-V error messages
If you get an error message on Windows about lack of Windows Home
support for Hyper-V when running `vagrant up`, the problem is that
Windows is incorrectly attempting to use Hyper-V rather than
Virtualbox as the virtualization provider. You can fix this by
explicitly passing the virtualbox provider to `vagrant up`:
```console
$ vagrant up --provide=virtualbox
```
#### Connection timeout on `vagrant up`
If you see the following error after running `vagrant up`:
```console
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default: Error: Connection timeout. Retrying...
default: Error: Connection timeout. Retrying...
default: Error: Connection timeout. Retrying...
```
A likely cause is that hardware virtualization is not enabled for your
computer. This must be done via your computer's BIOS settings. Look for a
setting called VT-x (Intel) or (AMD-V).
If this is already enabled in your BIOS, double-check that you are running a
64-bit operating system.
For further information about troubleshooting Vagrant timeout errors [see
this post](https://stackoverflow.com/questions/22575261/vagrant-stuck-connection-timeout-retrying#22575302).
#### VBoxManage errors related to VT-x or WHvSetupPartition
```console
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.
Command: ["startvm", "8924a681-b4e4-4b7a-96f2-4cb11619f123", "--type", "headless"]
Stderr: VBoxManage.exe: error: (VERR_NEM_MISSING_KERNEL_API).
VBoxManage.exe: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ConsoleWrap, interface IConsole
```
or
```console
Stderr: VBoxManage.exe: error: Call to WHvSetupPartition failed: ERROR_SUCCESS (Last=0xc000000d/87) (VERR_NEM_VM_CREATE_FAILED)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ConsoleWrap, interface IConsole
```
First, ensure that hardware virtualization support (VT-x or AMD-V) is
enabled in your BIOS.
If the error persists, you may have run into an incompatibility
between VirtualBox and Hyper-V on Windows. To disable Hyper-V, open
command prompt as administrator, run
`bcdedit /set hypervisorlaunchtype off`, and reboot. If you need to
enable it later, run `bcdedit /deletevalue hypervisorlaunchtype`, and
reboot.
#### OSError: [Errno 26] Text file busy
```console
default: Traceback (most recent call last):
…
default: File "/srv/zulip-py3-venv/lib/python3.6/shutil.py", line 426, in _rmtree_safe_fd
default: os.rmdir(name, dir_fd=topfd)
default: OSError: [Errno 26] Text file busy: 'baremetrics'
```
This error is caused by a
[bug](https://www.virtualbox.org/ticket/19004) in recent versions of
the VirtualBox Guest Additions for Linux on Windows hosts. You can
check the running version of VirtualBox Guest Additions with this
command:
```console
$ vagrant ssh -- 'sudo modinfo -F version vboxsf'
```
The bug has not been fixed upstream as of this writing, but you may be
able to work around it by downgrading VirtualBox Guest Additions to
5.2.44. To do this, create a `~/.zulip-vagrant-config` file and add
this line:
```text
VBOXADD_VERSION 5.2.44
```
Then run these commands (yes, reload is needed twice):
```console
$ vagrant plugin install vagrant-vbguest
$ vagrant reload
$ vagrant reload --provision
```

42
docs/development/setup/wsl-rebuild.md Normal file
View File

@@ -0,0 +1,42 @@
If you ever want to recreate your development environment again from
scratch (e.g., to test a change you've made to the provisioning
process, or because you think something is broken), you can do so
using the following steps:
1. To find the distribution name to unregister (delete), open Command
Prompt or PowerShell and use the following command:
```console
$ wsl --list --verbose
```
If you are unsure about which distribution to unregister, you can log
into the WSL distributions to ensure you are deleting the one
containing your development environment using the command:
```console
wsl -d <Distribution Name>
```
2. To uninstall your WSL distribution, enter the command:
```console
$ wsl --unregister <Distribution Name>
```
For more information, checkout the [official documentation for WSL
commands](https://learn.microsoft.com/en-us/windows/wsl/basic-commands#unregister-or-uninstall-a-linux-distribution)
3. **Next, follow the setup instructions**, starting from [[Step 1:
Install
prerequisites]](/development/setup-recommended.md#step-1-install-prerequisites)
If you just want to rebuild the development database, the following is
much faster:
```console
$ ./tools/rebuild-dev-database
```
For more details, see the [schema migration
documentation](/subsystems/schema-migrations.md#local-schema-rebuilds).

75
docs/development/setup/wsl-troubleshoot.md Normal file
View File

@@ -0,0 +1,75 @@
WSL2 users often encounter issues where services fail to start or remain inactive. Follow the steps below to diagnose and resolve such problems.
#### 1. Check the Status of the Service
To verify if a service is running, use the following command:
```console
$ systemctl status <service_name>
```
If the service is inactive, you can attempt to start it with:
```console
$ systemctl start <service_name>
```
#### 2. Diagnose Port Conflicts
Services like `postgresql` may fail to start due to port conflicts. These conflicts can be caused by:
- Other services running in Windows.
- Services running in another WSL2 instance.
#### Resolving Port Conflicts with Other WSL Instances
To resolve port conflicts with another WSL2 instance, stop the conflicting instance using the following command:
```console
> wsl -t <WSL_Instance_Name>
```
After stopping the conflicting instance, restart your WSL instance with:
```console
> wsl -d <Your_Zulip_Instance_Name>
```
#### Resolving Port Conflicts with Services Running on Windows
To resolve conflicts caused by Windows processes:
1. Identify the process using the conflicting port by running:
```console
> Get-Process -Id (Get-NetTCPConnection -LocalPort <your_port_number>).OwningProcess
```
2. If a process is found, terminate it using:
```console
> taskkill /PID <pid> /F
```
3. Restart the Service or Enable Auto-Start
After resolving port conflicts, try restarting the service using:
```console
$ systemctl start <service_name>
```
To ensure the service always starts on boot, enable it with:
```console
$ systemctl enable <service_name>
```
---
#### Additional Tips
- Use `wsl --list` to view all running WSL2 instances and their states.
- Avoid overlapping port usage between WSL2 instances and Windows processes.
- Keep a record of services and their associated port numbers to prevent conflicts in the future.
- Ensure that you use a fresh WSL instance to setup the Zulip development environment to avoid dependency conflicts.

133
docs/development/test-install.md Normal file
View File

@@ -0,0 +1,133 @@
# Testing the installer
Zulip's install process is tested as part of [its continuous
integrations suite][ci], but that only tests the most common
configurations; when making changes to more complicated [installation
options][installer-docs], Zulip provides tooling to repeatedly test
the installation process in a clean environment each time.
[ci]: https://github.com/zulip/zulip/actions/workflows/production-suite.yml?query=branch%3Amain
[installer-docs]: ../production/install.md
## Configuring
Using the test installer framework requires a Linux operating system;
it will not work on WSL, for instance. It requires at least 3G of
RAM, in order to accommodate the VMs and the steps which build the
release assets.
To begin, install the LXC toolchain:
```bash
sudo apt-get install lxc lxc-utils
```
All LXC commands (and hence many parts of the test installer) will
need to be run as root.
## Running a test install
### Build and unpack a release tarball
You only need to do this step once per time you work on a set of
changes, to refresh the package that the installer uses. The installer
doesn't work cleanly out of a source checkout; it wants a release
checkout, so we build a tarball of one of those first:
```bash
./tools/build-release-tarball test-installer
```
This will produce a file in /tmp, which it will print out the path to
as the last step; for example,
`/tmp/tmp.fepqqNBWxp/zulip-server-test-installer.tar.gz`
Next, unpack that file into a local directory; we will make any
changes we want in our source checkout and copy them into this
directory. The test installer needs the release directory to be named
`zulip-server`, so we rename it and move it appropriately. In the
first line, you'll need to substitute the actual path that you got for
the tarball, above:
```bash
tar xzf /tmp/tmp.fepqqNBWxp/zulip-server-test-installer.tar.gz
mkdir zulip-test-installer
mv zulip-server-test-installer zulip-test-installer/zulip-server
```
You should delete and re-create this `zulip-test-installer` directory
(using these steps) if you are working on a different installer
branch, or a significant time has passed since you last used it.
### Test an install
The `test-install` tooling takes a distribution release name
(e.g., "jammy"), the path to an unpacked release directory
or tarball, and then any of the normal options you want to pass down
into the installer.
For example, to test an install onto Ubuntu 22.04 "Jammy", we might
call:
```bash
sudo ./tools/test-install/install \
-r jammy \
./zulip-test-installer/ \
--hostname=zulip.example.net \
--email=username@example.net
```
The first time you run this command for a given distribution, it will
build a "base" image for that to use on subsequent times; this will
take a while.
### See running containers after installation
Regardless of if the install succeeds or fails, it will stay running
so you can inspect it. You can see all of the containers which are
running, and their randomly-generated names, by running:
```bash
sudo lxc-ls -f
```
### Connect to a running container
After using `lxc-ls` to list containers, you can choose one of them
and connect to its terminal:
```bash
sudo lxc-attach --clear-env -n zulip-install-jammy-PUvff
```
### Stopping and destroying containers
To destroy all containers (but leave the base containers, which speed
up the initial install):
```bash
sudo ./tools/test-install/destroy-all -f
```
To destroy just one container:
```bash
sudo lxc-destroy -f -n zulip-install-jammy-PUvff
```
### Iterating on the installer
Iterate on the installer by making changes to your source tree,
copying them into the release directory, and re-running the installer,
which will start up a new container. Here, we update just the
`scripts` and `puppet` directories of the release directory:
```bash
rsync -az scripts puppet zulip-test-installer/zulip-server/
sudo ./tools/test-install/install \
-r jammy \
./zulip-test-installer/ \
--hostname=zulip.example.net \
--email=username@example.net
```

103
docs/development/using.md Normal file
View File

@@ -0,0 +1,103 @@
# Using the development environment
This page describes the basic edit/refresh workflows for working with
the Zulip development environment. Generally, the development
environment will automatically update as soon as you save changes
using your editor. Details for work on the [server](#server),
[web app](#web), and [mobile apps](#mobile) are below.
If you're working on authentication methods or need to use the [Zulip
REST API][rest-api], which requires an API key, see [authentication in
the development environment][authentication-dev-server].
## Common
- Zulip's `main` branch moves quickly, and you should rebase
constantly with, for example,
`git fetch upstream; git rebase upstream/main` to avoid developing
on an old version of the Zulip codebase (leading to unnecessary
merge conflicts).
- Remember to run `tools/provision` to update your development
environment after switching branches; it will run in under a second
if no changes are required.
- After making changes, you'll often want to run the
[linters](../testing/linters.md) and relevant [test
suites](../testing/testing.md). Consider using our [Git pre-commit
hook](../git/zulip-tools.md#set-up-git-repo-script) to
automatically lint whenever you make a commit.
- All of our test suites are designed to support quickly testing just
a single file or test case, which you should take advantage of to
save time.
- Many useful development tools, including tools for rebuilding the
database with different test data, are documented in-app at
`https://localhost:9991/devtools`.
- If you want to restore your development environment's database to a
pristine state, you can use `./tools/rebuild-dev-database`.
## Server
- For changes that don't affect the database model, the Zulip
development environment will automatically detect changes and
restart:
- The main Django/Tornado server processes are run on top of
Django's [manage.py runserver][django-runserver], which will
automatically restart them when you save changes to Python code
they use. You can watch this happen in the `run-dev` console
to make sure the backend has reloaded.
- The Python queue workers will also automatically restart when you
save changes, as long as they haven't crashed (which can happen if
they reloaded into a version with a syntax error).
- If you change the database schema (`zerver/models/*.py`), you'll need
to use the [Django migrations
process](../subsystems/schema-migrations.md); see also the [new
feature tutorial][new-feature-tutorial] for an example.
- While testing server changes, it's helpful to watch the `run-dev`
console output, which will show tracebacks for any 500 errors your
Zulip development server encounters (which are probably caused by
bugs in your code).
- To manually query Zulip's database interactively, use
`./manage.py shell` or `manage.py dbshell`.
- The database(s) used for the automated tests are independent from
the one you use for manual testing in the UI, so changes you make to
the database manually will never affect the automated tests.
## Web
- Once the development server (`run-dev`) is running, you can visit
<http://localhost:9991/> in your browser.
- By default, the development server homepage just shows a list of the
users that exist on the server and you can log in as any of them by
just clicking on a user.
- This setup saves time for the common case where you want to test
something other than the login process.
- You can test the login or registration process by clicking the
links for the normal login page.
- Most changes will take effect automatically. Details:
- If you change CSS files, your changes will appear immediately via
webpack hot module replacement.
- If you change JavaScript code (`web/src`) or Handlebars
templates (`web/templates`), the browser window will be
reloaded automatically.
- For Jinja2 backend templates (`templates/*`), you'll need to reload
the browser window to see your changes.
- Any JavaScript exceptions encountered while using the web app in a
development environment will be displayed as a large notice, so you
don't need to watch the JavaScript console for exceptions.
- Both Chrome and Firefox have great debuggers, inspectors, and
profilers in their built-in developer tools.
- `debug.js` has some occasionally useful JavaScript profiling code.
## Mobile
See the mobile project's documentation on [getting set up to develop
and contribute to the mobile app][mobile-development-guide], including
how to [run the app against your development
server][mobile-dev-server-guide].
[rest-api]: https://zulip.com/api/rest
[authentication-dev-server]: authentication.md
[django-runserver]: https://docs.djangoproject.com/en/5.0/ref/django-admin/#runserver
[new-feature-tutorial]: ../tutorials/new-feature-tutorial.md
[testing-docs]: ../testing/testing.md
[mobile-development-guide]: https://github.com/zulip/zulip-flutter/blob/main/docs/setup.md
[mobile-dev-server-guide]: https://github.com/zulip/zulip-flutter/blob/main/docs/howto/dev-server.md