upstream-zulip/docs/webhooks/incoming-webhooks-walkthrough.md

# Writing an incoming webhook

This guide goes through the process of creating a simple incoming webhook
integration, using the [Zulip Hello World](https://zulip.com/integrations/helloworld)
integration as an example.

The example integration receives an HTTP POST request with JSON data
about Wikipedia's featured article of the day from a fictional
third-party service, formats that data into a "hello" message, and sends
that message to the specified conversation in Zulip.

Before you get started, you'll want to set up the [Zulip development
environment](../development/overview.md).

## Step 0: Create fixtures

The first step in creating an incoming webhook integration is to examine
the data that the third-party service you're working with will send to
Zulip.

Use [Zulip's JSON integration](https://zulip.com/integrations/json),
<https://webhook.site/>, or a similar tool to capture outgoing webhook
payload(s) from the service. Examining this data allows you to:

- Determine how you will structure your webhook integration code,
  including what event types your integration should support and how.
- Create fixtures. A test fixture is a small file containing test data
  for one kind of event or scenario. Fixtures enable the testing of
  webhook integration code without the need to actually contact the
  service being integrated.

You'll want to write a test for each distinct event type your incoming
webhook integration supports, and you'll need fixtures for these
tests. Depending on the type of data the third-party service sends,
your fixtures may contain JSON, URL encoded text, or some other kind
of data. [Step 5: Create automated tests](#step-5-create-automated-tests)
and [our testing documentation](../testing/testing.md) have further
details about writing tests and using test fixtures.

Because the Zulip Hello World is very simple and does only one thing,
it requires only one fixture,
`zerver/webhooks/helloworld/fixtures/hello.json`.

```json
{
  "featured_title":"Marilyn Monroe",
  "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe"
}
```

### Fixture guidelines

Fixtures for a real third-party service must always be actual
captured payloads, or payloads from the service's official API
documentation, never hand-written, invented, or AI-generated. A
fabricated payload does not accurately reflect what the service
really sends, so an integration developed against one may not work
with real events.

Collect only the fixtures your integration's tests need. If you want
to test how the integration handles different values of a URL query
parameter, prefer writing tests that reuse one fixture, instead of
capturing several fixtures that are almost identical.

When setting up the integration, make sure not to use any personal
details that you would not want to be part of the fixtures. Use
fixtures exactly as you captured them; do not edit their contents. An
edited fixture no longer records what the service really sends, and
subtle mistakes are easy to introduce. If you need a variation,
capture a new fixture instead.

If you absolutely must remove personal details from a captured
payload, commit the captured fixture locally first, then replace the
personal details with equally realistic strings and amend the commit.
This makes the scrub a reviewable diff, and the original data never
reaches the public history.

Fixture data appears in the messages and topics that your integration
produces, and in the auto-generated example screenshots in the
integration's documentation, so when setting up the third-party
service, use realistic names for projects, tasks, and other entities.
Where you can, use the same names and email addresses as the shared
example content in `zerver/webhooks/fixtureless_integrations.py`, so
that the example screenshots are consistent across integrations.

Fixture files are named descriptively, in snake_case, after the event
(and variant) they capture. The Hello World integration's fixture is
`hello.json`; for a richer service, a fixture might be named
something like `issue_created_with_assignee.json`.

For integrations that send the event type or other important
information as part of the HTTP header, the value of the header is
encoded in the first part of the fixture's filename, separated from
the rest by a double underscore. Refer to
[the section on custom HTTP headers](incoming-webhooks-reference.md#custom-http-headers)
in the reference guide for more details.

Once captured, fixtures rarely need to be updated: as long as the
fields your integration reads are still what the service sends, there
is no need to update fixtures just because the service has added new
fields over the years.

## Step 1: Initialize the python package

In the `zerver/webhooks/` directory, create new subdirectory for the
incoming webhook integration. In our example, it is `helloworld`. The new
directory will be a python package, so you should also create an empty
`__init__.py` file in that directory via, for
example, `touch zerver/webhooks/helloworld/__init__.py`.

## Step 2: Write the main webhook code

The majority of the code for your new integration will be in a single
python file named `view.py`. The Zulip Hello World integration code is
found in `zerver/webhooks/helloworld/view.py`:

```python
from django.http import HttpRequest, HttpResponse

from zerver.decorator import webhook_view
from zerver.lib.response import json_success
from zerver.lib.typed_endpoint import JsonBodyPayload, typed_endpoint
from zerver.lib.validator import WildValue, check_string
from zerver.lib.webhooks.common import check_send_webhook_message
from zerver.models import UserProfile


@webhook_view("HelloWorld")
@typed_endpoint
def api_helloworld_webhook(
    request: HttpRequest,
    user_profile: UserProfile,
    *,
    payload: JsonBodyPayload[WildValue],
) -> HttpResponse:
    # construct the body of the message
    body = "Hello! I am happy to be here! :smile:"

    # try to add the Wikipedia article of the day
    body_template = (
        "\nThe Wikipedia featured article for today is **[{featured_title}]({featured_url})**"
    )
    body += body_template.format(
        featured_title=payload["featured_title"].tame(check_string),
        featured_url=payload["featured_url"].tame(check_string),
    )

    topic_name = "Hello World"

    # send the message
    check_send_webhook_message(request, user_profile, topic_name, body)

    return json_success(request)
```

The above code imports the required functions and defines the main
webhook handler, `api_helloworld_webhook`, decorating it with
`webhook_view` and `typed_endpoint`.

### Decorators

The `typed_endpoint` decorator allows the integration to access request
variables with `JsonBodyPayload[WildValue]`. You can find more about
`JsonBodyPayload` and request variables in [the tutorial on writing
views](../tutorials/writing-views.md#request-variables).

You must pass the name of the integration to the `webhook_view` decorator.
That name will be used to describe the integration in Zulip's analytics
pages. In the example, the integration's name is `HelloWorld`. To be
consistent with other incoming webhook integrations, you should use the
name of the third-party service in camel case, spelled as the service
spells its own name, with the exception that the first letter should be
upper case even if it's lower case in the service's name/brand.

The `webhook_view` decorator indicates that the third-party service will
send the authorization as an API key in the query parameters. If the
third-party service uses HTTP basic authentication, you would instead use
the `authenticated_rest_api_view` decorator.

### Main webhook handler

The webhook function should be named as in the example above,
`api_helloworld_webhook`, replacing `helloworld` with the name of the
integration, in lower case.

At minimum, the webhook function must accept `request` (Django
[HttpRequest](https://docs.djangoproject.com/en/5.0/ref/request-response/#django.http.HttpRequest)
object), and `user_profile` (Zulip's user object). You can also define
additional parameters using the `typed_endpoint` decorator.

In the Zulip Hello World example above, there is also a `payload`
parameter, which is populated from the body of the HTTP POST request.

The main function defines the body of the message using [Zulip's message
formatting](https://zulip.com/api/message-formatting). For example,
`:smile:` in the initial sentence of the message body indicates an emoji.
The data from the JSON payload is used for the link to the Wikipedia
featured article of the day.

Sometimes, it may occur that a JSON payload does not contain all the
required keys that an integration checks for. In such a case, any
`KeyError` that is thrown is handled by the Zulip server's backend, which
will create an appropriate response.

A default topic name should be defined for cases when the [webhook
URL](incoming-webhooks-overview.md#url-specification) specifies a
channel recipient, via the `stream` query parameter, but does not specify
a topic.

The formatted message is sent with `check_send_webhook_message`, which
will validate the message and do the following:

- Send a channel message if the `stream` query parameter is specified in
  the [webhook URL](incoming-webhooks-overview.md#url-specification).
- Send a direct message to the owner of the webhook bot if the `stream`
  query parameter isn't specified.

Finally, a response is sent with a 200 HTTP status with a JSON format
success message via `json_success(request)`.

## Step 3: Create an API endpoint for the webhook

In order for an incoming webhook integration to be externally available,
it must be mapped to a URL. This "registration" is done in
`zerver/lib/integrations.py`.

Look for the lines beginning with:

```python
INCOMING_WEBHOOK_INTEGRATIONS: List[IncomingWebhookIntegration] = [
```

In that list, find the entry for the example Hello World integration:

```python
    IncomingWebhookIntegration(
        "helloworld", ["misc"], [WebhookScreenshotConfig("hello.json")], display_name="Hello World"
    ),
```

This tells the Zulip API to call the `api_helloworld_webhook` function in
`zerver/webhooks/helloworld/view.py` when it receives a request at
`/api/v1/external/helloworld`.

This entry also adds the example Hello World integration to Zulip's main
[integrations documentation](https://zulip.com/integrations/). The second
argument defines the categories that include the integration in said
documentation. The third argument defines the configuration for
generating (and updating) the example screenshot included in the
integration's documentation page. And the final argument defines the
display name for the integration in the documentation. [Step 6: Create
documentation](#step-6-document-the-integration) has more details about
creating end-user documentation.

In cases where an incoming webhook integration supports optional URL
parameters, use the `url_options` argument described in
[the custom URL parameters section](incoming-webhooks-reference.md#custom-url-query-parameters)
of the reference guide to create the corresponding UI element in the modal
for [generating an integration URL](https://zulip.com/help/generate-integration-url).

## Step 4: Manually test the webhook

You'll want to manually test your webhook implementation in the Zulip
development environment.

### Command line tools

There are two command line tools you can use. For either one of them,
you'll need to [get an API key](https://zulip.com/api/api-keys) for
an [incoming webhook bot](https://zulip.com/help/add-a-bot-or-integration)
in your Zulip development environment. Replace `<api_key>` with the
bot's API key in the examples below. This is how the server knows that
the request was made by an authorized user.

#### curl

Using curl, with the Zulip development server running in a separate
console window:

```bash
curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://localhost:9991/api/v1/external/helloworld\?api_key\=<api_key>
```

After running the above command, you should see something similar to:

```json
{"msg":"","result":"success"}
```

And if you log-in to the web app as the bot's owner, you should see a new
direct message from the bot.

#### `send_webhook_fixture_message` management command

Using `manage.py` from within the Zulip development environment:

```console
(zulip-server) vagrant@vagrant:/srv/zulip$
./manage.py send_webhook_fixture_message \
    --fixture=zerver/webhooks/helloworld/fixtures/hello.json \
    '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'
```

After running the above command, you should see something similar to:

```
2016-07-07 15:06:59,187 INFO     127.0.0.1       POST    200 143ms (mem: 6ms/13) (md: 43ms/1) (db: 20ms/9q) (+start: 147ms) /api/v1/external/helloworld (helloworld-bot@zulip.com via ZulipHelloWorldWebhook)
```

Some webhooks require custom HTTP headers, which can be passed using
`./manage.py send_webhook_fixture_message --custom-headers`. For
example:

    --custom-headers='{"X-Custom-Header": "value"}'

The format is a JSON object, so make sure that the header names do
not contain any spaces in them and that you use the precise quoting
approach shown above.

For more information about `manage.py` command-line tools in Zulip, see
the [management commands](../production/management-commands.md)
documentation.

### Integrations dev panel

This is a GUI tool that you can use to test your webhook in the Zulip
development environment.

1. Run `./tools/run-dev`, and use a web browser to go to
   `http://localhost:9991/devtools/integrations/`.
1. Select a bot, an integration and a fixture from the dropdown menus.
1. Click **Send**. The webhook notification message will be sent to the
   default Zulip organization in your development environment.

By having Zulip open in one browser tab and this tool in another, you can
quickly tweak your webhook code and send sample messages for different
test fixtures.

Custom HTTP headers must be entered as a JSON object, if you want to
use any. Feel free to use 4-spaces as tabs for indentation if you'd like.

### Testing with the third-party service

:::{warning}
The development environment is not hardened against hostile traffic. Only
expose it while you are actively testing.
:::

To have the third-party service deliver the webhooks itself, it needs to
be able to reach your development server over the internet.

You don't need to reconfigure the development server for this. You can
use a tunneling tool, such as [UltraHook](https://www.ultrahook.com/) or
[localtunnel](https://github.com/localtunnel/localtunnel), to expose
`http://localhost:9991` at a temporary public URL. When you configure the
webhook in the third-party service, use that temporary public URL with the
path for your incoming webhook integration, e.g.,
`<temporary-public-url>/api/v1/external/helloworld?api_key=<api_key>`.

End-to-end testing with live data can help you spot why something isn't
working, or whether the service is using [custom HTTP
headers](incoming-webhooks-reference.md#custom-http-headers) that your
integration needs to handle.

If you instead need the development server to be reachable from another
device on your own network, the mobile project's guide to [running the app
against a development
server](https://github.com/zulip/zulip-flutter/blob/main/docs/howto/dev-server.md)
covers configuring the server to listen on all network interfaces.

## Step 5: Create automated tests

Every incoming webhook integration should have a corresponding `tests.py`
file. The Hello World integration's tests are in
`zerver/webhooks/helloworld/tests.py`.

The test class should be named `<WebhookName>HookTests`, and it should
inherit the `WebhookTestCase` base class. For Zulip's Hello World
integration, the test class is `HelloWorldHookTests`.

```python
class HelloWorldHookTests(WebhookTestCase):
    DIRECT_MESSAGE_URL_TEMPLATE = "/api/v1/external/helloworld?&api_key={api_key}"

    # Note: Include a test function per each distinct message condition your integration supports
    def test_hello_message(self) -> None:
        expected_topic_name = "Hello World"
        expected_message = "Hello! I am happy to be here! :smile:\nThe Wikipedia featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/Marilyn_Monroe)**"

        # use fixture named hello.json
        self.check_webhook(
            "hello",
            expected_topic_name,
            expected_message,
            content_type="application/x-www-form-urlencoded",
        )

    def test_pm_to_bot_owner(self) -> None:
        # Note that this is really just a test for check_send_webhook_message
        self.url_template = self.DIRECT_MESSAGE_URL_TEMPLATE
        self.url = self.build_webhook_url()
        expected_message = "Hello! I am happy to be here! :smile:\nThe Wikipedia featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**"

        self.send_and_test_private_message(
            "goodbye",
            expected_message=expected_message,
            content_type="application/x-www-form-urlencoded",
        )
```

When writing tests, you'll want to include one test function for each
distinct event type and condition that the integration supports.

If, for example, we added support for sending a goodbye message to the
Hello World webhook, then we would add another test function to
`HelloWorldHookTests` class called something like `test_goodbye_message`:

```python
    def test_goodbye_message(self) -> None:
        expected_topic_name = "Hello World"
        expected_message = "Hello! I am happy to be here! :smile:\nThe Wikipedia featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**"

        # use fixture named goodbye.json
        self.check_webhook(
            "goodbye",
            expected_topic_name,
            expected_message,
            content_type="application/x-www-form-urlencoded",
        )
```

As well as a new fixture `goodbye.json` in
`zerver/webhooks/helloworld/fixtures/`:

```json
{
  "featured_title":"Goodbye",
  "featured_url":"https://en.wikipedia.org/wiki/Goodbye"
}
```

Also, consider if the integration should have negative tests, i.e., tests
where the data from the test fixture should result in an error. For
details, see [negative tests](incoming-webhooks-reference.md#negative-tests).

Once you have written some tests, you can run just these new tests from
within the Zulip development environment with this command:

```console
./tools/test-backend zerver/webhooks/helloworld
```

You will see some script output in the console. And if all the tests have
passed, you will see:

```console
Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_goodbye_message
Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_hello_message
DONE!
```

## Step 6: Document the integration

In order for an incoming webhook integration to be used, there needs to
be documentation for end-users. All incoming webhooks are included in
Zulip's main [integrations documentation](https://zulip.com/integrations).

There are two parts to the user-facing integrations documentation.

The first is the lozenge in the grid of integrations, which shows the
integration logo and name, and is a link to the detailed documentation
for each integration.

Each integration needs a square, svg logo of the third-party service that
has been integrated with Zulip, which is saved in the
`static/images/integrations/logos` directory. The Zulip Hello World logo
can be found at `static/images/integrations/logos/helloworld.svg`.

The lozenge is generated automatically once the integration is added to
`INCOMING_WEBHOOK_INTEGRATIONS` in `zerver/lib/integrations.py`, which
supports some customization via options to the `IncomingWebhookIntegration`
class.

Second, is the detailed documentation content for the integration, which
is in a file named `doc.md` in the integration's directory. The Zulip
Hello World documentation can be found at `zerver/webhooks/helloworld/doc.md`.

Zulip has a macro-based Markdown/Jinja2 framework that includes macros for
common instructions in Zulip's webhooks/integrations documentation (e.g.,
`{!create-an-incoming-webhook.md!}` and `{!congrats.md!}`).

See [the guide on documenting an integration](../documentation/integrations.md)
for further details, including how to easily create the example
screenshot for the documentation page.

## Step 7: Prepare a pull request

When you have finished your incoming webhook integration, follow these
guidelines before pushing the code to your Zulip fork and submitting a
pull request to [zulip/zulip](https://github.com/zulip/zulip):

- Run tests, including linters, to ensure you have addressed any issues
  they report. See [testing](../testing/testing.md) and
  [linters](../testing/linters.md) for details.
- Read through [code styles and conventions](../contributing/code-style.md)
  and review your code to double-check that you've followed Zulip's
  guidelines.
- Take a look at your Git history to ensure your commits have been clear
  and logical (see [commit discipline](../contributing/commit-discipline.md)
  for tips). If not, consider revising them with `git rebase --interactive`.
  For most incoming webhook integrations, you'll want to submit a pull
  request with a single commit that has a good, clear commit message.

If you would like feedback on your integration as you go, feel free to
post a message in the [Zulip development community][czo-integrations-channel]
You can also create a [draft pull request][github-draft-pull-requests]
while you are still working on your integration. See the [git
guide](../git/pull-requests.md#create-a-pull-request) for more on Zulip's
pull request process.

[czo-integrations-channel]: https://chat.zulip.org/#narrow/channel/integrations
[github-draft-pull-requests]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#draft-pull-requests

## Advanced topics

More complex implementation or testing needs may require additional code,
beyond what the standard helper functions provide. This section discusses
some of these situations.

### Handling unexpected webhook event types

Many third-party services have dozens of different event types. In some
cases, we may choose to explicitly ignore specific events. In other cases,
there may be events that are new or events that we don't know about. In
such cases, we recommend raising `UnsupportedWebhookEventTypeError`
(found in `zerver/lib/exceptions.py`), with a string describing the
unsupported event type, like so:

```
raise UnsupportedWebhookEventTypeError(event_type)
```