upstream/upstream-watermill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

snapshot of https://github.com/ThreeDotsLabs/watermill.git (master, 2026-09-15; blobs > 131072B omitted)
Some checks failed
master / ci (push) Failing after 0s
Details

Browse Source
This commit is contained in:
upstream-mirror
2026-09-15 21:27:46 +05:00
commit ce933d1277
538 changed files with 46504 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
docs/content/advanced/delayed-messages.md Normal file
View File

@@ -0,0 +1,44 @@
+++
title = "Delayed Messages"
description = "Receive messages with a delay"
weight = -40
draft = false
bref = "Receive messages with a delay"
+++
Delaying events or commands is a common use case in many applications.
For example, you may want to send the user a reminder after a few days of signing up.
It's not a complex logic to implement, but you can leverage messages to use it out of the box.
## Delay Metadata
Watermill's [`delay`](https://github.com/ThreeDotsLabs/watermill/tree/master/components/delay) package allows you to
*add delay metadata* to messages.
{{< callout "danger" >}}
**The delay metadata does nothing by itself. You need to use a Pub/Sub implementation that supports it to make it work.**
See below for supported Pub/Subs.
{{< /callout >}}
There are two APIs you can use. If you work with raw messages, use `delay.Message`:
```go
msg := message.NewMessage(watermill.NewUUID(), []byte("hello"))
delay.Message(msg, delay.For(time.Second * 10))
```
If you use the CQRS component, use `delay.WithContext` instead (since you can't access the message directly):
{{% load-snippet-partial file="src-link/_examples/real-world-examples/delayed-messages/main.go" first_line_contains="cmd := SendFeedbackForm" last_line_contains="return err" padding_after="1" %}}
You can also use `delay.Until` instead of `delay.For` to specify `time.Time` instead of `time.Duration`.
## Supported Pub/Subs
* [PostgreSQL](/pubsubs/sql/)
* [MySQL](/pubsubs/sql/)
## Full Example
See the [full example](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/delayed-messages) in the Watermill repository.

55
docs/content/advanced/fanin.md Normal file
View File

@@ -0,0 +1,55 @@
+++
title = "FanIn (merging topics)"
description = "Merging two topics into one with the FanIn component"
date = 2023-01-21T12:47:30+01:00
weight = -100
draft = false
bref = "Merging two topics into one with the FanIn component"
+++
## FanIn component
The FanIn component merges two topics into one.
### Configuring
{{% load-snippet-partial file="src-link/components/fanin/fanin.go" first_line_contains="type Config struct {" last_line_contains="CloseTimeout time.Duration" padding_after="1" %}}
### Running
You need to provide a Publisher and a Subscriber implementation for the FanIn component.
You can find the list of supported Pub/Subs on [Supported Pub/Subs page](/pubsubs/).
The Publisher and subscriber can be implemented by different message brokers (for example, you can merge a Kafka topic with a RabbitMQ topic).
```go
logger := watermill.NewStdLogger(false, false)
// create Publisher and Subscriber
pub, err := // ...
sub, err := // ...
fi, err := fanin.NewFanIn(
sub,
pub,
fanin.Config{
SourceTopics: upstreamTopics,
TargetTopic: downstreamTopic,
},
logger,
)
if err != nil {
panic(err)
}
if err := fi.Run(context.Background()); err != nil {
panic(err)
}
```
### Controlling FanIn component
The FanIn component can be stopped by cancelling the context passed to the `Run` method or by calling the `Close` method.
{{% load-snippet-partial file="src-link/components/fanin/fanin.go" first_line_contains="func (f *FanIn) Run" last_line_contains=" Close() error" padding_after="2" %}}

30
docs/content/advanced/fanout.md Normal file
View File

@@ -0,0 +1,30 @@
+++
title = "FanOut (multiplying messages)"
description = "FanOut is a component that receives messages from the subscriber and passes them to all publishers."
date = 2024-10-09T02:47:30+01:00
weight = -50
draft = false
bref = "FanOut is a component that receives messages from the subscriber and passes them to all publishers."
+++
## FanOut component
FanOut is a component that receives messages from a topic and passes them to all subscribers. In effect, messages are "multiplied".
A typical use case for using FanOut is having one external subscription and multiple workers
inside the process.
### Configuring
{{% load-snippet-partial file="src-link/pubsub/gochannel/fanout.go" first_line_contains="// NewFanOut" last_line_contains=")" padding_after="0" %}}
You need to call AddSubscription method for all topics that you want to listen to.
This needs to be done *before* starting the FanOut.
{{% load-snippet-partial file="src-link/pubsub/gochannel/fanout.go" first_line_contains="// AddSubscription" last_line_contains=")" padding_after="0" %}}
### Running
{{% load-snippet-partial file="src-link/pubsub/gochannel/fanout.go" first_line_contains="// Run" last_line_contains=")" padding_after="0" %}}
Then, use it as any other `message.Subscriber`.

119
docs/content/advanced/forwarder.md Normal file
View File

@@ -0,0 +1,119 @@
+++
title = "Forwarder (the outbox pattern)"
description = "Implement outbox pattern by publishing messages in transactional way"
date = 2021-01-13T12:47:30+01:00
weight = -300
draft = false
bref = "Emitting events along with storing data in a database in one transaction"
+++
## Publishing messages in transactions (and why we should care)
While working with an event-driven application, you may in some point need to store an application state and publish a message
telling the rest of the system about what's just happened. In a perfect scenario, you'd want to persist the application state
and publish the message **in a transaction**, as not doing so might get you easily into troubles with data consistency. In
order to commit both storing data and emitting an event in one transaction, you'd have to be able to publish
messages to the same database you use for the data storage,
or implement [2PC](https://martinfowler.com/articles/patterns-of-distributed-systems/two-phase-commit.html)
on your own. If you don't want to change your message broker to a database, nor invent the wheel once again,
you can make your life easier by using Watermill's [Forwarder component](https://github.com/ThreeDotsLabs/watermill/blob/master/components/forwarder/forwarder.go)!
## Forwarder component
You can think of the Forwarder as a background running daemon which awaits messages that are published to a database, and makes sure they eventually reach a message broker.
<img src="/img/publishing-with-forwarder.svg" alt="Watermill Forwarder component" style="width:100%;" />
In order to make the Forwarder universal and usable transparently, it listens to a single topic on an intermediate
database based Pub/Sub, where enveloped messages are sent with help of a decorated [Forwarder Publisher](https://github.com/ThreeDotsLabs/watermill/blob/9e04bfefbd6fef9f9ffa59956654277005fa2e8a/components/forwarder/publisher.go#L30).
The Forwarder unwraps them, and sends to a specified destined topic on the message broker.
<img src="/img/forwarder-envelope.svg" alt="Forwarder envelope" style="width:100%; background-color: white; padding: 2rem;" />
## Example
Let's consider a following example: there's a command which responsibility is to run a lottery. It has to pick
a random user that's registered in the system as a winner. While it does so, it should also persist the decision it made by
storing a database entry associating a unique lottery ID with a picked user's ID. Additionally, as it's an
event-driven system, it should emit a `LotteryConcluded` event, so that other components could react to that appropriately.
To be precise - there will be component responsible for sending prizes to lottery winners. It will receive `LotteryConcluded`
events, and using the lottery ID embedded in the event, verify who was the winner, checking with the database entry.
In our case, the database is MySQL and the message broker is Google Pub/Sub, but it could be any two other technologies.
Approaching to implementation of such a command, we could go various ways. Below we're going to cover three possible
attempts, pointing their vulnerabilities.
### Publishing an event first, storing data next
In this approach, the command is going to publish an event first and store data just after that. While in most of the
cases that approach will probably work just fine, let's try to find out what could possibly go wrong.
There are three basic actions that the command has to do:
1. Pick a random user `A` as a lottery winner.
2. Publish a `LotteryConcluded` event telling that lottery `B` has been concluded.
3. Store in the database that the lottery `B` has been won by the user `A`.
Every of these steps could potentially fail, breaking the flow of our command. The first point wouldn't have huge
repercussions in case of its failure - we would just return an error and consider the whole command failed. No data would
be stored, no event would be emitted. We can simply rerun the command.
In case the second point fails, we'll still have no event emitted and no data stored in the database. We can rerun the
command and try once again.
What's most interesting is what could happen in case the third point fails. We'd already have the event emitted after
the second point, but no data would be stored eventually in the database. Other components would get a signal
that the lottery had been concluded, but no winner would be associated to the lottery ID sent in the event. They wouldn't
be able to verify who's the winner, so their action would have to be considered failed as well.
We still can get out of this situation, but most probably it will require some manual action, i.e., rerunning the command
with the lottery ID that the emitted event has.
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 1. Publishes event" last_line_contains="// In case this fails" padding_after="9" %}}
### Storing data first, publishing an event next
In the second approach, we're going to try address first approach's drawbacks. We won't leak our failure to outer
components by not emitting an event in case we don't have the state persisted properly in database. That means
we'll change the order of our actions to following:
1. Pick a random user `A` as a lottery winner.
2. Store in the database that the lottery `B` has been won by the user `A`.
3. Publish a `LotteryConcluded` event telling that lottery `B` has been concluded.
Having two first actions failed, we have no repercussions, just as in the first approach. In case of failure of the 3rd
point, we'd have data persisted in the database, but no event emitted. In this case, we wouldn't leak our failure
outside the lottery component. Although, considering the expected system behavior, we'd have no prize sent to our winner,
because no event would be delivered to the component responsible for this action.
That probably can be fixed by some manual action as well, i.e., emitting the event manually. We still can do better.
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 2. Persists data" last_line_contains="// In case this fails" padding_after="9" %}}
### Storing data and publishing an event in one transaction
Let's imagine our command could do the 2nd, and the 3rd point at the same time. They would be committed atomically,
meaning that any of them can't succeed having the other failed. This can be achieved by leveraging a transaction
mechanism which happens to be implemented by most of the databases in today's world. One of them is MySQL used in our
example.
In order to commit both storing data and emitting an event in one transaction, we'd have to be able to publish our
messages to MySQL. Because we don't want to change our message broker to be backed by MySQL in the whole system,
we have to find a way to do that differently.
There's a good news: Watermill provides all the tools straight away! In case the database you're using is one among MySQL,
PostgreSQL (or any other SQL), Firestore or Bolt, you can publish messages to them. **Forwarder** component will help
you with picking all the messages you publish to the database and forwarding them to a message broker of yours.
Everything you have to do is to make sure that:
1. Your command uses a publisher working in a context of a database transaction (i.e. [SQL](https://github.com/ThreeDotsLabs/watermill-sql/blob/4f39bf82b6180ca2191c791e7cb220fff22b9255/pkg/sql/publisher.go#L53),
[Firestore](https://github.com/ThreeDotsLabs/watermill-firestore/blob/b7bd31b3458884dc76076196cdc8942d18b5ab61/pkg/firestore/transactional.go#L14), [Bolt](https://github.com/ThreeDotsLabs/watermill-bolt/blob/0652f3602f6adbe4e3e39b97308fbed16dcbe29e/pkg/bolt/tx_publisher.go#L24)).
2. **Forwarder** component is running, using a database subscriber, and a message broker publisher.
The command could look like following in this case:
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 3. Persists data" last_line_contains="err = publisher.Publish(googleCloudEventTopic" padding_after="5" %}}
In order to make the **Forwarder** component work in background for you and forward messages from MySQL to Google Pub/Sub,
you'd have to set it up as follows:
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// Setup the Forwarder " last_line_contains="err := fwd.Run" padding_after="3" %}}
If you wish to explore the example more, you can find it implemented [here](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/transactional-events-forwarder/main.go).

138
docs/content/advanced/metrics.md Normal file
View File

@@ -0,0 +1,138 @@
+++
title = "Metrics"
description = "Monitor Watermill in realtime"
date = 2019-02-12T21:00:00+01:00
weight = -400
draft = false
bref = "Monitor Watermill in realtime using Prometheus"
+++
Monitoring of Watermill may be performed by using decorators for publishers/subscribers and middlewares for handlers.
We provide a default implementation using Prometheus, based on the official [Prometheus client](https://github.com/prometheus/client_golang) for Go.
The `components/metrics` package exports `PrometheusMetricsBuilder`, which provides convenience functions to wrap publishers, subscribers and handlers so that they update the relevant Prometheus registry:
{{% load-snippet-partial file="src-link/components/metrics/builder.go" first_line_contains="// PrometheusMetricsBuilder" last_line_contains="func (b PrometheusMetricsBuilder)" %}}
## Wrapping publishers, subscribers and handlers
If you are using Watermill's [router](/docs/messages-router) (which is recommended in most cases), you can use a single convenience function `AddPrometheusRouterMetrics` to ensure that all the handlers added to this router are wrapped to update the Prometheus registry, together with their publishers and subscribers:
{{% load-snippet-partial file="src-link/components/metrics/builder.go" first_line_contains="// AddPrometheusRouterMetrics" last_line_contains="AddMiddleware" padding_after="1" %}}
Example use of `AddPrometheusRouterMetrics`:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="// we leave the namespace" last_line_contains="metricsBuilder.AddPrometheusRouterMetrics" %}}
In the snippet above, we have left the `namespace` and `subsystem` arguments empty. The Prometheus client library [uses these](https://godoc.org/github.com/prometheus/client_golang/prometheus#BuildFQName) to prefix the metric names. You may want to use namespace or subsystem, but be aware that this will impact the metric names and you will have to adjust the Grafana dashboard accordingly.
The `PrometheusMetricsBuilder` allows for custom configuration of histogram buckets by setting the `PublishBuckets` or `HandlerBuckets` field.
If `HandlerBuckets` is not provided, default watermill's values will be used, which are one order of magnitude smaller than default buckets (5ms~10s), because the handler execution times are typically shorter (µs~ms range).
For `PublishBuckets`, the default values are the same as the default Prometheus buckets (5ms~10s).
Standalone publishers and subscribers may also be decorated through the use of dedicated methods of `PrometheusMetricBuilder`:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="subWithMetrics, err := " last_line_contains="pubWithMetrics, err := " padding_after="3" %}}
## Exposing the /metrics endpoint
In accordance with how Prometheus works, the service needs to expose a HTTP endpoint for scraping. By convention, it is a GET endpoint, and its path is usually `/metrics`.
To serve this endpoint, there are two convenience functions, one using a previously created Prometheus Registry, while the other also creates a new registry:
{{% load-snippet-partial file="src-link/components/metrics/http.go" first_line_contains="// CreateRegistryAndServeHTTP" last_line_contains="func ServeHTTP(" %}}
Here is an example of its use in practice:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="prometheusRegistry, closeMetricsServer :=" last_line_contains="metricsBuilder.AddPrometheusRouterMetrics" %}}
## Example application
To see how the metrics dashboard works in practice, you can check out the [metrics example](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/4-metrics).
Follow the instructions in the example's [README](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/4-metrics/README.md) to make it run and add the Prometheus data source to Grafana.
## Grafana dashboard
We have prepared a [Grafana dashboard](https://grafana.com/grafana/dashboards/9777-watermill/) to use with the metrics implementation described above. It provides basic information about the throughput, failure rates and publish/handler durations.
If you want to check out the dashboard on your machine, you can use the [Example application](#example-application).
To find out more about the metrics that are exported to Prometheus, see [Exported metrics](#exported-metrics).
<a target="_blank" href="https://threedots.tech/watermill-io/grafana_dashboard.png"><img src="https://threedots.tech/watermill-io/grafana_dashboard_small.png" /></a>
### Importing the dashboard
To import the Grafana dashboard, select Dashboard/Manage from the left menu, and then click on `+Import`.
Enter the dashboard URL https://grafana.com/dashboards/9777 (or just the ID, 9777), and click on Load.
![Importing the dashboard](https://threedots.tech/watermill-io/grafana_import_dashboard.png)
Then select your Prometheus data source that scrapes the `/metrics` endpoint. Click on `Import`, and you're done!
## Exported metrics
Listed below are all the metrics that are registered on the Prometheus Registry by `PrometheusMetricsBuilder`.
For more information on Prometheus metric types, please refer to [Prometheus docs](https://prometheus.io/docs/concepts/metric_types).
<table>
<tr>
<th>Object</th>
<th>Metric</th>
<th>Description</th>
<th>Labels/Values</th>
</tr>
<tr>
<td rowspan="3">Subscriber</td>
<td rowspan="3"><code>subscriber_messages_received_total</code></td>
<td rowspan="3">A Prometheus Counter.<br>Counts the number of messages obtained by the subscriber.</td>
<td><code>acked</code> is either "acked" or "nacked".</td>
</tr>
<tr>
<td><code>handler_name</code> is set if the subscriber operates within a handler; "&lt;no handler&gt;" otherwise.</td>
</tr>
<tr>
<td><code>subscriber_name</code> identifies the subscriber. If it implements <code>fmt.Stringer</code>, it is the result of `String()`, <code>package.structName</code> otherwise.</td>
</tr>
<tr>
<td rowspan="2">Handler</td>
<td rowspan="2"><code>handler_execution_time_seconds</code></td>
<td rowspan="2">A Prometheus Histogram. <br>Registers the execution time of the handler function wrapped by the middleware.</td>
<td><code>handler_name</code> is the name of the handler.</td>
</tr>
<tr>
<td><code>success</code> is either "true" or "false", depending on whether the wrapped handler function returned an error or not.</td>
</tr>
<tr>
<td rowspan="3">Publisher</td>
<td rowspan="3"><code>publish_time_seconds</code></td>
<td rowspan="3">A Prometheus Histogram.<br>Registers the time of execution of the Publish function of the decorated publisher.</td>
<td><code>success</code> is either "true" or "false", depending on whether the decorated publisher returned an error or not.</td>
</tr>
<tr>
<td><code>handler_name</code> is set if the publisher operates within a handler; "&lt;no handler&gt;" otherwise.</td>
</tr>
<tr>
<td><code>publisher_name</code> identifies the publisher. If it implements <code>fmt.Stringer</code>, it is the result of `String()`, <code>package.structName</code> otherwise.</td>
</tr>
</table>
Additionally, every metric has the `node` label, provided by Prometheus, with value corresponding to the instance that the metric comes from, and `job`, which is the job name specified in the [Prometheus configuration file](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/4-metrics/prometheus.yml).
**NOTE**: As described [above](#wrapping-publishers-subscribers-and-handlers), using non-empty `namespace` or `subsystem` will result in prefixed metric names. You might need to adjust for it, for example in the definitions of panels in the Grafana dashboard.
## Customization
If you feel like some metric is missing, you can easily expand this basic implementation. The best way to do so is to use the prometheus registry that is used with the [ServeHTTP method](#exposing-the-metrics-endpoint) and register a metric according to [the documentation](https://godoc.org/github.com/prometheus/client_golang/prometheus) of the Prometheus client.
An elegant way to update these metrics would be through the use of decorators:
{{% load-snippet-partial file="src-link/message/decorator.go" first_line_contains="// MessageTransformSubscriberDecorator" last_line_contains="type messageTransformSubscriberDecorator" %}}
and/or [router middlewares](/docs/messages-router/#middleware).
A more simplistic approach would be to just update the metric that you want in the handler function.

55
docs/content/advanced/requeuing-after-error.md Normal file
View File

@@ -0,0 +1,55 @@
+++
title = "Requeuing After Error"
description = "How to requeue a message after it fails to process"
weight = -20
draft = false
bref = "How to requeue a message after it fails to process"
+++
When a message fails to process (a nack is sent), it usually blocks other messages on the same topic (within the same consumer group or partition).
Depending on your setup, it may be useful to requeue the failed message back to the tail of the queue.
Consider this if:
* You don't care about the order of messages.
* Your system isn't resilient to blocked messages.
## Requeuer
The `Requeuer` component is a wrapper on the `Router` that moves messages from one topic to another.
{{% load-snippet-partial file="src-link/components/requeuer/requeuer.go" first_line_contains="type Config" last_line_contains="}" %}}
A trivial usage can look like this. It requeues messages from one topic to the same topic after a delay.
{{< callout "danger" >}}
Using the delay this way is not recommended, as it blocks the entire requeue process for the given time.
{{< /callout >}}
```go
req, err := requeuer.NewRequeuer(requeuer.Config{
Subscriber: sub,
SubscribeTopic: "topic",
Publisher: pub,
GeneratePublishTopic: func(params requeuer.GeneratePublishTopicParams) (string, error) {
return "topic", nil
},
Delay: time.Millisecond * 200,
}, logger)
if err != nil {
return err
}
err := req.Run(context.Background())
if err != nil {
return err
}
```
A better way to use the `Requeuer` is to combine it with the `Poison` middleware.
The middleware moves messages to a separate "poison" topic.
Then, the requeuer moves them back to the original topic based on the metadata.
You combine this with a Pub/Sub that [supports delayed messages](/advanced/delayed-messages/#supported-pubsubs).
See the [full example based on PostgreSQL](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/real-world-examples/delayed-requeue/main.go).