upstream/upstream-watermill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

snapshot of https://github.com/ThreeDotsLabs/watermill.git (master, 2026-09-15; blobs > 131072B omitted)
Some checks failed
master / ci (push) Failing after 0s
Details

Browse Source
This commit is contained in:
upstream-mirror
2026-09-15 21:27:46 +05:00
commit ce933d1277
538 changed files with 46504 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
docs/content/_index.md Normal file
View File

@@ -0,0 +1,13 @@
---
title: "Watermill"
description: "Building event-driven applications the easy way in Go."
lead: "Building event-driven applications the easy way in Go."
date: 2023-09-07T16:33:54+02:00
lastmod: 2023-09-07T16:33:54+02:00
draft: false
seo:
title: "Watermill" # custom title (optional)
description: "" # custom description (recommended)
canonical: "" # custom canonical URL (optional)
noindex: false # false (default) or true
---

44
docs/content/advanced/delayed-messages.md Normal file
View File

@@ -0,0 +1,44 @@
+++
title = "Delayed Messages"
description = "Receive messages with a delay"
weight = -40
draft = false
bref = "Receive messages with a delay"
+++
Delaying events or commands is a common use case in many applications.
For example, you may want to send the user a reminder after a few days of signing up.
It's not a complex logic to implement, but you can leverage messages to use it out of the box.
## Delay Metadata
Watermill's [`delay`](https://github.com/ThreeDotsLabs/watermill/tree/master/components/delay) package allows you to
*add delay metadata* to messages.
{{< callout "danger" >}}
**The delay metadata does nothing by itself. You need to use a Pub/Sub implementation that supports it to make it work.**
See below for supported Pub/Subs.
{{< /callout >}}
There are two APIs you can use. If you work with raw messages, use `delay.Message`:
```go
msg := message.NewMessage(watermill.NewUUID(), []byte("hello"))
delay.Message(msg, delay.For(time.Second * 10))
```
If you use the CQRS component, use `delay.WithContext` instead (since you can't access the message directly):
{{% load-snippet-partial file="src-link/_examples/real-world-examples/delayed-messages/main.go" first_line_contains="cmd := SendFeedbackForm" last_line_contains="return err" padding_after="1" %}}
You can also use `delay.Until` instead of `delay.For` to specify `time.Time` instead of `time.Duration`.
## Supported Pub/Subs
* [PostgreSQL](/pubsubs/sql/)
* [MySQL](/pubsubs/sql/)
## Full Example
See the [full example](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/delayed-messages) in the Watermill repository.

55
docs/content/advanced/fanin.md Normal file
View File

@@ -0,0 +1,55 @@
+++
title = "FanIn (merging topics)"
description = "Merging two topics into one with the FanIn component"
date = 2023-01-21T12:47:30+01:00
weight = -100
draft = false
bref = "Merging two topics into one with the FanIn component"
+++
## FanIn component
The FanIn component merges two topics into one.
### Configuring
{{% load-snippet-partial file="src-link/components/fanin/fanin.go" first_line_contains="type Config struct {" last_line_contains="CloseTimeout time.Duration" padding_after="1" %}}
### Running
You need to provide a Publisher and a Subscriber implementation for the FanIn component.
You can find the list of supported Pub/Subs on [Supported Pub/Subs page](/pubsubs/).
The Publisher and subscriber can be implemented by different message brokers (for example, you can merge a Kafka topic with a RabbitMQ topic).
```go
logger := watermill.NewStdLogger(false, false)
// create Publisher and Subscriber
pub, err := // ...
sub, err := // ...
fi, err := fanin.NewFanIn(
sub,
pub,
fanin.Config{
SourceTopics: upstreamTopics,
TargetTopic: downstreamTopic,
},
logger,
)
if err != nil {
panic(err)
}
if err := fi.Run(context.Background()); err != nil {
panic(err)
}
```
### Controlling FanIn component
The FanIn component can be stopped by cancelling the context passed to the `Run` method or by calling the `Close` method.
{{% load-snippet-partial file="src-link/components/fanin/fanin.go" first_line_contains="func (f *FanIn) Run" last_line_contains=" Close() error" padding_after="2" %}}

30
docs/content/advanced/fanout.md Normal file
View File

@@ -0,0 +1,30 @@
+++
title = "FanOut (multiplying messages)"
description = "FanOut is a component that receives messages from the subscriber and passes them to all publishers."
date = 2024-10-09T02:47:30+01:00
weight = -50
draft = false
bref = "FanOut is a component that receives messages from the subscriber and passes them to all publishers."
+++
## FanOut component
FanOut is a component that receives messages from a topic and passes them to all subscribers. In effect, messages are "multiplied".
A typical use case for using FanOut is having one external subscription and multiple workers
inside the process.
### Configuring
{{% load-snippet-partial file="src-link/pubsub/gochannel/fanout.go" first_line_contains="// NewFanOut" last_line_contains=")" padding_after="0" %}}
You need to call AddSubscription method for all topics that you want to listen to.
This needs to be done *before* starting the FanOut.
{{% load-snippet-partial file="src-link/pubsub/gochannel/fanout.go" first_line_contains="// AddSubscription" last_line_contains=")" padding_after="0" %}}
### Running
{{% load-snippet-partial file="src-link/pubsub/gochannel/fanout.go" first_line_contains="// Run" last_line_contains=")" padding_after="0" %}}
Then, use it as any other `message.Subscriber`.

119
docs/content/advanced/forwarder.md Normal file
View File

@@ -0,0 +1,119 @@
+++
title = "Forwarder (the outbox pattern)"
description = "Implement outbox pattern by publishing messages in transactional way"
date = 2021-01-13T12:47:30+01:00
weight = -300
draft = false
bref = "Emitting events along with storing data in a database in one transaction"
+++
## Publishing messages in transactions (and why we should care)
While working with an event-driven application, you may in some point need to store an application state and publish a message
telling the rest of the system about what's just happened. In a perfect scenario, you'd want to persist the application state
and publish the message **in a transaction**, as not doing so might get you easily into troubles with data consistency. In
order to commit both storing data and emitting an event in one transaction, you'd have to be able to publish
messages to the same database you use for the data storage,
or implement [2PC](https://martinfowler.com/articles/patterns-of-distributed-systems/two-phase-commit.html)
on your own. If you don't want to change your message broker to a database, nor invent the wheel once again,
you can make your life easier by using Watermill's [Forwarder component](https://github.com/ThreeDotsLabs/watermill/blob/master/components/forwarder/forwarder.go)!
## Forwarder component
You can think of the Forwarder as a background running daemon which awaits messages that are published to a database, and makes sure they eventually reach a message broker.
<img src="/img/publishing-with-forwarder.svg" alt="Watermill Forwarder component" style="width:100%;" />
In order to make the Forwarder universal and usable transparently, it listens to a single topic on an intermediate
database based Pub/Sub, where enveloped messages are sent with help of a decorated [Forwarder Publisher](https://github.com/ThreeDotsLabs/watermill/blob/9e04bfefbd6fef9f9ffa59956654277005fa2e8a/components/forwarder/publisher.go#L30).
The Forwarder unwraps them, and sends to a specified destined topic on the message broker.
<img src="/img/forwarder-envelope.svg" alt="Forwarder envelope" style="width:100%; background-color: white; padding: 2rem;" />
## Example
Let's consider a following example: there's a command which responsibility is to run a lottery. It has to pick
a random user that's registered in the system as a winner. While it does so, it should also persist the decision it made by
storing a database entry associating a unique lottery ID with a picked user's ID. Additionally, as it's an
event-driven system, it should emit a `LotteryConcluded` event, so that other components could react to that appropriately.
To be precise - there will be component responsible for sending prizes to lottery winners. It will receive `LotteryConcluded`
events, and using the lottery ID embedded in the event, verify who was the winner, checking with the database entry.
In our case, the database is MySQL and the message broker is Google Pub/Sub, but it could be any two other technologies.
Approaching to implementation of such a command, we could go various ways. Below we're going to cover three possible
attempts, pointing their vulnerabilities.
### Publishing an event first, storing data next
In this approach, the command is going to publish an event first and store data just after that. While in most of the
cases that approach will probably work just fine, let's try to find out what could possibly go wrong.
There are three basic actions that the command has to do:
1. Pick a random user `A` as a lottery winner.
2. Publish a `LotteryConcluded` event telling that lottery `B` has been concluded.
3. Store in the database that the lottery `B` has been won by the user `A`.
Every of these steps could potentially fail, breaking the flow of our command. The first point wouldn't have huge
repercussions in case of its failure - we would just return an error and consider the whole command failed. No data would
be stored, no event would be emitted. We can simply rerun the command.
In case the second point fails, we'll still have no event emitted and no data stored in the database. We can rerun the
command and try once again.
What's most interesting is what could happen in case the third point fails. We'd already have the event emitted after
the second point, but no data would be stored eventually in the database. Other components would get a signal
that the lottery had been concluded, but no winner would be associated to the lottery ID sent in the event. They wouldn't
be able to verify who's the winner, so their action would have to be considered failed as well.
We still can get out of this situation, but most probably it will require some manual action, i.e., rerunning the command
with the lottery ID that the emitted event has.
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 1. Publishes event" last_line_contains="// In case this fails" padding_after="9" %}}
### Storing data first, publishing an event next
In the second approach, we're going to try address first approach's drawbacks. We won't leak our failure to outer
components by not emitting an event in case we don't have the state persisted properly in database. That means
we'll change the order of our actions to following:
1. Pick a random user `A` as a lottery winner.
2. Store in the database that the lottery `B` has been won by the user `A`.
3. Publish a `LotteryConcluded` event telling that lottery `B` has been concluded.
Having two first actions failed, we have no repercussions, just as in the first approach. In case of failure of the 3rd
point, we'd have data persisted in the database, but no event emitted. In this case, we wouldn't leak our failure
outside the lottery component. Although, considering the expected system behavior, we'd have no prize sent to our winner,
because no event would be delivered to the component responsible for this action.
That probably can be fixed by some manual action as well, i.e., emitting the event manually. We still can do better.
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 2. Persists data" last_line_contains="// In case this fails" padding_after="9" %}}
### Storing data and publishing an event in one transaction
Let's imagine our command could do the 2nd, and the 3rd point at the same time. They would be committed atomically,
meaning that any of them can't succeed having the other failed. This can be achieved by leveraging a transaction
mechanism which happens to be implemented by most of the databases in today's world. One of them is MySQL used in our
example.
In order to commit both storing data and emitting an event in one transaction, we'd have to be able to publish our
messages to MySQL. Because we don't want to change our message broker to be backed by MySQL in the whole system,
we have to find a way to do that differently.
There's a good news: Watermill provides all the tools straight away! In case the database you're using is one among MySQL,
PostgreSQL (or any other SQL), Firestore or Bolt, you can publish messages to them. **Forwarder** component will help
you with picking all the messages you publish to the database and forwarding them to a message broker of yours.
Everything you have to do is to make sure that:
1. Your command uses a publisher working in a context of a database transaction (i.e. [SQL](https://github.com/ThreeDotsLabs/watermill-sql/blob/4f39bf82b6180ca2191c791e7cb220fff22b9255/pkg/sql/publisher.go#L53),
[Firestore](https://github.com/ThreeDotsLabs/watermill-firestore/blob/b7bd31b3458884dc76076196cdc8942d18b5ab61/pkg/firestore/transactional.go#L14), [Bolt](https://github.com/ThreeDotsLabs/watermill-bolt/blob/0652f3602f6adbe4e3e39b97308fbed16dcbe29e/pkg/bolt/tx_publisher.go#L24)).
2. **Forwarder** component is running, using a database subscriber, and a message broker publisher.
The command could look like following in this case:
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// 3. Persists data" last_line_contains="err = publisher.Publish(googleCloudEventTopic" padding_after="5" %}}
In order to make the **Forwarder** component work in background for you and forward messages from MySQL to Google Pub/Sub,
you'd have to set it up as follows:
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events-forwarder/main.go" first_line_contains="// Setup the Forwarder " last_line_contains="err := fwd.Run" padding_after="3" %}}
If you wish to explore the example more, you can find it implemented [here](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/transactional-events-forwarder/main.go).

138
docs/content/advanced/metrics.md Normal file
View File

@@ -0,0 +1,138 @@
+++
title = "Metrics"
description = "Monitor Watermill in realtime"
date = 2019-02-12T21:00:00+01:00
weight = -400
draft = false
bref = "Monitor Watermill in realtime using Prometheus"
+++
Monitoring of Watermill may be performed by using decorators for publishers/subscribers and middlewares for handlers.
We provide a default implementation using Prometheus, based on the official [Prometheus client](https://github.com/prometheus/client_golang) for Go.
The `components/metrics` package exports `PrometheusMetricsBuilder`, which provides convenience functions to wrap publishers, subscribers and handlers so that they update the relevant Prometheus registry:
{{% load-snippet-partial file="src-link/components/metrics/builder.go" first_line_contains="// PrometheusMetricsBuilder" last_line_contains="func (b PrometheusMetricsBuilder)" %}}
## Wrapping publishers, subscribers and handlers
If you are using Watermill's [router](/docs/messages-router) (which is recommended in most cases), you can use a single convenience function `AddPrometheusRouterMetrics` to ensure that all the handlers added to this router are wrapped to update the Prometheus registry, together with their publishers and subscribers:
{{% load-snippet-partial file="src-link/components/metrics/builder.go" first_line_contains="// AddPrometheusRouterMetrics" last_line_contains="AddMiddleware" padding_after="1" %}}
Example use of `AddPrometheusRouterMetrics`:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="// we leave the namespace" last_line_contains="metricsBuilder.AddPrometheusRouterMetrics" %}}
In the snippet above, we have left the `namespace` and `subsystem` arguments empty. The Prometheus client library [uses these](https://godoc.org/github.com/prometheus/client_golang/prometheus#BuildFQName) to prefix the metric names. You may want to use namespace or subsystem, but be aware that this will impact the metric names and you will have to adjust the Grafana dashboard accordingly.
The `PrometheusMetricsBuilder` allows for custom configuration of histogram buckets by setting the `PublishBuckets` or `HandlerBuckets` field.
If `HandlerBuckets` is not provided, default watermill's values will be used, which are one order of magnitude smaller than default buckets (5ms~10s), because the handler execution times are typically shorter (µs~ms range).
For `PublishBuckets`, the default values are the same as the default Prometheus buckets (5ms~10s).
Standalone publishers and subscribers may also be decorated through the use of dedicated methods of `PrometheusMetricBuilder`:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="subWithMetrics, err := " last_line_contains="pubWithMetrics, err := " padding_after="3" %}}
## Exposing the /metrics endpoint
In accordance with how Prometheus works, the service needs to expose a HTTP endpoint for scraping. By convention, it is a GET endpoint, and its path is usually `/metrics`.
To serve this endpoint, there are two convenience functions, one using a previously created Prometheus Registry, while the other also creates a new registry:
{{% load-snippet-partial file="src-link/components/metrics/http.go" first_line_contains="// CreateRegistryAndServeHTTP" last_line_contains="func ServeHTTP(" %}}
Here is an example of its use in practice:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="prometheusRegistry, closeMetricsServer :=" last_line_contains="metricsBuilder.AddPrometheusRouterMetrics" %}}
## Example application
To see how the metrics dashboard works in practice, you can check out the [metrics example](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/4-metrics).
Follow the instructions in the example's [README](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/4-metrics/README.md) to make it run and add the Prometheus data source to Grafana.
## Grafana dashboard
We have prepared a [Grafana dashboard](https://grafana.com/grafana/dashboards/9777-watermill/) to use with the metrics implementation described above. It provides basic information about the throughput, failure rates and publish/handler durations.
If you want to check out the dashboard on your machine, you can use the [Example application](#example-application).
To find out more about the metrics that are exported to Prometheus, see [Exported metrics](#exported-metrics).
<a target="_blank" href="https://threedots.tech/watermill-io/grafana_dashboard.png"><img src="https://threedots.tech/watermill-io/grafana_dashboard_small.png" /></a>
### Importing the dashboard
To import the Grafana dashboard, select Dashboard/Manage from the left menu, and then click on `+Import`.
Enter the dashboard URL https://grafana.com/dashboards/9777 (or just the ID, 9777), and click on Load.
![Importing the dashboard](https://threedots.tech/watermill-io/grafana_import_dashboard.png)
Then select your Prometheus data source that scrapes the `/metrics` endpoint. Click on `Import`, and you're done!
## Exported metrics
Listed below are all the metrics that are registered on the Prometheus Registry by `PrometheusMetricsBuilder`.
For more information on Prometheus metric types, please refer to [Prometheus docs](https://prometheus.io/docs/concepts/metric_types).
<table>
<tr>
<th>Object</th>
<th>Metric</th>
<th>Description</th>
<th>Labels/Values</th>
</tr>
<tr>
<td rowspan="3">Subscriber</td>
<td rowspan="3"><code>subscriber_messages_received_total</code></td>
<td rowspan="3">A Prometheus Counter.<br>Counts the number of messages obtained by the subscriber.</td>
<td><code>acked</code> is either "acked" or "nacked".</td>
</tr>
<tr>
<td><code>handler_name</code> is set if the subscriber operates within a handler; "&lt;no handler&gt;" otherwise.</td>
</tr>
<tr>
<td><code>subscriber_name</code> identifies the subscriber. If it implements <code>fmt.Stringer</code>, it is the result of `String()`, <code>package.structName</code> otherwise.</td>
</tr>
<tr>
<td rowspan="2">Handler</td>
<td rowspan="2"><code>handler_execution_time_seconds</code></td>
<td rowspan="2">A Prometheus Histogram. <br>Registers the execution time of the handler function wrapped by the middleware.</td>
<td><code>handler_name</code> is the name of the handler.</td>
</tr>
<tr>
<td><code>success</code> is either "true" or "false", depending on whether the wrapped handler function returned an error or not.</td>
</tr>
<tr>
<td rowspan="3">Publisher</td>
<td rowspan="3"><code>publish_time_seconds</code></td>
<td rowspan="3">A Prometheus Histogram.<br>Registers the time of execution of the Publish function of the decorated publisher.</td>
<td><code>success</code> is either "true" or "false", depending on whether the decorated publisher returned an error or not.</td>
</tr>
<tr>
<td><code>handler_name</code> is set if the publisher operates within a handler; "&lt;no handler&gt;" otherwise.</td>
</tr>
<tr>
<td><code>publisher_name</code> identifies the publisher. If it implements <code>fmt.Stringer</code>, it is the result of `String()`, <code>package.structName</code> otherwise.</td>
</tr>
</table>
Additionally, every metric has the `node` label, provided by Prometheus, with value corresponding to the instance that the metric comes from, and `job`, which is the job name specified in the [Prometheus configuration file](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/4-metrics/prometheus.yml).
**NOTE**: As described [above](#wrapping-publishers-subscribers-and-handlers), using non-empty `namespace` or `subsystem` will result in prefixed metric names. You might need to adjust for it, for example in the definitions of panels in the Grafana dashboard.
## Customization
If you feel like some metric is missing, you can easily expand this basic implementation. The best way to do so is to use the prometheus registry that is used with the [ServeHTTP method](#exposing-the-metrics-endpoint) and register a metric according to [the documentation](https://godoc.org/github.com/prometheus/client_golang/prometheus) of the Prometheus client.
An elegant way to update these metrics would be through the use of decorators:
{{% load-snippet-partial file="src-link/message/decorator.go" first_line_contains="// MessageTransformSubscriberDecorator" last_line_contains="type messageTransformSubscriberDecorator" %}}
and/or [router middlewares](/docs/messages-router/#middleware).
A more simplistic approach would be to just update the metric that you want in the handler function.

55
docs/content/advanced/requeuing-after-error.md Normal file
View File

@@ -0,0 +1,55 @@
+++
title = "Requeuing After Error"
description = "How to requeue a message after it fails to process"
weight = -20
draft = false
bref = "How to requeue a message after it fails to process"
+++
When a message fails to process (a nack is sent), it usually blocks other messages on the same topic (within the same consumer group or partition).
Depending on your setup, it may be useful to requeue the failed message back to the tail of the queue.
Consider this if:
* You don't care about the order of messages.
* Your system isn't resilient to blocked messages.
## Requeuer
The `Requeuer` component is a wrapper on the `Router` that moves messages from one topic to another.
{{% load-snippet-partial file="src-link/components/requeuer/requeuer.go" first_line_contains="type Config" last_line_contains="}" %}}
A trivial usage can look like this. It requeues messages from one topic to the same topic after a delay.
{{< callout "danger" >}}
Using the delay this way is not recommended, as it blocks the entire requeue process for the given time.
{{< /callout >}}
```go
req, err := requeuer.NewRequeuer(requeuer.Config{
Subscriber: sub,
SubscribeTopic: "topic",
Publisher: pub,
GeneratePublishTopic: func(params requeuer.GeneratePublishTopicParams) (string, error) {
return "topic", nil
},
Delay: time.Millisecond * 200,
}, logger)
if err != nil {
return err
}
err := req.Run(context.Background())
if err != nil {
return err
}
```
A better way to use the `Requeuer` is to combine it with the `Poison` middleware.
The middleware moves messages to a separate "poison" topic.
Then, the requeuer moves them back to the original topic based on the metadata.
You combine this with a Pub/Sub that [supports delayed messages](/advanced/delayed-messages/#supported-pubsubs).
See the [full example based on PostgreSQL](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/real-world-examples/delayed-requeue/main.go).

12
docs/content/development/benchmark.md Normal file
View File

@@ -0,0 +1,12 @@
+++
title = "Benchmark"
description = "Watermill Benchmark"
weight = 250
draft = false
bref = "Watermill Benchmark"
+++
You can find benchmarking tools and results in the [github.com/ThreeDotsLabs/watermill-benchmark](https://github.com/ThreeDotsLabs/watermill-benchmark) repository.
**Note, they are meant as rough estimate and should not be used to decide which Pub/Sub is the best choice.**
Performance depends on many factors, including configuration, and it's always best to test performance in your own environment.

53
docs/content/development/contributing.md Normal file
View File

@@ -0,0 +1,53 @@
+++
title = "Contributing Guide"
description = "Contribute to Watermill"
weight = 100
bref = "Contribute to Watermill"
+++
## How can I help?
We are always happy to help you in contributing to Watermill. If you have any ideas, please let us know on our [Discord server](https://watermill.io/support/).
There are multiple ways in which you can help us.
### Existing issues
You can pick one of the existing issues. Most of the issues should have an estimation (S - small, M - medium, L - large).
- [Good first issues list](https://github.com/ThreeDotsLabs/watermill/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) - simple issues to begin with
- [Help wanted issues list](https://github.com/ThreeDotsLabs/watermill/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) - tasks that are already more or less clear, and you can start to implement them pretty quickly
### New Pub/Sub implementations
If you have an idea to create a Pub/Sub based on some technology and it is not listed yet in our issues (because we don't know it, or it is just some crazy idea, like physical mail based Pub/Sub), feel free to add your own implementation.
You can do it in your private repository and later, if you want, we can move it to `ThreeDotsLabs/watermill-[name]`.
You will keep the maintainer permissions to the repository, and we'll invite you to a maintainers-only discord channel.
When adding a new Pub/Sub implementation, you should start with [Implementing a new Pub/Sub]({{< ref "pub-sub-implementing" >}}).
### New ideas
If you have any idea that is not covered in the issues list, please post a new issue describing it.
It's recommended to discuss your idea on [Discord](https://discord.gg/QV6VFg4YQE)/GitHub before creating production-ready implementation - in some situations, it may save a lot of your time before implementing something that can be simplified or done more easily. :)
In general, it's helpful to discuss a Proof of Concept to align with the idea.
## Local development
Makefile and docker-compose (for Pub/Subs) are your friends. You can run all tests locally (they are running in CI in the same way).
Useful commands:
- `make up` - docker-compose up
- `make test` - tests
- `make test_short` - run short tests (useful to perform a very fast check after changes)
- `make fmt` - do goimports
## Code standards
- you should run `make fmt`
- [CodeReviewComments](https://github.com/golang/go/wiki/CodeReviewComments)
- [Effective Go](https://golang.org/doc/effective_go.html)
- SOLID
- code should be open for configuration and not coupled to any serialization method (for example: [AMQP marshaler](https://github.com/ThreeDotsLabs/watermill-amqp/blob/master/pkg/amqp/marshaler.go), [AMQP Config](https://github.com/ThreeDotsLabs/watermill-amqp/blob/master/pkg/amqp/config.go)

40
docs/content/development/pub-sub-implementing.md Normal file
View File

@@ -0,0 +1,40 @@
+++
title = "Implementing a new Pub/Sub"
description = "Bring Your Own Pub/Sub"
date = 2018-12-05T12:48:34+01:00
weight = 200
draft = false
bref = "Bring Your Own Pub/Sub"
+++
## The Pub/Sub interface
To add support for a custom Pub/Sub, you have to implement both `message.Publisher` and `message.Subscriber` interfaces.
{{% load-snippet-partial file="src-link/message/pubsub.go" first_line_contains="type Publisher interface" last_line_contains="type SubscribeInitializer" padding_after="0" %}}
## Testing
Watermill provides [a set of test scenarios](https://github.com/ThreeDotsLabs/watermill/blob/master/pubsub/tests/test_pubsub.go)
that any Pub/Sub implementation can use. Each test suite needs to declare what features it supports and how to construct a new Pub/Sub.
These scenarios check both basic usage and more uncommon use cases. Stress tests are also included.
## TODO list
Here are a few things you shouldn't forget about:
1. Logging (good messages and proper levels).
2. Replaceable and configurable message marshaler.
3. `Close()` implementation for the publisher and subscriber that is:
- idempotent
- working correctly even when the publisher or the subscriber is blocked (for example, waiting for an Ack).
- working correctly when the subscriber output channel is blocked (because nothing is listening on it).
4. `Ack()` **and** `Nack()` support for consumed messages.
5. Redelivery on `Nack()` for a consumed message.
6. Use [Universal Pub/Sub tests]({{< ref "/docs/pub-sub#universal-tests" >}}). For debugging tips, you should check [tests troubleshooting guide](/docs/troubleshooting/#debugging-pubsub-tests).
7. Performance optimizations.
8. GoDocs, [Markdown docs]({{< ref "/pubsubs" >}}) and [Getting Started examples](/learn/getting-started).
We will also be thankful for submitting a [pull request](https://github.com/ThreeDotsLabs/watermill/pulls) with the new Pub/Sub implementation.
If anything is not clear, feel free to use any of our [support channels]({{< ref "/support" >}}) to reach us, we will be glad to help.

10
docs/content/development/releases.md Normal file
View File

@@ -0,0 +1,10 @@
+++
title = "Releases"
description = "Watermill Releases"
weight = 300
bref = "Watermill Releases"
+++
You can read about the historical Watermill releases in [the posts on the Three Dots Labs blog](https://threedots.tech/series/watermill-release-post/).
<iframe src="https://releases.threedots.tech" frameborder="0" style="border: 0; width: 100%; height: 1000px"></iframe>

15
docs/content/docs/_index.md Normal file
View File

@@ -0,0 +1,15 @@
---
title: "Docs"
description: ""
summary: ""
date: 2023-09-07T16:12:03+02:00
lastmod: 2023-09-07T16:12:03+02:00
draft: false
weight: 999
toc: true
seo:
title: "" # custom title (optional)
description: "" # custom description (recommended)
canonical: "" # custom canonical URL (optional)
noindex: false # false (default) or true
---

15
docs/content/docs/articles.md Normal file
View File

@@ -0,0 +1,15 @@
+++
title = "Articles"
description = "In-depth articles mentioning Watermill"
weight = 100
draft = false
bref = "In-depth articles mentioning Watermill"
+++
You can find more in-depth tips on Watermill in these articles:
* [Distributed Transactions in Go: Read Before You Try](https://threedots.tech/post/distributed-transactions-in-go/)
* [Live website updates with Go, SSE, and htmx](https://threedots.tech/post/live-website-updates-go-sse-htmx/)
* [Using MySQL as a Pub/Sub](https://threedots.tech/post/when-sql-database-makes-great-pub-sub/)
* [Creating local Go dev environment with Docker and live code reloading](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/)

67
docs/content/docs/awesome.md Normal file
View File

@@ -0,0 +1,67 @@
+++
title = "Awesome Watermill"
description = "Selected unofficial libraries"
weight = 200
draft = false
bref = "Selected unofficial libraries"
+++
Below is a list of libraries that are not maintained by Three Dots Labs, but you may find them useful.
**Please note we can't provide support or guarantee they work correctly**. Do your own research.
If you know another library or are an author of one, please [add it to the list](https://github.com/ThreeDotsLabs/watermill/edit/master/docs/content/docs/awesome.md).
## Examples
* https://github.com/minghsu0107/golang-taipei-watermill-example
* https://github.com/minghsu0107/Kafka-PubSub
* https://github.com/pperaltaisern/go-example-financing
## Pub/Subs
* AMQP 1.0 https://github.com/kahowell/watermill-amqp10
* Apache Pulsar https://github.com/AlexCuse/watermill-pulsar
* Apache RocketMQ https://github.com/yflau/watermill-rocketmq
* CockroachDB https://github.com/cockroachdb/watermill-crdb
* Ensign https://github.com/rotationalio/watermill-ensign
* GoogleCloud Pub/Sub HTTP Push https://github.com/dentech-floss/watermill-googlecloud-http
* MongoDB https://github.com/cunyat/watermill-mongodb
* MQTT https://github.com/perfect13/watermill-mqtt
* NSQ https://github.com/chennqqi/watermill-nsq
* Redis Zset https://github.com/stong1994/watermill-rediszset
* SQLite https://github.com/davidroman0O/watermill-comfymill
If you want to find out how to implement your own Pub/Sub adapter,
check out [Implementing custom Pub/Sub](/development/pub-sub-implementing).
## Logging
* logrus
* https://github.com/ma-hartma/watermill-logrus-adapter
* https://github.com/UNIwise/walrus
* zap
* https://github.com/garsue/watermillzap
* https://github.com/pperaltaisern/watermillzap
* zerolog
* https://github.com/alexdrl/zerowater
* https://github.com/bogatyr285/watermillzlog
* https://github.com/vsvp21/zerolog-watermill-adapter
## Observability
* OpenCensus
* https://github.com/czeslavo/watermill-opencensus
* https://github.com/sagikazarmark/ocwatermill
* OpenTelemetry
* https://github.com/voi-oss/watermill-opentelemetry
* https://github.com/dentech-floss/watermill-opentelemetry-go-extra
* https://github.com/nkonev/watermill-opentelemetry
* AMQP https://github.com/hpcslag/otel-watermill-amqp
* GoChannel https://github.com/hpcslag/watermill-otel-tracable-gochannel
## Other
* https://github.com/asyncapi/go-watermill-template
* https://github.com/goph/watermillx
* https://github.com/voi-oss/protoc-gen-event

263
docs/content/docs/cqrs.md Normal file
View File

@@ -0,0 +1,263 @@
+++
title = "CQRS Component"
description = "Build CQRS and Event-Driven applications"
date = 2019-02-12T12:47:30+01:00
weight = -400
draft = false
bref = "Go CQRS implementation in Watermill"
+++
The CQRS component is a high-level API that lets you work with Go structs instead of messages.
Once you configure the EventBus and EventProcessor (or the command equivalents), publishing and handling events becomes very straightforward.
```go
event := UserRegistered{
UserID: id,
Email: email,
JoinedAt: time.Now(),
}
err := eventBus.Publish(ctx, event)
```
```go
eventProcessor.AddHandlers(
cqrs.NewEventHandler("SendWelcomeEmail", sendWelcomeEmail),
)
func sendWelcomeEmail(ctx context.Context, event *UserRegistered) error {
return emailService.Send(event.Email, "Welcome!")
}
```
## CQRS
> CQRS means "Command-query responsibility segregation". We segregate the responsibility between commands (write requests) and queries (read requests). The write requests and the read requests are handled by different objects.
>
> That's it. We can further split up the data storage, having separate read and write stores. Once that happens, there may be many read stores, optimized for handling different types of queries or spanning many bounded contexts. Though separate read/write stores are often discussed in relation with CQRS, this is not CQRS itself. CQRS is just the first split of commands and queries.
>
> Source: [www.cqrs.nu FAQ](http://www.cqrs.nu/Faq/command-query-responsibility-segregation)
<img src="https://threedots.tech/watermill-io/cqrs-big-picture.svg" alt="CQRS Schema" style="width:100%; margin-bottom: 3rem; background-color: white; padding: 2rem;" />
The `cqrs` component provides some useful abstractions built on top of Pub/Sub and Router that help to implement the CQRS pattern.
You don't need to implement the entire CQRS. It's very common to use just the event part of this component to build event-driven applications.
### Building blocks
#### Event
The event represents something that already took place. Events are immutable.
#### Event Bus
{{% load-snippet-partial file="src-link/components/cqrs/event_bus.go" first_line_contains="// EventBus" last_line_contains="type EventBus" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_bus.go" first_line_contains="type EventBusConfig" last_line_contains="func (c *EventBusConfig) setDefaults()" padding_after="4" %}}
#### Event Processor
{{% load-snippet-partial file="src-link/components/cqrs/event_processor.go" first_line_contains="// EventProcessor" last_line_contains="type EventProcessor" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_processor.go" first_line_contains="type EventProcessorConfig" last_line_contains="func (c *EventProcessorConfig) setDefaults()" padding_after="4" %}}
#### Event Group Processor
{{% load-snippet-partial file="src-link/components/cqrs/event_processor_group.go" first_line_contains="// EventGroupProcessor" last_line_contains="type EventGroupProcessor" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_processor_group.go" first_line_contains="type EventGroupProcessorConfig" last_line_contains="func (c *EventGroupProcessorConfig) setDefaults()" padding_after="4" %}}
Learn more in [Event Group Processor](#event-handler-groups).
#### Event Handler
{{% load-snippet-partial file="src-link/components/cqrs/event_handler.go" first_line_contains="// EventHandler" last_line_contains="type EventHandler" padding_after="0" %}}
#### Command
The command is a simple data structure, representing the request for executing some operation.
#### Command Bus
{{% load-snippet-partial file="src-link/components/cqrs/command_bus.go" first_line_contains="// CommandBus" last_line_contains="type CommandBus" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/command_bus.go" first_line_contains="type CommandBusConfig" last_line_contains="func (c *CommandBusConfig) setDefaults()" padding_after="4" %}}
#### Command Processor
{{% load-snippet-partial file="src-link/components/cqrs/command_processor.go" first_line_contains="// CommandProcessor" last_line_contains="type CommandProcessor" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/command_processor.go" first_line_contains="type CommandProcessorConfig" last_line_contains="func (c *CommandProcessorConfig) setDefaults()" padding_after="4" %}}
#### Command Handler
{{% load-snippet-partial file="src-link/components/cqrs/command_handler.go" first_line_contains="// CommandHandler" last_line_contains="type CommandHandler" padding_after="0" %}}
#### Command and Event Marshaler
{{% load-snippet-partial file="src-link/components/cqrs/marshaler.go" first_line_contains="// CommandEventMarshaler" last_line_contains="NameFromMessage(" padding_after="1" %}}
#### Command and Event Marshaler Decorator
Sometimes it's useful to add extra metadata to each command or event after marshaling it to a message. For example, you may want to add a partition key to each message using Kafka.
You can use `CommandEventMarshalerDecorator` to extend a marshaler with an extra step.
{{% load-snippet-partial file="src-link/components/cqrs/marshaler.go" first_line_contains="// CommandEventMarshalerDecorator" last_line_contains="}" padding_after="0" %}}
```go
type Event interface {
PartitionKey() string
}
// ...
cqrsMarshaler := CommandEventMarshalerDecorator{
CommandEventMarshaler: cqrs.JSONMarshaler{},
DecorateFunc: func(v any, msg *message.Message) error {
pm, ok := v.(Event)
if !ok {
return fmt.Errorf("%T does not implement Event and can't be marshaled", v)
}
partitionKey := pm.PartitionKey()
if partitionKey == "" {
return fmt.Errorf("PartitionKey is empty")
}
msg.Metadata.Set(PartitionKeyMetadataField, partitionKey)
return nil
},
}
```
## Usage
### Example domain
As an example, we will use a simple domain, that is responsible for handing room booking in a hotel.
We will use **Event Storming** notation to show the model of this domain.
Legend:
- **blue** post-its are commands
- **orange** post-its are events
- **green** post-its are read models, asynchronously generated from events
- **violet** post-its are policies, which are triggered by events and produce commands
- **pink** post its are hot-spots; we mark places where problems often occur
![CQRS Event Storming](https://threedots.tech/watermill-io/cqrs-example-storming.png)
The domain is simple:
- A Guest is able to **book a room**.
- **Whenever a room is booked, we order a beer** for the guest (because we love our guests).
- We know that sometimes there are **not enough beers**.
- We generate a **financial report** based on the bookings.
### Sending a command
For the beginning, we need to simulate the guest's action.
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="bookRoomCmd := &BookRoom{" last_line_contains="panic(err)" padding_after="1" %}}
### Command handler
`BookRoomHandler` will handle our command.
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="// BookRoomHandler is a command handler" last_line_contains="// OrderBeerOnRoomBooked is an event handler" padding_after="0" %}}
### Event handler
As mentioned before, we want to order a beer every time when a room is booked (*"Whenever a Room is booked"* post-it). We do it by using the `OrderBeer` command.
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="// OrderBeerOnRoomBooked is an event handler" last_line_contains="// OrderBeerHandler is a command handler" padding_after="0" %}}
`OrderBeerHandler` is very similar to `BookRoomHandler`. The only difference is, that it sometimes returns an error when there are not enough beers, which causes redelivery of the command.
You can find the entire implementation in the [example source code](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/5-cqrs-protobuf/?utm_source=cqrs_doc).
### Event Handler groups
By default, each event handler has a separate subscriber instance.
It works fine, if just one event type is sent to the topic.
In the scenario, when we have multiple event types on one topic, you have two options:
1. You can set `EventConfig.AckOnUnknownEvent` to true - it will acknowledge all events that are not handled by handler,
2. You can use Event Handler groups mechanism.
**Key differences between `EventProcessor` and `EventGroupProcessor`:**
1. `EventProcessor`:
- Each handler has its own subscriber instance
- One handler per event type
- Simple one-to-one matching of events to handlers
2. `EventGroupProcessor`:
- Group of handlers share a single subscriber instance (and one consumer group, if such mechanism is supported -- allows to maintain order of events),
- One handler group can support multiple event types,
- When message arrives to the topic, Watermill will match it to the handler in the group based on event type
<img src="/img/group-handlers.svg" alt="Group Handlers" style="width:100%; background-color: white; padding: ;">
**Event Handler groups are helpful when you have multiple event types on one topic and you want to maintain order of events.**
Thanks to using one subscriber instance and consumer group, events will be processed in the order they were sent.
{{< callout context="note" title="Note" icon="outline/info-circle" >}}
It's supported to have multiple handlers for the same event type in one group, but we recommend to not do that.
Please keep in mind that those handlers will be processed within the same message.
If first handler succeeds and the second fails, the message will be re-delivered and the first will be re-executed.
{{< /callout >}}
To use event groups, you need to set `GenerateHandlerGroupSubscribeTopic` and `GroupSubscriberConstructor` options in [`EventConfig`](#event-config).
After that, you can use `AddHandlersGroup` on [`EventProcessor`](#event-processor).
{{% load-snippet-partial file="src-link/_examples/basic/6-cqrs-ordered-events/main.go" first_line_contains="eventProcessor.AddHandlersGroup(" last_line_contains="if err != nil {" padding_after="0" %}}
Both `GenerateHandlerGroupSubscribeTopic` and `GroupSubscriberConstructor` receives information about group name in function arguments.
You can see a fully working example with event groups in our [examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/6-cqrs-ordered-events/).
### Generic handlers
Since Watermill v1.3 it's possible to use generic handlers for commands and events. It's useful when you have a lot of commands/events and you don't want to create a handler for each of them.
{{% load-snippet-partial file="src-link/_examples/basic/6-cqrs-ordered-events/main.go" first_line_contains="cqrs.NewGroupEventHandler" last_line_contains=")," padding_after="0" %}}
Under the hood, it creates EventHandler or CommandHandler implementation.
It's available for all kind of handlers.
{{% load-snippet-partial file="src-link/components/cqrs/command_handler.go" first_line_contains="// NewCommandHandler" last_line_contains="func NewCommandHandler" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_handler.go" first_line_contains="// NewEventHandler" last_line_contains="func NewEventHandler" padding_after="0" %}}
{{% load-snippet-partial file="src-link/components/cqrs/event_handler.go" first_line_contains="// NewGroupEventHandler" last_line_contains="func NewGroupEventHandler" padding_after="0" %}}
### Building a read model with the event handler
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="// BookingsFinancialReport is a read model" last_line_contains="func main() {" padding_after="0" %}}
### Wiring it up
We have all the blocks to build our CQRS application.
We will use the AMQP (RabbitMQ) as our message broker: [AMQP]({{< ref "/pubsubs/amqp" >}}).
Under the hood, CQRS is using Watermill's message router. If you are not familiar with it and want to learn how it works, you should check [Getting Started guide]({{< ref "getting-started" >}}).
It will also show you how to use some standard messaging patterns, like metrics, poison queue, throttling, correlation and other tools used by every message-driven application. Those come built-in with Watermill.
Let's go back to the CQRS. As you already know, CQRS is built from multiple components, like Command or Event buses, handlers, processors, etc.
{{% load-snippet-partial file="src-link/_examples/basic/5-cqrs-protobuf/main.go" first_line_contains="main() {" last_line_contains="err := router.Run(" padding_after="3" %}}
And that's all. We have a working CQRS application.
### What's next?
As mentioned before, if you are not familiar with Watermill, we highly recommend reading [Getting Started guide]({{< ref "getting-started" >}}).

43
docs/content/docs/message.md Normal file
View File

@@ -0,0 +1,43 @@
+++
title = "Message"
description = "Message is one of core parts of Watermill"
date = 2018-12-05T12:42:40+01:00
weight = -1000
draft = false
bref = "Message is one of core parts of Watermill"
+++
Message is one of the core parts of Watermill. Messages are emitted by [*Publishers*]({{< ref "/docs/pub-sub#publisher" >}}) and received by [*Subscribers*]({{< ref "/docs/pub-sub#subscriber" >}}).
When a message is processed, you should send an [`Ack()`]({{< ref "#ack" >}}) or a [`Nack()`]({{< ref "#nack" >}}) when the processing failed.
`Acks` and `Nacks` are processed by Subscribers (in default implementations, the subscribers are waiting for an `Ack` or a `Nack`).
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="type Message struct {" last_line_contains="ctx context.Context" padding_after="2" %}}
## Ack
#### Sending `Ack`
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="// Ack" last_line_contains="func (m *Message) Ack() bool {" padding_after="0" %}}
## Nack
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="// Nack" last_line_contains="func (m *Message) Nack() bool {" padding_after="0" %}}
#### Receiving `Ack/Nack`
{{% load-snippet-partial file="docs/message/receiving-ack.go" first_line_contains="select {" last_line_contains="}" padding_after="0" %}}
## Context
Message contains the standard library context, just like an HTTP request.
This context is used to propagate cancellation and deadlines when publishing and processing messages.
You can set the context using the `SetContext` method or the `NewMessageWithContext` constructor.
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="// NewMessageWithContext" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/message/message.go" first_line_contains="// Context" last_line_contains="func (m *Message) SetContext" padding_after="2" %}}

3
docs/content/docs/message/.validate_example.yml Normal file
View File

@@ -0,0 +1,3 @@
validation_cmd: "go run receiving-ack.go"
timeout: 30
expected_output: "ack received"

12
docs/content/docs/message/go.mod Normal file
View File

@@ -0,0 +1,12 @@
module receiving-ack.go
require github.com/ThreeDotsLabs/watermill v1.5.1
require (
github.com/google/uuid v1.6.0 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
)
go 1.25

19
docs/content/docs/message/go.sum Normal file
View File

@@ -0,0 +1,19 @@
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.11.0 h1:ib4sjIrwZKxE5u/Japgo/7SJV3PvgjGiRNAvTVGqQl8=
github.com/stretchr/testify v1.11.0/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

24
docs/content/docs/message/receiving-ack.go Normal file
View File

@@ -0,0 +1,24 @@
package main
import (
"log"
"time"
"github.com/ThreeDotsLabs/watermill/message"
)
func main() {
msg := message.NewMessage("1", []byte("foo"))
go func() {
time.Sleep(time.Millisecond * 10)
msg.Ack()
}()
select {
case <-msg.Acked():
log.Print("ack received")
case <-msg.Nacked():
log.Print("nack received")
}
}

124
docs/content/docs/messages-router.md Normal file
View File

@@ -0,0 +1,124 @@
+++
title = "Router"
description = "The Magic Glue of Watermill"
date = 2018-12-05T12:48:04+01:00
weight = -850
draft = false
bref = "The Magic Glue of Watermill"
toc = true
+++
[*Publishers and Subscribers*]({{< ref "/docs/pub-sub" >}}) are rather low-level parts of Watermill.
In production use, you'd usually want to use a high-level interface and features like [correlation, metrics, poison queue, retrying, throttling, etc.]({{< ref "/docs/messages-router#middleware" >}}).
You also might not want to send an Ack when processing was successful. Sometimes, you'd like to send a message after processing of another message finishes.
To handle these requirements, there is a component named **Router**.
<img src="/img/watermill-router.svg" alt="Watermill Router" style="width:100%; background-color: white; padding: ;">
## Configuration
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="type RouterConfig struct {" last_line_contains="RouterConfig) Validate()" padding_after="2" %}}
## Handler
At the beginning you need to implement `HandlerFunc`:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// HandlerFunc is" last_line_contains="type HandlerFunc func" padding_after="1" %}}
Next, you have to add a new handler with `Router.AddHandler`:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// AddHandler" last_line_contains=") {" padding_after="0" %}}
See an example usage from [Getting Started]({{< ref "/learn/getting-started#using-messages-router" >}}):
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="// AddHandler returns a handler" last_line_contains="return h(message)" padding_after="3" %}}
## Consumer handler
Not every handler will produce new messages. You can add this kind of handler by using `Router.AddConsumerHandler`:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// AddConsumerHandler" last_line_contains=") {" padding_after="0" %}}
## Ack
By default, `msg.Ack()` is called when `HandlerFunc` doesn't return an error. If an error is returned, `msg.Nack()` will be called.
Because of this, you don't have to call `msg.Ack()` or `msg.Nack()` after a message is processed (you can if you want, of course).
## Producing messages
When returning multiple messages from a handler, be aware that most Publisher implementations don't support [atomic publishing of messages]({{< ref "/docs/pub-sub#publishing-multiple-messages" >}}). It may end up producing only some messages and sending `msg.Nack()` if the broker or the storage are not available.
If it is an issue, consider publishing just one message with each handler.
## Running the Router
To run the Router, you need to call `Run()`.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// Run" last_line_contains="func (r *Router) Run(ctx context.Context) (err error) {" padding_after="0" %}}
### Ensuring that the Router is running
It can be useful to know if the router is running. You can use the `Running()` method for this.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// Running" last_line_contains="func (r *Router) Running()" padding_after="0" %}}
You can also use `IsRunning` function, that returns bool:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// IsRunning" last_line_contains="func (r *Router) IsRunning()" padding_after="0" %}}
### Closing the Router
To close the Router, you need to call `Close()`.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// Close gracefully" last_line_contains="func (r *Router) Close()" padding_after="1" %}}
`Close()` will close all publishers and subscribers, and wait for all handlers to finish.
`Close()` will wait for a timeout configured in `RouterConfig.CloseTimeout`.
If the timeout is reached, `Close()` will return an error.
## Adding handler after the router has started
You can add a new handler while the router is already running.
To do that, you need to call `AddConsumerHandler` or `AddHandler` and call `RunHandlers`.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// RunHandlers" last_line_contains="func (r *Router) RunHandlers" padding_after="0" %}}
## Stopping running handler
It is possible to stop **just one running handler** by calling `Stop()`.
Please keep in mind, that router will be closed when there are no running handlers.
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// Stop" last_line_contains="func (h *Handler) Stop()" padding_after="0" %}}
## Execution models
*Subscribers* can consume either one message at a time or multiple messages in parallel.
* **Single stream of messages** is the simplest approach and it means that until a `msg.Ack()` is called, the subscriber
will not receive any new messages.
* **Multiple message streams** are supported only by some subscribers. By subscribing to multiple topic partitions at once,
several messages can be consumed in parallel, even previous messages that were not acked (for example, the Kafka subscriber
works like this). Router handles this model by running concurrent `HandlerFunc`s, one for each partition.
See the chosen Pub/Sub documentation for supported execution models.
## Middleware
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// HandlerMiddleware" last_line_contains="type HandlerMiddleware" padding_after="1" %}}
A full list of standard middleware can be found in [Middleware]({{< ref "/docs/middlewares" >}}).
## Plugin
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// RouterPlugin" last_line_contains="type RouterPlugin" padding_after="1" %}}
A full list of standard plugins can be found in [message/router/plugin](https://github.com/ThreeDotsLabs/watermill/tree/master/message/router/plugin).
## Context
Each message received by handler holds some useful values in the `context`:
{{% load-snippet-partial file="src-link/message/router_context.go" first_line_contains="// HandlerNameFromCtx" last_line_contains="func PublishTopicFromCtx" padding_after="2" %}}

36
docs/content/docs/middlewares.md Normal file
View File

@@ -0,0 +1,36 @@
+++
title = "Middleware"
description = "Add generic functionalities to your handlers in an unobtrusive way"
date = 2019-06-01T19:00:00+01:00
weight = -500
draft = false
bref = "Add functionality to handlers"
+++
## Introduction
Middleware wraps handlers with functionality that is important, but not relevant for the primary handler's logic.
Examples include retrying the handler after an error was returned, or recovering from a panic in the handler
and capturing the stacktrace.
Middleware wraps the handler function like this:
{{% load-snippet-partial file="src-link/message/router.go" first_line_contains="// HandlerMiddleware" last_line_contains="type HandlerMiddleware" %}}
## Usage
Middleware can be executed for all as well as for a specific handler in a router. When middleware is added directly
to a router it will be executed for all handlers provided for a router. If a middleware should be executed only
for a specific handler, it needs to be added to handler in the router.
Example usage is shown below:
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="router, err := message.NewRouter(message.RouterConfig{}, logger)" last_line_contains="// Now that all handlers are registered, we're running the Router." padding_after="1" %}}
## Available middleware
Below are the middleware provided by Watermill and ready to use. You can also easily implement your own.
For example, if you'd like to store every received message in some kind of log, it's the best way to do it.
{{% readfile file="/content/src-link/middleware-defs.md" %}}

73
docs/content/docs/pub-sub.md Normal file
View File

@@ -0,0 +1,73 @@
+++
title = "Publisher & Subscriber"
description = "Publishers and Subscribers"
date = 2018-12-05T12:47:30+01:00
weight = -900
draft = false
bref = "Publishers and Subscribers"
+++
## Publisher
{{% load-snippet-partial file="src-link/message/pubsub.go" first_line_contains="Publisher interface {" last_line_contains="Close() error" padding_after="1" %}}
### Publishing multiple messages
Most publisher implementations don't support atomic publishing of messages.
This means that if publishing one of the messages fails, the remaining messages won't be published.
### Async publish
Publish can be synchronous or asynchronous - it depends on the implementation.
#### `Close()`
`Close` should flush unsent messages if the publisher is asynchronous.
**It is important not to forget to close the publisher**. Otherwise you may lose some of the messages.
## Subscriber
{{% load-snippet-partial file="src-link/message/pubsub.go" first_line_contains="Subscriber interface {" last_line_contains="Close() error" padding_after="1" %}}
### Ack/Nack mechanism
It is the *Subscriber's* responsibility to handle an `Ack` and a `Nack` from a message.
A proper implementation should wait for an `Ack` or a `Nack` before consuming the next message.
**Important Subscriber's implementation notice**:
Ack/offset to message's storage/broker **must** be sent after Ack from Watermill's message.
Otherwise there is a risk of losing messages if the process dies before the messages have been processed.
#### `Close()`
`Close` shuts down all subscriptions, closes their output channels, and flushes offsets when needed.
## At-least-once delivery
Watermill is built with [at-least-once delivery](http://www.cloudcomputingpatterns.org/at_least_once_delivery/) semantics.
This means that if an error occurs while processing a message and an Ack cannot be sent, the message will be redelivered.
You need to keep it in mind and build your application to be [idempotent](http://www.cloudcomputingpatterns.org/idempotent_processor/) or implement a deduplication mechanism.
Unfortunately, it's not possible to create a universal [*middleware*]({{< ref "/docs/messages-router#middleware" >}}) for deduplication, so we encourage you to build your own.
## Universal tests
Every Pub/Sub is similar in most aspects.
To avoid implementing separate tests for every Pub/Sub, we've created a test suite, which should be passed by any Pub/Sub
implementation.
These tests can be found in `pubsub/tests/test_pubsub.go`.
## Built-in implementations
To check available Pub/Sub implementations, see [Supported Pub/Subs]({{< ref "/pubsubs" >}}).
## Implementing custom Pub/Sub
See [Implementing custom Pub/Sub]({{< ref "/development/pub-sub-implementing" >}}) for instructions on how to introduce support for
a new Pub/Sub.
We will also be thankful for submitting [pull requests](https://github.com/ThreeDotsLabs/watermill/pulls) with the new Pub/Sub implementations.
You can also request a new Pub/Sub implementation by submitting a [new issue](https://github.com/ThreeDotsLabs/watermill/issues).

4
docs/content/docs/snippets/amqp-consumer-groups/.validate_example.yml Normal file
View File

@@ -0,0 +1,4 @@
validation_cmd: "docker compose up"
teardown_cmd: "docker compose down"
timeout: 120
expected_output: "payload: Hello, world!"

15
docs/content/docs/snippets/amqp-consumer-groups/docker-compose.yml Normal file
View File

@@ -0,0 +1,15 @@
services:
server:
image: golang:1.25
restart: unless-stopped
depends_on:
- rabbitmq
volumes:
- .:/app
- $GOPATH/pkg/mod:/go/pkg/mod
working_dir: /app
command: go run main.go
rabbitmq:
image: rabbitmq:3.7
restart: unless-stopped

19
docs/content/docs/snippets/amqp-consumer-groups/go.mod Normal file
View File

@@ -0,0 +1,19 @@
module main.go
require (
github.com/ThreeDotsLabs/watermill v1.5.1
github.com/ThreeDotsLabs/watermill-amqp/v2 v2.1.3
)
require (
github.com/cenkalti/backoff/v3 v3.2.2 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rabbitmq/amqp091-go v1.11.0 // indirect
)
go 1.25

32
docs/content/docs/snippets/amqp-consumer-groups/go.sum Normal file
View File

@@ -0,0 +1,32 @@
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/ThreeDotsLabs/watermill-amqp/v2 v2.1.3 h1:fkhmiBtaLn+rz5lbkPD1h8tXHfKy3gX0vMtGmxNtAsk=
github.com/ThreeDotsLabs/watermill-amqp/v2 v2.1.3/go.mod h1:xy2qXKcJpgrJURRT6YwgRyGL3qIi6/sOHrDI0MO/r5I=
github.com/cenkalti/backoff/v3 v3.2.2 h1:cfUAAO3yvKMYKPrvhDuHSwQnhZNk/RMHKdZqKTxfm6M=
github.com/cenkalti/backoff/v3 v3.2.2/go.mod h1:cIeZDE3IrqwwJl6VUwCN6trj1oXrTS4rc0ij+ULvLYs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rabbitmq/amqp091-go v1.11.0 h1:HxIctVm9Gid/Vtn706necmZ7Wj6pgGI2eqplRbEY8O8=
github.com/rabbitmq/amqp091-go v1.11.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
github.com/stretchr/testify v1.11.0 h1:ib4sjIrwZKxE5u/Japgo/7SJV3PvgjGiRNAvTVGqQl8=
github.com/stretchr/testify v1.11.0/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

80
docs/content/docs/snippets/amqp-consumer-groups/main.go Normal file
View File

@@ -0,0 +1,80 @@
package main
import (
"context"
"log"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill-amqp/v2/pkg/amqp"
"github.com/ThreeDotsLabs/watermill/message"
)
var amqpURI = "amqp://guest:guest@rabbitmq:5672/"
func createSubscriber(queueSuffix string) *amqp.Subscriber {
subscriber, err := amqp.NewSubscriber(
// This config is based on this example: https://www.rabbitmq.com/tutorials/tutorial-three-go.html
// to create just a simple queue, you can use NewDurableQueueConfig or create your own config.
amqp.NewDurablePubSubConfig(
amqpURI,
// Rabbit's queue name in this example is based on Watermill's topic passed to Subscribe
// plus provided suffix.
//
// Exchange is Rabbit's "fanout", so when subscribing with suffix other than "test_consumer_group",
// it will also receive all messages. It will work like separate consumer groups in Kafka.
amqp.GenerateQueueNameTopicNameWithSuffix(queueSuffix),
),
watermill.NewStdLogger(false, false),
)
if err != nil {
panic(err)
}
return subscriber
}
func main() {
subscriber1 := createSubscriber("test_consumer_group_1")
messages1, err := subscriber1.Subscribe(context.Background(), "example.topic")
if err != nil {
panic(err)
}
go process("subscriber_1", messages1)
subscriber2 := createSubscriber("test_consumer_group_2")
messages2, err := subscriber2.Subscribe(context.Background(), "example.topic")
if err != nil {
panic(err)
}
// subscriber2 will receive all messages independently from subscriber1
go process("subscriber_2", messages2)
publisher, err := amqp.NewPublisher(
amqp.NewDurablePubSubConfig(
amqpURI,
nil, // generateQueueName is not used with publisher
),
watermill.NewStdLogger(false, false),
)
if err != nil {
panic(err)
}
publishMessages(publisher)
}
func publishMessages(publisher message.Publisher) {
for {
msg := message.NewMessage(watermill.NewUUID(), []byte("Hello, world!"))
if err := publisher.Publish("example.topic", msg); err != nil {
panic(err)
}
}
}
func process(subscriber string, messages <-chan *message.Message) {
for msg := range messages {
log.Printf("[%s] received message: %s, payload: %s", subscriber, msg.UUID, string(msg.Payload))
msg.Ack()
}
}

17
docs/content/docs/snippets/tail-log-file/go.mod Normal file
View File

@@ -0,0 +1,17 @@
module github.com/ThreeDotsLabs/watermill/docs/content/docs/snippets/tail-log-file
go 1.25
require (
github.com/ThreeDotsLabs/watermill v1.5.1
github.com/ThreeDotsLabs/watermill-io v1.1.2
)
require (
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
)

26
docs/content/docs/snippets/tail-log-file/go.sum Normal file
View File

@@ -0,0 +1,26 @@
github.com/ThreeDotsLabs/watermill v1.5.1 h1:t5xMivyf9tpmU3iozPqyrCZXHvoV1XQDfihas4sV0fY=
github.com/ThreeDotsLabs/watermill v1.5.1/go.mod h1:Uop10dA3VeJWsSvis9qO3vbVY892LARrKAdki6WtXS4=
github.com/ThreeDotsLabs/watermill-io v1.1.2 h1:t3wismmE++6HbB+fDnSdvLtSKs0yYaSXRtLmyUcRyTk=
github.com/ThreeDotsLabs/watermill-io v1.1.2/go.mod h1:DF6rhoPWBOeWRW/1wWjNfLkke8rZsB5BUzBox/L6fRI=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.11.0 h1:ib4sjIrwZKxE5u/Japgo/7SJV3PvgjGiRNAvTVGqQl8=
github.com/stretchr/testify v1.11.0/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

53
docs/content/docs/snippets/tail-log-file/main.go Normal file
View File

@@ -0,0 +1,53 @@
package main
import (
"context"
"fmt"
"os"
"github.com/ThreeDotsLabs/watermill"
"github.com/ThreeDotsLabs/watermill-io/pkg/io"
"github.com/ThreeDotsLabs/watermill/message"
)
// this will `tail -f` a log file and publish an alert if a line fulfils some criterion
func main() {
// if an alert is raised, the offending line will be published on this
// this would be set to an actual publisher
var alertPublisher message.Publisher
if len(os.Args) < 2 {
panic(
fmt.Errorf("usage: %s /path/to/file.log", os.Args[0]),
)
}
logFile, err := os.OpenFile(os.Args[1], os.O_RDONLY, 0444)
if err != nil {
panic(err)
}
sub, err := io.NewSubscriber(logFile, io.SubscriberConfig{
UnmarshalFunc: io.PayloadUnmarshalFunc,
}, watermill.NewStdLogger(true, false))
if err != nil {
panic(err)
}
// for io.Subscriber, topic does not matter
lines, err := sub.Subscribe(context.Background(), "")
if err != nil {
panic(err)
}
for line := range lines {
if criterion(string(line.Payload)) {
_ = alertPublisher.Publish("alerts", line)
}
}
}
func criterion(line string) bool {
// decide whether an action needs to be taken
return false
}

103
docs/content/docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,103 @@
+++
title = "Troubleshooting"
description = "When something goes wrong"
weight = -90
draft = false
bref = "When something goes wrong"
+++
## Logging
In most cases, you will find the answer to your problem in the logs.
Watermill offers a significant amount of logs on different severity levels.
If you are using `StdLoggerAdapter`, just change `debug` and `trace` options to true:
```bash
logger := watermill.NewStdLogger(true, true)
```
## Debugging Pub/Sub tests
### Running a single test
```bash
make up
go test -v ./... -run TestPublishSubscribe/TestContinueAfterSubscribeClose
```
### Grep is your friend
Each executed test case has a unique UUID.
It's used in the topic name.
Thanks to that, you can easily grep the output of the test.
This gives you detailed information about the test execution.
```bash
> go test -v ./... > test.out
> less test.out
// ...
--- PASS: TestPublishSubscribe (0.00s)
--- PASS: TestPublishSubscribe/TestPublishSubscribe (2.38s)
--- PASS: TestPublishSubscribe/TestPublishSubscribe/81eeb56c-3336-4eb9-a0ac-13abda6f38ff (2.38s)
```
```bash
cat test.out | grep 81eeb56c-3336-4eb9-a0ac-13abda6f38ff | less
[watermill] 2020/08/18 14:51:46.283366 subscriber.go:300: level=TRACE msg="Msg acked" message_uuid=5c920330-5075-4870-8d86-9013771eee78 provider=google_cloud_pubsub subscription_name=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff topic=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff
[watermill] 2020/08/18 14:51:46.283405 subscriber.go:300: level=TRACE msg="Msg acked" message_uuid=46e04a08-994e-4c04-afff-7fd42fd67f95 provider=google_cloud_pubsub subscription_name=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff topic=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff
2020/08/18 14:51:46 all messages (100/100) received in bulk read after 110.04155ms of 45s (test ID: 81eeb56c-3336-4eb9-a0ac-13abda6f38ff)
[watermill] 2020/08/18 14:51:46.284569 subscriber.go:186: level=DEBUG msg="Closing message consumer" provider=google_cloud_pubsub subscription_name=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff topic=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff
[watermill] 2020/08/18 14:51:46.284828 subscriber.go:300: level=TRACE msg="Msg acked" message_uuid=2f409208-d4d2-46f6-b6b9-afb1aea0e59f provider=google_cloud_pubsub subscription_name=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff topic=topic_81eeb56c-3336-4eb9-a0ac-13abda6f38ff
--- PASS: TestPublishSubscribe/TestPublishSubscribe/81eeb56c-3336-4eb9-a0ac-13abda6f38ff (2.38s)
```
## I have a deadlock
When running locally, you can send a `SIGQUIT` to the running process:
- `CTRL + \` on Linux
- `kill -s SIGQUIT [pid]` on other UNIX systems
This will kill the process and print all goroutines along with lines on which they have stopped.
```bash
SIGQUIT: quit
PC=0x45e7c3 m=0 sigcode=128
goroutine 1 [runnable]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).sendMessage(0xc000024100, 0x7c5250, 0xd, 0xc000872d70, 0x0, 0x0)
/home/example/go/src/github.com/ThreeDotsLabs/watermill/pubsub/gochannel/pubsub.go:83 +0x36a
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Publish(0xc000024100, 0x7c5250, 0xd, 0xc000098530, 0x1, 0x1, 0x0, 0x0)
/home/example/go/src/github.com/ThreeDotsLabs/watermill/pubsub/gochannel/pubsub.go:53 +0x6d
main.publishMessages(0x7fdf7a317000, 0xc000024100)
/home/example/go/src/github.com/ThreeDotsLabs/watermill/docs/src-link/_examples/pubsubs/go-channel/main.go:43 +0x1ec
main.main()
/home/example/go/src/github.com/ThreeDotsLabs/watermill/docs/src-link/_examples/pubsubs/go-channel/main.go:36 +0x20a
// ...
```
When running in production and you don't want to kill the entire process, a better idea is to use [pprof](https://golang.org/pkg/net/http/pprof/).
You can visit [http://localhost:6060/debug/pprof/goroutine?debug=1](http://localhost:6060/debug/pprof/goroutine?debug=1)
on your local machine to see the status of all goroutines.
```bash
goroutine profile: total 5
1 @ 0x41024c 0x6a8311 0x6a9bcb 0x6a948d 0x7028bc 0x70260a 0x42f187 0x45c971
# 0x6a8310 github.com/ThreeDotsLabs/watermill.LogFields.Add+0xd0 /home/example/go/src/github.com/ThreeDotsLabs/watermill/log.go:15
# 0x6a9bca github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).sendMessage+0x6fa /home/example/go/src/github.com/ThreeDotsLabs/watermill/pubsub/gochannel/pubsub.go:75
# 0x6a948c github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Publish+0x6c /home/example/go/src/github.com/ThreeDotsLabs/watermill/pubsub/gochannel/pubsub.go:53
# 0x7028bb main.publishMessages+0x1eb /home/example/go/src/github.com/ThreeDotsLabs/watermill/docs/src-link/_examples/pubsubs/go-channel/main.go:43
# 0x702609 main.main+0x209 /home/example/go/src/github.com/ThreeDotsLabs/watermill/docs/src-link/_examples/pubsubs/go-channel/main.go:36
# 0x42f186 runtime.main+0x206 /usr/lib/go/src/runtime/proc.go:201
// ...
```

53
docs/content/learn/_index.md Normal file
View File

File diff suppressed because one or more lines are too long

395
docs/content/learn/getting-started.md Normal file
View File

@@ -0,0 +1,395 @@
+++
title = "Getting started"
description = "Watermill up and running"
draft = false
bref = "Watermill up and running"
weight = 10
+++
## What is Watermill?
Watermill is a Go library for working with messages the easy way.
You can use it to build message-driven and event-driven applications with Pub/Subs like Kafka, RabbitMQ, PostgreSQL, and many more.
Watermill comes with batteries included. It gives you tools used by every message-driven application.
## Why use Watermill?
When you run an HTTP server, you don't deal directly with TCP sockets, parsing HTTP requests, or managing connections.
Instead, you use a high-level library like `net/http` that handles all that complexity for you.
**It's what Watermill aims to be for messages**.
It provides all you need to build an application based on events or other asynchronous patterns.
There are many different message queues, each with different features, client libraries, and APIs.
Watermill hides all that complexity behind an API that is easy to use and understand.
**Watermill is NOT a framework**.
It's a lightweight library that's easy to plug in or remove from your project.
## Install
```bash
go get -u github.com/ThreeDotsLabs/watermill
```
{{< callout context="note" title="Learn in practice" icon="outline/info-circle" >}}
Docs too boring? Prefer learning by doing?
[**Try the free hands-on training**]({{< ref "/learn/quickstart/" >}}) where you'll solve exercises to learn how to use Watermill in your projects.
It'll guide you through the basics and a few advanced concepts like message ordering and the Outbox pattern.
{{< /callout >}}
## One-Minute Background
The idea behind event-driven applications is always the same: one part publishes messages, and another part subscribes to them.
Watermill supports this behavior for multiple [publishers and subscribers]({{< ref "/pubsubs" >}}).
### Three APIs
Watermill comes with three APIs for working with messages.
They build on top of each other, each step providing a higher-level API.
In this guide, we're going to start from the bottom and move up.
It's good to know the fundamentals, even if you're going to use the high-level APIs.
<div class="text-center">
<img src="/img/pyramid.png" alt="Watermill components pyramid" style="width:35rem;" />
</div>
## Publisher & Subscriber
Most Pub/Sub libraries come with complex features.
Watermill hides this complexity behind two interfaces: the `Publisher` and `Subscriber`.
```go
type Publisher interface {
Publish(topic string, messages ...*Message) error
Close() error
}
type Subscriber interface {
Subscribe(ctx context.Context, topic string) (<-chan *Message, error)
Close() error
}
```
### Creating Messages
**The core part of Watermill is the [Message]({{< ref "/docs/message" >}}).**
It is what `http.Request` is for the `net/http` package.
Most Watermill features work with this struct.
Watermill doesn't enforce any message format. `NewMessage` expects a slice of bytes as the payload.
You can use strings, JSON, protobuf, Avro, gob, or anything else that serializes to `[]byte`.
The message UUID is optional but recommended for debugging.
```go
msg := message.NewMessage(watermill.NewUUID(), []byte("Hello, world!"))
```
### Publishing Messages
`Publish` expects a topic and one or more `Message`s to be published.
```go
err := publisher.Publish("example.topic", msg)
if err != nil {
panic(err)
}
```
{{< tabs "publishing" >}}
{{< tab "Go Channel" "go-channel" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/go-channel/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{< /tab >}}
{{< tab "Kafka" "kafka" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{< /tab >}}
{{< tab "NATS JetStream" "nats" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-jetstream/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{< /tab >}}
{{< tab "Google Cloud Pub/Sub" "gcp" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{< /tab >}}
{{< tab "RabbitMQ (AMQP)" "amqp" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{< /tab >}}
{{< tab "SQL" "sql" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{< /tab >}}
{{< tab "AWS SQS" "aws-sqs" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/aws-sqs/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{< /tab >}}
{{< tab "AWS SNS" "aws-sns" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/aws-sns/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{< /tab >}}
{{< /tabs >}}
### Subscribing for Messages
`Subscribe` expects a topic name and returns a channel of incoming messages.
What _topic_ exactly means depends on the Pub/Sub implementation.
Usually, it needs to match the topic name used by the publisher.
Messages need to be acknowledged after processing by calling the `Ack()` method.
```go
messages, err := subscriber.Subscribe(ctx, "example.topic")
if err != nil {
panic(err)
}
for msg := range messages {
fmt.Printf("received message: %s, payload: %s\n", msg.UUID, string(msg.Payload))
msg.Ack()
}
```
See detailed examples below for supported PubSubs.
{{< tabs "getting-started" >}}
{{< tab "Go Channel" "go-channel" >}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/go-channel/main.go" first_line_contains="package main" last_line_contains="process(messages)" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/go-channel/main.go" first_line_contains="func process" %}}
{{< /tab >}}
{{< tab "Kafka" "kafka" >}}
<details>
<summary><strong>Running in Docker</strong></summary>
The easiest way to run Watermill locally with Kafka is by using Docker.
{{% load-snippet file="src-link/_examples/pubsubs/kafka/docker-compose.yml" type="yaml" %}}
The source should go to `main.go`.
To run, execute the `docker-compose up` command.
A more detailed explanation of how it works (and how to add live code reload) can be found in the [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
</details>
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="package main" last_line_contains="process(messages)" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="func process" %}}
{{< /tab >}}
{{< tab "NATS JetStream" "nats" >}}
<details>
<summary><strong>Running in Docker</strong></summary>
The easiest way to run Watermill locally with NATS is using Docker.
{{% load-snippet file="src-link/_examples/pubsubs/nats-jetstream/docker-compose.yml" type="yaml" %}}
The source should go to `main.go`.
To run, execute the `docker-compose up` command.
A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
</details>
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-jetstream/main.go" first_line_contains="package main" last_line_contains="process(messages)" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-jetstream/main.go" first_line_contains="func process" %}}
{{< /tab >}}
{{< tab "Google Cloud Pub/Sub" "gcp" >}}
<details>
<summary><strong>Running in Docker</strong></summary>
You can run the Google Cloud Pub/Sub emulator locally for development.
{{% load-snippet file="src-link/_examples/pubsubs/googlecloud/docker-compose.yml" type="yaml" %}}
The source should go to `main.go`.
To run, execute `docker-compose up`.
A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
</details>
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="package main" last_line_contains="process(messages)" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="func process" %}}
{{< /tab >}}
{{< tab "RabbitMQ (AMQP)" "amqp" >}}
<details>
<summary><strong>Running in Docker</strong></summary>
{{% load-snippet file="src-link/_examples/pubsubs/amqp/docker-compose.yml" type="yaml" %}}
The source should go to `main.go`.
To run, execute `docker-compose up`.
A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
</details>
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="package main" last_line_contains="process(messages)" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="func process" %}}
{{< /tab >}}
{{< tab "SQL" "sql" >}}
<details>
<summary><strong>Running in Docker</strong></summary>
{{% load-snippet file="src-link/_examples/pubsubs/sql/docker-compose.yml" type="yaml" %}}
The source should go to `main.go`.
To run, execute `docker-compose up`.
A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
</details>
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="package main" last_line_contains="process(messages)" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="func process" %}}
{{< /tab >}}
{{< tab "AWS SQS" "aws-sqs" >}}
<details>
<summary><strong>Running in Docker</strong></summary>
{{% load-snippet file="src-link/_examples/pubsubs/aws-sqs/docker-compose.yml" type="yaml" %}}
The source should go to `main.go`.
To run, execute `docker-compose up`.
A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
</details>
{{% load-snippet-partial file="src-link/_examples/pubsubs/aws-sqs/main.go" first_line_contains="package main" last_line_contains="process(messages)" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/aws-sqs/main.go" first_line_contains="func process" %}}
{{< /tab >}}
{{< tab "AWS SNS" "aws-sns" >}}
<details>
<summary><strong>Running in Docker</strong></summary>
{{% load-snippet file="src-link/_examples/pubsubs/aws-sns/docker-compose.yml" type="yaml" %}}
The source should go to `main.go`.
To run, execute `docker-compose up`.
A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
</details>
{{% load-snippet-partial file="src-link/_examples/pubsubs/aws-sns/main.go" first_line_contains="package main" last_line_contains="go process(" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/aws-sns/main.go" first_line_contains="func process" %}}
{{< /tab >}}
{{< /tabs >}}
## Router
[*Publishers and subscribers*]({{< ref "/docs/pub-sub" >}}) are the low-level parts of Watermill.
For most cases, you want to use a high-level API: the [*Router*]({{< ref "/docs/messages-router" >}}) component.
### Router configuration
Start with configuring the router and adding plugins and middlewares.
A middleware is a function executed for each incoming message.
You can use one of the existing ones for things like [correlation, metrics, poison queue, retrying, throttling, etc.]({{< ref "/docs/messages-router#middleware" >}}).
You can also create your own.
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="message.NewRouter" last_line_contains="middleware.Recoverer," padding_after="1" %}}
### Handlers
Set up handlers that the router uses.
Each handler independently handles incoming messages.
A handler listens to messages from the given subscriber and topic.
Any messages returned from the handler function will be published to the given publisher and topic.
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="AddHandler returns" last_line_contains=")" padding_after="0" %}}
*Note: the example above uses one `pubSub` argument for both the subscriber and publisher.
It's because we use the `GoChannel` implementation, which is a simple in-memory Pub/Sub.*
Alternatively, if you don't plan to publish messages from within the handler, you can use the simpler `AddConsumerHandler` method.
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="AddConsumerHandler" last_line_contains=")" padding_after="0" %}}
You can use two types of *handler functions*:
1. a function `func(msg *message.Message) ([]*message.Message, error)`
2. a struct method `func (c structHandler) Handler(msg *message.Message) ([]*message.Message, error)`
Use the first one if your handler is a function without any dependencies.
The second option is useful when your handler requires dependencies such as a database handle or a logger.
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="func printMessages" last_line_contains="return message.Messages{msg}, nil" padding_after="3" %}}
Finally, run the router.
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="router.Run" last_line_contains="}" padding_after="0" %}}
The complete example's source can be found at [/_examples/basic/3-router/main.go](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/3-router/main.go).
## Logging
To see Watermill's logs, pass any logger that implements the [LoggerAdapter](https://github.com/ThreeDotsLabs/watermill/blob/master/log.go).
For experimental development, you can use `NewStdLogger`.
Watermill provides ready-to-use `slog` adapter. You can create it with [`watermill.NewSlogLogger`](https://github.com/ThreeDotsLabs/watermill/blob/master/slog.go).
You can also map Watermill's log levels to `slog` levels with [`watermill.NewSlogLoggerWithLevelMapping`](https://github.com/ThreeDotsLabs/watermill/blob/master/slog.go).
## What's next?
See the [CQRS component](/docs/cqrs) for the generic high-level API.
For more details, see [documentation topics]({{< ref "/docs" >}}).
[The Outbox Pattern](/advanced/forwarder/) is a key pattern to know in event-driven applications.
We recommend checking the examples below to see how Watermill works in practice.
You can also try the [free hands-on training]({{< ref "/learn/quickstart/" >}}) to learn how to use Watermill in practice.
## Examples
Check out the [examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples) that will show you how to start using Watermill.
The recommended entry point is [Your first Watermill application](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/1-your-first-app).
It contains the entire environment in `docker-compose.yml`, including Go and Kafka, which you can run with one command.
After that, you can see the [Realtime feed](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/2-realtime-feed) example.
It uses more middlewares and contains two handlers.
For a different subscriber implementation (**HTTP**), see the [receiving-webhooks](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/receiving-webhooks) example.
It is a straightforward application that saves webhooks to Kafka.
You can find the complete list of examples in the [README](https://github.com/ThreeDotsLabs/watermill#examples).
## Support
If anything is not clear, feel free to use any of our [support channels]({{< ref "/support" >}}); we will be glad to help.

52
docs/content/learn/quickstart.md Normal file
View File

@@ -0,0 +1,52 @@
---
title: "Watermill Quickstart"
description: "Learn how to use Watermill in your project with a hands-on training."
draft: false
toc: false
weight: 20
hideBanner: true
seo:
title: "Watermill Quickstart - Hands-on Training"
description: "Learn how to use Watermill in your project with a hands-on training covering Publisher/Subscriber, Router, CQRS, Kafka, and more."
canonical: ""
noindex: false
---
{{< callout context="note" title="Just released 🎉" >}}
<p>Check our new, interactive quickstart that you can finish in your IDE.
Zero Docker setup, zero coding in browser.</p>
{{< /callout >}}
Watermill Quickstart is a hands-on training that will teach you how to use Watermill, a Go library for building event-driven applications, in your project.
Basic Go knowledge is all you need to get started.
You'll learn the basics of Watermill and a few more advanced concepts:
* The 3 core APIs: Publisher/Subscriber, Router, and CQRS.
* Working with Kafka.
* Topic topology, middleware, consumer groups.
* Message ordering.
* The Outbox pattern (with PostgreSQL).
* Switching the Pub/Sub (Redis).
<div class="text-center">
<a href="https://academy.threedots.tech/trainings/watermill-quickstart/start" class="btn btn-primary btn-lg my-3">Start the Watermill Quickstart</a>
</div>
<a href="https://academy.threedots.tech/trainings/watermill-quickstart/start" target="_blank">
<figure class="position-relative" style="width: 100%; margin-top:15px;">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960.000000 700.000000" style="width: 100%; height: auto;">
<!-- Clip path image -->
<defs>
<clipPath id="svgf77">
<rect y="0" x="5" width="735" height="680"></rect>
</clipPath>
</defs>
<image clip-path="url(#svgf77)" xlink:href="/img/exercise.png" height="85%" width="85%" style="transform: translate(104px, 84px); height: 492px;"></image>
<image xlink:href="/img/laptop.png" height="95%" width="95%" style="transform: translate(35px, 0px);"></image>
<image clip-path="url(#svgf77)" xlink:href="/img/ide.png" height="85%" width="85%" style="transform: translate(-103px, 320px); height: 381px;"></image>
</svg>
</figure>
</a>
</div>

4
docs/content/pubsubs/_index.md Normal file
View File

@@ -0,0 +1,4 @@
+++
title = "Supported Pub/Subs"
bref = "Watermill supports these Pub/Sub adapters out of the box:"
+++

77
docs/content/pubsubs/amqp.md Normal file
View File

@@ -0,0 +1,77 @@
+++
title = "RabbitMQ (AMQP)"
description = "The most widely deployed open source message broker"
date = 2019-07-06T22:30:00+02:00
bref = "The most widely deployed open source message broker"
weight = 100
+++
> RabbitMQ is the most widely deployed open source message broker.
We are providing Pub/Sub implementation based on [github.com/rabbitmq/amqp091-go](https://github.com/rabbitmq/amqp091-go) official library.
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/doc.go" first_line_contains="// AMQP" last_line_contains="package amqp" padding_after="0" %}}
You can find a fully functional example with RabbitMQ in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/amqp).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-amqp/v3
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes* | there are no literal consumer groups in AMQP, but we can achieve similar behaviour with `GenerateQueueNameTopicNameWithSuffix`. For more details please check [AMQP "Consumer Groups" section](#amqp-consumer-groups) |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | yes | yes, please check https://www.rabbitmq.com/semantics.html#ordering |
| Persistent | yes* | when using `NewDurablePubSubConfig` or `NewDurableQueueConfig` |
### Configuration
Our AMQP is shipped with some pre-created configurations:
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/config.go" first_line_contains="// NewDurablePubSubConfig" last_line_contains="type Config struct {" %}}
For detailed configuration description, please check [watermill-amqp/pkg/amqp/config.go](https://github.com/ThreeDotsLabs/watermill-amqp/tree/master/pkg/amqp/config.go)
#### TLS Config
TLS config can be passed to `Config.TLSConfig`.
#### Connecting
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="publisher, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="subscriber, err :=" last_line_contains="panic(err)" padding_after="1" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/subscriber.go" first_line_contains="// Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
Marshaler is responsible for mapping AMQP's messages to Watermill's messages.
Marshaller can be changed via the Configuration.
If you need to customize thing in `amqp.Delivery`, you can do it `PostprocessPublishing` function.
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/marshaler.go" first_line_contains="// Marshaler" last_line_contains="func (d DefaultMarshaler)" padding_after="0" %}}
### AMQP "Consumer Groups"
AMQP doesn't provide mechanism like Kafka's "consumer groups". You can still achieve similar behaviour with `GenerateQueueNameTopicNameWithSuffix` and `NewDurablePubSubConfig`.
{{% load-snippet-partial file="docs/snippets/amqp-consumer-groups/main.go" first_line_contains="func createSubscriber(" last_line_contains="go process(\"subscriber_2\", messages2)" %}}
In this example both `pubSub1` and `pubSub2` will receive some messages independently.
### AMQP `TopologyBuilder`
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/topology_builder.go" first_line_contains="// TopologyBuilder" last_line_contains="}" padding_after="0" %}}

250
docs/content/pubsubs/aws.md Normal file
View File

@@ -0,0 +1,250 @@
+++
title = "Amazon AWS SNS/SQS"
description = "AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple and scale microservices, distributed systems, and serverless applications."
date = 2024-10-19T15:30:00+02:00
bref = "AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple and scale microservices, distributed systems, and serverless applications."
weight = 10
+++
AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple
and scale microservices, distributed systems, and serverless applications.
Watermill provides a simple way to use AWS SQS and SNS with Go.
It handles all the AWS SDK internals and provides a simple API to publish and subscribe messages.
Official Documentation:
- [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html)
- [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html)
You can find a fully functional example with AWS SNS in the Watermill examples:
- [SNS](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/aws-sns)
- [SQS](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/aws-sqs)
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-aws
```
## SQS vs SNS
While both SQS and SNS are messaging services provided by AWS, they serve different purposes and are best suited for different scenarios in your Watermill applications.
### How SNS is connected with SQS
To use SNS as a Pub/Sub (to have multiple subscribers receiving the same message), you need to create an SNS topic and subscribe to SQS queues.
When a message is published to the SNS topic, it will be delivered to all subscribed SQS queues.
We implemented this logic in the `watermill-aws` package out of the box.
When you subscribe to an SNS topic, Watermill AWS creates an SQS queue and subscribes to it.
[![](https://mermaid.ink/img/pako:eNptkU1uwyAQRq-CWLWScwEW2TjbVnboru4Cw9hB5ccZoFIV5e7FxVRKUjaM5j0-jZgLlV4BZTTAOYGTcNBiRmEHR_JZBEYt9SJcJB0RgfBXTro0Gh1OgI_OijfrzS9a_mP0xchXnyABeXcfj1ZbU3gag0Q9AhaxqN1uv8-U1VGIhRDEDKHgjFahzwIHpyot0Hi_lKqtUueNIZPH-5ie77LSMnKEmNDd5rQFdehlblf2FJ7vwg9gIMLt2zV5w0ew_usPkwm9Jef1Y4qZx6cNtYBWaJW3dFnbA40nsDBQlksl8HOgg7tmT6To-beTlEVM0NC0KBHrRimbhAm5C0pHjy9l7b_bbyj6NJ824_oDU0CtBA?type=png)](https://mermaid.live/edit#pako:eNptkU1uwyAQRq-CWLWScwEW2TjbVnboru4Cw9hB5ccZoFIV5e7FxVRKUjaM5j0-jZgLlV4BZTTAOYGTcNBiRmEHR_JZBEYt9SJcJB0RgfBXTro0Gh1OgI_OijfrzS9a_mP0xchXnyABeXcfj1ZbU3gag0Q9AhaxqN1uv8-U1VGIhRDEDKHgjFahzwIHpyot0Hi_lKqtUueNIZPH-5ie77LSMnKEmNDd5rQFdehlblf2FJ7vwg9gIMLt2zV5w0ew_usPkwm9Jef1Y4qZx6cNtYBWaJW3dFnbA40nsDBQlksl8HOgg7tmT6To-beTlEVM0NC0KBHrRimbhAm5C0pHjy9l7b_bbyj6NJ824_oDU0CtBA)
We can say, that a single SQS queue acts as a consumer group or subscription in other Pub/Sub implementations.
The mechanism is detailed in [AWS documentation](https://docs.aws.amazon.com/sns/latest/dg/subscribe-sqs-queue-to-sns-topic.html).
### How to choose between SQS and SNS
#### SQS (Simple Queue Service)
- Use when you need a simple message queue with a single consumer.
- Great for task queues or background job processing.
- Supports exactly-once processing (with FIFO queues) and guaranteed order (mostly).
Example use case: Processing user uploads in the background.
[![](https://mermaid.ink/img/pako:eNplkT1uwzAMRq8icGoB5wIasjhrASdevdASHQvVj0NJAYogd69c2wmaaCL4HilI3w1U0AQSIl0yeUUHg2dG13lRzoScjDIT-iQagVG0x1Y0ubcmjsTvzoxX65gp07tRb7zNfVRs-nnNojW7_b4QuV0gHMWIZ4oLtiFMS1U_xGCtGAK_mIXtilJLcaKU2W_4OV1Qw0GV9sY-4ufL8gNZSvR_dt684hO5cH1gMXBw4vJ8M3kNFThih0aX773N7Q7SSI46kKXUyN8ddP5ePMwptD9egUycqYI8aUxbFCAHtLF0SZsU-GvJ6y-2Cjjk87ga91_dnpaW?type=png)](https://mermaid.live/edit#pako:eNplkT1uwzAMRq8icGoB5wIasjhrASdevdASHQvVj0NJAYogd69c2wmaaCL4HilI3w1U0AQSIl0yeUUHg2dG13lRzoScjDIT-iQagVG0x1Y0ubcmjsTvzoxX65gp07tRb7zNfVRs-nnNojW7_b4QuV0gHMWIZ4oLtiFMS1U_xGCtGAK_mIXtilJLcaKU2W_4OV1Qw0GV9sY-4ufL8gNZSvR_dt684hO5cH1gMXBw4vJ8M3kNFThih0aX773N7Q7SSI46kKXUyN8ddP5ePMwptD9egUycqYI8aUxbFCAHtLF0SZsU-GvJ6y-2Cjjk87ga91_dnpaW)
#### SNS (Simple Notification Service)
- Use when you need to broadcast messages to multiple subscribers.
- Perfect for implementing pub/sub patterns.
- Useful for event-driven architectures.
- Supports multiple types of subscribers (SQS, Lambda, HTTP/S, email, SMS, etc.).
Example use case: Notifying multiple services about a new user registration.
Our SNS implementation in Watermill automatically creates and manages SQS queues for each subscriber, simplifying the process of using SNS with multiple SQS queues.
Remember, you can use both in the same application where appropriate. For instance, you might use SNS to broadcast events and SQS to process specific tasks triggered by those events.
To learn how SNS and SQS work together, see the [How SNS is connected with SQS](#how-sns-is-connected-with-sqs) section.
## SQS
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ConsumerGroups | no | it's a queue, for consumer groups-like functionality use [SNS](#sns) |
| ExactlyOnceDelivery | no | [yes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) |
| GuaranteedOrder | yes\* | from [AWS Docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html): _"(...) due to the highly distributed architecture, more than one copy of a message might be delivered, and messages may occasionally arrive out of order. Despite this, standard queues make a best-effort attempt to maintain the order in which messages are sent."_ |
| Persistent | yes | |
### Required permissions
- `"sqs:ReceiveMessage"`
- `"sqs:DeleteMessage"`
- `"sqs:GetQueueUrl"`
- `"sqs:CreateQueue"`
- `"sqs:GetQueueAttributes"`
- `"sqs:SendMessage"`
- `"sqs:ChangeMessageVisibility"`
[todo - verify]
### SQS Configuration
{{% load-snippet-partial file="src-link/watermill-aws/sqs/config.go" first_line_contains="type SubscriberConfig struct " last_line_contains="type GenerateCreateQueueInputFunc" %}}
### Resolving Queue URL
In the Watermill model, we are normalizing the AWS queue url to `topic` used in the `Publish` and `Subscribe` methods.
To give you flexibility of what you want to use as a topic in Watermill, you can customize resolving the queue URL.
{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// QueueUrlResolver" last_line_contains="GenerateQueueUrlResolver" %}}
You can implement your own `QueueUrlResolver` or use one of the provided resolvers.
By default, `GetQueueUrlByNameUrlResolver` resolver is used:
{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// GetQueueUrlByNameUrlResolver " last_line_contains="NewGetQueueUrlByNameUrlResolver" %}}
There are two more resolvers available:
{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// GenerateQueueUrlResolver" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// TransparentUrlResolver" last_line_contains="}" %}}
### Using with SQS emulator
You may want to use [`goaws`](https://github.com/Admiral-Piett/goaws) or [`localstack`](https://hub.docker.com/r/localstack/localstack) for local development or testing.
You can override the endpoint using the `OptFns` option in the `SubscriberConfig` or `PublisherConfig`.
```go
package main
import (
amazonsqs "github.com/aws/aws-sdk-go-v2/service/sqs"
"github.com/ThreeDotsLabs/watermill-amazonsqs/sqs"
)
func main() {
// ...
sqsOpts := []func(*amazonsqs.Options){
amazonsqs.WithEndpointResolverV2(sqs.OverrideEndpointResolver{
Endpoint: transport.Endpoint{
URI: *lo.Must(url.Parse("http://localstack:4566")),
},
}),
}
sqsConfig := sqs.SubscriberConfig{
AWSConfig: cfg,
OptFns: sqsOpts,
}
sub, err := sqs.NewSubscriber(sqsConfig, logger)
if err != nil {
panic(fmt.Errorf("unable to create new subscriber: %w", err))
}
// ...
}
```
## SNS
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ConsumerGroups | yes | yes |
| ExactlyOnceDelivery | no | [yes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) |
| GuaranteedOrder | yes\* | from [AWS Docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html): _"(...) due to the highly distributed architecture, more than one copy of a message might be delivered, and messages may occasionally arrive out of order. Despite this, standard queues make a best-effort attempt to maintain the order in which messages are sent."_ |
| Persistent | yes | |
### Required permissions
- `sns:Subscribe`
- `sns:ConfirmSubscription`
- `sns:Receive`
- `sns:Unsubscribe`
and all permissions required for SQS:
- `sqs:ReceiveMessage`
- `sqs:DeleteMessage`
- `sqs:GetQueueUrl`
- `sqs:CreateQueue`
- `sqs:GetQueueAttributes`
- `sqs:SendMessage`
- `sqs:ChangeMessageVisibility`
- `sqs:SetQueueAttributes`
Additionally, if `sns.SubscriberConfig.DoNotSetQueueAccessPolicy` is not enabled, you should have the following:
- `sqs:SetQueueAttributes`
### SNS Configuration
{{% load-snippet-partial file="src-link/watermill-aws/sns/config.go" first_line_contains="type SubscriberConfig struct " last_line_contains="type GenerateSqsQueueNameFn" %}}
Additionally, because SNS Subscriber uses SQS queues as "subscriptions", you need to pass [SQS configuration](#sqs-configuration) as well.
### Resolving Queue URL
In the Watermill model, we normalise AWS Topic ARN to the `topic` used in the `Publish` and `Subscribe` methods.
{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// TopicResolver" last_line_contains="}" %}}
We are providing two out-of-the-box resolvers:
{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// TransparentTopicResolver" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// GenerateArnTopicResolver" last_line_contains="}" %}}
### Using with SNS emulator
You may want to use [`goaws`](https://github.com/Admiral-Piett/goaws) or [`localstack`](https://hub.docker.com/r/localstack/localstack) for local development or testing.
You can override the endpoint using the `OptFns` option in the `SubscriberConfig` or `PublisherConfig`.
```go
package main
import (
amazonsns "github.com/aws/aws-sdk-go-v2/service/sns"
"github.com/ThreeDotsLabs/watermill-amazonsns/sns"
)
func main() {
// ...
snsOpts := []func(*amazonsns.Options){
amazonsns.WithEndpointResolverV2(sns.OverrideEndpointResolver{
Endpoint: transport.Endpoint{
URI: *lo.Must(url.Parse("http://localstack:4566")),
},
}),
}
snsConfig := sns.SubscriberConfig{
AWSConfig: cfg,
OptFns: snsOpts,
}
sub, err := sns.NewSubscriber(snsConfig, sqsConfig, logger)
if err != nil {
panic(fmt.Errorf("unable to create new subscriber: %w", err))
}
// ...
}
```

65
docs/content/pubsubs/bolt.md Normal file
View File

@@ -0,0 +1,65 @@
+++
title = "Bolt Pub/Sub"
description = "A pure Go key/value store"
date = 2021-11-19T00:00:00+02:00
bref = "A pure Go key/value store"
weight = 20
+++
Bolt is a pure Go key/value store which provides a simple, fast, and reliable
database for projects that don't require a full database server such as
Postgres or MySQL.
Bolt backed Pub/Sub is good for simple applications which don't need a more
advanced Pub/Sub system with external dependencies or already use Bolt and
want to publish messages in transaction when saving other data.
Bolt documentation: https://github.com/etcd-io/bbolt
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-bolt
```
### Characteristics
| Feature | Implements | Note |
| ------------------- | ---------- | ---- |
| ConsumerGroups | no | |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | yes | |
### Configuration
{{% load-snippet-partial file="src-link/watermill-bolt/pkg/bolt/bolt.go" first_line_contains="type CommonConfig struct " last_line_equals="}" %}}
{{% load-snippet-partial file="src-link/watermill-bolt/pkg/bolt/bolt.go" first_line_contains="type PublisherConfig struct " last_line_equals="}" %}}
{{% load-snippet-partial file="src-link/watermill-bolt/pkg/bolt/bolt.go" first_line_contains="type SubscriberConfig struct " last_line_equals="}" %}}
#### Subscription name
To receive messages published to a topic, you must create a subscription to
that topic. Only messages published to the topic after the subscription is
created will be received by the subscriber.
A topic can have multiple subscriptions, but a given subscription belongs to
a single topic.
In Watermill, the subscription is created automatically during calling
`Subscribe()`. Subscription name is generated by calling the function set as
`SubscriberConfig.GenerateSubscriptionName`. By default, it is the topic name
with the string `_sub` appended to it.
#### Marshaler
Watermill's messages cannot be directly saved in Bolt which operates on byte
slices. Marshaller converts the messages to and from byte slices. The default
implementation marshals messages as JSON, a format which is human-readable
for easier debugging. The performance should be enough for most applications
unless a very large messages are used within your system. If that is the case
you may want to consider implementing a more efficient marshaler.
{{% load-snippet-partial file="src-link/watermill-bolt/pkg/bolt/marshaler.go" first_line_contains="// Marshaler" last_line_equals="}" %}}

78
docs/content/pubsubs/firestore.md Normal file
View File

@@ -0,0 +1,78 @@
+++
title = "Firestore Pub/Sub"
description = "A scalable document database from Google"
date = 2021-07-29T15:30:00+02:00
bref = "A scalable document database from Google"
weight = 30
+++
Cloud Firestore is a cloud-hosted, NoSQL database from Google.
This Pub/Sub comes with two publishers. To publish messages in a transaction
use the `TransactionalPublisher`. If you do not want to publish messages in
transaction use the normal `Publisher`.
Using Firestore as a Pub/Sub instead of using a dedicated Pub/Sub system can be
useful to publish messages in transaction while at the same time saving other
data in Firestore. Thanks to that the data and the messages can be consistently
persisted. If the messages and the data weren't being published transactionally
you could end up in situations where messages were emitted even though the data
wasn't saved or messages weren't emitted even though the data was saved. After
transactionally publishing messages in Firestore you can then subscribe to them
and relay them to a different Pub/Sub system.
Godoc: <https://pkg.go.dev/github.com/ThreeDotsLabs/watermill-firestore>
Firestore documentation: <https://firebase.google.com/docs/firestore/>
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-firestore
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes | |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | yes | |
### Configuration
#### Publisher configuration
{{% load-snippet-partial file="src-link/watermill-firestore/pkg/firestore/publisher.go" first_line_contains="type PublisherConfig struct {" last_line_equals="}" %}}
#### Subscriber configuration
{{% load-snippet-partial file="src-link/watermill-firestore/pkg/firestore/subscriber.go" first_line_contains="type SubscriberConfig struct {" last_line_equals="}" %}}
#### Subscription name
To receive messages published to a topic, you must create a subscription to
that topic. Only messages published to the topic after the subscription is
created will be received by the subscribers.
A topic can have multiple subscriptions, but a given subscription belongs to a
single topic.
In Watermill, the subscription is created automatically during calling
`Subscribe()`. Subscription name is generated by function passed to
`SubscriberConfig.GenerateSubscriptionName`. By default, it is just the topic
name with a suffix `_sub` appended to it.
If you want to consume messages from a topic with multiple subscribers
processing the incoming messages in a different way, you should use a custom
function to generate unique subscription names for each subscriber.
### Marshaler
Watermill's messages cannot be stored directly in Firestore. The marshaler is
responsible for converting them to a type which can be stored by Firestore.
The default implementation should be enough for most applications so it is
unlikely that you need to implement your own marshaler.
{{% load-snippet-partial file="src-link/watermill-firestore/pkg/firestore/marshaler.go" first_line_contains="// Marshaler" last_line_equals="}" padding_after="0" %}}

38
docs/content/pubsubs/gochannel.md Normal file
View File

@@ -0,0 +1,38 @@
+++
title = "Go Channel"
description = "A Pub/Sub implemented on Golang goroutines and channels"
date = 2019-07-06T22:30:00+02:00
bref = "A Pub/Sub implemented on Golang goroutines and channels"
weight = 40
+++
{{% load-snippet-partial file="src-link/pubsub/gochannel/pubsub.go" first_line_contains="// GoChannel" last_line_contains="type GoChannel struct {" %}}
You can find a fully functional example with Go Channels in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/go-channel).
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | no | |
| ExactlyOnceDelivery | yes | |
| GuaranteedOrder | yes | |
| Persistent | no| |
### Configuration
You can inject configuration via the constructor.
{{% load-snippet-partial file="src-link/pubsub/gochannel/pubsub.go" first_line_contains="func NewGoChannel" last_line_contains="logger:" %}}
### Publishing
{{% load-snippet-partial file="src-link/pubsub/gochannel/pubsub.go" first_line_contains="// Publish" last_line_contains="func (g *GoChannel) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/pubsub/gochannel/pubsub.go" first_line_contains="// Subscribe" last_line_contains="func (g *GoChannel) Subscribe" %}}
### Marshaler
No marshaling is needed when sending messages within the process.

85
docs/content/pubsubs/googlecloud.md Normal file
View File

@@ -0,0 +1,85 @@
+++
title = "Google Cloud Pub/Sub"
description = "The fully-managed real-time messaging service from Google"
date = 2019-07-06T22:30:00+02:00
bref = "The fully-managed real-time messaging service from Google"
weight = 50
+++
Cloud Pub/Sub brings the flexibility and reliability of enterprise message-oriented middleware to
the cloud.
At the same time, Cloud Pub/Sub is a scalable, durable event ingestion and delivery
system that serves as a foundation for modern stream analytics pipelines.
By providing many-to-many, asynchronous messaging that decouples senders and receivers,
it allows for secure and highly available communication among independently written applications.
Cloud Pub/Sub delivers low-latency, durable messaging that helps developers quickly integrate
systems hosted on the Google Cloud Platform and externally.
Official Documentation: [https://cloud.google.com/pubsub/docs/](https://cloud.google.com/pubsub/docs/overview)
You can find a fully functional example with Google Cloud Pub/Sub in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/googlecloud).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-googlecloud/v2
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes | multiple subscribers within the same Subscription name |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | yes* | maximum retention time is 7 days |
### Configuration
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/publisher.go" first_line_contains="type PublisherConfig struct " last_line_contains="func NewPublisher" %}}
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/subscriber.go" first_line_contains="type SubscriberConfig struct {" last_line_contains="func NewSubscriber(" %}}
#### Subscription name
To receive messages published to a topic, you must create a subscription to that topic.
Only messages published to the topic after the subscription is created are available to subscriber
applications.
The subscription connects the topic to a subscriber application that receives and processes
messages published to the topic.
A topic can have multiple subscriptions, but a given subscription belongs to a single topic.
In Watermill, the subscription is created automatically during calling `Subscribe()`.
Subscription name is generated by function passed to `SubscriberConfig.GenerateSubscriptionName`.
By default, it is just the topic name (`TopicSubscriptionName`).
When you want to consume messages from a topic with multiple subscribers, you should use
`TopicSubscriptionNameWithSuffix` or your custom function to generate the subscription name.
### Connecting
Watermill will connect to the instance of Google Cloud Pub/Sub indicated by the environment variables. For production setup, set the `GOOGLE_APPLICATION_CREDENTIALS` env, as described in [the official Google Cloud Pub/Sub docs](https://cloud.google.com/pubsub/docs/quickstart-client-libraries#pubsub-client-libraries-go). Note that you won't need to install the Cloud SDK, as Watermill will take care of the administrative tasks (creating topics/subscriptions) with the default settings and proper permissions.
For development, you can use a Docker image with the emulator and the `PUBSUB_EMULATOR_HOST` env ([check out the Getting Started guide]({{< ref "getting-started#subscribing_gcloud" >}})).
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="publisher, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="subscriber, err :=" last_line_contains="panic(err)" padding_after="1" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/subscriber.go" first_line_contains="// Subscribe " last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
Watermill's messages cannot be directly sent to Google Cloud Pub/Sub - they need to be marshaled. You can implement your marshaler or use the default implementation.
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/marshaler.go" first_line_contains="// Marshaler" last_line_contains="type DefaultMarshalerUnmarshaler " padding_after="0" %}}

74
docs/content/pubsubs/http.md Normal file
View File

@@ -0,0 +1,74 @@
+++
title = "HTTP"
description = "Call and listen to webhooks asynchronously"
date = 2019-07-06T22:30:00+02:00
bref = "Call and listen to webhooks asynchronously"
weight = 60
+++
The HTTP subscriber listens to HTTP requests (for example - webhooks) and outputs them as messages.
You can then post them to any Publisher. Here is an example with [sending HTTP messages to Kafka](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/real-world-examples/receiving-webhooks/main.go).
The HTTP publisher sends HTTP requests as specified in its configuration. Here is an example with [transforming Kafka messages into HTTP webhook requests](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/sending-webhooks).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-http/v2
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | no | |
| ExactlyOnceDelivery | yes | |
| GuaranteedOrder | yes | |
| Persistent | no| |
### Subscriber configuration
Subscriber configuration is done via the config struct passed to the constructor:
{{% load-snippet-partial file="src-link/watermill-http/pkg/http/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="}" %}}
You can use the `Router` config option to `SubscriberConfig` to pass your own `chi.Router` (see [chi](https://github.com/go-chi/chi)).
This may be helpful if you'd like to add your own HTTP handlers (e.g. a health check endpoint).
### Publisher configuration
Publisher configuration is done via the config struct passed to the constructor:
{{% load-snippet-partial file="src-link/watermill-http/pkg/http/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="}" %}}
How the message topic and body translate into the URL, method, headers, and payload of the HTTP request is highly configurable through the use of `MarshalMessageFunc`.
Use the provided `DefaultMarshalMessageFunc` to send POST requests to a specific url:
{{% load-snippet-partial file="src-link/watermill-http/pkg/http/publisher.go" first_line_contains="// MarshalMessageFunc" last_line_contains="return req, nil" padding_after="2" %}}
You can pass your own `http.Client` to execute the requests or use Golang's default client.
### Running
To run HTTP subscriber you need to run `StartHTTPServer()`. It needs to be run after `Subscribe()`.
When using with the router, you should wait for the router to start.
```go
<-r.Running()
httpSubscriber.StartHTTPServer()
```
### Subscribing
{{% load-snippet-partial file="src-link/watermill-http/pkg/http/subscriber.go" first_line_contains="// Subscribe adds" last_line_contains="func (s *Subscriber) Subscribe" %}}
#### Custom HTTP status codes
To specify a custom HTTP status code, which will returned as response, you can use following call during message handling:
```go
// msg is a *message.Message
http.SetResponseStatusCode(msg, http.StatusForbidden)
msg.Nack()
```

66
docs/content/pubsubs/io.md Normal file
View File

@@ -0,0 +1,66 @@
+++
title = "io.Writer/io.Reader"
description = "Pub/Sub implemented as Go stdlib's most loved interfaces"
date = 2019-07-06T22:30:00+02:00
bref = "Pub/Sub implemented as Go stdlib's most loved interfaces"
weight = 70
+++
This is an experimental Pub/Sub implementation that leverages the [standard library's](https://golang.org/pkg/io/) `io.Writer` and `io.Reader` interfaces as sources of Publisher and Subscriber, respectively.
Note that these aren't full-fledged Pub/Subs like Kafka, RabbitMQ, or the likes, but given the ubiquity of implementations of `Writer` and `Reader` they may come in handy, for uses like:
* Writing messages to file or stdout
* Subscribing for data on a file or stdin and packaging it as messages
* Interfacing with third-party libraries that implement `io.Writer` or `io.Reader`, like [github.com/colinmarc/hdfs](https://github.com/colinmarc/hdfs) or [github.com/mholt/archiver](https://github.com/mholt/archiver).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-io
```
### Characteristics
This is a very bare-bones implementation for now, so no extra features are supported. However, it is still sufficient for applications like a [CLI producer/consumer](https://github.com/ThreeDotsLabs/watermill/tree/master/tools/mill).
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | no | |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | no | |
### Configuration
The publisher configuration is relatively simple.
{{% load-snippet-partial file="src-link/watermill-io/pkg/io/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="// Publisher" %}}
The subscriber may work in two modes – either perform buffered reads of constant size from the io.Reader, or split the byte stream into messages using a delimiter byte.
The reading will continue even if the reads come up empty, but they will not be sent out as messages. The time to wait after an empty read is configured through the `PollInterval` parameter. As soon as a non-empty input is read, it will be packaged as a message and sent out.
{{% load-snippet-partial file="src-link/watermill-io/pkg/io/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="// Subscriber" %}}
The continuous reading may be used, for example, to emulate the behaviour of a `tail -f` command, like in this snippet:
{{% load-snippet-partial file="docs/snippets/tail-log-file/main.go" first_line_contains="// this will" last_line_contains="return false" padding_after="1" %}}
### Marshaling/Unmarshaling
The MarshalFunc is an important part of `io.Publisher`, because it fully controls the format in the underlying `io.Writer` will obtain the messages.
Correspondingly, the UnmarshalFunc regulates how the bytes read by the `io.Reader` will be interpreted as Watermill messages.
{{% load-snippet-partial file="src-link/watermill-io/pkg/io/marshal.go" first_line_contains="// MarshalMessageFunc" last_line_contains="// PayloadMarshalFunc" %}}
{{% load-snippet-partial file="src-link/watermill-io/pkg/io/marshal.go" first_line_contains="// UnmarshalMessageFunc" last_line_contains="// PayloadUnmarshalFunc" %}}
The package comes with some predefined marshal and unmarshal functions, but you might want to write your own marshaler/unmarshaler to work with the specific implementation of `io.Writer/io.Reader` that you are working with.
### Topic
For the Publisher/Subscriber implementation itself, the topic has no meaning. It is difficult to interpret the meaning of topic in the general context of `io.Writer` and `io.Reader` interfaces.
However, the topic is passed as a parameter to the marshal/unmarshal functions, so the adaptations to particular `Writer/Reader` implementation may take it into account.

85
docs/content/pubsubs/kafka.md Normal file
View File

@@ -0,0 +1,85 @@
+++
title = "Kafka"
description = "A distributed streaming platform from Apache"
date = 2019-07-06T22:30:00+02:00
bref = "A distributed streaming platform from Apache"
weight = 80
+++
Apache Kafka is one of the most popular Pub/Subs. We are providing Pub/Sub implementation based on [IBM Sarama](https://github.com/IBM/sarama).
You can find a fully functional example with Kafka in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/kafka).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-kafka/v3
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes | |
| ExactlyOnceDelivery | no | in theory can be achieved with [Transactions](https://www.confluent.io/blog/transactions-apache-kafka/), currently no support for any Golang client |
| GuaranteedOrder | yes | require [partition key usage](#partitioning) |
| Persistent | yes| |
### Configuration
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="// Subscribe" %}}
#### Passing custom `Sarama` config
You can pass [custom config](https://github.com/Shopify/sarama/blob/master/config.go#L20) parameters via `overwriteSaramaConfig *sarama.Config` in `NewSubscriber` and `NewPublisher`.
When `nil` is passed, default config is used (`DefaultSaramaSubscriberConfig`).
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// DefaultSaramaSubscriberConfig" last_line_contains="return config" padding_after="1" %}}
### Connecting
#### Publisher
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/publisher.go" first_line_contains="// NewPublisher" last_line_contains="(*Publisher, error)" padding_after="0" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="publisher, err := kafka.NewPublisher" last_line_contains="panic(err)" padding_after="1" %}}
#### Subscriber
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="(*Subscriber, error)" padding_after="0" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="saramaSubscriberConfig :=" last_line_contains="panic(err)" padding_after="1" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
Watermill's messages cannot be directly sent to Kafka - they need to be marshaled. You can implement your marshaler or use default implementation.
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/marshaler.go" first_line_contains="// Marshaler" last_line_contains="func (DefaultMarshaler)" padding_after="0" %}}
### Partitioning
Our Publisher has support for the partitioning mechanism.
It can be done with special Marshaler implementation:
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/marshaler.go" first_line_contains="type kafkaJsonWithPartitioning" last_line_contains="func (j kafkaJsonWithPartitioning) Marshal" padding_after="0" %}}
When using, you need to pass your function to generate partition key.
It's a good idea to pass this partition key with metadata to not unmarshal entire message.
```go
marshaler := kafka.NewWithPartitioningMarshaler(func(topic string, msg *message.Message) (string, error) {
return msg.Metadata.Get("partition"), nil
})
```
Please note that in the example above, if the `partition` key is missing from the message metadata, an empty string `""` will be used as the partitioning key. This will cause all such messages to be routed to the same partition, which may not be the desired behavior and could lead to uneven load distribution.

85
docs/content/pubsubs/nats.md Normal file
View File

@@ -0,0 +1,85 @@
+++
title = "NATS Jetstream"
description = "A simple, secure and high performance open source messaging system"
date = 2022-02-03T10:30:00+05:00
bref = "A simple, secure and high performance open source messaging system"
weight = 90
+++
NATS Jetstream is a data streaming system powered by NATS, and written in the Go programming language.
As of v2.1.0, the `pkg/jetstream` implementation, based on the
[nats.go Jetstream package](https://github.com/nats-io/nats.go/tree/main/jetstream) is considered stable.
For production use it is recommended to use the pubsub implementations in `pkg/nats` with Jetstream enabled.
You can find a fully functional example with NATS JetStream in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/nats-jetstream).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-nats/v2
```
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|-----------------------------------------------------------------------------------------------------------------------|
| ConsumerGroups | yes | you need to set `QueueGroupPrefix` name or provide an optional calculator |
| ExactlyOnceDelivery | yes | you need to ensure `AckAsync` has default false value and set `TrackMsgId` to true on the Jetstream configuration |
| GuaranteedOrder | no | [with the redelivery feature, order can't be guaranteed](https://github.com/nats-io/nats-streaming-server/issues/187) |
| Persistent | yes | |
### Configuration
Configuration is done through PublisherConfig and SubscriberConfig types. These share a common JetStreamConfig. To use the experimental nats-core support, set Disabled=true.
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/jetstream.go" first_line_contains="// JetStreamConfig contains" last_line_contains="type DurableCalculator =" %}}
PublisherConfig:
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="type Publisher struct {" %}}
SubscriberConfig:
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="type Subscriber struct" %}}
### Connecting
By default, the NATS client will try to connect to `localhost:4222`. If you are using different hostname or port you should specify using the URL property of `SubscriberConfig` and `PublisherConfig`.
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/publisher.go" first_line_contains="// NewPublisher" last_line_contains="func NewPublisher" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-jetstream/main.go" first_line_contains="publisher, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="func NewSubscriber" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-jetstream/main.go" first_line_contains="subscriber, err :=" last_line_contains="panic(err)" padding_after="1" %}}
You can also use `NewSubscriberWithNatsConn` and `NewPublisherWithNatsConn` to use a custom `*nats.Conn`.
### Publishing
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/publisher.go" first_line_contains="// Publish publishes" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/subscriber.go" first_line_contains="// Subscribe " last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
NATS provides a header passing mechanism that allows conveying the watermill message details as metadata. This is done by default with only the binary payload sent in the message body. The header `_watermill_message_uuid` is reserved.
Other builtin marshalers are based on Golang's [`gob`](https://golang.org/pkg/encoding/gob/) and [`json`](https://golang.org/packages/encoding/json) packages.
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/marshaler.go" first_line_contains="type Marshaler " last_line_contains="func defaultNatsMsg" padding_after="0" %}}
When you have your own format of the messages, you can implement your own Marshaler, which will serialize messages in your format. An example protobuf implementation with tests and benchmarks can be found [here](https://github.com/ThreeDotsLabs/watermill-nats/tree/master/_examples/marshalers/protobuf/)
When needed, you can bypass both [UUID]({{< ref "message#message" >}}) and [Metadata]({{< ref "message#message" >}}) and send just a `message.Payload`,
but some standard [middlewares]({{< ref "messages-router#middleware" >}}) may not work.
## Core NATS
This package also includes limited support for connecting to [Core NATS](https://docs.nats.io/nats-concepts/core-nats). While Core NATS does not support many of the streaming features needed for a perfect fit with watermill and most acks end up implemented as no-ops, in environments with a mix of jetstream and Core NATS messaging in play it can be nice to use watermill consistently on the application side.

64
docs/content/pubsubs/redisstream.md Normal file
View File

@@ -0,0 +1,64 @@
+++
title = "Redis Stream"
description = "A fast, open source, in-memory, key-value data store"
date = 2023-02-01T22:30:00+08:00
bref = "A fast, open source, in-memory, key-value data store"
weight = 110
+++
Redis is the open source, in-memory data store used by millions of developers. Redis stream is a data structure that acts like an append-only log in Redis. We are providing Pub/Sub implementation based on [redis/go-redis](https://github.com/redis/go-redis).
You can find a fully functional example with Redis Stream in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/redisstream).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-redisstream
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes | |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | yes | |
| FanOut | yes | use XREAD to fan out messages when there is no consumer group |
### Configuration
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="// Publish publishes message to redis stream" %}}
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="func (s *Subscriber) Subscribe" %}}
#### Passing `redis.UniversalClient`
You need to configure and pass your own go-redis client via `Client redis.UniversalClient` in `NewSubscriber` and `NewPublisher`. The client can be either `redis.Client` or `redis.ClusterClient`.
#### Publisher
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/publisher.go" first_line_contains="// NewPublisher" last_line_contains="(*Publisher, error)" padding_after="0" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/redisstream/main.go" first_line_contains="pubClient := redis.NewClient" last_line_contains="panic(err)" padding_after="1" %}}
#### Subscriber
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="(*Subscriber, error)" padding_after="0" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/redisstream/main.go" first_line_contains="subClient := redis.NewClient" last_line_contains="panic(err)" padding_after="1" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/subscriber.go" first_line_contains="func (s *Subscriber) Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
Watermill's messages cannot be directly sent to Redis - they need to be marshaled. You can implement your marshaler or use default implementation. The default implementation uses [MessagePack](https://msgpack.org/index.html) for efficient serialization.
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/marshaller.go" first_line_contains="const UUIDHeaderKey" last_line_contains="type DefaultMarshallerUnmarshaller" padding_after="0" %}}

144
docs/content/pubsubs/sql.md Normal file
View File

@@ -0,0 +1,144 @@
+++
title = "SQL (PostgreSQL, MySQL)"
description = "Pub/Sub based on MySQL or PostgreSQL."
date = 2019-07-06T22:30:00+02:00
bref = "Pub/Sub based on MySQL or PostgreSQL."
weight = 120
+++
SQL Pub/Sub executes queries on any SQL database, using it like a messaging system. At the moment, **MySQL** and **PostgreSQL** are supported.
It be useful for projects that are not using any specialized message queue at the moment, but have access to a SQL database.
If you are looking for SQLite Pub/Sub, check out the [SQLite Pub/Sub](/pubsubs/sqlite/) documentation.
The SQL subscriber runs a `SELECT` query within short periods, remembering the position of the last record. If it finds
any new records, they are returned. One handy use case is consuming events from a database table, that can be later published
on some kind of message queue.
The SQL publisher simply inserts consumed messages into the chosen table. A common approach would be to use it as a persistent
log of events that were published on a queue with short message expiration time.
SQL Pub/Sub is also a good choice for implementing Outbox pattern with [Forwarder](/docs/forwarder/) component.
See also the [SQL example](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/sql).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-sql/v4
```
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|-------------------------------------------------------------------------------|
| ConsumerGroups | yes | See `ConsumerGroup` in `SubscriberConfig` (not supported by the queue schema) |
| ExactlyOnceDelivery | yes |
| GuaranteedOrder | yes | |
| Persistent | yes | |
### Schema
SQL Pub/Sub uses user-defined schema to handle select and insert queries. You need to implement `SchemaAdapter` and pass
it to `SubscriberConfig` or `PublisherConfig`.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/schema_adapter_mysql.go" first_line_contains="// DefaultMySQLSchema" last_line_contains="type DefaultMySQLSchema" %}}
There is a default schema provided for each supported engine (`DefaultMySQLSchema` and `DefaultPostgreSQLSchema`).
It supports the most common use case (storing events in a table). You can base your schema on one of these, extending only chosen methods.
#### Extending schema
Consider an example project, where you're fine with using the default schema, but would like to use `BINARY(16)` for storing
the `uuid` column, instead of `VARCHAR(36)`. In that case, you have to define two methods:
* `SchemaInitializingQueries` that creates the table.
* `UnmarshalMessage` method that produces a `Message` from the database record.
Note that you don't have to use the initialization queries provided by Watermill. They will be run only if you set the
`InitializeSchema` field to `true` in the config. Otherwise, you can use your own solution for database migrations.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/schema_adapter_mysql.go" first_line_contains="// DefaultMySQLSchema" last_line_contains="type DefaultMySQLSchema" %}}
### Configuration
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="}" %}}
## Publishing
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/publisher.go" first_line_contains="func NewPublisher" last_line_contains="func NewPublisher" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="publisher, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/publisher.go" first_line_contains="// Publish " last_line_contains="func (p *Publisher) Publish" %}}
### Transactions
If you need to publish messages within a database transaction, you have to pass a `*sql.Tx` in the `NewPublisher`
constructor. You have to create one publisher for each transaction.
Example:
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events/main.go" first_line_contains="func simulateEvents" last_line_contains="return pub.Publish(" padding_after="3" %}}
## Subscribing
To create a subscriber, you need to pass not only proper schema adapter, but also an offsets adapter.
* For MySQL schema use `DefaultMySQLOffsetsAdapter`
* For PostgreSQL schema use `DefaultPostgreSQLOffsetsAdapter`
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/subscriber.go" first_line_contains="func NewSubscriber" last_line_contains="func NewSubscriber" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="subscriber, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/subscriber.go" first_line_contains="func (s *Subscriber) Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
## Offsets Adapter
The logic for storing offsets of messages is provided by the `OffsetsAdapter`. If your schema uses auto-incremented integer as the row ID,
it should work out of the box with default offset adapters.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/offsets_adapter.go" first_line_contains="type OffsetsAdapter" %}}
## Queue
Instead of the default Pub/Sub schema, you can use the *queue* schema and offsets adapters.
It's a simpler schema that doesn't support consumer groups.
However, it has other advantages.
It lets you specify a custom `WHERE` clause for getting the messages.
You can use it to filter messages by some condition in the payload or in the metadata.
Additionally, you can choose to delete messages from the table after they are acknowledged.
Thanks to this, the table doesn't grow in size with time.
This schema is supported by both PostgreSQL and MySQL.
The example below is based on PostgreSQL, but the same approach can be used with MySQL.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/queue_schema_adapter_postgresql.go" first_line_contains="// PostgreSQLQueueSchema" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/queue_offsets_adapter_postgresql.go" first_line_contains="// PostgreSQLQueueOffsetsAdapter" last_line_contains="}" %}}
## Caveats
### Using last processed transaction ID in PostgreSQL to ensure no messages are lost
In some cases, PostgreSQL `SERIAL` is not incremental.
The `SERIAL` value is generated while the transaction is in progress, not when it is committed.
If transactions are committed in a different order than they were started, message offsets based on `SERIAL` values will not be incremental.
To keep storing acknowledgment information efficient, Watermill keeps only the last message's acknowledgment information.
To ensure no messages are missed when a message order is not kept, Watermill also uses the transaction ID to ensure no message is lost.
For more details, see [Watermill#311](https://github.com/ThreeDotsLabs/watermill/issues/311).
It is important to note that very long-running transactions may result in delayed message delivery.
For instance, if a transaction is running for an hour, no messages will be delivered until the transaction is committed.
While we do not recommend the use of such long transactions, **if they are necessary, we advise the use of the [Queue schema adapter](#queue), which does not depend on the transaction ID.**
You have nothing to worry about if you don't have such long transactions.
If you are migrating your data to a new database, you may need to set `last_processed_transaction_id` in your offsets table.

135
docs/content/pubsubs/sqlite.md Normal file
View File

@@ -0,0 +1,135 @@
+++
title = "SQLite"
description = "A lightweight, file-based SQL database engine"
date = 2025-05-08T11:30:00+02:00
bref = "A lightweight, file-based SQL database engine"
weight = 121
+++
**Beta Version Warning: this Pub/Sub is stable, but it has not been widely tested in production environments. It may be sensitive to certain edge cases and combinations of configuration parameters.**
SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured SQL database engine. Our SQLite Pub/Sub implementation provides two **CGO-free** driver variants optimized for different use cases. Both drivers use pure Go implementations of SQLite, enabling cross-compilation and avoiding CGO dependencies while maintaining full SQLite functionality.
SQLite Pub/Subs provide the easiest way to publish and process events durably, since you do not have to set up or manage a separate database. The database is just a file on disk. Some cloud compute providers offer distributed SQLite clusters, which can provide both durability and unmatched read performance. Tuned SQLite is [~35% faster](https://sqlite.org/fasterthanfs.html) than the Linux file system.
You can find a fully functional example with SQLite in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/sqlite).
## ModernC vs ZombieZen Driver
The vanilla **ModernC driver** is compatible with the Golang standard library SQL package and works without CGO. It has fewer dependencies than the ZombieZen variant and uses the `modernc.org/sqlite` pure Go SQLite implementation. Most users should pick this driver for full compatibility with the standard library.
The advanced **ZombieZen driver** abandons the standard Golang library SQL conventions in favor of [the more orthogonal API and higher performance potential](https://crawshaw.io/blog/go-and-sqlite). Under the hood, it also uses the ModernC SQLite3 implementation and does not need CGO. Advanced SQLite users might prefer this driver for its performance benefits. It is about **6 times faster** than the ModernC variant. It is currently more stable due to lower level control. It is faster than even the CGO SQLite variants on standard library interfaces, and with some tuning should become the absolute speed champion of persistent message brokers over time.
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|---------------------------------------------------|
| ConsumerGroups | yes | See `ConsumerGroupMatcher` in `SubscriberOptions` |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | yes | |
| Persistent | yes | |
Like the [BoltDB](../bolt/) implementation, the SQLite drivers are imbeddable into your application. They do not require any additional infrastructure other than a mounted persistent disk. Their advantage over the BoltDB Pub/Sub is supporting consumer groups and guaranteed order.
## Vanilla ModernC Driver
### Installation
```bash
go get github.com/ThreeDotsLabs/watermill-sqlite/wmsqlitemodernc@latest
```
### Usage
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/main.go" first_line_contains="import (" last_line_contains="}" padding_after="1" %}}
### Configuration
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/publisher.go" first_line_contains="type PublisherOptions struct" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/subscriber.go" first_line_contains="type SubscriberOptions struct" last_line_contains="}" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/publisher.go" first_line_contains="// NewPublisher" last_line_contains="func NewPublisher" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/publisher.go" first_line_contains="// Publish " last_line_contains="func (p *publisher) Publish" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/main.go" first_line_contains="publisher, err := wmsqlitemodernc.NewPublisher(" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/main.go" first_line_contains="func publishMessages(" last_line_contains="panic(err)" padding_after="1" %}}
#### Publishing in transaction
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/transaction.go" first_line_contains="import (" padding_after="1" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="func NewSubscriber" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/main.go" first_line_contains="subscriber, err := wmsqlitemodernc.NewSubscriber(" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/subscriber.go" first_line_contains="// Subscribe " last_line_contains="func (s *subscriber) Subscribe" %}}
## Advanced ZombieZen Driver
### Installation
```bash
go get -u github.com/ThreeDotsLabs/watermill-sqlite/wmsqlitezombiezen@latest
```
### Usage
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/main.go" first_line_contains="import (" last_line_contains="}" padding_after="1" %}}
### Configuration
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/publisher.go" first_line_contains="type PublisherOptions struct" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/subscriber.go" first_line_contains="type SubscriberOptions struct" last_line_contains="}" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/publisher.go" first_line_contains="// NewPublisher" last_line_contains="func NewPublisher" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/publisher.go" first_line_contains="// Publish " last_line_contains="func (p *publisher) Publish" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/main.go" first_line_contains="publisher, err := wmsqlitezombiezen.NewPublisher(" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/main.go" first_line_contains="func publishMessages(" last_line_contains="panic(err)" padding_after="1" %}}
#### Publishing in transaction
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/transaction.go" first_line_contains="import (" padding_after="1" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="func NewSubscriber" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/main.go" first_line_contains="subscriber, err := wmsqlitezombiezen.NewSubscriber(" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/subscriber.go" first_line_contains="// Subscribe " last_line_contains="func (s *subscriber) Subscribe" %}}
## Marshaler
Watermill's messages are stored in SQLite using JSON serialization. Both drivers use the same marshaling approach - messages are automatically marshaled to and from JSON format when publishing and subscribing.
The default marshaler handles:
- Message payload (stored as JSON blob)
- Message metadata (stored as JSON object)
- Message UUID (stored as TEXT)
- Timestamps for ordering and consumer group management
Both drivers automatically handle message marshaling and unmarshaling, so no custom marshaler configuration is typically required.
## Caveats
SQLite3 does not support querying `FOR UPDATE`, which is used for row locking when subscribers in the same consumer group read an event batch in official Watermill SQL PubSub implementations. Current architectural decision is to lock a consumer group offset using `unixepoch()+lockTimeout` time stamp. While one consumed message is processing per group, the offset lock time is extended by `lockTimeout` periodically by `time.Ticker`. If the subscriber is unable to finish the consumer group batch, other subscribers will take over the lock as soon as the grace period runs out. A time lock fulfills the role of a traditional database network timeout that terminates transactions when its client disconnects.
All the normal SQLite limitations apply to Watermill. The connections are file handles. Create new connections for concurrent processing. If you must share a connection, protect it with a mutual exclusion lock. If you are writing within a transaction, create a connection for that transaction only.

14
docs/content/support.md Normal file
View File

@@ -0,0 +1,14 @@
+++
title = "Support"
description = ""
+++
### Community Support
Join us on the `#watermill` channel on the [Three Dots Labs discord](https://discord.gg/QV6VFg4YQE).
### Professional Support
For enterprise support, please contact us by e-mail: contact@threedotslabs.com
You can also use the [contact form on our website](https://threedots.tech/contact/?utm_source=watermill-docs).