upstream/upstream-watermill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

snapshot of https://github.com/ThreeDotsLabs/watermill.git (master, 2026-09-15; blobs > 131072B omitted)
Some checks failed
master / ci (push) Failing after 0s
Details

Browse Source
This commit is contained in:
upstream-mirror
2026-09-15 21:27:46 +05:00
commit ce933d1277
538 changed files with 46504 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/content/pubsubs/_index.md Normal file
View File

@@ -0,0 +1,4 @@
+++
title = "Supported Pub/Subs"
bref = "Watermill supports these Pub/Sub adapters out of the box:"
+++

77
docs/content/pubsubs/amqp.md Normal file
View File

@@ -0,0 +1,77 @@
+++
title = "RabbitMQ (AMQP)"
description = "The most widely deployed open source message broker"
date = 2019-07-06T22:30:00+02:00
bref = "The most widely deployed open source message broker"
weight = 100
+++
> RabbitMQ is the most widely deployed open source message broker.
We are providing Pub/Sub implementation based on [github.com/rabbitmq/amqp091-go](https://github.com/rabbitmq/amqp091-go) official library.
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/doc.go" first_line_contains="// AMQP" last_line_contains="package amqp" padding_after="0" %}}
You can find a fully functional example with RabbitMQ in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/amqp).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-amqp/v3
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes* | there are no literal consumer groups in AMQP, but we can achieve similar behaviour with `GenerateQueueNameTopicNameWithSuffix`. For more details please check [AMQP "Consumer Groups" section](#amqp-consumer-groups) |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | yes | yes, please check https://www.rabbitmq.com/semantics.html#ordering |
| Persistent | yes* | when using `NewDurablePubSubConfig` or `NewDurableQueueConfig` |
### Configuration
Our AMQP is shipped with some pre-created configurations:
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/config.go" first_line_contains="// NewDurablePubSubConfig" last_line_contains="type Config struct {" %}}
For detailed configuration description, please check [watermill-amqp/pkg/amqp/config.go](https://github.com/ThreeDotsLabs/watermill-amqp/tree/master/pkg/amqp/config.go)
#### TLS Config
TLS config can be passed to `Config.TLSConfig`.
#### Connecting
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="publisher, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="subscriber, err :=" last_line_contains="panic(err)" padding_after="1" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/subscriber.go" first_line_contains="// Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
Marshaler is responsible for mapping AMQP's messages to Watermill's messages.
Marshaller can be changed via the Configuration.
If you need to customize thing in `amqp.Delivery`, you can do it `PostprocessPublishing` function.
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/marshaler.go" first_line_contains="// Marshaler" last_line_contains="func (d DefaultMarshaler)" padding_after="0" %}}
### AMQP "Consumer Groups"
AMQP doesn't provide mechanism like Kafka's "consumer groups". You can still achieve similar behaviour with `GenerateQueueNameTopicNameWithSuffix` and `NewDurablePubSubConfig`.
{{% load-snippet-partial file="docs/snippets/amqp-consumer-groups/main.go" first_line_contains="func createSubscriber(" last_line_contains="go process(\"subscriber_2\", messages2)" %}}
In this example both `pubSub1` and `pubSub2` will receive some messages independently.
### AMQP `TopologyBuilder`
{{% load-snippet-partial file="src-link/watermill-amqp/pkg/amqp/topology_builder.go" first_line_contains="// TopologyBuilder" last_line_contains="}" padding_after="0" %}}

250
docs/content/pubsubs/aws.md Normal file
View File

@@ -0,0 +1,250 @@
+++
title = "Amazon AWS SNS/SQS"
description = "AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple and scale microservices, distributed systems, and serverless applications."
date = 2024-10-19T15:30:00+02:00
bref = "AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple and scale microservices, distributed systems, and serverless applications."
weight = 10
+++
AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple
and scale microservices, distributed systems, and serverless applications.
Watermill provides a simple way to use AWS SQS and SNS with Go.
It handles all the AWS SDK internals and provides a simple API to publish and subscribe messages.
Official Documentation:
- [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html)
- [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html)
You can find a fully functional example with AWS SNS in the Watermill examples:
- [SNS](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/aws-sns)
- [SQS](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/aws-sqs)
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-aws
```
## SQS vs SNS
While both SQS and SNS are messaging services provided by AWS, they serve different purposes and are best suited for different scenarios in your Watermill applications.
### How SNS is connected with SQS
To use SNS as a Pub/Sub (to have multiple subscribers receiving the same message), you need to create an SNS topic and subscribe to SQS queues.
When a message is published to the SNS topic, it will be delivered to all subscribed SQS queues.
We implemented this logic in the `watermill-aws` package out of the box.
When you subscribe to an SNS topic, Watermill AWS creates an SQS queue and subscribes to it.
[![](https://mermaid.ink/img/pako:eNptkU1uwyAQRq-CWLWScwEW2TjbVnboru4Cw9hB5ccZoFIV5e7FxVRKUjaM5j0-jZgLlV4BZTTAOYGTcNBiRmEHR_JZBEYt9SJcJB0RgfBXTro0Gh1OgI_OijfrzS9a_mP0xchXnyABeXcfj1ZbU3gag0Q9AhaxqN1uv8-U1VGIhRDEDKHgjFahzwIHpyot0Hi_lKqtUueNIZPH-5ie77LSMnKEmNDd5rQFdehlblf2FJ7vwg9gIMLt2zV5w0ew_usPkwm9Jef1Y4qZx6cNtYBWaJW3dFnbA40nsDBQlksl8HOgg7tmT6To-beTlEVM0NC0KBHrRimbhAm5C0pHjy9l7b_bbyj6NJ824_oDU0CtBA?type=png)](https://mermaid.live/edit#pako:eNptkU1uwyAQRq-CWLWScwEW2TjbVnboru4Cw9hB5ccZoFIV5e7FxVRKUjaM5j0-jZgLlV4BZTTAOYGTcNBiRmEHR_JZBEYt9SJcJB0RgfBXTro0Gh1OgI_OijfrzS9a_mP0xchXnyABeXcfj1ZbU3gag0Q9AhaxqN1uv8-U1VGIhRDEDKHgjFahzwIHpyot0Hi_lKqtUueNIZPH-5ie77LSMnKEmNDd5rQFdehlblf2FJ7vwg9gIMLt2zV5w0ew_usPkwm9Jef1Y4qZx6cNtYBWaJW3dFnbA40nsDBQlksl8HOgg7tmT6To-beTlEVM0NC0KBHrRimbhAm5C0pHjy9l7b_bbyj6NJ824_oDU0CtBA)
We can say, that a single SQS queue acts as a consumer group or subscription in other Pub/Sub implementations.
The mechanism is detailed in [AWS documentation](https://docs.aws.amazon.com/sns/latest/dg/subscribe-sqs-queue-to-sns-topic.html).
### How to choose between SQS and SNS
#### SQS (Simple Queue Service)
- Use when you need a simple message queue with a single consumer.
- Great for task queues or background job processing.
- Supports exactly-once processing (with FIFO queues) and guaranteed order (mostly).
Example use case: Processing user uploads in the background.
[![](https://mermaid.ink/img/pako:eNplkT1uwzAMRq8icGoB5wIasjhrASdevdASHQvVj0NJAYogd69c2wmaaCL4HilI3w1U0AQSIl0yeUUHg2dG13lRzoScjDIT-iQagVG0x1Y0ubcmjsTvzoxX65gp07tRb7zNfVRs-nnNojW7_b4QuV0gHMWIZ4oLtiFMS1U_xGCtGAK_mIXtilJLcaKU2W_4OV1Qw0GV9sY-4ufL8gNZSvR_dt684hO5cH1gMXBw4vJ8M3kNFThih0aX773N7Q7SSI46kKXUyN8ddP5ePMwptD9egUycqYI8aUxbFCAHtLF0SZsU-GvJ6y-2Cjjk87ga91_dnpaW?type=png)](https://mermaid.live/edit#pako:eNplkT1uwzAMRq8icGoB5wIasjhrASdevdASHQvVj0NJAYogd69c2wmaaCL4HilI3w1U0AQSIl0yeUUHg2dG13lRzoScjDIT-iQagVG0x1Y0ubcmjsTvzoxX65gp07tRb7zNfVRs-nnNojW7_b4QuV0gHMWIZ4oLtiFMS1U_xGCtGAK_mIXtilJLcaKU2W_4OV1Qw0GV9sY-4ufL8gNZSvR_dt684hO5cH1gMXBw4vJ8M3kNFThih0aX773N7Q7SSI46kKXUyN8ddP5ePMwptD9egUycqYI8aUxbFCAHtLF0SZsU-GvJ6y-2Cjjk87ga91_dnpaW)
#### SNS (Simple Notification Service)
- Use when you need to broadcast messages to multiple subscribers.
- Perfect for implementing pub/sub patterns.
- Useful for event-driven architectures.
- Supports multiple types of subscribers (SQS, Lambda, HTTP/S, email, SMS, etc.).
Example use case: Notifying multiple services about a new user registration.
Our SNS implementation in Watermill automatically creates and manages SQS queues for each subscriber, simplifying the process of using SNS with multiple SQS queues.
Remember, you can use both in the same application where appropriate. For instance, you might use SNS to broadcast events and SQS to process specific tasks triggered by those events.
To learn how SNS and SQS work together, see the [How SNS is connected with SQS](#how-sns-is-connected-with-sqs) section.
## SQS
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ConsumerGroups | no | it's a queue, for consumer groups-like functionality use [SNS](#sns) |
| ExactlyOnceDelivery | no | [yes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) |
| GuaranteedOrder | yes\* | from [AWS Docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html): _"(...) due to the highly distributed architecture, more than one copy of a message might be delivered, and messages may occasionally arrive out of order. Despite this, standard queues make a best-effort attempt to maintain the order in which messages are sent."_ |
| Persistent | yes | |
### Required permissions
- `"sqs:ReceiveMessage"`
- `"sqs:DeleteMessage"`
- `"sqs:GetQueueUrl"`
- `"sqs:CreateQueue"`
- `"sqs:GetQueueAttributes"`
- `"sqs:SendMessage"`
- `"sqs:ChangeMessageVisibility"`
[todo - verify]
### SQS Configuration
{{% load-snippet-partial file="src-link/watermill-aws/sqs/config.go" first_line_contains="type SubscriberConfig struct " last_line_contains="type GenerateCreateQueueInputFunc" %}}
### Resolving Queue URL
In the Watermill model, we are normalizing the AWS queue url to `topic` used in the `Publish` and `Subscribe` methods.
To give you flexibility of what you want to use as a topic in Watermill, you can customize resolving the queue URL.
{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// QueueUrlResolver" last_line_contains="GenerateQueueUrlResolver" %}}
You can implement your own `QueueUrlResolver` or use one of the provided resolvers.
By default, `GetQueueUrlByNameUrlResolver` resolver is used:
{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// GetQueueUrlByNameUrlResolver " last_line_contains="NewGetQueueUrlByNameUrlResolver" %}}
There are two more resolvers available:
{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// GenerateQueueUrlResolver" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// TransparentUrlResolver" last_line_contains="}" %}}
### Using with SQS emulator
You may want to use [`goaws`](https://github.com/Admiral-Piett/goaws) or [`localstack`](https://hub.docker.com/r/localstack/localstack) for local development or testing.
You can override the endpoint using the `OptFns` option in the `SubscriberConfig` or `PublisherConfig`.
```go
package main
import (
amazonsqs "github.com/aws/aws-sdk-go-v2/service/sqs"
"github.com/ThreeDotsLabs/watermill-amazonsqs/sqs"
)
func main() {
// ...
sqsOpts := []func(*amazonsqs.Options){
amazonsqs.WithEndpointResolverV2(sqs.OverrideEndpointResolver{
Endpoint: transport.Endpoint{
URI: *lo.Must(url.Parse("http://localstack:4566")),
},
}),
}
sqsConfig := sqs.SubscriberConfig{
AWSConfig: cfg,
OptFns: sqsOpts,
}
sub, err := sqs.NewSubscriber(sqsConfig, logger)
if err != nil {
panic(fmt.Errorf("unable to create new subscriber: %w", err))
}
// ...
}
```
## SNS
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ConsumerGroups | yes | yes |
| ExactlyOnceDelivery | no | [yes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) |
| GuaranteedOrder | yes\* | from [AWS Docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html): _"(...) due to the highly distributed architecture, more than one copy of a message might be delivered, and messages may occasionally arrive out of order. Despite this, standard queues make a best-effort attempt to maintain the order in which messages are sent."_ |
| Persistent | yes | |
### Required permissions
- `sns:Subscribe`
- `sns:ConfirmSubscription`
- `sns:Receive`
- `sns:Unsubscribe`
and all permissions required for SQS:
- `sqs:ReceiveMessage`
- `sqs:DeleteMessage`
- `sqs:GetQueueUrl`
- `sqs:CreateQueue`
- `sqs:GetQueueAttributes`
- `sqs:SendMessage`
- `sqs:ChangeMessageVisibility`
- `sqs:SetQueueAttributes`
Additionally, if `sns.SubscriberConfig.DoNotSetQueueAccessPolicy` is not enabled, you should have the following:
- `sqs:SetQueueAttributes`
### SNS Configuration
{{% load-snippet-partial file="src-link/watermill-aws/sns/config.go" first_line_contains="type SubscriberConfig struct " last_line_contains="type GenerateSqsQueueNameFn" %}}
Additionally, because SNS Subscriber uses SQS queues as "subscriptions", you need to pass [SQS configuration](#sqs-configuration) as well.
### Resolving Queue URL
In the Watermill model, we normalise AWS Topic ARN to the `topic` used in the `Publish` and `Subscribe` methods.
{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// TopicResolver" last_line_contains="}" %}}
We are providing two out-of-the-box resolvers:
{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// TransparentTopicResolver" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// GenerateArnTopicResolver" last_line_contains="}" %}}
### Using with SNS emulator
You may want to use [`goaws`](https://github.com/Admiral-Piett/goaws) or [`localstack`](https://hub.docker.com/r/localstack/localstack) for local development or testing.
You can override the endpoint using the `OptFns` option in the `SubscriberConfig` or `PublisherConfig`.
```go
package main
import (
amazonsns "github.com/aws/aws-sdk-go-v2/service/sns"
"github.com/ThreeDotsLabs/watermill-amazonsns/sns"
)
func main() {
// ...
snsOpts := []func(*amazonsns.Options){
amazonsns.WithEndpointResolverV2(sns.OverrideEndpointResolver{
Endpoint: transport.Endpoint{
URI: *lo.Must(url.Parse("http://localstack:4566")),
},
}),
}
snsConfig := sns.SubscriberConfig{
AWSConfig: cfg,
OptFns: snsOpts,
}
sub, err := sns.NewSubscriber(snsConfig, sqsConfig, logger)
if err != nil {
panic(fmt.Errorf("unable to create new subscriber: %w", err))
}
// ...
}
```

65
docs/content/pubsubs/bolt.md Normal file
View File

@@ -0,0 +1,65 @@
+++
title = "Bolt Pub/Sub"
description = "A pure Go key/value store"
date = 2021-11-19T00:00:00+02:00
bref = "A pure Go key/value store"
weight = 20
+++
Bolt is a pure Go key/value store which provides a simple, fast, and reliable
database for projects that don't require a full database server such as
Postgres or MySQL.
Bolt backed Pub/Sub is good for simple applications which don't need a more
advanced Pub/Sub system with external dependencies or already use Bolt and
want to publish messages in transaction when saving other data.
Bolt documentation: https://github.com/etcd-io/bbolt
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-bolt
```
### Characteristics
| Feature | Implements | Note |
| ------------------- | ---------- | ---- |
| ConsumerGroups | no | |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | yes | |
### Configuration
{{% load-snippet-partial file="src-link/watermill-bolt/pkg/bolt/bolt.go" first_line_contains="type CommonConfig struct " last_line_equals="}" %}}
{{% load-snippet-partial file="src-link/watermill-bolt/pkg/bolt/bolt.go" first_line_contains="type PublisherConfig struct " last_line_equals="}" %}}
{{% load-snippet-partial file="src-link/watermill-bolt/pkg/bolt/bolt.go" first_line_contains="type SubscriberConfig struct " last_line_equals="}" %}}
#### Subscription name
To receive messages published to a topic, you must create a subscription to
that topic. Only messages published to the topic after the subscription is
created will be received by the subscriber.
A topic can have multiple subscriptions, but a given subscription belongs to
a single topic.
In Watermill, the subscription is created automatically during calling
`Subscribe()`. Subscription name is generated by calling the function set as
`SubscriberConfig.GenerateSubscriptionName`. By default, it is the topic name
with the string `_sub` appended to it.
#### Marshaler
Watermill's messages cannot be directly saved in Bolt which operates on byte
slices. Marshaller converts the messages to and from byte slices. The default
implementation marshals messages as JSON, a format which is human-readable
for easier debugging. The performance should be enough for most applications
unless a very large messages are used within your system. If that is the case
you may want to consider implementing a more efficient marshaler.
{{% load-snippet-partial file="src-link/watermill-bolt/pkg/bolt/marshaler.go" first_line_contains="// Marshaler" last_line_equals="}" %}}

78
docs/content/pubsubs/firestore.md Normal file
View File

@@ -0,0 +1,78 @@
+++
title = "Firestore Pub/Sub"
description = "A scalable document database from Google"
date = 2021-07-29T15:30:00+02:00
bref = "A scalable document database from Google"
weight = 30
+++
Cloud Firestore is a cloud-hosted, NoSQL database from Google.
This Pub/Sub comes with two publishers. To publish messages in a transaction
use the `TransactionalPublisher`. If you do not want to publish messages in
transaction use the normal `Publisher`.
Using Firestore as a Pub/Sub instead of using a dedicated Pub/Sub system can be
useful to publish messages in transaction while at the same time saving other
data in Firestore. Thanks to that the data and the messages can be consistently
persisted. If the messages and the data weren't being published transactionally
you could end up in situations where messages were emitted even though the data
wasn't saved or messages weren't emitted even though the data was saved. After
transactionally publishing messages in Firestore you can then subscribe to them
and relay them to a different Pub/Sub system.
Godoc: <https://pkg.go.dev/github.com/ThreeDotsLabs/watermill-firestore>
Firestore documentation: <https://firebase.google.com/docs/firestore/>
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-firestore
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes | |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | yes | |
### Configuration
#### Publisher configuration
{{% load-snippet-partial file="src-link/watermill-firestore/pkg/firestore/publisher.go" first_line_contains="type PublisherConfig struct {" last_line_equals="}" %}}
#### Subscriber configuration
{{% load-snippet-partial file="src-link/watermill-firestore/pkg/firestore/subscriber.go" first_line_contains="type SubscriberConfig struct {" last_line_equals="}" %}}
#### Subscription name
To receive messages published to a topic, you must create a subscription to
that topic. Only messages published to the topic after the subscription is
created will be received by the subscribers.
A topic can have multiple subscriptions, but a given subscription belongs to a
single topic.
In Watermill, the subscription is created automatically during calling
`Subscribe()`. Subscription name is generated by function passed to
`SubscriberConfig.GenerateSubscriptionName`. By default, it is just the topic
name with a suffix `_sub` appended to it.
If you want to consume messages from a topic with multiple subscribers
processing the incoming messages in a different way, you should use a custom
function to generate unique subscription names for each subscriber.
### Marshaler
Watermill's messages cannot be stored directly in Firestore. The marshaler is
responsible for converting them to a type which can be stored by Firestore.
The default implementation should be enough for most applications so it is
unlikely that you need to implement your own marshaler.
{{% load-snippet-partial file="src-link/watermill-firestore/pkg/firestore/marshaler.go" first_line_contains="// Marshaler" last_line_equals="}" padding_after="0" %}}

38
docs/content/pubsubs/gochannel.md Normal file
View File

@@ -0,0 +1,38 @@
+++
title = "Go Channel"
description = "A Pub/Sub implemented on Golang goroutines and channels"
date = 2019-07-06T22:30:00+02:00
bref = "A Pub/Sub implemented on Golang goroutines and channels"
weight = 40
+++
{{% load-snippet-partial file="src-link/pubsub/gochannel/pubsub.go" first_line_contains="// GoChannel" last_line_contains="type GoChannel struct {" %}}
You can find a fully functional example with Go Channels in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/go-channel).
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | no | |
| ExactlyOnceDelivery | yes | |
| GuaranteedOrder | yes | |
| Persistent | no| |
### Configuration
You can inject configuration via the constructor.
{{% load-snippet-partial file="src-link/pubsub/gochannel/pubsub.go" first_line_contains="func NewGoChannel" last_line_contains="logger:" %}}
### Publishing
{{% load-snippet-partial file="src-link/pubsub/gochannel/pubsub.go" first_line_contains="// Publish" last_line_contains="func (g *GoChannel) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/pubsub/gochannel/pubsub.go" first_line_contains="// Subscribe" last_line_contains="func (g *GoChannel) Subscribe" %}}
### Marshaler
No marshaling is needed when sending messages within the process.

85
docs/content/pubsubs/googlecloud.md Normal file
View File

@@ -0,0 +1,85 @@
+++
title = "Google Cloud Pub/Sub"
description = "The fully-managed real-time messaging service from Google"
date = 2019-07-06T22:30:00+02:00
bref = "The fully-managed real-time messaging service from Google"
weight = 50
+++
Cloud Pub/Sub brings the flexibility and reliability of enterprise message-oriented middleware to
the cloud.
At the same time, Cloud Pub/Sub is a scalable, durable event ingestion and delivery
system that serves as a foundation for modern stream analytics pipelines.
By providing many-to-many, asynchronous messaging that decouples senders and receivers,
it allows for secure and highly available communication among independently written applications.
Cloud Pub/Sub delivers low-latency, durable messaging that helps developers quickly integrate
systems hosted on the Google Cloud Platform and externally.
Official Documentation: [https://cloud.google.com/pubsub/docs/](https://cloud.google.com/pubsub/docs/overview)
You can find a fully functional example with Google Cloud Pub/Sub in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/googlecloud).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-googlecloud/v2
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes | multiple subscribers within the same Subscription name |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | yes* | maximum retention time is 7 days |
### Configuration
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/publisher.go" first_line_contains="type PublisherConfig struct " last_line_contains="func NewPublisher" %}}
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/subscriber.go" first_line_contains="type SubscriberConfig struct {" last_line_contains="func NewSubscriber(" %}}
#### Subscription name
To receive messages published to a topic, you must create a subscription to that topic.
Only messages published to the topic after the subscription is created are available to subscriber
applications.
The subscription connects the topic to a subscriber application that receives and processes
messages published to the topic.
A topic can have multiple subscriptions, but a given subscription belongs to a single topic.
In Watermill, the subscription is created automatically during calling `Subscribe()`.
Subscription name is generated by function passed to `SubscriberConfig.GenerateSubscriptionName`.
By default, it is just the topic name (`TopicSubscriptionName`).
When you want to consume messages from a topic with multiple subscribers, you should use
`TopicSubscriptionNameWithSuffix` or your custom function to generate the subscription name.
### Connecting
Watermill will connect to the instance of Google Cloud Pub/Sub indicated by the environment variables. For production setup, set the `GOOGLE_APPLICATION_CREDENTIALS` env, as described in [the official Google Cloud Pub/Sub docs](https://cloud.google.com/pubsub/docs/quickstart-client-libraries#pubsub-client-libraries-go). Note that you won't need to install the Cloud SDK, as Watermill will take care of the administrative tasks (creating topics/subscriptions) with the default settings and proper permissions.
For development, you can use a Docker image with the emulator and the `PUBSUB_EMULATOR_HOST` env ([check out the Getting Started guide]({{< ref "getting-started#subscribing_gcloud" >}})).
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="publisher, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="subscriber, err :=" last_line_contains="panic(err)" padding_after="1" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/subscriber.go" first_line_contains="// Subscribe " last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
Watermill's messages cannot be directly sent to Google Cloud Pub/Sub - they need to be marshaled. You can implement your marshaler or use the default implementation.
{{% load-snippet-partial file="src-link/watermill-googlecloud/pkg/googlecloud/marshaler.go" first_line_contains="// Marshaler" last_line_contains="type DefaultMarshalerUnmarshaler " padding_after="0" %}}

74
docs/content/pubsubs/http.md Normal file
View File

@@ -0,0 +1,74 @@
+++
title = "HTTP"
description = "Call and listen to webhooks asynchronously"
date = 2019-07-06T22:30:00+02:00
bref = "Call and listen to webhooks asynchronously"
weight = 60
+++
The HTTP subscriber listens to HTTP requests (for example - webhooks) and outputs them as messages.
You can then post them to any Publisher. Here is an example with [sending HTTP messages to Kafka](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/real-world-examples/receiving-webhooks/main.go).
The HTTP publisher sends HTTP requests as specified in its configuration. Here is an example with [transforming Kafka messages into HTTP webhook requests](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/sending-webhooks).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-http/v2
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | no | |
| ExactlyOnceDelivery | yes | |
| GuaranteedOrder | yes | |
| Persistent | no| |
### Subscriber configuration
Subscriber configuration is done via the config struct passed to the constructor:
{{% load-snippet-partial file="src-link/watermill-http/pkg/http/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="}" %}}
You can use the `Router` config option to `SubscriberConfig` to pass your own `chi.Router` (see [chi](https://github.com/go-chi/chi)).
This may be helpful if you'd like to add your own HTTP handlers (e.g. a health check endpoint).
### Publisher configuration
Publisher configuration is done via the config struct passed to the constructor:
{{% load-snippet-partial file="src-link/watermill-http/pkg/http/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="}" %}}
How the message topic and body translate into the URL, method, headers, and payload of the HTTP request is highly configurable through the use of `MarshalMessageFunc`.
Use the provided `DefaultMarshalMessageFunc` to send POST requests to a specific url:
{{% load-snippet-partial file="src-link/watermill-http/pkg/http/publisher.go" first_line_contains="// MarshalMessageFunc" last_line_contains="return req, nil" padding_after="2" %}}
You can pass your own `http.Client` to execute the requests or use Golang's default client.
### Running
To run HTTP subscriber you need to run `StartHTTPServer()`. It needs to be run after `Subscribe()`.
When using with the router, you should wait for the router to start.
```go
<-r.Running()
httpSubscriber.StartHTTPServer()
```
### Subscribing
{{% load-snippet-partial file="src-link/watermill-http/pkg/http/subscriber.go" first_line_contains="// Subscribe adds" last_line_contains="func (s *Subscriber) Subscribe" %}}
#### Custom HTTP status codes
To specify a custom HTTP status code, which will returned as response, you can use following call during message handling:
```go
// msg is a *message.Message
http.SetResponseStatusCode(msg, http.StatusForbidden)
msg.Nack()
```

66
docs/content/pubsubs/io.md Normal file
View File

@@ -0,0 +1,66 @@
+++
title = "io.Writer/io.Reader"
description = "Pub/Sub implemented as Go stdlib's most loved interfaces"
date = 2019-07-06T22:30:00+02:00
bref = "Pub/Sub implemented as Go stdlib's most loved interfaces"
weight = 70
+++
This is an experimental Pub/Sub implementation that leverages the [standard library's](https://golang.org/pkg/io/) `io.Writer` and `io.Reader` interfaces as sources of Publisher and Subscriber, respectively.
Note that these aren't full-fledged Pub/Subs like Kafka, RabbitMQ, or the likes, but given the ubiquity of implementations of `Writer` and `Reader` they may come in handy, for uses like:
* Writing messages to file or stdout
* Subscribing for data on a file or stdin and packaging it as messages
* Interfacing with third-party libraries that implement `io.Writer` or `io.Reader`, like [github.com/colinmarc/hdfs](https://github.com/colinmarc/hdfs) or [github.com/mholt/archiver](https://github.com/mholt/archiver).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-io
```
### Characteristics
This is a very bare-bones implementation for now, so no extra features are supported. However, it is still sufficient for applications like a [CLI producer/consumer](https://github.com/ThreeDotsLabs/watermill/tree/master/tools/mill).
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | no | |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | no | |
### Configuration
The publisher configuration is relatively simple.
{{% load-snippet-partial file="src-link/watermill-io/pkg/io/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="// Publisher" %}}
The subscriber may work in two modes – either perform buffered reads of constant size from the io.Reader, or split the byte stream into messages using a delimiter byte.
The reading will continue even if the reads come up empty, but they will not be sent out as messages. The time to wait after an empty read is configured through the `PollInterval` parameter. As soon as a non-empty input is read, it will be packaged as a message and sent out.
{{% load-snippet-partial file="src-link/watermill-io/pkg/io/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="// Subscriber" %}}
The continuous reading may be used, for example, to emulate the behaviour of a `tail -f` command, like in this snippet:
{{% load-snippet-partial file="docs/snippets/tail-log-file/main.go" first_line_contains="// this will" last_line_contains="return false" padding_after="1" %}}
### Marshaling/Unmarshaling
The MarshalFunc is an important part of `io.Publisher`, because it fully controls the format in the underlying `io.Writer` will obtain the messages.
Correspondingly, the UnmarshalFunc regulates how the bytes read by the `io.Reader` will be interpreted as Watermill messages.
{{% load-snippet-partial file="src-link/watermill-io/pkg/io/marshal.go" first_line_contains="// MarshalMessageFunc" last_line_contains="// PayloadMarshalFunc" %}}
{{% load-snippet-partial file="src-link/watermill-io/pkg/io/marshal.go" first_line_contains="// UnmarshalMessageFunc" last_line_contains="// PayloadUnmarshalFunc" %}}
The package comes with some predefined marshal and unmarshal functions, but you might want to write your own marshaler/unmarshaler to work with the specific implementation of `io.Writer/io.Reader` that you are working with.
### Topic
For the Publisher/Subscriber implementation itself, the topic has no meaning. It is difficult to interpret the meaning of topic in the general context of `io.Writer` and `io.Reader` interfaces.
However, the topic is passed as a parameter to the marshal/unmarshal functions, so the adaptations to particular `Writer/Reader` implementation may take it into account.

85
docs/content/pubsubs/kafka.md Normal file
View File

@@ -0,0 +1,85 @@
+++
title = "Kafka"
description = "A distributed streaming platform from Apache"
date = 2019-07-06T22:30:00+02:00
bref = "A distributed streaming platform from Apache"
weight = 80
+++
Apache Kafka is one of the most popular Pub/Subs. We are providing Pub/Sub implementation based on [IBM Sarama](https://github.com/IBM/sarama).
You can find a fully functional example with Kafka in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/kafka).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-kafka/v3
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes | |
| ExactlyOnceDelivery | no | in theory can be achieved with [Transactions](https://www.confluent.io/blog/transactions-apache-kafka/), currently no support for any Golang client |
| GuaranteedOrder | yes | require [partition key usage](#partitioning) |
| Persistent | yes| |
### Configuration
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="// Subscribe" %}}
#### Passing custom `Sarama` config
You can pass [custom config](https://github.com/Shopify/sarama/blob/master/config.go#L20) parameters via `overwriteSaramaConfig *sarama.Config` in `NewSubscriber` and `NewPublisher`.
When `nil` is passed, default config is used (`DefaultSaramaSubscriberConfig`).
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// DefaultSaramaSubscriberConfig" last_line_contains="return config" padding_after="1" %}}
### Connecting
#### Publisher
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/publisher.go" first_line_contains="// NewPublisher" last_line_contains="(*Publisher, error)" padding_after="0" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="publisher, err := kafka.NewPublisher" last_line_contains="panic(err)" padding_after="1" %}}
#### Subscriber
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="(*Subscriber, error)" padding_after="0" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="saramaSubscriberConfig :=" last_line_contains="panic(err)" padding_after="1" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
Watermill's messages cannot be directly sent to Kafka - they need to be marshaled. You can implement your marshaler or use default implementation.
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/marshaler.go" first_line_contains="// Marshaler" last_line_contains="func (DefaultMarshaler)" padding_after="0" %}}
### Partitioning
Our Publisher has support for the partitioning mechanism.
It can be done with special Marshaler implementation:
{{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/marshaler.go" first_line_contains="type kafkaJsonWithPartitioning" last_line_contains="func (j kafkaJsonWithPartitioning) Marshal" padding_after="0" %}}
When using, you need to pass your function to generate partition key.
It's a good idea to pass this partition key with metadata to not unmarshal entire message.
```go
marshaler := kafka.NewWithPartitioningMarshaler(func(topic string, msg *message.Message) (string, error) {
return msg.Metadata.Get("partition"), nil
})
```
Please note that in the example above, if the `partition` key is missing from the message metadata, an empty string `""` will be used as the partitioning key. This will cause all such messages to be routed to the same partition, which may not be the desired behavior and could lead to uneven load distribution.

85
docs/content/pubsubs/nats.md Normal file
View File

@@ -0,0 +1,85 @@
+++
title = "NATS Jetstream"
description = "A simple, secure and high performance open source messaging system"
date = 2022-02-03T10:30:00+05:00
bref = "A simple, secure and high performance open source messaging system"
weight = 90
+++
NATS Jetstream is a data streaming system powered by NATS, and written in the Go programming language.
As of v2.1.0, the `pkg/jetstream` implementation, based on the
[nats.go Jetstream package](https://github.com/nats-io/nats.go/tree/main/jetstream) is considered stable.
For production use it is recommended to use the pubsub implementations in `pkg/nats` with Jetstream enabled.
You can find a fully functional example with NATS JetStream in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/nats-jetstream).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-nats/v2
```
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|-----------------------------------------------------------------------------------------------------------------------|
| ConsumerGroups | yes | you need to set `QueueGroupPrefix` name or provide an optional calculator |
| ExactlyOnceDelivery | yes | you need to ensure `AckAsync` has default false value and set `TrackMsgId` to true on the Jetstream configuration |
| GuaranteedOrder | no | [with the redelivery feature, order can't be guaranteed](https://github.com/nats-io/nats-streaming-server/issues/187) |
| Persistent | yes | |
### Configuration
Configuration is done through PublisherConfig and SubscriberConfig types. These share a common JetStreamConfig. To use the experimental nats-core support, set Disabled=true.
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/jetstream.go" first_line_contains="// JetStreamConfig contains" last_line_contains="type DurableCalculator =" %}}
PublisherConfig:
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="type Publisher struct {" %}}
SubscriberConfig:
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="type Subscriber struct" %}}
### Connecting
By default, the NATS client will try to connect to `localhost:4222`. If you are using different hostname or port you should specify using the URL property of `SubscriberConfig` and `PublisherConfig`.
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/publisher.go" first_line_contains="// NewPublisher" last_line_contains="func NewPublisher" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-jetstream/main.go" first_line_contains="publisher, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="func NewSubscriber" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-jetstream/main.go" first_line_contains="subscriber, err :=" last_line_contains="panic(err)" padding_after="1" %}}
You can also use `NewSubscriberWithNatsConn` and `NewPublisherWithNatsConn` to use a custom `*nats.Conn`.
### Publishing
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/publisher.go" first_line_contains="// Publish publishes" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/subscriber.go" first_line_contains="// Subscribe " last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
NATS provides a header passing mechanism that allows conveying the watermill message details as metadata. This is done by default with only the binary payload sent in the message body. The header `_watermill_message_uuid` is reserved.
Other builtin marshalers are based on Golang's [`gob`](https://golang.org/pkg/encoding/gob/) and [`json`](https://golang.org/packages/encoding/json) packages.
{{% load-snippet-partial file="src-link/watermill-nats/pkg/nats/marshaler.go" first_line_contains="type Marshaler " last_line_contains="func defaultNatsMsg" padding_after="0" %}}
When you have your own format of the messages, you can implement your own Marshaler, which will serialize messages in your format. An example protobuf implementation with tests and benchmarks can be found [here](https://github.com/ThreeDotsLabs/watermill-nats/tree/master/_examples/marshalers/protobuf/)
When needed, you can bypass both [UUID]({{< ref "message#message" >}}) and [Metadata]({{< ref "message#message" >}}) and send just a `message.Payload`,
but some standard [middlewares]({{< ref "messages-router#middleware" >}}) may not work.
## Core NATS
This package also includes limited support for connecting to [Core NATS](https://docs.nats.io/nats-concepts/core-nats). While Core NATS does not support many of the streaming features needed for a perfect fit with watermill and most acks end up implemented as no-ops, in environments with a mix of jetstream and Core NATS messaging in play it can be nice to use watermill consistently on the application side.

64
docs/content/pubsubs/redisstream.md Normal file
View File

@@ -0,0 +1,64 @@
+++
title = "Redis Stream"
description = "A fast, open source, in-memory, key-value data store"
date = 2023-02-01T22:30:00+08:00
bref = "A fast, open source, in-memory, key-value data store"
weight = 110
+++
Redis is the open source, in-memory data store used by millions of developers. Redis stream is a data structure that acts like an append-only log in Redis. We are providing Pub/Sub implementation based on [redis/go-redis](https://github.com/redis/go-redis).
You can find a fully functional example with Redis Stream in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/redisstream).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-redisstream
```
### Characteristics
| Feature | Implements | Note |
| ------- | ---------- | ---- |
| ConsumerGroups | yes | |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | no | |
| Persistent | yes | |
| FanOut | yes | use XREAD to fan out messages when there is no consumer group |
### Configuration
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="// Publish publishes message to redis stream" %}}
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="func (s *Subscriber) Subscribe" %}}
#### Passing `redis.UniversalClient`
You need to configure and pass your own go-redis client via `Client redis.UniversalClient` in `NewSubscriber` and `NewPublisher`. The client can be either `redis.Client` or `redis.ClusterClient`.
#### Publisher
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/publisher.go" first_line_contains="// NewPublisher" last_line_contains="(*Publisher, error)" padding_after="0" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/redisstream/main.go" first_line_contains="pubClient := redis.NewClient" last_line_contains="panic(err)" padding_after="1" %}}
#### Subscriber
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="(*Subscriber, error)" padding_after="0" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/redisstream/main.go" first_line_contains="subClient := redis.NewClient" last_line_contains="panic(err)" padding_after="1" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/subscriber.go" first_line_contains="func (s *Subscriber) Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
### Marshaler
Watermill's messages cannot be directly sent to Redis - they need to be marshaled. You can implement your marshaler or use default implementation. The default implementation uses [MessagePack](https://msgpack.org/index.html) for efficient serialization.
{{% load-snippet-partial file="src-link/watermill-redisstream/pkg/redisstream/marshaller.go" first_line_contains="const UUIDHeaderKey" last_line_contains="type DefaultMarshallerUnmarshaller" padding_after="0" %}}

144
docs/content/pubsubs/sql.md Normal file
View File

@@ -0,0 +1,144 @@
+++
title = "SQL (PostgreSQL, MySQL)"
description = "Pub/Sub based on MySQL or PostgreSQL."
date = 2019-07-06T22:30:00+02:00
bref = "Pub/Sub based on MySQL or PostgreSQL."
weight = 120
+++
SQL Pub/Sub executes queries on any SQL database, using it like a messaging system. At the moment, **MySQL** and **PostgreSQL** are supported.
It be useful for projects that are not using any specialized message queue at the moment, but have access to a SQL database.
If you are looking for SQLite Pub/Sub, check out the [SQLite Pub/Sub](/pubsubs/sqlite/) documentation.
The SQL subscriber runs a `SELECT` query within short periods, remembering the position of the last record. If it finds
any new records, they are returned. One handy use case is consuming events from a database table, that can be later published
on some kind of message queue.
The SQL publisher simply inserts consumed messages into the chosen table. A common approach would be to use it as a persistent
log of events that were published on a queue with short message expiration time.
SQL Pub/Sub is also a good choice for implementing Outbox pattern with [Forwarder](/docs/forwarder/) component.
See also the [SQL example](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/sql).
## Installation
```bash
go get github.com/ThreeDotsLabs/watermill-sql/v4
```
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|-------------------------------------------------------------------------------|
| ConsumerGroups | yes | See `ConsumerGroup` in `SubscriberConfig` (not supported by the queue schema) |
| ExactlyOnceDelivery | yes |
| GuaranteedOrder | yes | |
| Persistent | yes | |
### Schema
SQL Pub/Sub uses user-defined schema to handle select and insert queries. You need to implement `SchemaAdapter` and pass
it to `SubscriberConfig` or `PublisherConfig`.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/schema_adapter_mysql.go" first_line_contains="// DefaultMySQLSchema" last_line_contains="type DefaultMySQLSchema" %}}
There is a default schema provided for each supported engine (`DefaultMySQLSchema` and `DefaultPostgreSQLSchema`).
It supports the most common use case (storing events in a table). You can base your schema on one of these, extending only chosen methods.
#### Extending schema
Consider an example project, where you're fine with using the default schema, but would like to use `BINARY(16)` for storing
the `uuid` column, instead of `VARCHAR(36)`. In that case, you have to define two methods:
* `SchemaInitializingQueries` that creates the table.
* `UnmarshalMessage` method that produces a `Message` from the database record.
Note that you don't have to use the initialization queries provided by Watermill. They will be run only if you set the
`InitializeSchema` field to `true` in the config. Otherwise, you can use your own solution for database migrations.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/schema_adapter_mysql.go" first_line_contains="// DefaultMySQLSchema" last_line_contains="type DefaultMySQLSchema" %}}
### Configuration
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/publisher.go" first_line_contains="type PublisherConfig struct" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="}" %}}
## Publishing
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/publisher.go" first_line_contains="func NewPublisher" last_line_contains="func NewPublisher" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="publisher, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/publisher.go" first_line_contains="// Publish " last_line_contains="func (p *Publisher) Publish" %}}
### Transactions
If you need to publish messages within a database transaction, you have to pass a `*sql.Tx` in the `NewPublisher`
constructor. You have to create one publisher for each transaction.
Example:
{{% load-snippet-partial file="src-link/_examples/real-world-examples/transactional-events/main.go" first_line_contains="func simulateEvents" last_line_contains="return pub.Publish(" padding_after="3" %}}
## Subscribing
To create a subscriber, you need to pass not only proper schema adapter, but also an offsets adapter.
* For MySQL schema use `DefaultMySQLOffsetsAdapter`
* For PostgreSQL schema use `DefaultPostgreSQLOffsetsAdapter`
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/subscriber.go" first_line_contains="func NewSubscriber" last_line_contains="func NewSubscriber" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="subscriber, err :=" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/subscriber.go" first_line_contains="func (s *Subscriber) Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
## Offsets Adapter
The logic for storing offsets of messages is provided by the `OffsetsAdapter`. If your schema uses auto-incremented integer as the row ID,
it should work out of the box with default offset adapters.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/offsets_adapter.go" first_line_contains="type OffsetsAdapter" %}}
## Queue
Instead of the default Pub/Sub schema, you can use the *queue* schema and offsets adapters.
It's a simpler schema that doesn't support consumer groups.
However, it has other advantages.
It lets you specify a custom `WHERE` clause for getting the messages.
You can use it to filter messages by some condition in the payload or in the metadata.
Additionally, you can choose to delete messages from the table after they are acknowledged.
Thanks to this, the table doesn't grow in size with time.
This schema is supported by both PostgreSQL and MySQL.
The example below is based on PostgreSQL, but the same approach can be used with MySQL.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/queue_schema_adapter_postgresql.go" first_line_contains="// PostgreSQLQueueSchema" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/queue_offsets_adapter_postgresql.go" first_line_contains="// PostgreSQLQueueOffsetsAdapter" last_line_contains="}" %}}
## Caveats
### Using last processed transaction ID in PostgreSQL to ensure no messages are lost
In some cases, PostgreSQL `SERIAL` is not incremental.
The `SERIAL` value is generated while the transaction is in progress, not when it is committed.
If transactions are committed in a different order than they were started, message offsets based on `SERIAL` values will not be incremental.
To keep storing acknowledgment information efficient, Watermill keeps only the last message's acknowledgment information.
To ensure no messages are missed when a message order is not kept, Watermill also uses the transaction ID to ensure no message is lost.
For more details, see [Watermill#311](https://github.com/ThreeDotsLabs/watermill/issues/311).
It is important to note that very long-running transactions may result in delayed message delivery.
For instance, if a transaction is running for an hour, no messages will be delivered until the transaction is committed.
While we do not recommend the use of such long transactions, **if they are necessary, we advise the use of the [Queue schema adapter](#queue), which does not depend on the transaction ID.**
You have nothing to worry about if you don't have such long transactions.
If you are migrating your data to a new database, you may need to set `last_processed_transaction_id` in your offsets table.

135
docs/content/pubsubs/sqlite.md Normal file
View File

@@ -0,0 +1,135 @@
+++
title = "SQLite"
description = "A lightweight, file-based SQL database engine"
date = 2025-05-08T11:30:00+02:00
bref = "A lightweight, file-based SQL database engine"
weight = 121
+++
**Beta Version Warning: this Pub/Sub is stable, but it has not been widely tested in production environments. It may be sensitive to certain edge cases and combinations of configuration parameters.**
SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured SQL database engine. Our SQLite Pub/Sub implementation provides two **CGO-free** driver variants optimized for different use cases. Both drivers use pure Go implementations of SQLite, enabling cross-compilation and avoiding CGO dependencies while maintaining full SQLite functionality.
SQLite Pub/Subs provide the easiest way to publish and process events durably, since you do not have to set up or manage a separate database. The database is just a file on disk. Some cloud compute providers offer distributed SQLite clusters, which can provide both durability and unmatched read performance. Tuned SQLite is [~35% faster](https://sqlite.org/fasterthanfs.html) than the Linux file system.
You can find a fully functional example with SQLite in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/sqlite).
## ModernC vs ZombieZen Driver
The vanilla **ModernC driver** is compatible with the Golang standard library SQL package and works without CGO. It has fewer dependencies than the ZombieZen variant and uses the `modernc.org/sqlite` pure Go SQLite implementation. Most users should pick this driver for full compatibility with the standard library.
The advanced **ZombieZen driver** abandons the standard Golang library SQL conventions in favor of [the more orthogonal API and higher performance potential](https://crawshaw.io/blog/go-and-sqlite). Under the hood, it also uses the ModernC SQLite3 implementation and does not need CGO. Advanced SQLite users might prefer this driver for its performance benefits. It is about **6 times faster** than the ModernC variant. It is currently more stable due to lower level control. It is faster than even the CGO SQLite variants on standard library interfaces, and with some tuning should become the absolute speed champion of persistent message brokers over time.
### Characteristics
| Feature | Implements | Note |
|---------------------|------------|---------------------------------------------------|
| ConsumerGroups | yes | See `ConsumerGroupMatcher` in `SubscriberOptions` |
| ExactlyOnceDelivery | no | |
| GuaranteedOrder | yes | |
| Persistent | yes | |
Like the [BoltDB](../bolt/) implementation, the SQLite drivers are imbeddable into your application. They do not require any additional infrastructure other than a mounted persistent disk. Their advantage over the BoltDB Pub/Sub is supporting consumer groups and guaranteed order.
## Vanilla ModernC Driver
### Installation
```bash
go get github.com/ThreeDotsLabs/watermill-sqlite/wmsqlitemodernc@latest
```
### Usage
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/main.go" first_line_contains="import (" last_line_contains="}" padding_after="1" %}}
### Configuration
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/publisher.go" first_line_contains="type PublisherOptions struct" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/subscriber.go" first_line_contains="type SubscriberOptions struct" last_line_contains="}" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/publisher.go" first_line_contains="// NewPublisher" last_line_contains="func NewPublisher" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/publisher.go" first_line_contains="// Publish " last_line_contains="func (p *publisher) Publish" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/main.go" first_line_contains="publisher, err := wmsqlitemodernc.NewPublisher(" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/main.go" first_line_contains="func publishMessages(" last_line_contains="panic(err)" padding_after="1" %}}
#### Publishing in transaction
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/transaction.go" first_line_contains="import (" padding_after="1" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="func NewSubscriber" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite/main.go" first_line_contains="subscriber, err := wmsqlitemodernc.NewSubscriber(" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitemodernc/subscriber.go" first_line_contains="// Subscribe " last_line_contains="func (s *subscriber) Subscribe" %}}
## Advanced ZombieZen Driver
### Installation
```bash
go get -u github.com/ThreeDotsLabs/watermill-sqlite/wmsqlitezombiezen@latest
```
### Usage
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/main.go" first_line_contains="import (" last_line_contains="}" padding_after="1" %}}
### Configuration
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/publisher.go" first_line_contains="type PublisherOptions struct" last_line_contains="}" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/subscriber.go" first_line_contains="type SubscriberOptions struct" last_line_contains="}" %}}
### Publishing
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/publisher.go" first_line_contains="// NewPublisher" last_line_contains="func NewPublisher" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/publisher.go" first_line_contains="// Publish " last_line_contains="func (p *publisher) Publish" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/main.go" first_line_contains="publisher, err := wmsqlitezombiezen.NewPublisher(" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/main.go" first_line_contains="func publishMessages(" last_line_contains="panic(err)" padding_after="1" %}}
#### Publishing in transaction
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/transaction.go" first_line_contains="import (" padding_after="1" %}}
### Subscribing
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="func NewSubscriber" %}}
Example:
{{% load-snippet-partial file="src-link/_examples/pubsubs/sqlite-zombiezen/main.go" first_line_contains="subscriber, err := wmsqlitezombiezen.NewSubscriber(" last_line_contains="panic(err)" padding_after="1" %}}
{{% load-snippet-partial file="src-link/watermill-sqlite/wmsqlitezombiezen/subscriber.go" first_line_contains="// Subscribe " last_line_contains="func (s *subscriber) Subscribe" %}}
## Marshaler
Watermill's messages are stored in SQLite using JSON serialization. Both drivers use the same marshaling approach - messages are automatically marshaled to and from JSON format when publishing and subscribing.
The default marshaler handles:
- Message payload (stored as JSON blob)
- Message metadata (stored as JSON object)
- Message UUID (stored as TEXT)
- Timestamps for ordering and consumer group management
Both drivers automatically handle message marshaling and unmarshaling, so no custom marshaler configuration is typically required.
## Caveats
SQLite3 does not support querying `FOR UPDATE`, which is used for row locking when subscribers in the same consumer group read an event batch in official Watermill SQL PubSub implementations. Current architectural decision is to lock a consumer group offset using `unixepoch()+lockTimeout` time stamp. While one consumed message is processing per group, the offset lock time is extended by `lockTimeout` periodically by `time.Ticker`. If the subscriber is unable to finish the consumer group batch, other subscribers will take over the lock as soon as the grace period runs out. A time lock fulfills the role of a traditional database network timeout that terminates transactions when its client disconnects.
All the normal SQLite limitations apply to Watermill. The connections are file handles. Create new connections for concurrent processing. If you must share a connection, protect it with a mutual exclusion lock. If you are writing within a transaction, create a connection for that transaction only.