snapshot of https://github.com/tldraw/tldraw.git (main, 2026-09-23; blobs > 131072B omitted)

internal/grafana/README.md

@@ -0,0 +1,77 @@
+# Grafana dashboards and alerts
+
+**TL;DR:** `resources/` is the source of truth for our Grafana Cloud dashboards and alert rules. CI pushes it with `gcx` on every merge to `main` that touches these files. Edit here, not in Grafana — UI edits get overwritten by the next deploy.
+
+This directory currently provisions:
+
+- Eleven dashboards: `Zero on Fly.io — service health` (`zero-fly-health`), `Zero slow queries` (`slow-queries`), and `Zero-cache HTTP edge (Fly.io)` (`ni7hhgd`) in the `Zero` folder; `Anthropic scrape overview` in the `Anthropic` folder; `Dotcom events` (`ni5k8zc`), `Dotcom events V1` (`adgumkhou3chsd`), `File effect outbox`, `MCP server (shared boards)` (`mcp-server`), `MCP app sessions` (`kotx6wj`), and `Bemo analytics` in the `Dotcom` folder; and `Cloudflare platform metrics` (`nivs2rl`), which sits at the root rather than in a folder. The events pipeline currently reports from **staging** — set the environment variable accordingly when a dashboard looks empty.
+- All 13 Grafana-managed alert rules (Zero replication/errors/backups, Fly.io container health, Anthropic cost/usage, file room DO exceptions)
+- The folders `Zero` (dashboards + 9 rules), `Dotcom` (6 dashboards + 1 rule), and `Anthropic` (1 dashboard + 3 rules, see below)
+- The `discord` and `email` notification templates (`templates/*.gotmpl`, defining `tldraw.discord.title` / `tldraw.discord.text` and `tldraw.email.subject` / `tldraw.email.message`), which compress alert notifications to severity + summary + links (no Silence link on Discord: silence URLs carry one matcher per label and blow Discord’s 2000-char message cap on multi-instance alerts, truncating mid-link) and explain `DatasourceNoData` as a likely metrics-source outage. Templates can't go through `gcx resources push` — gcx ignores the `notifications.alerting.grafana.app` API group — so they live in `templates/` instead of `resources/` and the deploy workflow PUTs them through the classic provisioning API. Each contact point's optional Title/Subject and Message fields reference these templates; those references are set by hand (contact points stay hand-managed), so the templates only take effect while a contact point points at them.
+
+There are two MCP dashboards, for two different servers. `MCP server (shared boards)` covers the public server on the sync worker at `POST /app/mcp` (`apps/dotcom/sync-worker`), which reads its protocol metrics from the `MEASURE` dataset. `MCP app sessions` covers the separate tldraw MCP app worker (`apps/mcp-app`) and its own `MCP_ANALYTICS` dataset. Neither one's panels belong on `Dotcom events` — that dashboard covers the sync worker's own events.
+
+`Dotcom events V1` holds only the catch-all `Room events` panel and `max attempts before persistence success`; new panels go on `Dotcom events`.
+
+Contact points, notification policies, and data sources are hand-managed in Grafana on purpose.
+
+## Conventions
+
+Every provisioned dashboard carries the tags `provisioned` and `repo:tldraw/tldraw`, so anyone looking at it in Grafana knows where it's managed from. Add both tags when adopting a new dashboard into this tree.
+
+## Anthropic rules
+
+The three rules in the `Anthropic` folder watch the Grafana Anthropic integration's scrape of the Anthropic Admin API. The two metrics behave very differently, and the rules are shaped around that:
+
+- `gen_ai_usage_tokens_total` is near real time and is labelled by API key, model, and token type (`input`, `output`, `cache_creation_5m`, `cache_creation_1h`, `input_cache_read`). About 70 series at the time of writing. This is the only intraday signal.
+- `gen_ai_cost` is **month-to-date** spend in cents, with no API key label, updated **once a day at about 03:00 UTC**. It resets on the 1st, and has had no data at all on the 1st. Anything built on it describes the last completed day, sees a runaway up to 27 hours late, and is blind for a day or two at each month rollover.
+
+`AnthropicTokenBurst` (warning) fires when the summed rate across all series, excluding cache reads, exceeds 1000 tokens/sec over an hour (about 3.6M tokens/hour). Three deliberate choices, each learned the hard way:
+
+- **Summed, not per series.** A per-series rule makes one alert instance per key × model × token type, and every lightly used series has a tiny baseline that any real session clears.
+- **No baseline.** Usage is bursty and mostly idle (the median hour is under 10 tokens/sec), so a 7-day average is close to zero and "5x the average" is a normal working hour. #10579 and #10582 tuned the multiplier and floor twice and it still paged on ordinary sessions.
+- **Cache reads excluded.** They cost a tenth of uncached input and dominate agent and eval traffic. On 2026-09-01 and 09-02 the rule paged repeatedly on Opus 5 cache reads from the internal evals key, which spent about $50 that day. The floor is chosen so that workload stays quiet (it peaks around 500 non-cache-read tokens/sec) while a big-numbers bulk cold-outreach research run (`tldraw-internal`, `crm-dashboard-api`), roughly 2000 tokens/sec of uncached Sonnet 4.5 input at $20 to $25 an hour and about $500 over 2026-08-21 to 08-22, clears it within the hour. Those runs are expected work, so a page from this rule usually means "check whether someone started a bulk run" rather than "something is broken".
+
+`AnthropicDailyCostSpike` (warning) fires when the last completed day cost more than 1.5x the day before, with a $50 floor. `AnthropicHighCostThreshold` (critical) fires when the last completed day cost more than $1000. Both use `increase(gen_ai_cost[1d])` rather than `offset` subtraction so the monthly reset reads as a counter reset and counts from zero instead of going negative. The scrape gap on the 1st is still a blind spot: the first day of the month reads as roughly $0 because there is nothing to diff against. Because the metric only steps once a day, a firing rule stays firing until the next 03:00 update.
+
+All three are filter queries (`expr > threshold`), so the alert series only exists while the condition holds. `missingSeriesEvalsToResolve: 10` is what stops a single quiet minute from resolving and re-firing them.
+
+## Making a change
+
+1. Edit the files under `resources/`.
+2. Open a PR. CI validates the resources and dry-runs the push so you can see what would change. (The alerting API doesn't support server-side dry-run, so alert rules show as "skipped: client-side check only" — that's expected, not an error.)
+3. Merge. The deploy workflow (`.github/workflows/deploy-grafana.yml`) pushes to Grafana Cloud.
+
+## Adding a new alert rule
+
+A rule file carrying `grafana.com/group` labels cannot be _created_ by the push. Grafana rejects it with `cannot set group when creating a new rule` — the resource API can only update a rule that is already in a group. Nothing catches this earlier: alerting has no server-side dry-run, so the PR reports the rule as "skipped" and goes green.
+
+Bootstrap the group once by hand, then let the push adopt it:
+
+    gcx api /api/v1/provisioning/folder/<folder uid>/rule-groups/<group> \
+      -X PUT -H 'X-Disable-Provenance: true' -d @group.json
+
+`group.json` uses the classic provisioning shape (`{title, folderUid, interval, rules: [...]}`), which is not the shape of the file in this tree: `spec.expressions` becomes a `data` array, the expression carrying `source: true` becomes `condition`, and `spec.trigger.interval` becomes the group's `interval` in seconds. `X-Disable-Provenance: true` is what leaves provenance clear so the push can take ownership afterwards.
+
+`dotcom-file-do-exceptions` was created this way on 2026-08-18, after PR #10052's deploy failed on it.
+
+## Pulling fresh state
+
+If you prototyped a dashboard in the Grafana UI and want to adopt it, pull it into the tree with a local gcx context (`gcx login`):
+
+    gcx resources pull dashboards/<uid> -p internal/grafana/resources -o yaml
+    gcx resources pull alertrules -p internal/grafana/resources -o yaml
+
+Note the alert rules selector is `alertrules` — bare `rules` routes to the Asserts product and 403s on our stack.
+
+Don't hand-reformat gcx output; it needs to round-trip. Exception: `zero-fly-health` is stored as JSON because gcx 1.0.0's YAML writer emits an invalid escape for emoji variation selectors in row titles (its own reader then rejects the file). If a pulled YAML file fails `gcx resources validate`, re-pull that resource with `-o json`.
+
+Alert rules created through Grafana's "import Prometheus rules" flow can't be pushed: they carry `converted_prometheus` provenance, and Grafana only accepts writes to a rule from the tool that owns its provenance. To adopt one, strip the `alerting.grafana.app/prometheus-rule-definition` annotation and set `grafana.com/provenance: ''` in the pulled file, then recreate the rule in Grafana with clear provenance so the push can take ownership: delete the imported group (`gcx api /api/convert/prometheus/config/v1/rules/<folder title>/<group> -X DELETE`), recreate it via `gcx api /api/v1/provisioning/folder/<folder uid>/rule-groups/<group> -X PUT -H 'X-Disable-Provenance: true' -d @group.json`, and push. Recreating grouped rules must go through that group endpoint — the resource API gcx pushes with can't add rules to groups, only update rules already in one. The three Anthropic cost rules went through this adoption on 2026-08-10 and are repo-owned now — don't re-run the integration's rule import for them, which would recreate the imported group alongside the adopted one and bring the provenance clash back.
+
+## Deleting a resource
+
+Deleting a file here does **not** delete it in Grafana. Remove the file and run `gcx resources delete <type>/<uid>` manually.
+
+## Service account access
+
+CI authenticates with a Grafana service account token (`GRAFANA_SERVER` / `GRAFANA_TOKEN` repo secrets). The `zero-fly-health` dashboard has restricted permissions: the service account needs an explicit grant on it (or admin role) or pulls and pushes will 403.

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/8451fbba-5835-56e2-b2f5-7f5e44c520b8.yaml

@@ -0,0 +1,112 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: anthropic
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-03-20T11:22:55Z'
+    grafana.com/updatedBy: bf1wptf89bpq8d
+  labels:
+    grafana.app/folder: anthropic
+    grafana.com/group: anthropic
+    grafana.com/group-index: '1'
+  name: 8451fbba-5835-56e2-b2f5-7f5e44c520b8
+  namespace: stacks-890472
+spec:
+  annotations:
+    description: 'Anthropic API usage over the last hour averaged {{ $value | humanize }}
+      tokens/sec excluding cache reads, above the 1000 tokens/sec floor (about 3.6M
+      tokens/hour). Check the scrape overview dashboard for which key and model.'
+    summary: Anthropic API token burst detected.
+  execErrState: Ok
+  expressions:
+    prometheus_math:
+      model:
+        datasource:
+          IsPrunable: false
+          access: ''
+          apiVersion: ''
+          basicAuth: false
+          basicAuthUser: ''
+          created: '0001-01-01T00:00:00Z'
+          database: ''
+          id: -100
+          isDefault: false
+          jsonData: {}
+          name: __expr__
+          readOnly: false
+          secureJsonData: {}
+          type: __expr__
+          uid: __expr__
+          updated: '0001-01-01T00:00:00Z'
+          url: ''
+          user: ''
+          withCredentials: false
+        expression: is_number($query) || is_nan($query) || is_inf($query)
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: prometheus_math
+        type: math
+      queryType: math
+    query:
+      datasourceUID: grafanacloud-prom
+      model:
+        datasource:
+          type: prometheus
+          uid: grafanacloud-prom
+        expr: |
+          sum(
+            rate(gen_ai_usage_tokens_total{job=~"integrations/anthropic/.*", gen_ai_token_type!="input_cache_read"}[1h])
+          ) > 1000
+        instant: true
+        intervalMs: 1000
+        maxDataPoints: 43200
+        range: false
+        refId: query
+      queryType: prometheus
+      relativeTimeRange:
+        from: 11m0s
+        to: 1m0s
+    threshold:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0
+              type: gt
+        datasource:
+          IsPrunable: false
+          access: ''
+          apiVersion: ''
+          basicAuth: false
+          basicAuthUser: ''
+          created: '0001-01-01T00:00:00Z'
+          database: ''
+          id: -100
+          isDefault: false
+          jsonData: {}
+          name: __expr__
+          readOnly: false
+          secureJsonData: {}
+          type: __expr__
+          uid: __expr__
+          updated: '0001-01-01T00:00:00Z'
+          url: ''
+          user: ''
+          withCredentials: false
+        expression: prometheus_math
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: threshold
+        type: threshold
+      queryType: threshold
+      source: true
+  for: 15m0s
+  labels:
+    __converted_prometheus_rule__: 'true'
+    severity: warning
+  missingSeriesEvalsToResolve: 10
+  noDataState: Ok
+  title: AnthropicTokenBurst
+  trigger:
+    interval: 1m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/afjn4kz1g4jk0a.yaml

@@ -0,0 +1,66 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-21T09:07:11Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-alerts-fast
+    grafana.com/group-index: '2'
+  name: afjn4kz1g4jk0a
+  namespace: stacks-890472
+spec:
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: grafanacloud-prom
+      model:
+        editorMode: code
+        expr: zero_replication_total_lag_millisecond{deployment_environment="production",
+          service_name="zero-rm"}
+        instant: true
+        intervalMs: 1000
+        legendFormat: __auto
+        maxDataPoints: 43200
+        range: false
+        refId: A
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 1000
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 5m0s
+  labels:
+    service: zero
+    severity: warning
+  noDataState: NoData
+  title: Zero replication lag elevated
+  trigger:
+    interval: 5m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/bfjyilovht2pse.yaml

@@ -0,0 +1,81 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-28T08:48:46Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-fly-alerts
+    grafana.com/group-index: '5'
+  name: bfjyilovht2pse
+  namespace: stacks-890472
+spec:
+  annotations:
+    description: |-
+      Fires when the 5-minute rolling count of logs with detected_level error or fatal is greater than zero for service_name matching zero-(rm|vs) and deployment_environment=production.
+
+      Investigate in Grafana Explore (grafanacloud-tldraw-logs):
+
+      {service_name=~"zero-(rm|vs)", deployment_environment="production"} | detected_level=~"error|fatal"
+
+      Typical causes include failed upstream connections, auth/sync failures, or crashes; use log fields (component, worker, status, message) to narrow the blast radius.
+    summary: 'Production Zero: error/fatal logs in the last evaluation window.'
+  execErrState: Ok
+  expressions:
+    A:
+      datasourceUID: grafanacloud-logs
+      model:
+        datasource:
+          type: loki
+          uid: grafanacloud-logs
+        editorMode: code
+        expr:
+          sum(count_over_time({service_name=~"zero-(rm|vs)", deployment_environment="production"}
+          | detected_level=~"error|fatal" [5m]))
+        instant: true
+        intervalMs: 1000
+        maxDataPoints: 43200
+        queryType: instant
+        refId: A
+      queryType: instant
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 1m0s
+  labels:
+    service: zero
+    severity: critical
+  noDataState: Ok
+  paused: true
+  title: Zero error log (production)
+  trigger:
+    interval: 1m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/cfjnb1whljcowe.yaml

@@ -0,0 +1,72 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-28T08:48:46Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-fly-alerts
+    grafana.com/group-index: '2'
+  name: cfjnb1whljcowe
+  namespace: stacks-890472
+spec:
+  annotations:
+    summary: Zero container disk utilization exceeds 80%
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: afj5re4yk9og0e
+      model:
+        datasource:
+          type: prometheus
+          uid: afj5re4yk9og0e
+        editorMode: code
+        expr: |-
+          (fly_instance_filesystem_blocks{app=~"production-zero-rm|production-zero-vs"} - fly_instance_filesystem_blocks_avail{app=~"production-zero-rm|production-zero-vs"}) * fly_instance_filesystem_block_size{app=~"production-zero-rm|production-zero-vs"} /
+            (fly_instance_filesystem_blocks{app=~"production-zero-rm|production-zero-vs"} * fly_instance_filesystem_block_size{app=~"production-zero-rm|production-zero-vs"})
+        instant: true
+        intervalMs: 1000
+        legendFormat: __auto
+        maxDataPoints: 43200
+        range: false
+        refId: A
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0.8
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 10m0s
+  labels:
+    service: zero
+    severity: critical
+  noDataState: NoData
+  title: Flyio zero container disk high
+  trigger:
+    interval: 1m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/cfjnbehzmlhxcc.yaml

@@ -0,0 +1,72 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-28T08:48:46Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-fly-alerts
+    grafana.com/group-index: '3'
+  name: cfjnbehzmlhxcc
+  namespace: stacks-890472
+spec:
+  annotations:
+    summary: Zero container CPU utilization exceeds 85%
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: afj5re4yk9og0e
+      model:
+        datasource:
+          type: prometheus
+          uid: afj5re4yk9og0e
+        editorMode: code
+        expr:
+          sum by (instance, app)(rate(fly_instance_cpu{app=~"production-zero-rm|production-zero-vs",mode!="idle"}[5m]))
+          / sum by (instance, app)(rate(fly_instance_cpu{app=~"production-zero-rm|production-zero-vs"}[5m]))
+        instant: true
+        intervalMs: 1000
+        legendFormat: __auto
+        maxDataPoints: 43200
+        range: false
+        refId: A
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0.85
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 15m0s
+  labels:
+    service: zero
+    severity: warning
+  noDataState: NoData
+  title: Flyio zero container cpu high
+  trigger:
+    interval: 1m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/cfjyl9tbrury8b.yaml

@@ -0,0 +1,80 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-28T08:48:46Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-fly-alerts
+    grafana.com/group-index: '6'
+  name: cfjyl9tbrury8b
+  namespace: stacks-890472
+spec:
+  annotations:
+    description: |-
+      Fires when the 5-minute rolling count of logs with detected_level error or fatal is greater than zero for service_name matching zero-(rm|vs) and deployment_environment=staging.
+
+      Investigate in Grafana Explore (grafanacloud-tldraw-logs):
+
+      {service_name=~"zero-(rm|vs)", deployment_environment="staging"} | detected_level=~"error|fatal"
+
+      Typical causes include failed upstream connections, auth/sync failures, or crashes; use log fields (component, worker, status, message) to narrow the blast radius.
+    summary: 'Staging Zero: error/fatal logs in the last evaluation window.'
+  execErrState: Ok
+  expressions:
+    A:
+      datasourceUID: grafanacloud-logs
+      model:
+        datasource:
+          type: loki
+          uid: grafanacloud-logs
+        editorMode: code
+        expr: sum(count_over_time({service_name=~"zero-(rm|vs)", deployment_environment="staging"}
+          | detected_level=~"error|fatal" [5m]))
+        instant: true
+        intervalMs: 1000
+        maxDataPoints: 43200
+        queryType: instant
+        refId: A
+      queryType: instant
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 5m0s
+  labels:
+    service: zero
+    severity: warning
+  noDataState: Ok
+  paused: true
+  title: Zero error log (staging)
+  trigger:
+    interval: 1m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/dfjn3qe3l7w8wc.yaml

@@ -0,0 +1,69 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-21T09:06:29Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-alerts
+    grafana.com/group-index: '1'
+  name: dfjn3qe3l7w8wc
+  namespace: stacks-890472
+spec:
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: grafanacloud-prom
+      model:
+        editorMode: code
+        expr: zero_replica_backup_lag_millisecond{deployment_environment="production",
+          service_name="zero-rm"}
+        instant: true
+        intervalMs: 1000
+        legendFormat: __auto
+        maxDataPoints: 43200
+        range: false
+        refId: A
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 1200000
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 20m0s
+  labels:
+    service: zero
+    severity: critical
+  noDataState: NoData
+  notificationSettings:
+    receiver: Discord
+    type: SimplifiedRouting
+  title: Zero replica backup lag
+  trigger:
+    interval: 20m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/dfjn9xellhibka.yaml

@@ -0,0 +1,72 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-28T08:48:46Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-fly-alerts
+    grafana.com/group-index: '1'
+  name: dfjn9xellhibka
+  namespace: stacks-890472
+spec:
+  annotations:
+    summary: Zero container memory utilization exceeds 90%
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: afj5re4yk9og0e
+      model:
+        datasource:
+          type: prometheus
+          uid: afj5re4yk9og0e
+        editorMode: code
+        expr: 1 - (fly_instance_memory_mem_available{app=~"production-zero-rm|production-zero-vs"}
+          / fly_instance_memory_mem_total{app=~"production-zero-rm|production-zero-vs"})
+        instant: true
+        intervalMs: 1000
+        legendFormat: __auto
+        maxDataPoints: 43200
+        range: false
+        refId: A
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0.9
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 1m0s
+  keepFiringFor: 5m0s
+  labels:
+    service: zero
+    severity: warning
+  noDataState: NoData
+  title: Flyio zero container memory high
+  trigger:
+    interval: 1m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/dotcom-file-do-exceptions.yaml

@@ -0,0 +1,94 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: dotcom
+    grafana.com/provenance: ''
+  labels:
+    grafana.app/folder: dotcom
+    grafana.com/group: dotcom-alerts
+    grafana.com/group-index: '1'
+  name: dotcom-file-do-exceptions
+  namespace: stacks-890472
+spec:
+  annotations:
+    description: TLFileDurableObject threw more than 300 uncaught exceptions in the
+      last hour. Quiet hours run 1-50; the bursts that followed hibernation being
+      enabled on 2026-08-13 run 700-1300. Check the "DO exceptions by namespace"
+      panel on the Cloudflare platform metrics dashboard, then Sentry for the
+      matching source tag.
+    summary: File room durable objects are throwing uncaught exceptions
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: ffuyblsuemj28b
+      model:
+        columns:
+          - selector: sum.requests
+            text: exceptions
+            type: number
+        format: table
+        global_query_id: ''
+        parser: backend
+        refId: A
+        root_selector: data.viewer.accounts[0].durableObjectsInvocationsAdaptiveGroups
+        source: url
+        type: graphql
+        url: https://api.cloudflare.com/client/v4/graphql
+        url_options:
+          body_content_type: application/json
+          body_graphql_query: |-
+            query {
+              viewer {
+                accounts(filter: {accountTag: "c34edc4e76350954b63adebde86d5eb1"}) {
+                  durableObjectsInvocationsAdaptiveGroups(limit: 1, filter: {datetime_geq: "${__from:date:iso}", datetime_leq: "${__to:date:iso}", namespaceId: "5864db4344ac4c55bfd94e81dd25a043", status: "scriptThrewException"}) {
+                    sum { requests }
+                  }
+                }
+              }
+            }
+          body_type: graphql
+          method: POST
+      relativeTimeRange:
+        from: 1h0m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 300
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 10m0s
+  labels:
+    service: dotcom
+    severity: warning
+  # The query filters to scriptThrewException, so a healthy hour returns no rows at
+  # all rather than a zero. NoData here means "nothing threw" — alerting on it would
+  # invert the rule.
+  noDataState: Ok
+  notificationSettings:
+    receiver: Discord
+    type: SimplifiedRouting
+  title: File room DO exceptions
+  trigger:
+    interval: 10m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/dotcom-version-chain-verify-failures.yaml

@@ -0,0 +1,95 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: dotcom
+    grafana.com/provenance: ''
+  labels:
+    grafana.app/folder: dotcom
+    grafana.com/group: version-chain-alerts
+    grafana.com/group-index: '1'
+  name: dotcom-version-chain-verify-failures
+  namespace: stacks-890472
+spec:
+  annotations:
+    description: More than two retired version chains failed to reproduce the state
+      the durable object persisted, in the last hour. Check the "Verify results"
+      panel on the Dotcom events dashboard for the reason. head-mismatch is the
+      serious one — the delta encoding is producing a different board — while
+      missing and legacy-fallback mean the chain did not answer at all. Until the
+      legacy full copies are turned off these versions still read; after that they
+      are unreadable and there is no rebuild path, so this should block the
+      rollout advancing.
+    summary: Version chain reconstructions are not reproducing persisted state
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: bdgu7tk03irr4e
+      model:
+        dateTimeType: DATETIME
+        editorMode: code
+        extrapolate: true
+        format: table
+        formatAs: table
+        intervalFactor: 1
+        # rawQuery must be the boolean, not a copy of the SQL. The dashboard panels
+        # carry the SQL in both fields, but that shape reaches the plugin as
+        # already-interpolated text and dies on "unsupported value type: $from".
+        # Matches 'fail' rather than negating 'ok': a check the durable object
+        # declined to run (outcome 'skipped') is not a failing chain, and on a
+        # board big enough to skip, a negated match would page on every retire.
+        query: |-
+          SELECT count() as fails
+          FROM 'MEASURE'
+          WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+          AND blob1 = 'version_chain_verify'
+          AND blob2 = 'production-tldraw-multiplayer'
+          AND blob3 = 'fail'
+        rawQuery: true
+        refId: A
+        round: 0s
+        skip_comments: true
+        step: ''
+      relativeTimeRange:
+        from: 1h0m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 2
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 10m0s
+  labels:
+    service: dotcom
+    severity: warning
+  # count() with no GROUP BY returns a row holding 0 rather than no rows, so a
+  # healthy hour evaluates a real zero. NoData here means the Analytics Engine
+  # datasource itself is down, which is not a version chain problem.
+  noDataState: Ok
+  notificationSettings:
+    receiver: Discord
+    type: SimplifiedRouting
+  title: Version chain verify failures
+  trigger:
+    interval: 10m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/dotcom-version-chain-write-errors.yaml

@@ -0,0 +1,85 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: dotcom
+    grafana.com/provenance: ''
+  labels:
+    grafana.app/folder: dotcom
+    grafana.com/group: version-chain-alerts
+    grafana.com/group-index: '2'
+  name: dotcom-version-chain-write-errors
+  namespace: stacks-890472
+spec:
+  annotations:
+    description: A chain write failed non-transiently in the last hour. These have
+      already retried three times and fallen back to writing a legacy full copy,
+      so no version is lost and boards are unaffected — but the chain is unhealthy
+      and the rollout should not advance while this fires. Check the "Chain write
+      errors" panel on the Dotcom events dashboard, then Sentry for the underlying
+      exception. Expected rate is zero.
+    summary: Version chain writes are failing and falling back to legacy copies
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: bdgu7tk03irr4e
+      model:
+        dateTimeType: DATETIME
+        editorMode: code
+        extrapolate: true
+        format: table
+        formatAs: table
+        intervalFactor: 1
+        # See the verify-failures rule: rawQuery is the boolean, and the SQL lives
+        # only in `query`, or $from reaches the plugin uninterpolated.
+        query: |-
+          SELECT count() as errors
+          FROM 'MEASURE'
+          WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+          AND blob1 = 'version_chain_error'
+          AND blob2 = 'production-tldraw-multiplayer'
+        rawQuery: true
+        refId: A
+        round: 0s
+        skip_comments: true
+        step: ''
+      relativeTimeRange:
+        from: 1h0m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 10m0s
+  labels:
+    service: dotcom
+    severity: warning
+  noDataState: Ok
+  notificationSettings:
+    receiver: Discord
+    type: SimplifiedRouting
+  title: Version chain write errors
+  trigger:
+    interval: 10m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/e75d2f6c-44cf-54bf-8a74-ced2e045fcb6.yaml

@@ -0,0 +1,116 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: anthropic
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-03-20T11:22:55Z'
+    grafana.com/updatedBy: bf1wptf89bpq8d
+  labels:
+    grafana.app/folder: anthropic
+    grafana.com/group: anthropic
+    grafana.com/group-index: '0'
+  name: e75d2f6c-44cf-54bf-8a74-ced2e045fcb6
+  namespace: stacks-890472
+spec:
+  annotations:
+    description: 'The last completed day cost ${{ $value | humanize }}, more than
+      1.5x the day before. The cost metric updates once a day at about 03:00 UTC,
+      so this describes yesterday.'
+    summary: Anthropic API daily cost spike detected.
+  execErrState: Ok
+  expressions:
+    prometheus_math:
+      model:
+        datasource:
+          IsPrunable: false
+          access: ''
+          apiVersion: ''
+          basicAuth: false
+          basicAuthUser: ''
+          created: '0001-01-01T00:00:00Z'
+          database: ''
+          id: -100
+          isDefault: false
+          jsonData: {}
+          name: __expr__
+          readOnly: false
+          secureJsonData: {}
+          type: __expr__
+          uid: __expr__
+          updated: '0001-01-01T00:00:00Z'
+          url: ''
+          user: ''
+          withCredentials: false
+        expression: is_number($query) || is_nan($query) || is_inf($query)
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: prometheus_math
+        type: math
+      queryType: math
+    query:
+      datasourceUID: grafanacloud-prom
+      model:
+        datasource:
+          type: prometheus
+          uid: grafanacloud-prom
+        expr: |
+          (
+            sum(increase(gen_ai_cost{job=~"integrations/anthropic/.*"}[1d])) / 100
+            >
+            1.5 * sum(increase(gen_ai_cost{job=~"integrations/anthropic/.*"}[1d] offset 1d)) / 100
+          )
+          and
+          sum(increase(gen_ai_cost{job=~"integrations/anthropic/.*"}[1d])) / 100 > 50
+        instant: true
+        intervalMs: 1000
+        maxDataPoints: 43200
+        range: false
+        refId: query
+      queryType: prometheus
+      relativeTimeRange:
+        from: 11m0s
+        to: 1m0s
+    threshold:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0
+              type: gt
+        datasource:
+          IsPrunable: false
+          access: ''
+          apiVersion: ''
+          basicAuth: false
+          basicAuthUser: ''
+          created: '0001-01-01T00:00:00Z'
+          database: ''
+          id: -100
+          isDefault: false
+          jsonData: {}
+          name: __expr__
+          readOnly: false
+          secureJsonData: {}
+          type: __expr__
+          uid: __expr__
+          updated: '0001-01-01T00:00:00Z'
+          url: ''
+          user: ''
+          withCredentials: false
+        expression: prometheus_math
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: threshold
+        type: threshold
+      queryType: threshold
+      source: true
+  for: 5m0s
+  labels:
+    __converted_prometheus_rule__: 'true'
+    severity: warning
+  missingSeriesEvalsToResolve: 10
+  noDataState: Ok
+  title: AnthropicDailyCostSpike
+  trigger:
+    interval: 1m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/eec2ad9e-30b9-5c8f-802f-53e26ab8fdc9.yaml

@@ -0,0 +1,110 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: anthropic
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-03-20T11:22:55Z'
+    grafana.com/updatedBy: bf1wptf89bpq8d
+  labels:
+    grafana.app/folder: anthropic
+    grafana.com/group: anthropic
+    grafana.com/group-index: '2'
+  name: eec2ad9e-30b9-5c8f-802f-53e26ab8fdc9
+  namespace: stacks-890472
+spec:
+  annotations:
+    description: 'The last completed day cost ${{ $value | humanize }}, above the
+      $1000 threshold. The cost metric updates once a day at about 03:00 UTC, so
+      this describes yesterday.'
+    summary: Anthropic API cost threshold exceeded.
+  execErrState: Ok
+  expressions:
+    prometheus_math:
+      model:
+        datasource:
+          IsPrunable: false
+          access: ''
+          apiVersion: ''
+          basicAuth: false
+          basicAuthUser: ''
+          created: '0001-01-01T00:00:00Z'
+          database: ''
+          id: -100
+          isDefault: false
+          jsonData: {}
+          name: __expr__
+          readOnly: false
+          secureJsonData: {}
+          type: __expr__
+          uid: __expr__
+          updated: '0001-01-01T00:00:00Z'
+          url: ''
+          user: ''
+          withCredentials: false
+        expression: is_number($query) || is_nan($query) || is_inf($query)
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: prometheus_math
+        type: math
+      queryType: math
+    query:
+      datasourceUID: grafanacloud-prom
+      model:
+        datasource:
+          type: prometheus
+          uid: grafanacloud-prom
+        expr: |
+          sum(increase(gen_ai_cost{job=~"integrations/anthropic/.*"}[1d])) / 100 > 1000
+        instant: true
+        intervalMs: 1000
+        maxDataPoints: 43200
+        range: false
+        refId: query
+      queryType: prometheus
+      relativeTimeRange:
+        from: 11m0s
+        to: 1m0s
+    threshold:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0
+              type: gt
+        datasource:
+          IsPrunable: false
+          access: ''
+          apiVersion: ''
+          basicAuth: false
+          basicAuthUser: ''
+          created: '0001-01-01T00:00:00Z'
+          database: ''
+          id: -100
+          isDefault: false
+          jsonData: {}
+          name: __expr__
+          readOnly: false
+          secureJsonData: {}
+          type: __expr__
+          uid: __expr__
+          updated: '0001-01-01T00:00:00Z'
+          url: ''
+          user: ''
+          withCredentials: false
+        expression: prometheus_math
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: threshold
+        type: threshold
+      queryType: threshold
+      source: true
+  for: 15m0s
+  labels:
+    __converted_prometheus_rule__: 'true'
+    severity: critical
+  missingSeriesEvalsToResolve: 10
+  noDataState: Ok
+  title: AnthropicHighCostThreshold
+  trigger:
+    interval: 1m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/efjn4hgdlhc00f.yaml

@@ -0,0 +1,66 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-21T09:07:11Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-alerts-fast
+    grafana.com/group-index: '1'
+  name: efjn4hgdlhc00f
+  namespace: stacks-890472
+spec:
+  execErrState: Error
+  expressions:
+    A:
+      datasourceUID: grafanacloud-prom
+      model:
+        editorMode: code
+        expr: zero_replication_total_lag_millisecond{deployment_environment="production",
+          service_name="zero-rm"}
+        instant: true
+        intervalMs: 1000
+        legendFormat: __auto
+        maxDataPoints: 43200
+        range: false
+        refId: A
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 2000
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  for: 5m0s
+  labels:
+    service: zero
+    severity: critical
+  noDataState: NoData
+  title: Zero replication lag high
+  trigger:
+    interval: 5m

internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/efjnbq9h4iha8a.yaml

@@ -0,0 +1,69 @@
+apiVersion: rules.alerting.grafana.app/v0alpha1
+kind: AlertRule
+metadata:
+  annotations:
+    grafana.app/folder: f6hsw6
+    grafana.com/provenance: ''
+    grafana.com/updateTimestamp: '2026-04-28T08:48:46Z'
+    grafana.com/updatedBy: ae5t47muqbegwf
+  labels:
+    grafana.app/folder: f6hsw6
+    grafana.com/group: zero-fly-alerts
+    grafana.com/group-index: '4'
+  name: efjnbq9h4iha8a
+  namespace: stacks-890472
+spec:
+  annotations:
+    summary: Zero container OOM kill detected
+  execErrState: Ok
+  expressions:
+    A:
+      datasourceUID: afj5re4yk9og0e
+      model:
+        datasource:
+          type: prometheus
+          uid: afj5re4yk9og0e
+        editorMode: code
+        expr: fly_instance_exit_oom{app=~"production-zero-rm|production-zero-vs"}
+        instant: true
+        intervalMs: 1000
+        legendFormat: __auto
+        maxDataPoints: 43200
+        range: false
+        refId: A
+      relativeTimeRange:
+        from: 10m0s
+        to: 0s
+    C:
+      model:
+        conditions:
+          - evaluator:
+              params:
+                - 0
+              type: gt
+            operator:
+              type: and
+            query:
+              params:
+                - C
+            reducer:
+              params: []
+              type: last
+            type: query
+        datasource:
+          type: __expr__
+          uid: __expr__
+        expression: A
+        intervalMs: 1000
+        maxDataPoints: 43200
+        refId: C
+        type: threshold
+      queryType: expression
+      source: true
+  labels:
+    service: zero
+    severity: critical
+  noDataState: Ok
+  title: Flyio zero container oom kill
+  trigger:
+    interval: 1m

internal/grafana/resources/dashboards.v0alpha1.dashboard.grafana.app/adgumkhou3chsd.yaml

@@ -0,0 +1,254 @@
+apiVersion: dashboard.grafana.app/v0alpha1
+kind: Dashboard
+metadata:
+  annotations:
+    grafana.app/folder: dotcom
+    grafana.app/saved-from-ui: Grafana Cloud
+  labels: {}
+  name: adgumkhou3chsd
+  namespace: stacks-890472
+spec:
+  annotations:
+    list:
+      - builtIn: 1
+        datasource:
+          type: grafana
+          uid: -- Grafana --
+        enable: true
+        hide: true
+        iconColor: rgba(0, 211, 255, 1)
+        name: Annotations & Alerts
+        type: dashboard
+  editable: true
+  fiscalYearStartMonth: 0
+  graphTooltip: 0
+  liveNow: false
+  panels:
+    - datasource:
+        type: vertamedia-clickhouse-datasource
+        uid: bdgu7tk03irr4e
+      fieldConfig:
+        defaults:
+          color:
+            mode: palette-classic
+          custom:
+            axisBorderShow: false
+            axisCenteredZero: false
+            axisColorMode: text
+            axisLabel: ''
+            axisPlacement: auto
+            barAlignment: 0
+            barWidthFactor: 0.6
+            drawStyle: line
+            fillOpacity: 0
+            gradientMode: none
+            hideFrom:
+              legend: false
+              tooltip: false
+              viz: false
+            insertNulls: false
+            lineInterpolation: linear
+            lineWidth: 1
+            pointSize: 5
+            scaleDistribution:
+              type: linear
+            showPoints: auto
+            showValues: false
+            spanNulls: false
+            stacking:
+              group: A
+              mode: none
+            thresholdsStyle:
+              mode: 'off'
+          thresholds:
+            mode: absolute
+            steps:
+              - color: green
+                value: 0
+              - color: red
+                value: 80
+      gridPos:
+        h: 21
+        w: 24
+        x: 0
+        y: 0
+      id: 1
+      options:
+        annotations:
+          clustering: -1
+          multiLane: false
+        legend:
+          calcs: []
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          hideZeros: false
+          maxHeight: 600
+          mode: single
+          sort: none
+      pluginVersion: 13.1.0-24556818486
+      targets:
+        - datasource:
+            type: vertamedia-clickhouse-datasource
+            uid: bdgu7tk03irr4e
+          dateTimeType: DATETIME
+          editorMode: builder
+          extrapolate: true
+          format: time_series
+          formatAs: time_series
+          formattedQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval' SECOND)\
+            \ as date, blob1 as event, count() as total\n\tFROM 'MEASURE'\n    WHERE NOT\
+            \ endsWith(event, 'tldraw-multiplayer')\n\tAND timestamp >= toDateTime($from)\
+            \ AND timestamp <= toDateTime($to)\n\tGROUP BY date, event\n\tORDER BY date\
+            \ ASC"
+          intervalFactor: 1
+          query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval' SECOND) as\
+            \ date, blob1 as event, count() as total\n\tFROM 'MEASURE'\n\tWHERE timestamp\
+            \ >= toDateTime($from) AND timestamp <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+            \tGROUP BY date, event\n\tORDER BY date ASC"
+          rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120' SECOND) as date,\
+            \ blob1 as event, count() as total\n\tFROM 'MEASURE'\n\tWHERE timestamp >=\
+            \ toDateTime(1733213050) AND timestamp <= toDateTime(1733385850)\n\tAND blob2\
+            \ = 'pr-5028-tldraw-multiplayer'\n\tGROUP BY date, event\n\tORDER BY date ASC"
+          refId: A
+          round: 0s
+          showFormattedSQL: true
+          showHelp: true
+          skip_comments: true
+          step: ''
+      title: Room events
+      type: timeseries
+    - datasource:
+        type: vertamedia-clickhouse-datasource
+        uid: bdgu7tk03irr4e
+      fieldConfig:
+        defaults:
+          color:
+            mode: palette-classic
+          custom:
+            axisBorderShow: false
+            axisCenteredZero: false
+            axisColorMode: text
+            axisLabel: ''
+            axisPlacement: right
+            barAlignment: 0
+            barWidthFactor: 0.6
+            drawStyle: line
+            fillOpacity: 0
+            gradientMode: none
+            hideFrom:
+              legend: false
+              tooltip: false
+              viz: false
+            insertNulls: 3600000
+            lineInterpolation: stepAfter
+            lineStyle:
+              fill: solid
+            lineWidth: 1
+            pointSize: 6
+            scaleDistribution:
+              type: linear
+            showPoints: auto
+            showValues: false
+            spanNulls: false
+            stacking:
+              group: A
+              mode: none
+            thresholdsStyle:
+              mode: 'off'
+          min: 0
+          thresholds:
+            mode: absolute
+            steps:
+              - color: green
+                value: 0
+              - color: red
+                value: 80
+      gridPos:
+        h: 8
+        w: 12
+        x: 0
+        y: 21
+      id: 13
+      interval: 5m
+      options:
+        annotations:
+          clustering: -1
+          multiLane: false
+        legend:
+          calcs: []
+          displayMode: table
+          placement: bottom
+          showLegend: false
+        tooltip:
+          hideZeros: false
+          mode: single
+          sort: none
+      pluginVersion: 13.1.0-24556818486
+      targets:
+        - datasource:
+            type: vertamedia-clickhouse-datasource
+            uid: bdgu7tk03irr4e
+          dateTimeType: DATETIME
+          editorMode: builder
+          extrapolate: true
+          format: time_series
+          formatAs: time_series
+          formattedQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval' SECOND)\
+            \ as date, blob1 as name, max(double1) as avg_attempts\n\tFROM 'MEASURE'\n\
+            \tWHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n  \
+            \  AND name = 'persist_success'\n\tGROUP BY date, name\n\tORDER BY date ASC"
+          intervalFactor: 1
+          query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval' SECOND) as\
+            \ date, blob1 as name, max(double1) as avg_attempts\n\tFROM 'MEASURE'\n\t\
+            WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n  \
+            \  AND name = 'persist_success'\n\tGROUP BY date, name\n\tORDER BY date ASC"
+          rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '300' SECOND) as date,\
+            \ blob1 as name, max(double1) as avg_attempts\n\tFROM 'MEASURE'\n\tWHERE timestamp\
+            \ >= toDateTime(1762169567) AND timestamp <= toDateTime(1762180367)\n    AND\
+            \ name = 'persist_success'\n\tGROUP BY date, name\n\tORDER BY date ASC"
+          refId: A
+          round: 0s
+          showFormattedSQL: false
+          showHelp: false
+          skip_comments: true
+          step: ''
+      title: max attempts before persistence success
+      type: timeseries
+  preload: false
+  refresh: 5s
+  schemaVersion: 42
+  templating:
+    list:
+      - current:
+          text: production
+          value: production
+        description: Environment to filter by. Use production, staging, or your pr number
+          (pr-42).
+        hide: 0
+        label: 'Environment: production | staging | pr-42'
+        name: environment
+        query: production
+        skipUrlSync: false
+        type: textbox
+  time:
+    from: now-3h
+    to: now
+  timepicker:
+    refresh_intervals:
+      - 5s
+      - 10s
+      - 30s
+      - 1m
+      - 5m
+      - 15m
+      - 30m
+      - 1h
+      - 2h
+      - 1d
+  timezone: browser
+  title: Dotcom events V1
+  tags:
+    - provisioned
+    - repo:tldraw/tldraw

internal/grafana/resources/dashboards.v0alpha1.dashboard.grafana.app/anthropic-integration-scrape-overview.yaml

@@ -0,0 +1,195 @@
+apiVersion: dashboard.grafana.app/v0alpha1
+kind: Dashboard
+metadata:
+  annotations:
+    grafana.app/folder: anthropic
+    grafana.app/message: creating dashboard from the Cloud Connections plugin
+  labels: {}
+  name: anthropic-integration-scrape-overview
+  namespace: stacks-890472
+spec:
+  editable: false
+  fiscalYearStartMonth: 0
+  graphTooltip: 0
+  id: null
+  links:
+    - asDropdown: false
+      icon: external link
+      includeVars: true
+      keepTime: true
+      tags:
+        - anthropic-integration
+      targetBlank: true
+      title: Dashboards
+      type: dashboards
+  liveNow: false
+  panels:
+    - gridPos:
+        h: 12
+        w: 7
+        x: 0
+        y: 0
+      id: 1
+      options:
+        code:
+          language: plaintext
+          showLineNumbers: false
+          showMiniMap: false
+        content: <center><img style="width:auto;max-height:320px;" src="https://storage.googleapis.com/grafanalabs-integration-logos/anthropic-logo-dark.svg"></center>
+        mode: markdown
+      pluginVersion: 9.4.3
+      title: Logo
+      type: text
+    - datasource:
+        type: prometheus
+        uid: $datasource
+      description: Shows the number of samples received by metric name received for
+        this integration.
+      fieldConfig:
+        defaults:
+          color:
+            mode: palette-classic
+          custom:
+            axisCenteredZero: false
+            axisColorMode: text
+            axisLabel: ''
+            axisPlacement: auto
+            barAlignment: 0
+            drawStyle: line
+            fillOpacity: 0
+            gradientMode: none
+            hideFrom:
+              legend: false
+              tooltip: false
+              viz: false
+            lineInterpolation: linear
+            lineStyle:
+              fill: solid
+            lineWidth: 1
+            pointSize: 5
+            scaleDistribution:
+              type: linear
+            showPoints: auto
+            spanNulls: false
+            stacking:
+              group: A
+              mode: none
+            thresholdsStyle:
+              mode: 'off'
+          mappings: []
+          thresholds:
+            mode: absolute
+            steps:
+              - color: green
+                value: null
+              - color: red
+                value: 80
+          unit: pps
+        overrides: []
+      gridPos:
+        h: 12
+        w: 17
+        x: 7
+        y: 0
+      id: 7
+      options:
+        legend:
+          calcs: []
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: single
+          sort: none
+      targets:
+        - datasource:
+            type: prometheus
+            uid: $datasource
+          editorMode: code
+          expr: count by (__name__) ({__name__=~".+", job=~"integrations/anthropic/.+",
+            scrape_job=~"$scrape_job"})
+          legendFormat: __auto
+          range: true
+          refId: A
+      title: Samples Received
+      type: timeseries
+  refresh: 30s
+  revision: 1
+  schemaVersion: 38
+  style: dark
+  tags:
+    - anthropic-integration
+    - provisioned
+    - repo:tldraw/tldraw
+  templating:
+    list:
+      - current:
+          selected: false
+          text: default
+          value: default
+        hide: 0
+        includeAll: false
+        label: Data source
+        multi: false
+        name: datasource
+        options: []
+        query: prometheus
+        queryValue: ''
+        refresh: 1
+        regex: (?!grafanacloud-usage|grafanacloud-ml-metrics).+
+        skipUrlSync: false
+        type: datasource
+      - allValue: .+
+        current:
+          text: All
+          value:
+            - $__all
+        datasource: $datasource
+        definition: label_values({__name__=~".+", job=~"integrations/anthropic/.+"},
+          scrape_job)
+        hide: 0
+        includeAll: true
+        label: scrape_job
+        multi: true
+        name: scrape_job
+        options: []
+        query: label_values({__name__=~".+", job=~"integrations/anthropic/.+"}, scrape_job)
+        refresh: 2
+        regex: ''
+        skipUrlSync: false
+        sort: 0
+        tagValuesQuery: ''
+        tags: []
+        tagsQuery: ''
+        type: query
+        useTags: false
+      - allValue: integrations/anthropic/.+
+        current:
+          text: All
+          value:
+            - $__all
+        datasource: $datasource
+        definition: label_values({job=~"integrations/anthropic/.+"}, job)
+        hide: 0
+        includeAll: true
+        label: job
+        multi: true
+        name: job
+        options: []
+        query: label_values({job=~"integrations/anthropic/.+"}, job)
+        refresh: 2
+        regex: ''
+        skipUrlSync: false
+        sort: 0
+        tagValuesQuery: ''
+        tags: []
+        tagsQuery: ''
+        type: query
+        useTags: false
+  time:
+    from: now-30m
+    to: now
+  timepicker: {}
+  timezone: ''
+  title: Anthropic scrape overview
+  weekStart: ''

internal/grafana/resources/dashboards.v0alpha1.dashboard.grafana.app/edrigjn435v5sf.yaml

@@ -0,0 +1,375 @@
+apiVersion: dashboard.grafana.app/v0alpha1
+kind: Dashboard
+metadata:
+  annotations:
+    grafana.app/folder: dotcom
+  labels: {}
+  name: edrigjn435v5sf
+  namespace: stacks-890472
+spec:
+  annotations:
+    list:
+      - builtIn: 1
+        datasource:
+          type: grafana
+          uid: -- Grafana --
+        enable: true
+        hide: true
+        iconColor: rgba(0, 211, 255, 1)
+        name: Annotations & Alerts
+        type: dashboard
+  editable: true
+  fiscalYearStartMonth: 0
+  graphTooltip: 0
+  links: []
+  panels:
+    - datasource:
+        type: vertamedia-clickhouse-datasource
+        uid: bdgu7tk03irr4e
+      fieldConfig:
+        defaults:
+          color:
+            mode: thresholds
+          mappings: []
+          thresholds:
+            mode: absolute
+            steps:
+              - color: green
+                value: null
+              - color: red
+                value: 80
+        overrides: []
+      gridPos:
+        h: 8
+        w: 12
+        x: 0
+        y: 0
+      id: 3
+      options:
+        minVizHeight: 75
+        minVizWidth: 75
+        orientation: auto
+        reduceOptions:
+          calcs:
+            - sum
+          fields: ''
+          values: false
+        showThresholdLabels: false
+        showThresholdMarkers: true
+        sizing: auto
+      pluginVersion: 11.3.0-77222
+      targets:
+        - datasource:
+            type: vertamedia-clickhouse-datasource
+            uid: bdgu7tk03irr4e
+          dateTimeType: DATETIME
+          editorMode: builder
+          extrapolate: true
+          format: time_series
+          formatAs: time_series
+          formattedQuery: "SELECT count(*)\n\tFROM 'BEMO_ANALYICS'\n    WHERE blob1 =\
+            \ 'connect'\n\tAND timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n\
+            \tGROUP BY date\n\tORDER BY date ASC"
+          intervalFactor: 1
+          query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval' MINUTE) as\
+            \ date, count() as numConnects\n\tFROM 'BEMO_ANALYTICS'\n    WHERE blob1 =\
+            \ 'connect'\n\tAND timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n\
+            \tGROUP BY date\n\tORDER BY date ASC"
+          rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '1800' MINUTE) as date,\
+            \ count() as numConnects\n\tFROM 'BEMO_ANALYTICS'\n    WHERE blob1 = 'connect'\n\
+            \tAND timestamp >= toDateTime(1722620600) AND timestamp <= toDateTime(1725212600)\n\
+            \tGROUP BY date\n\tORDER BY date ASC"
+          refId: A
+          round: 0s
+          showFormattedSQL: false
+          showHelp: false
+          skip_comments: true
+          step: ''
+      title: Num Connects
+      type: gauge
+    - datasource:
+        type: vertamedia-clickhouse-datasource
+        uid: bdgu7tk03irr4e
+      fieldConfig:
+        defaults:
+          color:
+            mode: palette-classic
+          custom:
+            axisBorderShow: false
+            axisCenteredZero: false
+            axisColorMode: text
+            axisLabel: ''
+            axisPlacement: auto
+            barAlignment: 0
+            barWidthFactor: 0.6
+            drawStyle: line
+            fillOpacity: 0
+            gradientMode: none
+            hideFrom:
+              legend: false
+              tooltip: false
+              viz: false
+            insertNulls: false
+            lineInterpolation: linear
+            lineWidth: 1
+            pointSize: 5
+            scaleDistribution:
+              type: linear
+            showPoints: auto
+            spanNulls: false
+            stacking:
+              group: A
+              mode: none
+            thresholdsStyle:
+              mode: 'off'
+          mappings: []
+          thresholds:
+            mode: absolute
+            steps:
+              - color: green
+                value: null
+              - color: red
+                value: 80
+        overrides: []
+      gridPos:
+        h: 8
+        w: 12
+        x: 12
+        y: 0
+      id: 1
+      options:
+        legend:
+          calcs: []
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: single
+          sort: none
+      pluginVersion: 11.3.0-77222
+      targets:
+        - datasource:
+            type: vertamedia-clickhouse-datasource
+            uid: bdgu7tk03irr4e
+          dateTimeType: DATETIME
+          editorMode: builder
+          extrapolate: true
+          format: time_series
+          formatAs: time_series
+          formattedQuery: "SELECT count(*)\n\tFROM 'BEMO_ANALYICS'\n    WHERE blob1 =\
+            \ 'connect'\n\tAND timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n\
+            \tGROUP BY date\n\tORDER BY date ASC"
+          intervalFactor: 1
+          query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval' MINUTE) as\
+            \ date, count() as numConnects\n\tFROM 'BEMO_ANALYTICS'\n    WHERE blob1 =\
+            \ 'connect'\n\tAND timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n\
+            \tGROUP BY date\n\tORDER BY date ASC"
+          rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '3600' MINUTE) as date,\
+            \ count() as numConnects\n\tFROM 'BEMO_ANALYTICS'\n    WHERE blob1 = 'connect'\n\
+            \tAND timestamp >= toDateTime(1720773373) AND timestamp <= toDateTime(1726292377)\n\
+            \tGROUP BY date\n\tORDER BY date ASC"
+          refId: A
+          round: 0s
+          showFormattedSQL: false
+          showHelp: false
+          skip_comments: true
+          step: ''
+      title: Num Connects
+      type: timeseries
+    - datasource:
+        uid: bdgu7tk03irr4e
+      fieldConfig:
+        defaults:
+          color:
+            mode: palette-classic
+          custom:
+            axisBorderShow: false
+            axisCenteredZero: false
+            axisColorMode: text
+            axisLabel: ''
+            axisPlacement: auto
+            barAlignment: 0
+            barWidthFactor: 0.6
+            drawStyle: line
+            fillOpacity: 0
+            gradientMode: none
+            hideFrom:
+              legend: false
+              tooltip: false
+              viz: false
+            insertNulls: false
+            lineInterpolation: linear
+            lineWidth: 1
+            pointSize: 5
+            scaleDistribution:
+              type: linear
+            showPoints: auto
+            spanNulls: false
+            stacking:
+              group: A
+              mode: none
+            thresholdsStyle:
+              mode: 'off'
+          mappings: []
+          thresholds:
+            mode: absolute
+            steps:
+              - color: green
+                value: null
+              - color: red
+                value: 80
+        overrides: []
+      gridPos:
+        h: 8
+        w: 12
+        x: 0
+        y: 8
+      id: 5
+      options:
+        legend:
+          calcs: []
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: single
+          sort: none
+      pluginVersion: 11.3.0-77222
+      targets:
+        - dateTimeType: DATETIME
+          editorMode: builder
+          extrapolate: true
+          format: time_series
+          formattedQuery: SELECT $timeSeries as t, count() FROM $table WHERE $timeFilter
+            GROUP BY t ORDER BY t
+          intervalFactor: 1
+          query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval' MINUTE) as\
+            \ date, count() as numConnects,\n\tif(\n\t\tstartsWith(blob2, 'http://localhost')\
+            \ or startsWith(blob2, 'http://127.0.0.1'),\n\t\t'localhost',\n\t\tif(\n\t\
+            \t\tendsWith(blob2, '.csb.app'),\n\t\t\t'code sandbox',\n\t\t\tblob2\n\t\t\
+            )\n\t) as host\n\tFROM 'BEMO_ANALYTICS'\n    WHERE blob1 = 'connect'\n\tAND\
+            \ timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n\t-- AND\
+            \ NOT(startsWith(host, 'http://localhost')) AND NOT(startsWith(host, 'http://127.0.0.1'))\n\
+            \tGROUP BY date, host\n    ORDER BY date ASC"
+          rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '30' MINUTE) as date,\
+            \ count() as numConnects,\n\tif(\n\t\tstartsWith(blob2, 'http://localhost')\
+            \ or startsWith(blob2, 'http://127.0.0.1'),\n\t\t'localhost',\n\t\tif(\n\t\
+            \t\tendsWith(blob2, '.csb.app'),\n\t\t\t'code sandbox',\n\t\t\tblob2\n\t\t\
+            )\n\t) as host\n\tFROM 'BEMO_ANALYTICS'\n    WHERE blob1 = 'connect'\n\tAND\
+            \ timestamp >= toDateTime(1728620743) AND timestamp <= toDateTime(1728642343)\n\
+            \t\n\tGROUP BY date, host\n    ORDER BY date ASC"
+          refId: A
+          round: 0s
+          skip_comments: true
+      title: Num Connects
+      type: timeseries
+    - datasource:
+        type: vertamedia-clickhouse-datasource
+        uid: bdgu7tk03irr4e
+      fieldConfig:
+        defaults:
+          color:
+            mode: palette-classic
+          custom:
+            axisBorderShow: false
+            axisCenteredZero: false
+            axisColorMode: text
+            axisLabel: ''
+            axisPlacement: auto
+            barAlignment: 0
+            barWidthFactor: 0.6
+            drawStyle: line
+            fillOpacity: 0
+            gradientMode: none
+            hideFrom:
+              legend: false
+              tooltip: false
+              viz: false
+            insertNulls: false
+            lineInterpolation: smooth
+            lineWidth: 1
+            pointSize: 5
+            scaleDistribution:
+              type: linear
+            showPoints: auto
+            spanNulls: false
+            stacking:
+              group: A
+              mode: none
+            thresholdsStyle:
+              mode: 'off'
+          mappings: []
+          thresholds:
+            mode: absolute
+            steps:
+              - color: green
+                value: null
+              - color: red
+                value: 80
+        overrides: []
+      gridPos:
+        h: 19
+        w: 12
+        x: 12
+        y: 8
+      id: 4
+      options:
+        legend:
+          calcs: []
+          displayMode: table
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: single
+          sort: none
+      pluginVersion: 11.3.0-77222
+      targets:
+        - datasource:
+            type: vertamedia-clickhouse-datasource
+            uid: bdgu7tk03irr4e
+          dateTimeType: DATETIME
+          editorMode: builder
+          extrapolate: true
+          format: time_series
+          formatAs: time_series
+          formattedQuery: "SELECT count() as numConnects, blob2\n\tFROM 'BEMO_ANALYTICS'\n\
+            \    WHERE blob1 = 'connect'\n\tAND timestamp >= toDateTime($from) AND timestamp\
+            \ <= toDateTime($to)\n\tGROUP BY blob2\n    ORDER BY numConnects ASC"
+          intervalFactor: 1
+          query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval' MINUTE) as\
+            \ date, count() as numConnects,\n\tif(\n\t\tstartsWith(blob2, 'http://localhost')\
+            \ or startsWith(blob2, 'http://127.0.0.1'),\n\t\t'localhost',\n\t\tif(\n\t\
+            \t\tendsWith(blob2, '.csb.app'),\n\t\t\t'code sandbox',\n\t\t\tblob2\n\t\t\
+            )\n\t) as host\n\tFROM 'BEMO_ANALYTICS'\n    WHERE blob1 = 'connect'\n\tAND\
+            \ timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n\t-- AND\
+            \ NOT(startsWith(host, 'http://localhost')) AND NOT(startsWith(host, 'http://127.0.0.1'))\n\
+            \tGROUP BY date, host\n    ORDER BY date ASC"
+          rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '3600' MINUTE) as date,\
+            \ count() as numConnects,\n\tif(\n\t\tstartsWith(blob2, 'http://localhost')\
+            \ or startsWith(blob2, 'http://127.0.0.1'),\n\t\t'localhost',\n\t\tif(\n\t\
+            \t\tendsWith(blob2, '.csb.app'),\n\t\t\t'code sandbox',\n\t\t\tblob2\n\t\t\
+            )\n\t) as host\n\tFROM 'BEMO_ANALYTICS'\n    WHERE blob1 = 'connect'\n\tAND\
+            \ timestamp >= toDateTime(1720773373) AND timestamp <= toDateTime(1726292377)\n\
+            \t\n\tGROUP BY date, host\n    ORDER BY date ASC"
+          refId: A
+          round: 0s
+          showFormattedSQL: false
+          showHelp: false
+          skip_comments: true
+          step: ''
+      title: Panel Title
+      type: timeseries
+  preload: false
+  schemaVersion: 40
+  tags:
+    - provisioned
+    - repo:tldraw/tldraw
+  templating:
+    list: []
+  time:
+    from: now-6h
+    to: now
+  timepicker: {}
+  timezone: browser
+  title: Bemo analytics
+  weekStart: ''

internal/grafana/resources/dashboards.v0alpha1.dashboard.grafana.app/slow-queries.yaml

@@ -0,0 +1,605 @@
+apiVersion: dashboard.grafana.app/v0alpha1
+kind: Dashboard
+metadata:
+  annotations:
+    grafana.app/message: Add qtype split, env toggle, deploy annotations and restart
+      panel
+    grafana.app/folder: f6hsw6
+  labels: {}
+  name: slow-queries
+  namespace: stacks-890472
+spec:
+  annotations:
+    list:
+      - datasource:
+          type: loki
+          uid: grafanacloud-logs
+        enable: true
+        expr: '{service_name=~"$service", deployment_environment="$env"} |= "OpenTelemetry
+          SDK started" | process_worker="reaper"'
+        hide: false
+        iconColor: red
+        name: Deploys (machine restarts)
+        target:
+          expr: '{service_name=~"$service", deployment_environment="$env"} |= "OpenTelemetry
+            SDK started" | process_worker="reaper"'
+          queryType: range
+          refId: Anno
+        textFormat: '{{host_name}}'
+        titleFormat: zero-cache restart
+  editable: true
+  graphTooltip: 1
+  panels:
+    - collapsed: false
+      gridPos:
+        h: 1
+        w: 24
+        x: 0
+        y: 0
+      id: 1
+      panels: []
+      title: Overview
+      type: row
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: Total "Slow SQLite query" log lines from zero-cache in the selected
+        time range.
+      fieldConfig:
+        defaults:
+          unit: short
+        overrides: []
+      gridPos:
+        h: 5
+        w: 4
+        x: 0
+        y: 1
+      id: 2
+      options:
+        colorMode: value
+        graphMode: none
+        reduceOptions:
+          calcs:
+            - lastNotNull
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr: sum(count_over_time({service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" [$__range]))
+          instant: true
+          queryType: instant
+          range: false
+          refId: A
+      title: Slow queries in range
+      type: stat
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: Longest slow query seen in the selected time range.
+      fieldConfig:
+        defaults:
+          unit: ms
+        overrides: []
+      gridPos:
+        h: 5
+        w: 4
+        x: 4
+        y: 1
+      id: 3
+      options:
+        colorMode: value
+        graphMode: none
+        reduceOptions:
+          calcs:
+            - lastNotNull
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr: max(max_over_time({service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" | regexp "Slow SQLite query (?P<duration_ms>[0-9.]+)"
+            | unwrap duration_ms [$__range]))
+          instant: true
+          queryType: instant
+          range: false
+          refId: A
+      title: Max duration in range
+      type: stat
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: Slow SQLite query log lines per interval, by Zero service (zero-vs
+        = view-syncer, zero-rm = replication-manager).
+      fieldConfig:
+        defaults:
+          custom:
+            fillOpacity: 8
+            lineWidth: 1
+            showPoints: never
+          unit: short
+        overrides: []
+      gridPos:
+        h: 10
+        w: 8
+        x: 8
+        y: 1
+      id: 4
+      options:
+        legend:
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: multi
+          sort: desc
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr:
+            sum by (service_name) (count_over_time({service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" [$__auto]))
+          legendFormat: '{{service_name}}'
+          queryType: range
+          refId: A
+      title: Slow query rate
+      type: timeseries
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: Duration parsed from the log line, in ms. Only queries over zero-cache's
+        slow threshold are logged, so these are percentiles of the slow tail, not of
+        all queries.
+      fieldConfig:
+        defaults:
+          custom:
+            fillOpacity: 8
+            lineWidth: 1
+            showPoints: never
+          unit: ms
+        overrides: []
+      gridPos:
+        h: 10
+        w: 8
+        x: 16
+        y: 1
+      id: 5
+      options:
+        legend:
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: multi
+          sort: desc
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr: quantile_over_time(0.5, {service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" | regexp "Slow SQLite query (?P<duration_ms>[0-9.]+)"
+            | unwrap duration_ms [$__auto]) by ()
+          legendFormat: p50
+          queryType: range
+          refId: A
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr: quantile_over_time(0.95, {service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" | regexp "Slow SQLite query (?P<duration_ms>[0-9.]+)"
+            | unwrap duration_ms [$__auto]) by ()
+          legendFormat: p95
+          queryType: range
+          refId: B
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr: quantile_over_time(0.99, {service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" | regexp "Slow SQLite query (?P<duration_ms>[0-9.]+)"
+            | unwrap duration_ms [$__auto]) by ()
+          legendFormat: p99
+          queryType: range
+          refId: C
+      title: Slow query duration percentiles
+      type: timeseries
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: Which Fly machines are producing slow queries.
+      fieldConfig:
+        defaults:
+          custom:
+            fillOpacity: 8
+            lineWidth: 1
+            showPoints: never
+          unit: short
+        overrides: []
+      gridPos:
+        h: 5
+        w: 8
+        x: 0
+        y: 6
+      id: 6
+      options:
+        legend:
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: multi
+          sort: desc
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr:
+            sum by (host_name) (count_over_time({service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" [$__auto]))
+          legendFormat: '{{host_name}}'
+          queryType: range
+          refId: A
+      title: Slow queries by worker host
+      type: timeseries
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: Worker restarts (every zero-cache process logs one line per worker
+        on boot; a fleet-wide burst = a deploy) and post-restart auth failures from
+        the custom-query transform endpoint. Slow-query storms almost always line up
+        with these.
+      fieldConfig:
+        defaults:
+          custom:
+            drawStyle: bars
+            fillOpacity: 15
+            lineWidth: 1
+            showPoints: never
+          unit: short
+        overrides: []
+      gridPos:
+        h: 5
+        w: 16
+        x: 8
+        y: 11
+      id: 15
+      options:
+        legend:
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: multi
+          sort: desc
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr: sum(count_over_time({service_name=~"$service", deployment_environment="$env"}
+            |= "OpenTelemetry SDK started" [$__auto]))
+          legendFormat: worker restarts
+          queryType: range
+          refId: A
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr: sum(count_over_time({service_name=~"$service", deployment_environment="$env"}
+            |= "Connection auth validation failed" [$__auto]))
+          legendFormat: auth 401s on reconnect
+          queryType: range
+          refId: B
+      title: Deploys and reconnect storms
+      type: timeseries
+    - collapsed: false
+      gridPos:
+        h: 1
+        w: 24
+        x: 0
+        y: 16
+      id: 7
+      panels: []
+      title: By statement
+      type: row
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: The sql field is attached as structured metadata on each slow query
+        log line.
+      fieldConfig:
+        defaults:
+          custom:
+            align: auto
+            filterable: false
+            inspect: true
+          unit: short
+        overrides: []
+      gridPos:
+        h: 11
+        w: 12
+        x: 0
+        y: 17
+      id: 8
+      options:
+        showHeader: true
+        sortBy:
+          - desc: true
+            displayName: slow queries
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr:
+            topk(15, sum by (sql) (count_over_time({service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" [$__range])))
+          instant: true
+          queryType: instant
+          range: false
+          refId: A
+      title: Top statements by slow query count
+      transformations:
+        - id: organize
+          options:
+            excludeByName:
+              Time: true
+            renameByName:
+              Value: slow queries
+              'Value #A': slow queries
+      type: table
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: p95 of parsed duration, grouped by statement.
+      fieldConfig:
+        defaults:
+          custom:
+            align: auto
+            filterable: false
+            inspect: true
+          unit: ms
+        overrides: []
+      gridPos:
+        h: 11
+        w: 12
+        x: 12
+        y: 17
+      id: 9
+      options:
+        showHeader: true
+        sortBy:
+          - desc: true
+            displayName: p95 ms
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr:
+            topk(15, quantile_over_time(0.95, {service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" | regexp "Slow SQLite query (?P<duration_ms>[0-9.]+)"
+            | unwrap duration_ms [$__range]) by (sql))
+          instant: true
+          queryType: instant
+          range: false
+          refId: A
+      title: Top statements by p95 duration (ms)
+      transformations:
+        - id: organize
+          options:
+            excludeByName:
+              Time: true
+            renameByName:
+              Value: p95 ms
+              'Value #A': p95 ms
+      type: table
+    - collapsed: false
+      gridPos:
+        h: 1
+        w: 24
+        x: 0
+        y: 28
+      id: 10
+      panels: []
+      title: By table
+      type: row
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: "Table name extracted from the sql structured-metadata field. (Per-client
+        breakdowns are not possible here: more than 500 distinct clientGroupID values
+        per query window exceeds Loki's series limit.)"
+      fieldConfig:
+        defaults:
+          custom:
+            fillOpacity: 8
+            lineWidth: 1
+            showPoints: never
+            stacking:
+              group: A
+              mode: normal
+          unit: short
+        overrides: []
+      gridPos:
+        h: 9
+        w: 12
+        x: 0
+        y: 29
+      id: 11
+      options:
+        legend:
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: multi
+          sort: desc
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr:
+            sum by (tbl) (count_over_time({service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" | label_format tbl=`{{ regexReplaceAll
+            "(?s).*FROM .([a-zA-Z0-9._]+).*" .sql "$1" }}` [$__auto]))
+          legendFormat: '{{tbl}}'
+          queryType: range
+          refId: A
+      title: Slow queries by table
+      type: timeseries
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      description: zero-cache tags each statement event with class/method/type (e.g.
+        Statement/iterate/total).
+      fieldConfig:
+        defaults:
+          custom:
+            fillOpacity: 8
+            lineWidth: 1
+            showPoints: never
+          unit: short
+        overrides: []
+      gridPos:
+        h: 9
+        w: 12
+        x: 12
+        y: 29
+      id: 12
+      options:
+        legend:
+          displayMode: list
+          placement: bottom
+          showLegend: true
+        tooltip:
+          mode: multi
+          sort: desc
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr:
+            sum by (method, type) (count_over_time({service_name=~"$service", deployment_environment="$env"}
+            |~ "Slow SQLite query" | type=~"$qtype" [$__auto]))
+          legendFormat: '{{method}}/{{type}}'
+          queryType: range
+          refId: A
+      title: Slow queries by method / type
+      type: timeseries
+    - collapsed: false
+      gridPos:
+        h: 1
+        w: 24
+        x: 0
+        y: 38
+      id: 13
+      panels: []
+      title: Raw logs
+      type: row
+    - datasource:
+        type: loki
+        uid: grafanacloud-logs
+      gridPos:
+        h: 12
+        w: 24
+        x: 0
+        y: 39
+      id: 14
+      options:
+        dedupStrategy: none
+        enableLogDetails: true
+        showTime: true
+        sortOrder: Descending
+        wrapLogMessage: true
+      targets:
+        - datasource:
+            type: loki
+            uid: grafanacloud-logs
+          editorMode: code
+          expr: '{service_name=~"$service", deployment_environment="$env"} |~ "Slow SQLite
+            query" | type=~"$qtype"'
+          maxLines: 200
+          queryType: range
+          refId: A
+      title: Recent slow query logs
+      type: logs
+  refresh: ''
+  schemaVersion: 42
+  tags:
+    - zero
+    - loki
+    - slow-queries
+    - provisioned
+    - repo:tldraw/tldraw
+  templating:
+    list:
+      - allValue: zero-.+
+        current:
+          text:
+            - All
+          value:
+            - $__all
+        hide: 0
+        includeAll: true
+        label: Zero service
+        multi: true
+        name: service
+        options: []
+        query: zero-vs,zero-rm
+        type: custom
+      - current:
+          text: production
+          value: production
+        hide: 0
+        includeAll: false
+        label: Environment
+        multi: false
+        name: env
+        options:
+          - selected: true
+            text: production
+            value: production
+          - selected: false
+            text: staging
+            value: staging
+        query: production,staging
+        type: custom
+      - allValue: .*
+        current:
+          text: All
+          value: $__all
+        description: total = wall-clock across row iteration (includes syncer pipeline
+          stalls); sqlite = time inside the SQLite engine. Filter to sqlite to see whether
+          the database itself is slow.
+        hide: 0
+        includeAll: true
+        label: Query type
+        multi: false
+        name: qtype
+        options:
+          - selected: true
+            text: All
+            value: $__all
+          - selected: false
+            text: total
+            value: total
+          - selected: false
+            text: sqlite
+            value: sqlite
+        query: total,sqlite
+        type: custom
+  time:
+    from: now-6h
+    to: now
+  timezone: browser
+  title: Zero slow queries

internal/grafana/resources/dashboards.v2.dashboard.grafana.app/mix4slb.yaml

@@ -0,0 +1,819 @@
+apiVersion: dashboard.grafana.app/v2
+kind: Dashboard
+metadata:
+  annotations:
+    grafana.app/folder: dotcom
+    grafana.app/saved-from-ui: Grafana Cloud
+  labels: {}
+  name: mix4slb
+  namespace: stacks-890472
+spec:
+  annotations:
+    - kind: AnnotationQuery
+      spec:
+        builtIn: true
+        enable: true
+        hide: true
+        iconColor: rgba(0, 211, 255, 1)
+        name: Annotations & Alerts
+        query:
+          datasource:
+            name: -- Grafana --
+          group: grafana
+          kind: DataQuery
+          spec: {}
+          version: v0
+  cursorSync: 'Off'
+  editable: true
+  elements:
+    panel-1:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      formattedQuery: "SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, quantileWeighted(0.5)(double1,\
+                        \ _sample_interval) as p50_ms, quantileWeighted(0.95)(double1,\
+                        \ _sample_interval) as p95_ms, quantileWeighted(0.99)(double1,\
+                        \ _sample_interval) as p99_ms\n\tFROM 'MEASURE'\n\tWHERE timestamp\
+                        \ >= toDateTime($from) AND timestamp <= toDateTime($to)\n\t\
+                        AND blob2 = '${environment}-tldraw-multiplayer'\n\tAND blob1\
+                        \ = 'outbox_effect'\n\tGROUP BY date\n\tORDER BY date ASC"
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, quantileWeighted(0.5)(double1, _sample_interval)\
+                        \ as p50_ms, quantileWeighted(0.95)(double1, _sample_interval)\
+                        \ as p95_ms, quantileWeighted(0.99)(double1, _sample_interval)\
+                        \ as p99_ms\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_effect'\n\tGROUP BY date\n\tORDER BY date\
+                        \ ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, quantileWeighted(0.5)(double1, _sample_interval)\
+                        \ as p50_ms, quantileWeighted(0.95)(double1, _sample_interval)\
+                        \ as p95_ms, quantileWeighted(0.99)(double1, _sample_interval)\
+                        \ as p99_ms\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime(1762170000)\
+                        \ AND timestamp <= toDateTime(1762180000)\n\tAND blob2 = 'production-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_effect'\n\tGROUP BY date\n\tORDER BY date\
+                        \ ASC"
+                      round: 0s
+                      showFormattedSQL: true
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: 'End-to-end: time from a file change committing to its effect
+          completing (room notification / publish). double1 = ageMs.'
+        id: 1
+        links: []
+        title: Outbox effect latency (p50 / p95 / p99)
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: right
+                  barAlignment: 0
+                  barWidthFactor: 0.6
+                  drawStyle: line
+                  fillOpacity: 0
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 1
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: auto
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                min: 0
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+                    - color: red
+                      value: 80
+                unit: ms
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                placement: bottom
+                showLegend: true
+              tooltip:
+                hideZeros: false
+                mode: single
+                sort: none
+          version: 13.2.0-30616302309
+    panel-2:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      formattedQuery: "SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, sum(double1) as processed, sum(double2)\
+                        \ as failed\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_drain'\n\tGROUP BY date\n\tORDER BY date\
+                        \ ASC"
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, sum(double1) as processed, sum(double2) as\
+                        \ failed\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_drain'\n\tGROUP BY date\n\tORDER BY date\
+                        \ ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, sum(double1) as processed, sum(double2) as\
+                        \ failed\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime(1762170000)\
+                        \ AND timestamp <= toDateTime(1762180000)\n\tAND blob2 = 'production-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_drain'\n\tGROUP BY date\n\tORDER BY date\
+                        \ ASC"
+                      round: 0s
+                      showFormattedSQL: true
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: Per-drain summary. double1 = rows processed, double2 = rows failed
+          this drain.
+        id: 2
+        links: []
+        title: Outbox throughput (processed / failed)
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: right
+                  barAlignment: 0
+                  barWidthFactor: 0.6
+                  drawStyle: line
+                  fillOpacity: 0
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 1
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: auto
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                min: 0
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+                    - color: red
+                      value: 80
+                unit: short
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                placement: bottom
+                showLegend: true
+              tooltip:
+                hideZeros: false
+                mode: single
+                sort: none
+          version: 13.2.0-30616302309
+    panel-3:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      formattedQuery: "SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, blob4 as command, quantileWeighted(0.95)(double1,\
+                        \ _sample_interval) as p95_ms\n\tFROM 'MEASURE'\n\tWHERE timestamp\
+                        \ >= toDateTime($from) AND timestamp <= toDateTime($to)\n\t\
+                        AND blob2 = '${environment}-tldraw-multiplayer'\n\tAND blob1\
+                        \ = 'outbox_effect'\n\tGROUP BY date, command\n\tORDER BY date\
+                        \ ASC"
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, blob4 as command, quantileWeighted(0.95)(double1,\
+                        \ _sample_interval) as p95_ms\n\tFROM 'MEASURE'\n\tWHERE timestamp\
+                        \ >= toDateTime($from) AND timestamp <= toDateTime($to)\n\t\
+                        AND blob2 = '${environment}-tldraw-multiplayer'\n\tAND blob1\
+                        \ = 'outbox_effect'\n\tGROUP BY date, command\n\tORDER BY date\
+                        \ ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, blob4 as command, quantileWeighted(0.95)(double1,\
+                        \ _sample_interval) as p95_ms\n\tFROM 'MEASURE'\n\tWHERE timestamp\
+                        \ >= toDateTime(1762170000) AND timestamp <= toDateTime(1762180000)\n\
+                        \tAND blob2 = 'production-tldraw-multiplayer'\n\tAND blob1 =\
+                        \ 'outbox_effect'\n\tGROUP BY date, command\n\tORDER BY date\
+                        \ ASC"
+                      round: 0s
+                      showFormattedSQL: true
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: publish/unpublish ride the update command (awaitPersist) and
+          are the slowest. blob3=table, blob4=command.
+        id: 3
+        links: []
+        title: Outbox effect latency p95 by command
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: right
+                  barAlignment: 0
+                  barWidthFactor: 0.6
+                  drawStyle: line
+                  fillOpacity: 0
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 1
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: auto
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                min: 0
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+                    - color: red
+                      value: 80
+                unit: ms
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                placement: bottom
+                showLegend: true
+              tooltip:
+                hideZeros: false
+                mode: single
+                sort: none
+          version: 13.2.0-30616302309
+    panel-4:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      formattedQuery: "SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, quantileWeighted(0.95)(double3,\
+                        \ _sample_interval) as drain_p95_ms\n\tFROM 'MEASURE'\n\tWHERE\
+                        \ timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)\n\
+                        \tAND blob2 = '${environment}-tldraw-multiplayer'\n\tAND blob1\
+                        \ = 'outbox_drain'\n\tGROUP BY date\n\tORDER BY date ASC"
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, quantileWeighted(0.95)(double3, _sample_interval)\
+                        \ as drain_p95_ms\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_drain'\n\tGROUP BY date\n\tORDER BY date\
+                        \ ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, quantileWeighted(0.95)(double3, _sample_interval)\
+                        \ as drain_p95_ms\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime(1762170000)\
+                        \ AND timestamp <= toDateTime(1762180000)\n\tAND blob2 = 'production-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_drain'\n\tGROUP BY date\n\tORDER BY date\
+                        \ ASC"
+                      round: 0s
+                      showFormattedSQL: true
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: double3 = drainMs. Rising with flat throughput = individual effects
+          getting slower.
+        id: 4
+        links: []
+        title: Outbox drain duration p95
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: right
+                  barAlignment: 0
+                  barWidthFactor: 0.6
+                  drawStyle: line
+                  fillOpacity: 0
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 1
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: auto
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                min: 0
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+                    - color: red
+                      value: 80
+                unit: ms
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                placement: bottom
+                showLegend: true
+              tooltip:
+                hideZeros: false
+                mode: single
+                sort: none
+          version: 13.2.0-30616302309
+    panel-5:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      formattedQuery: "SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, blob4 as command, count() as\
+                        \ total\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_effect'\n\tGROUP BY date, command\n\t\
+                        ORDER BY date ASC"
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, blob4 as command, count() as total\n\tFROM\
+                        \ 'MEASURE'\n\tWHERE timestamp >= toDateTime($from) AND timestamp\
+                        \ <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_effect'\n\tGROUP BY date, command\n\t\
+                        ORDER BY date ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, blob4 as command, count() as total\n\tFROM\
+                        \ 'MEASURE'\n\tWHERE timestamp >= toDateTime(1762170000) AND\
+                        \ timestamp <= toDateTime(1762180000)\n\tAND blob2 = 'production-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_effect'\n\tGROUP BY date, command\n\t\
+                        ORDER BY date ASC"
+                      round: 0s
+                      showFormattedSQL: true
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: Volume of each effect type (insert / update / delete).
+        id: 5
+        links: []
+        title: Outbox effects by command
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: right
+                  barAlignment: 0
+                  barWidthFactor: 0.6
+                  drawStyle: line
+                  fillOpacity: 0
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 1
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: auto
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                min: 0
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+                    - color: red
+                      value: 80
+                unit: short
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                placement: bottom
+                showLegend: true
+              tooltip:
+                hideZeros: false
+                mode: single
+                sort: none
+          version: 13.2.0-30616302309
+    panel-6:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      formattedQuery: "SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, blob4 as command, count() as\
+                        \ parked\n\tFROM 'MEASURE'\n\tWHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_parked'\n\tGROUP BY date, command\n\t\
+                        ORDER BY date ASC"
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, blob4 as command, count() as parked\n\tFROM\
+                        \ 'MEASURE'\n\tWHERE timestamp >= toDateTime($from) AND timestamp\
+                        \ <= toDateTime($to)\n\tAND blob2 = '${environment}-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_parked'\n\tGROUP BY date, command\n\t\
+                        ORDER BY date ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, blob4 as command, count() as parked\n\tFROM\
+                        \ 'MEASURE'\n\tWHERE timestamp >= toDateTime(1762170000) AND\
+                        \ timestamp <= toDateTime(1762180000)\n\tAND blob2 = 'production-tldraw-multiplayer'\n\
+                        \tAND blob1 = 'outbox_parked'\n\tGROUP BY date, command\n\t\
+                        ORDER BY date ASC"
+                      round: 0s
+                      showFormattedSQL: true
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: A parked effect (10 failed attempts) is a data-loss event and
+          also fires Sentry. Any line here = investigate.
+        id: 6
+        links: []
+        title: Outbox parked effects (should be 0)
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: right
+                  barAlignment: 0
+                  barWidthFactor: 0.6
+                  drawStyle: line
+                  fillOpacity: 0
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 1
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: auto
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                min: 0
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+                    - color: red
+                      value: 80
+                unit: short
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                placement: bottom
+                showLegend: true
+              tooltip:
+                hideZeros: false
+                mode: single
+                sort: none
+          version: 13.2.0-30616302309
+  layout:
+    kind: GridLayout
+    spec:
+      items:
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-1
+            height: 8
+            width: 12
+            x: 0
+            y: 0
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-2
+            height: 8
+            width: 12
+            x: 12
+            y: 0
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-3
+            height: 8
+            width: 12
+            x: 0
+            y: 8
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-4
+            height: 8
+            width: 12
+            x: 12
+            y: 8
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-5
+            height: 8
+            width: 12
+            x: 0
+            y: 16
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-6
+            height: 8
+            width: 12
+            x: 12
+            y: 16
+  links: []
+  liveNow: false
+  preload: false
+  tags:
+    - dotcom
+    - outbox
+    - sync-worker
+    - provisioned
+    - repo:tldraw/tldraw
+  timeSettings:
+    autoRefresh: ''
+    autoRefreshIntervals:
+      - 5s
+      - 10s
+      - 30s
+      - 1m
+      - 5m
+      - 15m
+      - 30m
+      - 1h
+      - 2h
+      - 1d
+    fiscalYearStartMonth: 0
+    from: now-6h
+    hideTimepicker: false
+    timezone: browser
+    to: now
+  title: File effect outbox
+  variables:
+    - kind: TextVariable
+      spec:
+        current:
+          text: production
+          value: production
+        description: Environment to filter by. Use production, staging, or your pr number
+          (pr-42).
+        hide: dontHide
+        label: 'Environment: production | staging | pr-42'
+        name: environment
+        query: production
+        skipUrlSync: false

internal/grafana/resources/dashboards.v2beta1.dashboard.grafana.app/kotx6wj.yaml

@@ -0,0 +1,940 @@
+apiVersion: dashboard.grafana.app/v2beta1
+kind: Dashboard
+metadata:
+  annotations:
+    grafana.app/folder: dotcom
+    grafana.app/saved-from-ui: Grafana Cloud (steady)
+  labels: {}
+  name: kotx6wj
+  namespace: stacks-890472
+spec:
+  annotations:
+    - kind: AnnotationQuery
+      spec:
+        builtIn: true
+        enable: true
+        hide: true
+        iconColor: rgba(0, 211, 255, 1)
+        name: Annotations & Alerts
+        query:
+          datasource:
+            name: -- Grafana --
+          group: grafana
+          kind: DataQuery
+          spec: {}
+          version: v0
+  cursorSync: 'Off'
+  description: How many sessions have been created with the MCP App
+  editable: true
+  elements:
+    panel-1:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formattedQuery: "  SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, blob1 as event, count() as total\n\
+                        \  FROM 'MCP_ANALYTICS'\n  WHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n  AND blob1 = 'session_start'\n\
+                        \  GROUP BY date, event\n  ORDER BY date ASC\n"
+                      intervalFactor: 1
+                      query: "  SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, blob1 as event, count() as total\n  FROM\
+                        \ 'MCP_ANALYTICS'\n  WHERE timestamp >= toDateTime($from) AND\
+                        \ timestamp <= toDateTime($to)\n  AND blob1 = 'session_start'\n\
+                        \  GROUP BY date, event\n  ORDER BY date ASC\n"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, blob1 as event, count() as total\n  FROM\
+                        \ 'MCP_ANALYTICS'\n  WHERE timestamp >= toDateTime(1772623318)\
+                        \ AND timestamp <= toDateTime(1772796118)\n  AND blob1 = 'session_start'\n\
+                        \  GROUP BY date, event\n  ORDER BY date ASC"
+                      round: 0s
+                      skip_comments: true
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: Sessions are created when you make a request to the MCP worker
+        id: 1
+        links: []
+        title: Sessions by Time
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: auto
+                  barAlignment: 0
+                  barWidthFactor: 0.6
+                  drawStyle: line
+                  fillOpacity: 0
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 1
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: auto
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+              overrides: []
+            options:
+              legend:
+                calcs: []
+                displayMode: list
+                placement: bottom
+                showLegend: true
+              tooltip:
+                hideZeros: false
+                mode: single
+                sort: none
+          version: 13.0.0-22478626928
+    panel-4:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formattedQuery: "  SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, blob2 as tool, count() as total\n\
+                        \    FROM 'MCP_ANALYTICS'\n    WHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n    AND blob1 = 'tool_called'\n\
+                        \    and blob2 = 'exec'\n    GROUP BY date, tool\n    ORDER\
+                        \ BY date ASC\n"
+                      intervalFactor: 1
+                      query: "  SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, blob2 as tool, count() as total\n    FROM\
+                        \ 'MCP_ANALYTICS'\n    WHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n    AND blob1 = 'tool_called'\n\
+                        \    and blob2 = 'exec'\n    GROUP BY date, tool\n    ORDER\
+                        \ BY date ASC\n"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, blob2 as tool, count() as total\n    FROM\
+                        \ 'MCP_ANALYTICS'\n    WHERE timestamp >= toDateTime(1772623318)\
+                        \ AND timestamp <= toDateTime(1772796118)\n    AND blob1 = 'tool_called'\n\
+                        \    and blob2 = 'exec'\n    GROUP BY date, tool\n    ORDER\
+                        \ BY date ASC"
+                      round: 0s
+                      skip_comments: true
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: tool_called events for the exec tool.
+        id: 4
+        links: []
+        title: Tool call (exec)
+        vizConfig:
+          group: stat
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: thresholds
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: blue
+                      value: 0
+              overrides: []
+            options:
+              colorMode: value
+              graphMode: area
+              justifyMode: center
+              orientation: auto
+              percentChangeColorMode: standard
+              reduceOptions:
+                calcs:
+                  - sum
+                fields: ''
+                values: false
+              showPercentChange: false
+              textMode: auto
+              wideLayout: true
+          version: 13.0.0-22478626928
+    panel-2:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formattedQuery: "  SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, blob2 as tool, count() as total\n\
+                        \    FROM 'MCP_ANALYTICS'\n    WHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n    AND blob1 = 'tool_called'\n\
+                        \    and blob2 = 'search'\n    GROUP BY date, tool\n    ORDER\
+                        \ BY date ASC\n"
+                      intervalFactor: 1
+                      query: "  SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, blob2 as tool, count() as total\n    FROM\
+                        \ 'MCP_ANALYTICS'\n    WHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n    AND blob1 = 'tool_called'\n\
+                        \    and blob2 = 'search'\n    GROUP BY date, tool\n\
+                        \    ORDER BY date ASC\n"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, blob2 as tool, count() as total\n    FROM\
+                        \ 'MCP_ANALYTICS'\n    WHERE timestamp >= toDateTime(1772623318)\
+                        \ AND timestamp <= toDateTime(1772796118)\n    AND blob1 = 'tool_called'\n\
+                        \    and blob2 = 'search'\n    GROUP BY date, tool\n\
+                        \    ORDER BY date ASC"
+                      round: 0s
+                      skip_comments: true
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: tool_called events for the search tool.
+        id: 2
+        links: []
+        title: Tool call (search)
+        vizConfig:
+          group: stat
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: thresholds
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+              overrides: []
+            options:
+              colorMode: value
+              graphMode: area
+              justifyMode: center
+              orientation: auto
+              percentChangeColorMode: standard
+              reduceOptions:
+                calcs:
+                  - sum
+                fields: ''
+                values: false
+              showPercentChange: false
+              textMode: auto
+              wideLayout: true
+          version: 13.0.0-22478626928
+    panel-6:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: builder
+                      extrapolate: true
+                      format: time_series
+                      formattedQuery: "  SELECT toStartOfInterval(timestamp, INTERVAL\
+                        \ '$interval' SECOND) as date, blob1 as event, count() as total\n\
+                        \  FROM 'MCP_ANALYTICS'\n  WHERE timestamp >= toDateTime($from)\
+                        \ AND timestamp <= toDateTime($to)\n  AND blob1 = 'session_start'\n\
+                        \  GROUP BY date, event\n  ORDER BY date ASC\n"
+                      intervalFactor: 1
+                      query: "  SELECT toStartOfInterval(timestamp, INTERVAL '$interval'\
+                        \ SECOND) as date, blob1 as event, count() as total\n  FROM\
+                        \ 'MCP_ANALYTICS'\n  WHERE timestamp >= toDateTime($from) AND\
+                        \ timestamp <= toDateTime($to)\n  AND blob1 = 'session_start'\n\
+                        \  GROUP BY date, event\n  ORDER BY date ASC\n"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '120'\
+                        \ SECOND) as date, blob1 as event, count() as total\n  FROM\
+                        \ 'MCP_ANALYTICS'\n  WHERE timestamp >= toDateTime(1772622707)\
+                        \ AND timestamp <= toDateTime(1772795507)\n  AND blob1 = 'session_start'\n\
+                        \  GROUP BY date, event\n  ORDER BY date ASC"
+                      round: 0s
+                      skip_comments: true
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: Sessions are created when you make a request to the MCP worker
+        id: 6
+        links: []
+        title: Total Sessions
+        vizConfig:
+          group: stat
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: thresholds
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+              overrides: []
+            options:
+              colorMode: value
+              graphMode: area
+              justifyMode: auto
+              orientation: auto
+              percentChangeColorMode: standard
+              reduceOptions:
+                calcs:
+                  - sum
+                fields: ''
+                values: false
+              showPercentChange: false
+              textMode: auto
+              wideLayout: true
+          version: 13.0.0-22478626928
+    panel-7:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: code
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'
+                        SECOND) as date, count() as sessions
+
+                        FROM 'MCP_ANALYTICS'
+
+                        WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+
+                        AND blob1 = 'session_start'
+
+                        GROUP BY date
+
+                        ORDER BY date ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'
+                        SECOND) as date, count() as sessions
+
+                        FROM 'MCP_ANALYTICS'
+
+                        WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+
+                        AND blob1 = 'session_start'
+
+                        GROUP BY date
+
+                        ORDER BY date ASC"
+                      round: 0s
+                      showFormattedSQL: false
+                      showHelp: false
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: session_start events from the tldraw MCP server (MCP_ANALYTICS
+          dataset). Requires the MCP_ANALYTICS table in this datasource.
+        id: 7
+        links: []
+        title: MCP sessions started
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: auto
+                  barAlignment: 0
+                  barWidthFactor: 0.8
+                  drawStyle: bars
+                  fillOpacity: 80
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 0
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: never
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                enableFacetedFilter: false
+                overflow: ellipsis
+                placement: bottom
+                showLegend: false
+              tooltip:
+                hideZeros: false
+                maxHeight: 600
+                mode: multi
+                sort: desc
+          version: 13.2.0-29854286369
+    panel-10:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: code
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'
+                        SECOND) as date, count() as sessions
+
+                        FROM 'MCP_ANALYTICS'
+
+                        WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+
+                        AND blob1 = 'session_end'
+
+                        GROUP BY date
+
+                        ORDER BY date ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'
+                        SECOND) as date, count() as sessions
+
+                        FROM 'MCP_ANALYTICS'
+
+                        WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+
+                        AND blob1 = 'session_end'
+
+                        GROUP BY date
+
+                        ORDER BY date ASC"
+                      round: 0s
+                      showFormattedSQL: false
+                      showHelp: false
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: session_end events from the tldraw MCP server (MCP_ANALYTICS
+          dataset). Requires the MCP_ANALYTICS table in this datasource.
+        id: 10
+        links: []
+        title: MCP sessions ended
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: auto
+                  barAlignment: 0
+                  barWidthFactor: 0.8
+                  drawStyle: bars
+                  fillOpacity: 80
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 0
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: never
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                enableFacetedFilter: false
+                overflow: ellipsis
+                placement: bottom
+                showLegend: false
+              tooltip:
+                hideZeros: false
+                maxHeight: 600
+                mode: multi
+                sort: desc
+          version: 13.2.0-29854286369
+    panel-8:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: code
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'
+                        SECOND) as date, blob2 as tool, count() as calls
+
+                        FROM 'MCP_ANALYTICS'
+
+                        WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+
+                        AND blob1 = 'tool_called'
+
+                        GROUP BY date, tool
+
+                        ORDER BY date ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'
+                        SECOND) as date, blob2 as tool, count() as calls
+
+                        FROM 'MCP_ANALYTICS'
+
+                        WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+
+                        AND blob1 = 'tool_called'
+
+                        GROUP BY date, tool
+
+                        ORDER BY date ASC"
+                      round: 0s
+                      showFormattedSQL: false
+                      showHelp: false
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: tool_called events split by tool name (search, exec).
+        id: 8
+        links: []
+        title: MCP tool calls by tool
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: auto
+                  barAlignment: 0
+                  barWidthFactor: 0.8
+                  drawStyle: bars
+                  fillOpacity: 80
+                  gradientMode: none
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 0
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: never
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: normal
+                  thresholdsStyle:
+                    mode: 'off'
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                enableFacetedFilter: false
+                overflow: ellipsis
+                placement: bottom
+                showLegend: true
+              tooltip:
+                hideZeros: false
+                maxHeight: 600
+                mode: multi
+                sort: desc
+          version: 13.2.0-29854286369
+    panel-9:
+      kind: Panel
+      spec:
+        data:
+          kind: QueryGroup
+          spec:
+            queries:
+              - kind: PanelQuery
+                spec:
+                  hidden: false
+                  query:
+                    datasource:
+                      name: bdgu7tk03irr4e
+                    group: vertamedia-clickhouse-datasource
+                    kind: DataQuery
+                    spec:
+                      dateTimeType: DATETIME
+                      editorMode: code
+                      extrapolate: true
+                      format: time_series
+                      formatAs: time_series
+                      intervalFactor: 1
+                      query: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'
+                        SECOND) as date, count() as reads
+
+                        FROM 'MCP_ANALYTICS'
+
+                        WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+
+                        AND blob1 = 'resource_called'
+
+                        GROUP BY date
+
+                        ORDER BY date ASC"
+                      rawQuery: "SELECT toStartOfInterval(timestamp, INTERVAL '$interval'
+                        SECOND) as date, count() as reads
+
+                        FROM 'MCP_ANALYTICS'
+
+                        WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
+
+                        AND blob1 = 'resource_called'
+
+                        GROUP BY date
+
+                        ORDER BY date ASC"
+                      round: 0s
+                      showFormattedSQL: false
+                      showHelp: false
+                      skip_comments: true
+                      step: ''
+                    version: v0
+                  refId: A
+            queryOptions: {}
+            transformations: []
+        description: resource_called events for the tldraw-canvas UI resource.
+        id: 9
+        links: []
+        title: MCP canvas resource reads
+        vizConfig:
+          group: timeseries
+          kind: VizConfig
+          spec:
+            fieldConfig:
+              defaults:
+                color:
+                  mode: palette-classic
+                custom:
+                  axisBorderShow: false
+                  axisCenteredZero: false
+                  axisColorMode: text
+                  axisLabel: ''
+                  axisPlacement: auto
+                  barAlignment: 0
+                  barWidthFactor: 0.8
+                  drawStyle: line
+                  fillOpacity: 25
+                  gradientMode: opacity
+                  hideFrom:
+                    legend: false
+                    tooltip: false
+                    viz: false
+                  insertNulls: false
+                  lineInterpolation: linear
+                  lineWidth: 1.5
+                  pointSize: 5
+                  scaleDistribution:
+                    type: linear
+                  showPoints: never
+                  showValues: false
+                  spanNulls: false
+                  stacking:
+                    group: A
+                    mode: none
+                  thresholdsStyle:
+                    mode: 'off'
+                thresholds:
+                  mode: absolute
+                  steps:
+                    - color: green
+                      value: 0
+              overrides: []
+            options:
+              annotations:
+                clustering: -1
+                multiLane: false
+              legend:
+                calcs: []
+                displayMode: list
+                enableFacetedFilter: false
+                overflow: ellipsis
+                placement: bottom
+                showLegend: false
+              tooltip:
+                hideZeros: false
+                maxHeight: 600
+                mode: multi
+                sort: desc
+          version: 13.2.0-29854286369
+  layout:
+    kind: GridLayout
+    spec:
+      items:
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-6
+            height: 13
+            width: 12
+            x: 0
+            y: 0
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-1
+            height: 13
+            width: 12
+            x: 12
+            y: 0
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-4
+            height: 13
+            width: 12
+            x: 0
+            y: 13
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-2
+            height: 13
+            width: 12
+            x: 12
+            y: 13
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-7
+            height: 8
+            width: 6
+            x: 0
+            y: 26
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-10
+            height: 8
+            width: 6
+            x: 6
+            y: 26
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-8
+            height: 8
+            width: 6
+            x: 12
+            y: 26
+        - kind: GridLayoutItem
+          spec:
+            element:
+              kind: ElementReference
+              name: panel-9
+            height: 8
+            width: 6
+            x: 18
+            y: 26
+  links: []
+  liveNow: false
+  preload: false
+  tags:
+    - provisioned
+    - repo:tldraw/tldraw
+  timeSettings:
+    autoRefresh: ''
+    autoRefreshIntervals:
+      - 5s
+      - 10s
+      - 30s
+      - 1m
+      - 5m
+      - 15m
+      - 30m
+      - 1h
+      - 2h
+      - 1d
+    fiscalYearStartMonth: 0
+    from: now-6h
+    hideTimepicker: false
+    timezone: browser
+    to: now
+  title: MCP app sessions
+  variables: []

internal/grafana/resources/folders.v1.folder.grafana.app/anthropic.yaml

@@ -0,0 +1,9 @@
+apiVersion: folder.grafana.app/v1
+kind: Folder
+metadata:
+  annotations: {}
+  labels: {}
+  name: anthropic
+  namespace: stacks-890472
+spec:
+  title: Anthropic

internal/grafana/resources/folders.v1.folder.grafana.app/dotcom.yaml

@@ -0,0 +1,9 @@
+apiVersion: folder.grafana.app/v1
+kind: Folder
+metadata:
+  annotations: {}
+  labels: {}
+  name: dotcom
+  namespace: stacks-890472
+spec:
+  title: Dotcom

internal/grafana/resources/folders.v1.folder.grafana.app/f6hsw6.yaml

@@ -0,0 +1,9 @@
+apiVersion: folder.grafana.app/v1
+kind: Folder
+metadata:
+  annotations: {}
+  labels: {}
+  name: f6hsw6
+  namespace: stacks-890472
+spec:
+  title: Zero

internal/grafana/templates/discord.gotmpl

@@ -0,0 +1,22 @@
+{{ define "tldraw.discord.short" -}}
+{{ if eq .Labels.alertname "DatasourceNoData" }}No metrics: {{ .Labels.rulename }}{{ else if .Annotations.summary }}{{ .Annotations.summary }}{{ else }}{{ .Labels.alertname }}{{ end }}
+{{- end }}
+
+{{ define "tldraw.discord.title" -}}
+{{ if eq .Status "firing" }}🔴{{ else }}✅{{ end }} {{ if gt (len .Alerts) 1 }}[{{ len .Alerts }} alerts] {{ end }}{{ template "tldraw.discord.short" (index .Alerts 0) }}
+{{- end }}
+
+{{ define "tldraw.discord.alert" -}}
+{{ if eq .Labels.alertname "DatasourceNoData" -}}
+⚠️ **No metrics** for **{{ .Labels.rulename }}** (usually the metrics source is down - for Zero rules: Fly.io metrics - not the alert condition itself)
+{{- else -}}
+{{ with .Labels.severity }}**{{ . | toUpper }}** · {{ end }}{{ template "tldraw.discord.short" . }}
+{{- end }} · [View]({{ .GeneratorURL }}){{ with .DashboardURL }} · [Dashboard]({{ . }}){{ end }}
+{{- end }}
+
+{{ define "tldraw.discord.text" -}}
+{{ range .Alerts.Firing }}{{ template "tldraw.discord.alert" . }}
+{{ end -}}
+{{ range .Alerts.Resolved }}✅ Resolved: {{ template "tldraw.discord.alert" . }}
+{{ end -}}
+{{- end }}

internal/grafana/templates/email.gotmpl

@@ -0,0 +1,26 @@
+{{ define "tldraw.email.short" -}}
+{{ if eq .Labels.alertname "DatasourceNoData" }}No metrics: {{ .Labels.rulename }}{{ else if .Annotations.summary }}{{ .Annotations.summary }}{{ else }}{{ .Labels.alertname }}{{ end }}
+{{- end }}
+
+{{ define "tldraw.email.subject" -}}
+{{ if eq .Status "firing" }}🔴{{ else }}✅{{ end }} {{ if gt (len .Alerts) 1 }}[{{ len .Alerts }} alerts] {{ end }}{{ template "tldraw.email.short" (index .Alerts 0) }}
+{{- end }}
+
+{{ define "tldraw.email.alert" -}}
+{{ if eq .Labels.alertname "DatasourceNoData" -}}
+No metrics for {{ .Labels.rulename }} - the datasource returned no series. Usually the metrics source is down (for Zero rules: Fly.io metrics), not the alert condition itself.
+{{ else -}}
+{{ with .Labels.severity }}{{ . | toUpper }} · {{ end }}{{ template "tldraw.email.short" . }}
+{{ end -}}
+View: {{ .GeneratorURL }}
+Silence: {{ .SilenceURL }}
+{{- end }}
+
+{{ define "tldraw.email.message" -}}
+{{ range .Alerts.Firing }}FIRING · {{ template "tldraw.email.alert" . }}
+
+{{ end -}}
+{{ range .Alerts.Resolved }}RESOLVED · {{ template "tldraw.email.alert" . }}
+
+{{ end -}}
+{{- end }}