# Grafana dashboards and alerts

**TL;DR:** `resources/` is the source of truth for our Grafana Cloud dashboards and alert rules. CI pushes it with `gcx` on every merge to `main` that touches these files. Edit here, not in Grafana — UI edits get overwritten by the next deploy.

This directory currently provisions:

- Eleven dashboards: `Zero on Fly.io — service health` (`zero-fly-health`), `Zero slow queries` (`slow-queries`), and `Zero-cache HTTP edge (Fly.io)` (`ni7hhgd`) in the `Zero` folder; `Anthropic scrape overview` in the `Anthropic` folder; `Dotcom events` (`ni5k8zc`), `Dotcom events V1` (`adgumkhou3chsd`), `File effect outbox`, `MCP server (shared boards)` (`mcp-server`), `MCP app sessions` (`kotx6wj`), and `Bemo analytics` in the `Dotcom` folder; and `Cloudflare platform metrics` (`nivs2rl`), which sits at the root rather than in a folder. The events pipeline currently reports from **staging** — set the environment variable accordingly when a dashboard looks empty.
- All 13 Grafana-managed alert rules (Zero replication/errors/backups, Fly.io container health, Anthropic cost/usage, file room DO exceptions)
- The folders `Zero` (dashboards + 9 rules), `Dotcom` (6 dashboards + 1 rule), and `Anthropic` (1 dashboard + 3 rules, see below)
- The `discord` and `email` notification templates (`templates/*.gotmpl`, defining `tldraw.discord.title` / `tldraw.discord.text` and `tldraw.email.subject` / `tldraw.email.message`), which compress alert notifications to severity + summary + links (no Silence link on Discord: silence URLs carry one matcher per label and blow Discord’s 2000-char message cap on multi-instance alerts, truncating mid-link) and explain `DatasourceNoData` as a likely metrics-source outage. Templates can't go through `gcx resources push` — gcx ignores the `notifications.alerting.grafana.app` API group — so they live in `templates/` instead of `resources/` and the deploy workflow PUTs them through the classic provisioning API. Each contact point's optional Title/Subject and Message fields reference these templates; those references are set by hand (contact points stay hand-managed), so the templates only take effect while a contact point points at them.

There are two MCP dashboards, for two different servers. `MCP server (shared boards)` covers the public server on the sync worker at `POST /app/mcp` (`apps/dotcom/sync-worker`), which reads its protocol metrics from the `MEASURE` dataset. `MCP app sessions` covers the separate tldraw MCP app worker (`apps/mcp-app`) and its own `MCP_ANALYTICS` dataset. Neither one's panels belong on `Dotcom events` — that dashboard covers the sync worker's own events.

`Dotcom events V1` holds only the catch-all `Room events` panel and `max attempts before persistence success`; new panels go on `Dotcom events`.

Contact points, notification policies, and data sources are hand-managed in Grafana on purpose.

## Conventions

Every provisioned dashboard carries the tags `provisioned` and `repo:tldraw/tldraw`, so anyone looking at it in Grafana knows where it's managed from. Add both tags when adopting a new dashboard into this tree.

## Anthropic rules

The three rules in the `Anthropic` folder watch the Grafana Anthropic integration's scrape of the Anthropic Admin API. The two metrics behave very differently, and the rules are shaped around that:

- `gen_ai_usage_tokens_total` is near real time and is labelled by API key, model, and token type (`input`, `output`, `cache_creation_5m`, `cache_creation_1h`, `input_cache_read`). About 70 series at the time of writing. This is the only intraday signal.
- `gen_ai_cost` is **month-to-date** spend in cents, with no API key label, updated **once a day at about 03:00 UTC**. It resets on the 1st, and has had no data at all on the 1st. Anything built on it describes the last completed day, sees a runaway up to 27 hours late, and is blind for a day or two at each month rollover.

`AnthropicTokenBurst` (warning) fires when the summed rate across all series, excluding cache reads, exceeds 1000 tokens/sec over an hour (about 3.6M tokens/hour). Three deliberate choices, each learned the hard way:

- **Summed, not per series.** A per-series rule makes one alert instance per key × model × token type, and every lightly used series has a tiny baseline that any real session clears.
- **No baseline.** Usage is bursty and mostly idle (the median hour is under 10 tokens/sec), so a 7-day average is close to zero and "5x the average" is a normal working hour. #10579 and #10582 tuned the multiplier and floor twice and it still paged on ordinary sessions.
- **Cache reads excluded.** They cost a tenth of uncached input and dominate agent and eval traffic. On 2026-09-01 and 09-02 the rule paged repeatedly on Opus 5 cache reads from the internal evals key, which spent about $50 that day. The floor is chosen so that workload stays quiet (it peaks around 500 non-cache-read tokens/sec) while a big-numbers bulk cold-outreach research run (`tldraw-internal`, `crm-dashboard-api`), roughly 2000 tokens/sec of uncached Sonnet 4.5 input at $20 to $25 an hour and about $500 over 2026-08-21 to 08-22, clears it within the hour. Those runs are expected work, so a page from this rule usually means "check whether someone started a bulk run" rather than "something is broken".

`AnthropicDailyCostSpike` (warning) fires when the last completed day cost more than 1.5x the day before, with a $50 floor. `AnthropicHighCostThreshold` (critical) fires when the last completed day cost more than $1000. Both use `increase(gen_ai_cost[1d])` rather than `offset` subtraction so the monthly reset reads as a counter reset and counts from zero instead of going negative. The scrape gap on the 1st is still a blind spot: the first day of the month reads as roughly $0 because there is nothing to diff against. Because the metric only steps once a day, a firing rule stays firing until the next 03:00 update.

All three are filter queries (`expr > threshold`), so the alert series only exists while the condition holds. `missingSeriesEvalsToResolve: 10` is what stops a single quiet minute from resolving and re-firing them.

## Making a change

1. Edit the files under `resources/`.
2. Open a PR. CI validates the resources and dry-runs the push so you can see what would change. (The alerting API doesn't support server-side dry-run, so alert rules show as "skipped: client-side check only" — that's expected, not an error.)
3. Merge. The deploy workflow (`.github/workflows/deploy-grafana.yml`) pushes to Grafana Cloud.

## Adding a new alert rule

A rule file carrying `grafana.com/group` labels cannot be _created_ by the push. Grafana rejects it with `cannot set group when creating a new rule` — the resource API can only update a rule that is already in a group. Nothing catches this earlier: alerting has no server-side dry-run, so the PR reports the rule as "skipped" and goes green.

Bootstrap the group once by hand, then let the push adopt it:

    gcx api /api/v1/provisioning/folder/<folder uid>/rule-groups/<group> \
      -X PUT -H 'X-Disable-Provenance: true' -d @group.json

`group.json` uses the classic provisioning shape (`{title, folderUid, interval, rules: [...]}`), which is not the shape of the file in this tree: `spec.expressions` becomes a `data` array, the expression carrying `source: true` becomes `condition`, and `spec.trigger.interval` becomes the group's `interval` in seconds. `X-Disable-Provenance: true` is what leaves provenance clear so the push can take ownership afterwards.

`dotcom-file-do-exceptions` was created this way on 2026-08-18, after PR #10052's deploy failed on it.

## Pulling fresh state

If you prototyped a dashboard in the Grafana UI and want to adopt it, pull it into the tree with a local gcx context (`gcx login`):

    gcx resources pull dashboards/<uid> -p internal/grafana/resources -o yaml
    gcx resources pull alertrules -p internal/grafana/resources -o yaml

Note the alert rules selector is `alertrules` — bare `rules` routes to the Asserts product and 403s on our stack.

Don't hand-reformat gcx output; it needs to round-trip. Exception: `zero-fly-health` is stored as JSON because gcx 1.0.0's YAML writer emits an invalid escape for emoji variation selectors in row titles (its own reader then rejects the file). If a pulled YAML file fails `gcx resources validate`, re-pull that resource with `-o json`.

Alert rules created through Grafana's "import Prometheus rules" flow can't be pushed: they carry `converted_prometheus` provenance, and Grafana only accepts writes to a rule from the tool that owns its provenance. To adopt one, strip the `alerting.grafana.app/prometheus-rule-definition` annotation and set `grafana.com/provenance: ''` in the pulled file, then recreate the rule in Grafana with clear provenance so the push can take ownership: delete the imported group (`gcx api /api/convert/prometheus/config/v1/rules/<folder title>/<group> -X DELETE`), recreate it via `gcx api /api/v1/provisioning/folder/<folder uid>/rule-groups/<group> -X PUT -H 'X-Disable-Provenance: true' -d @group.json`, and push. Recreating grouped rules must go through that group endpoint — the resource API gcx pushes with can't add rules to groups, only update rules already in one. The three Anthropic cost rules went through this adoption on 2026-08-10 and are repo-owned now — don't re-run the integration's rule import for them, which would recreate the imported group alongside the adopted one and bring the provenance clash back.

## Deleting a resource

Deleting a file here does **not** delete it in Grafana. Remove the file and run `gcx resources delete <type>/<uid>` manually.

## Service account access

CI authenticates with a Grafana service account token (`GRAFANA_SERVER` / `GRAFANA_TOKEN` repo secrets). The `zero-fly-health` dashboard has restricted permissions: the service account needs an explicit grant on it (or admin role) or pulls and pushes will 403.