a774ac673a
Some checks failed
Deploy analytics worker / Deploy analytics worker to ${{ (github.ref == 'refs/heads/production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Failing after 1s
i18n upload strings / Upload latest strings (push) Failing after 1s
Deploy bemo / Deploy bemo to ${{ (github.ref == 'refs/heads/bemo-production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Failing after 1s
Deploy grafana resources / ${{ github.event_name == 'push' && 'Deploy grafana resources' || 'Validate grafana resources (dry run)' }} (push) Failing after 0s
End to end tests (dotcom) / Check for relevant changes (push) Failing after 0s
End to end tests (examples) / Check for relevant changes (push) Failing after 0s
End to end tests (mermaid readability) / Check for relevant changes (push) Failing after 1s
End to end tests (perf) / Check for relevant changes (push) Failing after 0s
Publish editor extensions / Publish editor extensions (push) Failing after 1s
Publish SDK packages / Remove the publish-packages label (push) Has been skipped
Checks / Tests & checks (push) Has been cancelled
Checks / Build all projects (push) Has been cancelled
Deploy templates / Deploy templates (push) Has been cancelled
End to end tests (dotcom) / End to end tests (dotcom) (push) Has been cancelled
End to end tests (mermaid readability) / End to end tests (mermaid readability) (push) Has been cancelled
End to end tests (perf) / End to end tests (perf) (push) Has been cancelled
Deploy .com / Deploy dotcom to ${{ (github.ref == 'refs/heads/production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Has been cancelled
End to end tests (examples) / End to end tests (examples) (push) Has been cancelled
Publish SDK packages / Publish SDK packages (push) Has been cancelled
Publish SDK packages / Tell user released version (push) Has been cancelled
Publish SDK packages / Publishes code templates to separate repositories (push) Has been cancelled
Publish SDK packages / Deploys the template demos with the new SDK version (push) Has been cancelled
86 lines
2.4 KiB
YAML
86 lines
2.4 KiB
YAML
apiVersion: rules.alerting.grafana.app/v0alpha1
|
|
kind: AlertRule
|
|
metadata:
|
|
annotations:
|
|
grafana.app/folder: dotcom
|
|
grafana.com/provenance: ''
|
|
labels:
|
|
grafana.app/folder: dotcom
|
|
grafana.com/group: version-chain-alerts
|
|
grafana.com/group-index: '2'
|
|
name: dotcom-version-chain-write-errors
|
|
namespace: stacks-890472
|
|
spec:
|
|
annotations:
|
|
description: A chain write failed non-transiently in the last hour. These have
|
|
already retried three times and fallen back to writing a legacy full copy,
|
|
so no version is lost and boards are unaffected — but the chain is unhealthy
|
|
and the rollout should not advance while this fires. Check the "Chain write
|
|
errors" panel on the Dotcom events dashboard, then Sentry for the underlying
|
|
exception. Expected rate is zero.
|
|
summary: Version chain writes are failing and falling back to legacy copies
|
|
execErrState: Error
|
|
expressions:
|
|
A:
|
|
datasourceUID: bdgu7tk03irr4e
|
|
model:
|
|
dateTimeType: DATETIME
|
|
editorMode: code
|
|
extrapolate: true
|
|
format: table
|
|
formatAs: table
|
|
intervalFactor: 1
|
|
# See the verify-failures rule: rawQuery is the boolean, and the SQL lives
|
|
# only in `query`, or $from reaches the plugin uninterpolated.
|
|
query: |-
|
|
SELECT count() as errors
|
|
FROM 'MEASURE'
|
|
WHERE timestamp >= toDateTime($from) AND timestamp <= toDateTime($to)
|
|
AND blob1 = 'version_chain_error'
|
|
AND blob2 = 'production-tldraw-multiplayer'
|
|
rawQuery: true
|
|
refId: A
|
|
round: 0s
|
|
skip_comments: true
|
|
step: ''
|
|
relativeTimeRange:
|
|
from: 1h0m0s
|
|
to: 0s
|
|
C:
|
|
model:
|
|
conditions:
|
|
- evaluator:
|
|
params:
|
|
- 0
|
|
type: gt
|
|
operator:
|
|
type: and
|
|
query:
|
|
params:
|
|
- C
|
|
reducer:
|
|
params: []
|
|
type: last
|
|
type: query
|
|
datasource:
|
|
type: __expr__
|
|
uid: __expr__
|
|
expression: A
|
|
intervalMs: 1000
|
|
maxDataPoints: 43200
|
|
refId: C
|
|
type: threshold
|
|
queryType: expression
|
|
source: true
|
|
for: 10m0s
|
|
labels:
|
|
service: dotcom
|
|
severity: warning
|
|
noDataState: Ok
|
|
notificationSettings:
|
|
receiver: Discord
|
|
type: SimplifiedRouting
|
|
title: Version chain write errors
|
|
trigger:
|
|
interval: 10m
|