upstream-seaweedfs/telemetry

SeaweedFS Telemetry System

A privacy-respecting telemetry system for SeaweedFS that collects cluster-level usage statistics and provides visualization through Prometheus and Grafana.

Features

• Privacy-First Design: Uses in-memory cluster IDs (regenerated on restart), no personal data collection
• Prometheus Integration: Native Prometheus metrics for monitoring and alerting
• Grafana Dashboards: Pre-built dashboards for data visualization
• Protocol Buffers: Efficient binary data transmission for optimal performance
• Opt-out: On by default, turned off with a single flag
• Docker Compose: Complete monitoring stack deployment
• Automatic Cleanup: Configurable data retention policies

Architecture

SeaweedFS Cluster → Telemetry Client → Telemetry Server → Prometheus → Grafana
                       (protobuf)         (metrics)      (queries)

Data Transmission

The telemetry system uses Protocol Buffers exclusively for efficient binary data transmission:

• Compact Format: 30-50% smaller than JSON
• Fast Serialization: Better performance than text-based formats
• Type Safety: Strong typing with generated Go structs
• Schema Evolution: Built-in versioning support

Protobuf Schema

message TelemetryData {
  string cluster_id = 1;           // In-memory generated UUID
  string version = 2;              // SeaweedFS version
  string os = 3;                   // Operating system
  // Field 4 reserved (was features)
  // Field 5 reserved (was deployment)
  int32 volume_server_count = 6;   // Number of volume servers
  uint64 total_disk_bytes = 7;     // Total disk usage
  int32 total_volume_count = 8;    // Total volume count
  int32 filer_count = 9;           // Number of filer servers
  int32 broker_count = 10;         // Number of broker servers
  int64 timestamp = 11;            // Collection timestamp
}

Privacy Approach

• No Personal Data: No hostnames, IP addresses, or user information
• In-Memory IDs: Cluster IDs are generated in-memory and change on restart
• Aggregated Data: Only cluster-level statistics, no individual file/user data
• Opt-out Anytime: -telemetry=false on the master stops all reporting
• Transparent: Open source implementation, clear data collection policy

Collected Data

Field	Description	Example
cluster_id	In-memory UUID (changes on restart)	a1b2c3d4-...
version	SeaweedFS version	3.45
os	Operating system and architecture	linux/amd64
volume_server_count	Number of volume servers	5
total_disk_bytes	Total disk usage across cluster	1073741824
total_volume_count	Total number of volumes	120
filer_count	Number of filer servers	2
broker_count	Number of broker servers	1
timestamp	When data was collected	1640995200

A master starts reporting once its cluster stores at least 10 GiB, and the server drops reports under that floor. Every fresh weed server, CI job and throwaway container mints its own cluster id, and at tens of thousands a day they were nearly all of the counted clusters and almost none of the bytes.

Quick Start

1. Deploy Telemetry Server

# Clone and start the complete monitoring stack
git clone https://github.com/seaweedfs/seaweedfs.git
cd seaweedfs
docker compose -f telemetry/docker-compose.yml up -d

# Or run the server directly
cd telemetry/server
go run . -port=8080 -dashboard=true

2. Configure SeaweedFS

# Reporting to telemetry.seaweedfs.com is on by default
weed master

# Send to your own telemetry server instead
weed master -telemetry.url=http://localhost:8080/api/collect

# Turn reporting off
weed master -telemetry=false
weed server -master.telemetry=false

3. Access Dashboards

• Telemetry Server: http://localhost:8080
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/admin)

Configuration

SeaweedFS Master/Server

# Disable telemetry (enabled by default)
-telemetry=false

# Set custom telemetry server URL (optional, defaults to telemetry.seaweedfs.com)
-telemetry.url=http://your-telemetry-server:8080/api/collect

In weed server and weed mini the flags are prefixed: -master.telemetry=false and -master.telemetry.url=....

Telemetry Server

# Server configuration
-port=8080                    # Server port
-dashboard=true               # Enable built-in dashboard
-cleanup=24h                  # Cleanup interval
-max-age=2160h                # Maximum data retention (90 days)
-state-file=data/telemetry-state.json  # Persist state across restarts (empty to disable)
-state-save=1h                # How often to save changed state

# Example
./telemetry-server -port=8080 -dashboard=true -cleanup=24h -max-age=2160h

Prometheus Metrics

The telemetry server exposes these Prometheus metrics:

Cluster Metrics

• seaweedfs_telemetry_total_clusters: Total unique clusters (30 days)
• seaweedfs_telemetry_active_clusters: Active clusters (7 days)
• seaweedfs_telemetry_confirmed_clusters: Active clusters seen on 7+ distinct days — one-shot reports don't count, and the version/OS distributions in /api/stats are computed over these

Per-Cluster Metrics

• seaweedfs_telemetry_volume_servers{cluster_id}: Volume servers per cluster
• seaweedfs_telemetry_disk_bytes{cluster_id}: Disk usage per cluster
• seaweedfs_telemetry_volume_count{cluster_id}: Volume count per cluster
• seaweedfs_telemetry_filer_count{cluster_id}: Filer servers per cluster
• seaweedfs_telemetry_broker_count{cluster_id}: Broker servers per cluster
• seaweedfs_telemetry_cluster_info{cluster_id, version, os}: Cluster metadata

Value gauges are keyed by cluster_id only, so a cluster keeps one continuous series across upgrades. To slice values by version or OS, join with cluster_info, e.g. seaweedfs_telemetry_disk_bytes * on(cluster_id) group_left(version, os) seaweedfs_telemetry_cluster_info.

Server Metrics

• seaweedfs_telemetry_reports_received_total: Total telemetry reports received
• seaweedfs_telemetry_reports_skipped_total: Reports dropped because the cluster stores less than 10 GiB

API Endpoints

Data Collection

# Submit telemetry data (protobuf only)
POST /api/collect
Content-Type: application/x-protobuf
[TelemetryRequest protobuf data]

Statistics (JSON for dashboard/debugging)

# Get aggregated statistics
GET /api/stats

# Get recent cluster instances
GET /api/instances?limit=100

# Get metrics over time
GET /api/metrics?days=30

# Get one cluster's daily usage history (disk bytes, volumes, volume servers)
GET /api/history?cluster_id=<uuid>&days=90

# Get per-cluster disk usage and volume servers over time, largest first,
# the rest summed as "other"
GET /api/cluster-sizes?days=30&limit=20

# Get how many clusters ran each version over time, oldest version first,
# the rest summed as "other"
GET /api/versions?days=30&limit=8

Monitoring

# Prometheus metrics
GET /metrics

Docker Deployment

Complete Stack (Recommended)

# docker-compose.yml
version: '3.8'
services:
  telemetry-server:
    build:
      context: ../
      dockerfile: telemetry/server/Dockerfile
    ports:
      - "8080:8080"
    command: ["-port=8080", "-dashboard=true", "-cleanup=24h"]
    
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning
      - ./grafana-dashboard.json:/var/lib/grafana/dashboards/seaweedfs.json

# Deploy the stack
docker compose -f telemetry/docker-compose.yml up -d

# Scale telemetry server if needed
docker compose -f telemetry/docker-compose.yml up -d --scale telemetry-server=3

Server Only

# Build and run telemetry server (build from repo root to include all sources)
docker build -t seaweedfs-telemetry -f telemetry/server/Dockerfile .
docker run -p 8080:8080 seaweedfs-telemetry -port=8080 -dashboard=true

Development

Protocol Buffer Development

# Generate protobuf code
cd telemetry
protoc --go_out=. --go_opt=paths=source_relative proto/telemetry.proto

# The generated code is already included in the repository

Build from Source

# Build telemetry server
cd telemetry/server
go build -o telemetry-server .

# Build SeaweedFS with telemetry support
cd ../..
go build -o weed ./weed

Testing

# Test telemetry server
cd telemetry/server
go test ./...

# Test protobuf communication (requires protobuf tools)
# See telemetry client code for examples

Grafana Dashboard

The included Grafana dashboard provides:

• Overview: Total and active clusters, version distribution
• Resource Usage: Volume servers and disk usage over time
• Infrastructure: Operating system distribution and server counts
• Growth Trends: Historical growth patterns

Custom Queries

# Total active clusters
seaweedfs_telemetry_active_clusters

# Disk usage per cluster
seaweedfs_telemetry_disk_bytes{cluster_id="<uuid>"}

# Disk usage by version (join with cluster_info for version/os)
sum by (version) (seaweedfs_telemetry_disk_bytes * on(cluster_id) group_left(version) seaweedfs_telemetry_cluster_info)

# Volume servers by operating system
sum by (os) (seaweedfs_telemetry_volume_servers * on(cluster_id) group_left(os) seaweedfs_telemetry_cluster_info)

# Broker servers across all clusters
sum(seaweedfs_telemetry_broker_count)

# Growth rate (weekly)
increase(seaweedfs_telemetry_total_clusters[7d])

Security Considerations

• Network Security: Use HTTPS in production environments
• Access Control: Implement authentication for Grafana and Prometheus
• Data Retention: Configure appropriate retention policies
• Monitoring: Monitor the telemetry infrastructure itself

Troubleshooting

Common Issues

SeaweedFS not sending data:

# Check telemetry configuration
weed master -h | grep telemetry

# Verify connectivity
curl -v http://your-telemetry-server:8080/api/collect

Server not receiving data:

# Check server logs
docker-compose logs telemetry-server

# Verify metrics endpoint
curl http://localhost:8080/metrics

Prometheus not scraping:

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Verify configuration
docker-compose logs prometheus

Debugging

# Enable verbose logging in SeaweedFS
weed master -v=2 -telemetry=true

# Check telemetry server metrics
curl http://localhost:8080/metrics | grep seaweedfs_telemetry

# Test data flow
curl http://localhost:8080/api/stats

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

This telemetry system is part of SeaweedFS and follows the same Apache 2.0 license.