snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

2026-09-16 03:23:42 +05:00
commit 89f5be005d
3997 changed files with 961970 additions and 0 deletions
--- a/telemetry/README.md
+++ b/telemetry/README.md
@@ -0,0 +1,379 @@
+# SeaweedFS Telemetry System
+
+A privacy-respecting telemetry system for SeaweedFS that collects cluster-level usage statistics and provides visualization through Prometheus and Grafana.
+
+## Features
+
+- **Privacy-First Design**: Uses in-memory cluster IDs (regenerated on restart), no personal data collection
+- **Prometheus Integration**: Native Prometheus metrics for monitoring and alerting
+- **Grafana Dashboards**: Pre-built dashboards for data visualization
+- **Protocol Buffers**: Efficient binary data transmission for optimal performance
+- **Opt-out**: On by default, turned off with a single flag
+- **Docker Compose**: Complete monitoring stack deployment
+- **Automatic Cleanup**: Configurable data retention policies
+
+## Architecture
+
+```
+SeaweedFS Cluster → Telemetry Client → Telemetry Server → Prometheus → Grafana
+                       (protobuf)         (metrics)      (queries)
+```
+
+## Data Transmission
+
+The telemetry system uses **Protocol Buffers exclusively** for efficient binary data transmission:
+
+- **Compact Format**: 30-50% smaller than JSON
+- **Fast Serialization**: Better performance than text-based formats
+- **Type Safety**: Strong typing with generated Go structs
+- **Schema Evolution**: Built-in versioning support
+
+### Protobuf Schema
+
+```protobuf
+message TelemetryData {
+  string cluster_id = 1;           // In-memory generated UUID
+  string version = 2;              // SeaweedFS version
+  string os = 3;                   // Operating system
+  // Field 4 reserved (was features)
+  // Field 5 reserved (was deployment)
+  int32 volume_server_count = 6;   // Number of volume servers
+  uint64 total_disk_bytes = 7;     // Total disk usage
+  int32 total_volume_count = 8;    // Total volume count
+  int32 filer_count = 9;           // Number of filer servers
+  int32 broker_count = 10;         // Number of broker servers
+  int64 timestamp = 11;            // Collection timestamp
+}
+```
+
+## Privacy Approach
+
+- **No Personal Data**: No hostnames, IP addresses, or user information
+- **In-Memory IDs**: Cluster IDs are generated in-memory and change on restart
+- **Aggregated Data**: Only cluster-level statistics, no individual file/user data
+- **Opt-out Anytime**: `-telemetry=false` on the master stops all reporting
+- **Transparent**: Open source implementation, clear data collection policy
+
+## Collected Data
+
+| Field | Description | Example |
+|-------|-------------|---------|
+| `cluster_id` | In-memory UUID (changes on restart) | `a1b2c3d4-...` |
+| `version` | SeaweedFS version | `3.45` |
+| `os` | Operating system and architecture | `linux/amd64` |
+| `volume_server_count` | Number of volume servers | `5` |
+| `total_disk_bytes` | Total disk usage across cluster | `1073741824` |
+| `total_volume_count` | Total number of volumes | `120` |
+| `filer_count` | Number of filer servers | `2` |
+| `broker_count` | Number of broker servers | `1` |
+| `timestamp` | When data was collected | `1640995200` |
+
+A master starts reporting once its cluster stores at least 10 GiB, and the server drops reports under that floor. Every fresh `weed server`, CI job and throwaway container mints its own cluster id, and at tens of thousands a day they were nearly all of the counted clusters and almost none of the bytes.
+
+## Quick Start
+
+### 1. Deploy Telemetry Server
+
+```bash
+# Clone and start the complete monitoring stack
+git clone https://github.com/seaweedfs/seaweedfs.git
+cd seaweedfs
+docker compose -f telemetry/docker-compose.yml up -d
+
+# Or run the server directly
+cd telemetry/server
+go run . -port=8080 -dashboard=true
+```
+
+### 2. Configure SeaweedFS
+
+```bash
+# Reporting to telemetry.seaweedfs.com is on by default
+weed master
+
+# Send to your own telemetry server instead
+weed master -telemetry.url=http://localhost:8080/api/collect
+
+# Turn reporting off
+weed master -telemetry=false
+weed server -master.telemetry=false
+```
+
+### 3. Access Dashboards
+
+- **Telemetry Server**: http://localhost:8080
+- **Prometheus**: http://localhost:9090
+- **Grafana**: http://localhost:3000 (admin/admin)
+
+## Configuration
+
+### SeaweedFS Master/Server
+
+```bash
+# Disable telemetry (enabled by default)
+-telemetry=false
+
+# Set custom telemetry server URL (optional, defaults to telemetry.seaweedfs.com)
+-telemetry.url=http://your-telemetry-server:8080/api/collect
+```
+
+In `weed server` and `weed mini` the flags are prefixed: `-master.telemetry=false` and `-master.telemetry.url=...`.
+
+### Telemetry Server
+
+```bash
+# Server configuration
+-port=8080                    # Server port
+-dashboard=true               # Enable built-in dashboard
+-cleanup=24h                  # Cleanup interval
+-max-age=2160h                # Maximum data retention (90 days)
+-state-file=data/telemetry-state.json  # Persist state across restarts (empty to disable)
+-state-save=1h                # How often to save changed state
+
+# Example
+./telemetry-server -port=8080 -dashboard=true -cleanup=24h -max-age=2160h
+```
+
+## Prometheus Metrics
+
+The telemetry server exposes these Prometheus metrics:
+
+### Cluster Metrics
+- `seaweedfs_telemetry_total_clusters`: Total unique clusters (30 days)
+- `seaweedfs_telemetry_active_clusters`: Active clusters (7 days)
+- `seaweedfs_telemetry_confirmed_clusters`: Active clusters seen on 7+ distinct days — one-shot reports don't count, and the version/OS distributions in `/api/stats` are computed over these
+
+### Per-Cluster Metrics
+- `seaweedfs_telemetry_volume_servers{cluster_id}`: Volume servers per cluster
+- `seaweedfs_telemetry_disk_bytes{cluster_id}`: Disk usage per cluster
+- `seaweedfs_telemetry_volume_count{cluster_id}`: Volume count per cluster
+- `seaweedfs_telemetry_filer_count{cluster_id}`: Filer servers per cluster
+- `seaweedfs_telemetry_broker_count{cluster_id}`: Broker servers per cluster
+- `seaweedfs_telemetry_cluster_info{cluster_id, version, os}`: Cluster metadata
+
+Value gauges are keyed by `cluster_id` only, so a cluster keeps one continuous
+series across upgrades. To slice values by version or OS, join with
+`cluster_info`, e.g.
+`seaweedfs_telemetry_disk_bytes * on(cluster_id) group_left(version, os) seaweedfs_telemetry_cluster_info`.
+
+### Server Metrics
+- `seaweedfs_telemetry_reports_received_total`: Total telemetry reports received
+- `seaweedfs_telemetry_reports_skipped_total`: Reports dropped because the cluster stores less than 10 GiB
+
+## API Endpoints
+
+### Data Collection
+```bash
+# Submit telemetry data (protobuf only)
+POST /api/collect
+Content-Type: application/x-protobuf
+[TelemetryRequest protobuf data]
+```
+
+### Statistics (JSON for dashboard/debugging)
+```bash
+# Get aggregated statistics
+GET /api/stats
+
+# Get recent cluster instances
+GET /api/instances?limit=100
+
+# Get metrics over time
+GET /api/metrics?days=30
+
+# Get one cluster's daily usage history (disk bytes, volumes, volume servers)
+GET /api/history?cluster_id=<uuid>&days=90
+
+# Get per-cluster disk usage and volume servers over time, largest first,
+# the rest summed as "other"
+GET /api/cluster-sizes?days=30&limit=20
+
+# Get how many clusters ran each version over time, oldest version first,
+# the rest summed as "other"
+GET /api/versions?days=30&limit=8
+```
+
+### Monitoring
+```bash
+# Prometheus metrics
+GET /metrics
+```
+
+## Docker Deployment
+
+### Complete Stack (Recommended)
+
+```yaml
+# docker-compose.yml
+version: '3.8'
+services:
+  telemetry-server:
+    build:
+      context: ../
+      dockerfile: telemetry/server/Dockerfile
+    ports:
+      - "8080:8080"
+    command: ["-port=8080", "-dashboard=true", "-cleanup=24h"]
+    
+  prometheus:
+    image: prom/prometheus:latest
+    ports:
+      - "9090:9090"
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml
+      
+  grafana:
+    image: grafana/grafana:latest
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+    volumes:
+      - ./grafana-provisioning:/etc/grafana/provisioning
+      - ./grafana-dashboard.json:/var/lib/grafana/dashboards/seaweedfs.json
+```
+
+```bash
+# Deploy the stack
+docker compose -f telemetry/docker-compose.yml up -d
+
+# Scale telemetry server if needed
+docker compose -f telemetry/docker-compose.yml up -d --scale telemetry-server=3
+```
+
+### Server Only
+
+```bash
+# Build and run telemetry server (build from repo root to include all sources)
+docker build -t seaweedfs-telemetry -f telemetry/server/Dockerfile .
+docker run -p 8080:8080 seaweedfs-telemetry -port=8080 -dashboard=true
+```
+
+## Development
+
+### Protocol Buffer Development
+
+```bash
+# Generate protobuf code
+cd telemetry
+protoc --go_out=. --go_opt=paths=source_relative proto/telemetry.proto
+
+# The generated code is already included in the repository
+```
+
+### Build from Source
+
+```bash
+# Build telemetry server
+cd telemetry/server
+go build -o telemetry-server .
+
+# Build SeaweedFS with telemetry support
+cd ../..
+go build -o weed ./weed
+```
+
+### Testing
+
+```bash
+# Test telemetry server
+cd telemetry/server
+go test ./...
+
+# Test protobuf communication (requires protobuf tools)
+# See telemetry client code for examples
+```
+
+## Grafana Dashboard
+
+The included Grafana dashboard provides:
+
+- **Overview**: Total and active clusters, version distribution
+- **Resource Usage**: Volume servers and disk usage over time  
+- **Infrastructure**: Operating system distribution and server counts
+- **Growth Trends**: Historical growth patterns
+
+### Custom Queries
+
+```promql
+# Total active clusters
+seaweedfs_telemetry_active_clusters
+
+# Disk usage per cluster
+seaweedfs_telemetry_disk_bytes{cluster_id="<uuid>"}
+
+# Disk usage by version (join with cluster_info for version/os)
+sum by (version) (seaweedfs_telemetry_disk_bytes * on(cluster_id) group_left(version) seaweedfs_telemetry_cluster_info)
+
+# Volume servers by operating system
+sum by (os) (seaweedfs_telemetry_volume_servers * on(cluster_id) group_left(os) seaweedfs_telemetry_cluster_info)
+
+# Broker servers across all clusters
+sum(seaweedfs_telemetry_broker_count)
+
+# Growth rate (weekly)
+increase(seaweedfs_telemetry_total_clusters[7d])
+```
+
+## Security Considerations
+
+- **Network Security**: Use HTTPS in production environments
+- **Access Control**: Implement authentication for Grafana and Prometheus
+- **Data Retention**: Configure appropriate retention policies
+- **Monitoring**: Monitor the telemetry infrastructure itself
+
+## Troubleshooting
+
+### Common Issues
+
+**SeaweedFS not sending data:**
+```bash
+# Check telemetry configuration
+weed master -h | grep telemetry
+
+# Verify connectivity
+curl -v http://your-telemetry-server:8080/api/collect
+```
+
+**Server not receiving data:**
+```bash
+# Check server logs
+docker-compose logs telemetry-server
+
+# Verify metrics endpoint
+curl http://localhost:8080/metrics
+```
+
+**Prometheus not scraping:**
+```bash
+# Check Prometheus targets
+curl http://localhost:9090/api/v1/targets
+
+# Verify configuration
+docker-compose logs prometheus
+```
+
+### Debugging
+
+```bash
+# Enable verbose logging in SeaweedFS
+weed master -v=2 -telemetry=true
+
+# Check telemetry server metrics
+curl http://localhost:8080/metrics | grep seaweedfs_telemetry
+
+# Test data flow
+curl http://localhost:8080/api/stats
+```
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Add tests if applicable
+5. Submit a pull request
+
+## License
+
+This telemetry system is part of SeaweedFS and follows the same Apache 2.0 license.