upstream-seaweedfs/telemetry/DEPLOYMENT.md

SeaweedFS Telemetry Server Deployment

This document describes how to deploy the SeaweedFS telemetry server to a remote server using GitHub Actions, or via Docker.

Prerequisites

1. A remote Linux server with:

   • SSH access
   • systemd (for service management)
   • Optional: Prometheus and Grafana (for monitoring)

2. GitHub repository secrets configured (see Setup GitHub Secrets below):

   • TELEMETRY_SSH_PRIVATE_KEY: SSH private key for accessing the remote server
   • TELEMETRY_HOST: Remote server hostname or IP address
   • TELEMETRY_USER: Username for SSH access

Setup GitHub Secrets

Before using the deployment workflow, you need to configure the required secrets in your GitHub repository.

Step 1: Generate SSH Key Pair

On your local machine, generate a new SSH key pair specifically for deployment:

# Generate a new SSH key pair
ssh-keygen -t ed25519 -C "seaweedfs-telemetry-deploy" -f ~/.ssh/seaweedfs_telemetry_deploy

# This creates two files:
# ~/.ssh/seaweedfs_telemetry_deploy     (private key)
# ~/.ssh/seaweedfs_telemetry_deploy.pub (public key)

Step 2: Configure Remote Server

Copy the public key to your remote server:

# Copy public key to remote server
ssh-copy-id -i ~/.ssh/seaweedfs_telemetry_deploy.pub user@your-server.com

# Or manually append to authorized_keys
cat ~/.ssh/seaweedfs_telemetry_deploy.pub | ssh user@your-server.com "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Test the SSH connection:

# Test SSH connection with the new key
ssh -i ~/.ssh/seaweedfs_telemetry_deploy user@your-server.com "echo 'SSH connection successful'"

Step 3: Add Secrets to GitHub Repository

1. Go to your GitHub repository
2. Click on Settings tab
3. In the sidebar, click Secrets and variables → Actions
4. Click New repository secret for each of the following:

TELEMETRY_SSH_PRIVATE_KEY

# Display the private key content
cat ~/.ssh/seaweedfs_telemetry_deploy

• Name: TELEMETRY_SSH_PRIVATE_KEY
• Value: Copy the entire private key content, including the -----BEGIN OPENSSH PRIVATE KEY----- and -----END OPENSSH PRIVATE KEY----- lines

TELEMETRY_HOST

• Name: TELEMETRY_HOST
• Value: Your server's hostname or IP address (e.g., telemetry.example.com or 192.168.1.100)

TELEMETRY_USER

• Name: TELEMETRY_USER
• Value: The username on the remote server (e.g., ubuntu, deploy, or your username)

Step 4: Verify Configuration

Create a simple test workflow or manually trigger the deployment to verify the secrets are working correctly.

Security Best Practices

1. Dedicated SSH Key: Use a separate SSH key only for deployment
2. Limited Permissions: Create a dedicated user on the remote server with minimal required permissions
3. Key Rotation: Regularly rotate SSH keys
4. Server Access: Restrict SSH access to specific IP ranges if possible

Example Server Setup

If you're setting up a new server, here's a basic configuration:

# On the remote server, create a dedicated user for deployment
sudo useradd -m -s /bin/bash seaweedfs-deploy
sudo usermod -aG sudo seaweedfs-deploy  # Only if sudo access is needed

# Switch to the deployment user
sudo su - seaweedfs-deploy

# Create SSH directory
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# Add your public key (paste the content of seaweedfs_telemetry_deploy.pub)
nano ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Troubleshooting

SSH Connection Issues

# Test SSH connection manually
ssh -i ~/.ssh/seaweedfs_telemetry_deploy -v user@your-server.com

# Check SSH key permissions
ls -la ~/.ssh/seaweedfs_telemetry_deploy*
# Should show: -rw------- for private key, -rw-r--r-- for public key

GitHub Actions Fails

1. Check secrets: Ensure all three secrets are properly set in GitHub
2. Verify SSH key: Make sure the entire private key (including headers/footers) is copied
3. Test connectivity: Manually SSH to the server from your local machine
4. Check user permissions: Ensure the remote user has necessary permissions

GitHub Actions Workflow

The deployment workflow (.github/workflows/deploy_telemetry.yml) provides two main operations:

1. First-time Setup

Run this once to set up the remote server:

1. Go to GitHub Actions in your repository
2. Select "Deploy Telemetry Server" workflow
3. Click "Run workflow"
4. Check "Run first-time server setup"
5. Click "Run workflow"

This will:

• Create necessary directories on the remote server
• Set up systemd service configuration
• Configure log rotation
• Upload Grafana dashboard and Prometheus configuration
• Enable the telemetry service (but not start it yet)

Note: The setup only prepares the infrastructure. You need to run a deployment afterward to install and start the telemetry server.

2. Deploy Updates

To deploy updates, manually trigger deployment:

1. Go to GitHub Actions in your repository
2. Select "Deploy Telemetry Server" workflow
3. Click "Run workflow"
4. Check "Deploy telemetry server to remote server"
5. Click "Run workflow"

Docker Deployment

You can build and run the telemetry server using Docker locally or on a remote host.

Build

• Using Docker Compose (recommended):

docker compose -f telemetry/docker-compose.yml build telemetry-server

• Using docker build directly (from the repository root):

docker build -t seaweedfs-telemetry \
  -f telemetry/server/Dockerfile \
  .

Run

• With Docker Compose:

docker compose -f telemetry/docker-compose.yml up -d telemetry-server

• With docker run:

docker run -d --name telemetry-server \
  -p 8080:8080 \
  seaweedfs-telemetry

Notes:

• The container runs as a non-root user by default.
• The image listens on port 8080 inside the container. Map it with -p <host_port>:8080.
• You can pass flags to the server by appending them after the image name, e.g. docker run -d -p 8353:8080 seaweedfs-telemetry -port=8353 -dashboard=false.

Server Directory Structure

After setup, the remote server will have:

~/seaweedfs-telemetry/
├── bin/
│   └── telemetry-server          # Binary executable
├── logs/
│   ├── telemetry.log            # Application logs
│   └── telemetry.error.log      # Error logs
├── data/                        # State snapshots (telemetry-state.json), restored on restart
├── grafana-dashboard.json       # Grafana dashboard configuration
└── prometheus.yml               # Prometheus configuration

Service Management

The telemetry server runs as a systemd service:

# Check service status
sudo systemctl status telemetry.service

# View logs
sudo journalctl -u telemetry.service -f

# Restart service
sudo systemctl restart telemetry.service

# Stop/start service
sudo systemctl stop telemetry.service
sudo systemctl start telemetry.service

Accessing the Service

After deployment, the telemetry server will be available at (default ports shown; adjust if you override with -port):

• Docker default: 8080

  • Dashboard: http://your-server:8080
  • API: http://your-server:8080/api/*
  • Metrics: http://your-server:8080/metrics
  • Health Check: http://your-server:8080/health

• Systemd example (if you configured a different port, e.g. 8353):

  • Dashboard: http://your-server:8353
  • API: http://your-server:8353/api/*
  • Metrics: http://your-server:8353/metrics
  • Health Check: http://your-server:8353/health

Optional: Prometheus and Grafana Integration

Prometheus Setup

1. Install Prometheus on your server
2. Update /etc/prometheus/prometheus.yml to include:

   scrape_configs:
     - job_name: 'seaweedfs-telemetry'
       static_configs:
         - targets: ['localhost:8353']
       metrics_path: '/metrics'
   

Grafana Setup

1. Install Grafana on your server
2. Import the dashboard from ~/seaweedfs-telemetry/grafana-dashboard.json
3. Configure Prometheus as a data source pointing to your Prometheus instance

Troubleshooting

Deployment Fails

1. Check GitHub Actions logs for detailed error messages
2. Verify SSH connectivity: ssh user@host
3. Ensure all required secrets are configured in GitHub

Service Won't Start

1. Check service logs: sudo journalctl -u telemetry.service
2. Verify binary permissions: ls -la ~/seaweedfs-telemetry/bin/
3. Test binary manually: ~/seaweedfs-telemetry/bin/telemetry-server -help

Port Conflicts

If port 8353 is already in use:

1. Edit the systemd service: sudo systemctl edit telemetry.service
2. Add override configuration:

   [Service]
   ExecStart=
   ExecStart=/home/user/seaweedfs-telemetry/bin/telemetry-server -port=8354
   

3. Reload and restart: sudo systemctl daemon-reload && sudo systemctl restart telemetry.service

Security Considerations

1. Firewall: Consider restricting access to telemetry ports
2. SSH Keys: Use dedicated SSH keys with minimal permissions
3. User Permissions: Run the service as a non-privileged user
4. Network: Consider running on internal networks only

Monitoring

Monitor the deployment and service health:

• GitHub Actions: Check workflow runs for deployment status
• System Logs: sudo journalctl -u telemetry.service
• Application Logs: tail -f ~/seaweedfs-telemetry/logs/telemetry.log
• Health Endpoint: curl http://localhost:8353/health
• Metrics: curl http://localhost:8353/metrics