upstream-seaweedfs/test/s3/iam/STS_DISTRIBUTED.md

Distributed STS Service for SeaweedFS S3 Gateway

This document explains how to configure and deploy the STS (Security Token Service) for distributed SeaweedFS S3 Gateway deployments with consistent identity provider configurations.

Problem Solved

Previously, identity providers had to be manually registered on each S3 gateway instance, leading to:

• ❌ Inconsistent authentication: Different instances might have different providers
• ❌ Manual synchronization: No guarantee all instances have same provider configs
• ❌ Authentication failures: Users getting different responses from different instances
• ❌ Operational complexity: Difficult to manage provider configurations at scale

Solution: Configuration-Driven Providers

The STS service now supports automatic provider loading from configuration files, ensuring:

• ✅ Consistent providers: All instances load identical providers from config
• ✅ Automatic synchronization: Configuration-driven, no manual registration needed
• ✅ Reliable authentication: Same behavior from all instances
• ✅ Easy management: Update config file, restart services

Configuration Schema

Basic STS Configuration

{
  "sts": {
    "tokenDuration": "1h",
    "maxSessionLength": "12h", 
    "issuer": "seaweedfs-sts",
    "signingKey": "base64-encoded-signing-key-32-chars-min"
  }
}

Note: The STS service uses a stateless JWT design where all session information is embedded directly in the JWT token. No external session storage is required.

Configuration-Driven Providers

{
  "sts": {
    "tokenDuration": "1h",
    "maxSessionLength": "12h",
    "issuer": "seaweedfs-sts",
    "signingKey": "base64-encoded-signing-key",
    "providers": [
      {
        "name": "keycloak-oidc",
        "type": "oidc", 
        "enabled": true,
        "config": {
          "issuer": "https://keycloak.company.com/realms/seaweedfs",
          "clientId": "seaweedfs-s3",
          "clientSecret": "super-secret-key",
          "jwksUri": "https://keycloak.company.com/realms/seaweedfs/protocol/openid-connect/certs",
          "scopes": ["openid", "profile", "email", "roles"],
          "claimsMapping": {
            "usernameClaim": "preferred_username",
            "groupsClaim": "roles"
          }
        }
      },
      {
        "name": "backup-oidc",
        "type": "oidc",
        "enabled": false,
        "config": {
          "issuer": "https://backup-oidc.company.com",
          "clientId": "seaweedfs-backup"
        }
      },
      {
        "name": "dev-mock-provider",
        "type": "mock",
        "enabled": true,
        "config": {
          "issuer": "http://localhost:9999",
          "clientId": "mock-client"
        }
      }
    ]
  }
}

Supported Provider Types

1. OIDC Provider ("type": "oidc")

For production authentication with OpenID Connect providers like Keycloak, Auth0, Google, etc.

Required Configuration:

• issuer: OIDC issuer URL
• clientId: OAuth2 client ID

Optional Configuration:

• clientSecret: OAuth2 client secret (for confidential clients)
• jwksUri: JSON Web Key Set URI (auto-discovered if not provided)
• userInfoUri: UserInfo endpoint URI (auto-discovered if not provided)
• scopes: OAuth2 scopes to request (default: ["openid"])
• claimsMapping: Map OIDC claims to identity attributes

Example:

{
  "name": "corporate-keycloak",
  "type": "oidc",
  "enabled": true,
  "config": {
    "issuer": "https://sso.company.com/realms/production",
    "clientId": "seaweedfs-prod",
    "clientSecret": "confidential-secret", 
    "scopes": ["openid", "profile", "email", "groups"],
    "claimsMapping": {
      "usernameClaim": "preferred_username",
      "groupsClaim": "groups",
      "emailClaim": "email"
    }
  }
}

2. Mock Provider ("type": "mock")

For development, testing, and staging environments.

Configuration:

• issuer: Mock issuer URL (default: http://localhost:9999)
• clientId: Mock client ID

Example:

{
  "name": "dev-mock",
  "type": "mock", 
  "enabled": true,
  "config": {
    "issuer": "http://dev-mock:9999",
    "clientId": "dev-client"
  }
}

Built-in Test Tokens:

• valid_test_token: Returns test user with developer groups
• valid-oidc-token: Compatible with integration tests
• expired_token: Returns token expired error
• invalid_token: Returns invalid token error

3. Future Provider Types

The factory pattern supports easy addition of new provider types:

• "type": "ldap": LDAP/Active Directory authentication
• "type": "saml": SAML 2.0 authentication
• "type": "oauth2": Generic OAuth2 providers
• "type": "custom": Custom authentication backends

Deployment Patterns

Single Instance (Development)

# Standard deployment with config-driven providers
weed s3 -filer=localhost:8888 -port=8333 -iam.config=/path/to/sts_config.json

Multiple Instances (Production)

# Instance 1 
weed s3 -filer=prod-filer:8888 -port=8333 -iam.config=/shared/sts_distributed.json

# Instance 2
weed s3 -filer=prod-filer:8888 -port=8334 -iam.config=/shared/sts_distributed.json

# Instance N
weed s3 -filer=prod-filer:8888 -port=833N -iam.config=/shared/sts_distributed.json

Critical Requirements for Distributed Deployment:

1. Identical Configuration Files: All instances must use the exact same configuration file
2. Same Signing Keys: All instances must have identical signingKey values
3. Same Issuer: All instances must use the same issuer value

Note: STS now uses stateless JWT tokens, eliminating the need for shared session storage.

High Availability Setup

# docker-compose.yml for production deployment
services:
  filer:
    image: seaweedfs/seaweedfs:latest
    command: "filer -master=master:9333"
    volumes:
      - filer-data:/data
    
  s3-gateway-1:
    image: seaweedfs/seaweedfs:latest
    command: "s3 -filer=filer:8888 -port=8333 -iam.config=/config/sts_distributed.json"
    ports:
      - "8333:8333"
    volumes:
      - ./sts_distributed.json:/config/sts_distributed.json:ro
    depends_on: [filer]
    
  s3-gateway-2:
    image: seaweedfs/seaweedfs:latest 
    command: "s3 -filer=filer:8888 -port=8333 -iam.config=/config/sts_distributed.json"
    ports:
      - "8334:8333"
    volumes:
      - ./sts_distributed.json:/config/sts_distributed.json:ro
    depends_on: [filer]
    
  s3-gateway-3:
    image: seaweedfs/seaweedfs:latest
    command: "s3 -filer=filer:8888 -port=8333 -iam.config=/config/sts_distributed.json"
    ports:
      - "8335:8333"
    volumes:
      - ./sts_distributed.json:/config/sts_distributed.json:ro
    depends_on: [filer]
    
  load-balancer:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on: [s3-gateway-1, s3-gateway-2, s3-gateway-3]

Authentication Flow

1. OIDC Authentication Flow

1. User authenticates with OIDC provider (Keycloak, Auth0, etc.)
   ↓
2. User receives OIDC JWT token from provider
   ↓  
3. User calls SeaweedFS STS AssumeRoleWithWebIdentity
   POST /sts/assume-role-with-web-identity
   {
     "RoleArn": "arn:aws:iam::role/S3AdminRole",
     "WebIdentityToken": "eyJ0eXAiOiJKV1QiLCJhbGc...",
     "RoleSessionName": "user-session"
   }
   ↓
4. STS validates OIDC token with configured provider
   - Verifies JWT signature using provider's JWKS
   - Validates issuer, audience, expiration
   - Extracts user identity and groups
   ↓
5. STS checks role trust policy
   - Verifies user/groups can assume the requested role
   - Validates conditions in trust policy
   ↓
6. STS generates temporary credentials
   - Creates temporary access key, secret key, session token
   - Session token is signed JWT with all session information embedded (stateless)
   ↓
7. User receives temporary credentials
   {
     "Credentials": {
       "AccessKeyId": "AKIA...",
       "SecretAccessKey": "base64-secret",
       "SessionToken": "eyJ0eXAiOiJKV1QiLCJhbGc...",
       "Expiration": "2024-01-01T12:00:00Z"
     }
   }
   ↓
8. User makes S3 requests with temporary credentials
   - AWS SDK signs requests with temporary credentials
   - SeaweedFS S3 gateway validates session token
   - Gateway checks permissions via policy engine

2. Cross-Instance Token Validation

User Request → Load Balancer → Any S3 Gateway Instance
                                      ↓
                              Extract JWT Session Token
                                      ↓
                              Validate JWT Token
                              (Self-contained - no external storage needed)
                                      ↓
                              Check Permissions
                              (Shared policy engine)
                                      ↓
                              Allow/Deny Request

Trust Policy Conditions

Step 5 above evaluates the role's trust policy against context keys derived from the OIDC token's claims. The available keys are:

Condition key	Source
oidc:iss	iss claim (issuer URL)
oidc:sub	sub claim
oidc:aud	aud claim
oidc:<claim>	any other token claim, e.g. oidc:roles, oidc:groups, oidc:email
aws:FederatedProvider	the provider name (e.g. keycloak-oidc) when its configured issuer matches the token, otherwise the raw issuer URL
aws:userid	sub claim (same value as oidc:sub during trust-policy evaluation)
sts:DurationSeconds	requested session duration, when supplied

During trust-policy evaluation aws:userid is the raw sub claim. Once the role has been assumed, the keys seen by request authorization differ: there aws:userid is a stable per-identity hash of sub and iss (see ComputeParentUser), so do not assume the two contexts carry the same value.

Custom claims are always exposed under the oidc: prefix, so a trust policy must use oidc:roles (not a bare roles) to match a roles claim:

"Condition": {
  "StringEquals": {
    "oidc:roles": "s3-admin"
  }
}

A multi-valued claim (such as a roles array) matches when any of its values equals the condition value. The same oidc: keys can be interpolated into policy resources, e.g. arn:aws:s3:::bucket/${oidc:sub}/*.

roleMapping vs. trust policy

A provider's roleMapping and a role's trust policy apply to two different entry points and are not interchangeable:

• Direct OIDC — an S3 request carrying Authorization: Bearer <OIDC-JWT>. The gateway applies roleMapping to choose the caller's role from the token claims; the first matching rule (or defaultRole) wins.
• STS AssumeRoleWithWebIdentity — the caller names the role explicitly via RoleArn, and that role's trust policy decides whether the assumption is allowed. roleMapping does not select the role on this path; instead the token claims are surfaced as the oidc: condition keys above for the trust policy to evaluate.

Configuration Management

Development Environment

{
  "sts": {
    "tokenDuration": "1h",
    "maxSessionLength": "12h",
    "issuer": "seaweedfs-dev-sts",
    "signingKey": "ZGV2LXNpZ25pbmcta2V5LTMyLWNoYXJhY3RlcnMtbG9uZw==",
    "providers": [
      {
        "name": "dev-mock",
        "type": "mock",
        "enabled": true,
        "config": {
          "issuer": "http://localhost:9999",
          "clientId": "dev-mock-client"
        }
      }
    ]
  }
}

Production Environment

{
  "sts": {
    "tokenDuration": "1h",
    "maxSessionLength": "12h",
    "issuer": "seaweedfs-prod-sts",
    "signingKey": "cHJvZC1zaWduaW5nLWtleS0zMi1jaGFyYWN0ZXJzLWxvbmctcmFuZG9t",
    "providers": [
      {
        "name": "corporate-sso",
        "type": "oidc",
        "enabled": true,
        "config": {
          "issuer": "https://sso.company.com/realms/production",
          "clientId": "seaweedfs-prod",
          "clientSecret": "${SSO_CLIENT_SECRET}",
          "scopes": ["openid", "profile", "email", "groups"],
          "claimsMapping": {
            "usernameClaim": "preferred_username",
            "groupsClaim": "groups"
          }
        }
      },
      {
        "name": "backup-auth",
        "type": "oidc", 
        "enabled": false,
        "config": {
          "issuer": "https://backup-sso.company.com",
          "clientId": "seaweedfs-backup"
        }
      }
    ]
  }
}

Operational Best Practices

1. Configuration Management

• Version Control: Store configurations in Git with proper versioning
• Environment Separation: Use separate configs for dev/staging/production
• Secret Management: Use environment variable substitution for secrets
• Configuration Validation: Test configurations before deployment

2. Security Considerations

• Signing Key Security: Use strong, randomly generated signing keys (32+ bytes)
• Key Rotation: Implement signing key rotation procedures
• Secret Storage: Store client secrets in secure secret management systems
• TLS Encryption: Always use HTTPS for OIDC providers in production

3. Monitoring and Troubleshooting

• Provider Health: Monitor OIDC provider availability and response times
• Session Metrics: Track active sessions, token validation errors
• Configuration Drift: Alert on configuration inconsistencies between instances
• Authentication Logs: Log authentication attempts for security auditing

4. Capacity Planning

• Provider Performance: Monitor OIDC provider response times and rate limits
• Token Validation: Monitor JWT validation performance and caching
• Memory Usage: Monitor JWT token validation caching and provider metadata

Migration Guide

From Manual Provider Registration

Before (Manual Registration):

// Each instance needs this code
keycloakProvider := oidc.NewOIDCProvider("keycloak-oidc")
keycloakProvider.Initialize(keycloakConfig)
stsService.RegisterProvider(keycloakProvider)

After (Configuration-Driven):

{
  "sts": {
    "providers": [
      {
        "name": "keycloak-oidc",
        "type": "oidc",
        "enabled": true,
        "config": {
          "issuer": "https://keycloak.company.com/realms/seaweedfs",
          "clientId": "seaweedfs-s3"
        }
      }
    ]
  }
}

Migration Steps

1. Create Configuration File: Convert manual provider registrations to JSON config
2. Test Single Instance: Deploy config to one instance and verify functionality
3. Validate Consistency: Ensure all instances load identical providers
4. Rolling Deployment: Update instances one by one with new configuration
5. Remove Manual Code: Clean up manual provider registration code

Troubleshooting

Common Issues

1. Provider Inconsistency

Symptoms: Authentication works on some instances but not others Diagnosis:

# Check provider counts on each instance
curl http://instance1:8333/sts/providers | jq '.providers | length'
curl http://instance2:8334/sts/providers | jq '.providers | length'

Solution: Ensure all instances use identical configuration files

2. Token Validation Failures

Symptoms: "Invalid signature" or "Invalid issuer" errors Diagnosis: Check signing key and issuer consistency Solution: Verify signingKey and issuer are identical across all instances

3. Provider Loading Failures

Symptoms: Providers not loaded at startup Diagnosis: Check logs for provider initialization errors Solution: Validate provider configuration against schema

4. OIDC Provider Connectivity

Symptoms: "Failed to fetch JWKS" errors Diagnosis: Test OIDC provider connectivity from all instances Solution: Check network connectivity, DNS resolution, certificates

Debug Commands

# Test configuration loading
weed s3 -iam.config=/path/to/config.json -test.config

# Validate JWT tokens
curl -X POST http://localhost:8333/sts/validate-token \
  -H "Content-Type: application/json" \
  -d '{"sessionToken": "eyJ0eXAiOiJKV1QiLCJhbGc..."}'

# List loaded providers
curl http://localhost:8333/sts/providers

# Check session store
curl http://localhost:8333/sts/sessions/count

Performance Considerations

Token Validation Performance

• JWT Validation: ~1-5ms per token validation
• JWKS Caching: Cache JWKS responses to reduce OIDC provider load
• Session Lookup: Filer session lookup adds ~10-20ms latency
• Concurrent Requests: Each instance can handle 1000+ concurrent validations

Scaling Recommendations

• Horizontal Scaling: Add more S3 gateway instances behind load balancer
• Session Store Optimization: Use SSD storage for filer session store
• Provider Caching: Implement JWKS caching to reduce provider load
• Connection Pooling: Use connection pooling for filer communication

Summary

The configuration-driven provider system solves critical distributed deployment issues:

• ✅ Automatic Provider Loading: No manual registration code required
• ✅ Configuration Consistency: All instances load identical providers from config
• ✅ Easy Management: Update config file, restart services
• ✅ Production Ready: Supports OIDC, proper session management, distributed storage
• ✅ Backwards Compatible: Existing manual registration still works

This enables SeaweedFS S3 Gateway to scale horizontally with consistent authentication across all instances, making it truly production-ready for enterprise deployments.