snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

2026-09-16 03:23:42 +05:00
commit 89f5be005d
3997 changed files with 961970 additions and 0 deletions
--- a/test/s3/iam/STS_DISTRIBUTED.md
+++ b/test/s3/iam/STS_DISTRIBUTED.md
@@ -0,0 +1,559 @@
+# Distributed STS Service for SeaweedFS S3 Gateway
+
+This document explains how to configure and deploy the STS (Security Token Service) for distributed SeaweedFS S3 Gateway deployments with consistent identity provider configurations.
+
+## Problem Solved
+
+Previously, identity providers had to be **manually registered** on each S3 gateway instance, leading to:
+
+- ❌ **Inconsistent authentication**: Different instances might have different providers
+- ❌ **Manual synchronization**: No guarantee all instances have same provider configs
+- ❌ **Authentication failures**: Users getting different responses from different instances
+- ❌ **Operational complexity**: Difficult to manage provider configurations at scale
+
+## Solution: Configuration-Driven Providers
+
+The STS service now supports **automatic provider loading** from configuration files, ensuring:
+
+- ✅ **Consistent providers**: All instances load identical providers from config
+- ✅ **Automatic synchronization**: Configuration-driven, no manual registration needed
+- ✅ **Reliable authentication**: Same behavior from all instances
+- ✅ **Easy management**: Update config file, restart services
+
+## Configuration Schema
+
+### Basic STS Configuration
+
+```json
+{
+  "sts": {
+    "tokenDuration": "1h",
+    "maxSessionLength": "12h", 
+    "issuer": "seaweedfs-sts",
+    "signingKey": "base64-encoded-signing-key-32-chars-min"
+  }
+}
+```
+
+**Note**: The STS service uses a **stateless JWT design** where all session information is embedded directly in the JWT token. No external session storage is required.
+
+### Configuration-Driven Providers
+
+```json
+{
+  "sts": {
+    "tokenDuration": "1h",
+    "maxSessionLength": "12h",
+    "issuer": "seaweedfs-sts",
+    "signingKey": "base64-encoded-signing-key",
+    "providers": [
+      {
+        "name": "keycloak-oidc",
+        "type": "oidc", 
+        "enabled": true,
+        "config": {
+          "issuer": "https://keycloak.company.com/realms/seaweedfs",
+          "clientId": "seaweedfs-s3",
+          "clientSecret": "super-secret-key",
+          "jwksUri": "https://keycloak.company.com/realms/seaweedfs/protocol/openid-connect/certs",
+          "scopes": ["openid", "profile", "email", "roles"],
+          "claimsMapping": {
+            "usernameClaim": "preferred_username",
+            "groupsClaim": "roles"
+          }
+        }
+      },
+      {
+        "name": "backup-oidc",
+        "type": "oidc",
+        "enabled": false,
+        "config": {
+          "issuer": "https://backup-oidc.company.com",
+          "clientId": "seaweedfs-backup"
+        }
+      },
+      {
+        "name": "dev-mock-provider",
+        "type": "mock",
+        "enabled": true,
+        "config": {
+          "issuer": "http://localhost:9999",
+          "clientId": "mock-client"
+        }
+      }
+    ]
+  }
+}
+```
+
+## Supported Provider Types
+
+### 1. OIDC Provider (`"type": "oidc"`)
+
+For production authentication with OpenID Connect providers like Keycloak, Auth0, Google, etc.
+
+**Required Configuration:**
+- `issuer`: OIDC issuer URL
+- `clientId`: OAuth2 client ID
+
+**Optional Configuration:**
+- `clientSecret`: OAuth2 client secret (for confidential clients)
+- `jwksUri`: JSON Web Key Set URI (auto-discovered if not provided)
+- `userInfoUri`: UserInfo endpoint URI (auto-discovered if not provided)
+- `scopes`: OAuth2 scopes to request (default: `["openid"]`)
+- `claimsMapping`: Map OIDC claims to identity attributes
+
+**Example:**
+```json
+{
+  "name": "corporate-keycloak",
+  "type": "oidc",
+  "enabled": true,
+  "config": {
+    "issuer": "https://sso.company.com/realms/production",
+    "clientId": "seaweedfs-prod",
+    "clientSecret": "confidential-secret", 
+    "scopes": ["openid", "profile", "email", "groups"],
+    "claimsMapping": {
+      "usernameClaim": "preferred_username",
+      "groupsClaim": "groups",
+      "emailClaim": "email"
+    }
+  }
+}
+```
+
+### 2. Mock Provider (`"type": "mock"`)
+
+For development, testing, and staging environments.
+
+**Configuration:**
+- `issuer`: Mock issuer URL (default: `http://localhost:9999`)
+- `clientId`: Mock client ID
+
+**Example:**
+```json
+{
+  "name": "dev-mock",
+  "type": "mock", 
+  "enabled": true,
+  "config": {
+    "issuer": "http://dev-mock:9999",
+    "clientId": "dev-client"
+  }
+}
+```
+
+**Built-in Test Tokens:**
+- `valid_test_token`: Returns test user with developer groups
+- `valid-oidc-token`: Compatible with integration tests
+- `expired_token`: Returns token expired error
+- `invalid_token`: Returns invalid token error
+
+### 3. Future Provider Types
+
+The factory pattern supports easy addition of new provider types:
+
+- `"type": "ldap"`: LDAP/Active Directory authentication
+- `"type": "saml"`: SAML 2.0 authentication  
+- `"type": "oauth2"`: Generic OAuth2 providers
+- `"type": "custom"`: Custom authentication backends
+
+## Deployment Patterns
+
+### Single Instance (Development)
+
+```bash
+# Standard deployment with config-driven providers
+weed s3 -filer=localhost:8888 -port=8333 -iam.config=/path/to/sts_config.json
+```
+
+### Multiple Instances (Production)
+
+```bash
+# Instance 1 
+weed s3 -filer=prod-filer:8888 -port=8333 -iam.config=/shared/sts_distributed.json
+
+# Instance 2
+weed s3 -filer=prod-filer:8888 -port=8334 -iam.config=/shared/sts_distributed.json
+
+# Instance N
+weed s3 -filer=prod-filer:8888 -port=833N -iam.config=/shared/sts_distributed.json
+```
+
+**Critical Requirements for Distributed Deployment:**
+
+1. **Identical Configuration Files**: All instances must use the exact same configuration file
+2. **Same Signing Keys**: All instances must have identical `signingKey` values
+3. **Same Issuer**: All instances must use the same `issuer` value
+
+**Note**: STS now uses stateless JWT tokens, eliminating the need for shared session storage.
+
+### High Availability Setup
+
+```yaml
+# docker-compose.yml for production deployment
+services:
+  filer:
+    image: seaweedfs/seaweedfs:latest
+    command: "filer -master=master:9333"
+    volumes:
+      - filer-data:/data
+    
+  s3-gateway-1:
+    image: seaweedfs/seaweedfs:latest
+    command: "s3 -filer=filer:8888 -port=8333 -iam.config=/config/sts_distributed.json"
+    ports:
+      - "8333:8333"
+    volumes:
+      - ./sts_distributed.json:/config/sts_distributed.json:ro
+    depends_on: [filer]
+    
+  s3-gateway-2:
+    image: seaweedfs/seaweedfs:latest 
+    command: "s3 -filer=filer:8888 -port=8333 -iam.config=/config/sts_distributed.json"
+    ports:
+      - "8334:8333"
+    volumes:
+      - ./sts_distributed.json:/config/sts_distributed.json:ro
+    depends_on: [filer]
+    
+  s3-gateway-3:
+    image: seaweedfs/seaweedfs:latest
+    command: "s3 -filer=filer:8888 -port=8333 -iam.config=/config/sts_distributed.json"
+    ports:
+      - "8335:8333"
+    volumes:
+      - ./sts_distributed.json:/config/sts_distributed.json:ro
+    depends_on: [filer]
+    
+  load-balancer:
+    image: nginx:alpine
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx.conf:/etc/nginx/nginx.conf:ro
+    depends_on: [s3-gateway-1, s3-gateway-2, s3-gateway-3]
+```
+
+## Authentication Flow
+
+### 1. OIDC Authentication Flow
+
+```
+1. User authenticates with OIDC provider (Keycloak, Auth0, etc.)
+   ↓
+2. User receives OIDC JWT token from provider
+   ↓  
+3. User calls SeaweedFS STS AssumeRoleWithWebIdentity
+   POST /sts/assume-role-with-web-identity
+   {
+     "RoleArn": "arn:aws:iam::role/S3AdminRole",
+     "WebIdentityToken": "eyJ0eXAiOiJKV1QiLCJhbGc...",
+     "RoleSessionName": "user-session"
+   }
+   ↓
+4. STS validates OIDC token with configured provider
+   - Verifies JWT signature using provider's JWKS
+   - Validates issuer, audience, expiration
+   - Extracts user identity and groups
+   ↓
+5. STS checks role trust policy
+   - Verifies user/groups can assume the requested role
+   - Validates conditions in trust policy
+   ↓
+6. STS generates temporary credentials
+   - Creates temporary access key, secret key, session token
+   - Session token is signed JWT with all session information embedded (stateless)
+   ↓
+7. User receives temporary credentials
+   {
+     "Credentials": {
+       "AccessKeyId": "AKIA...",
+       "SecretAccessKey": "base64-secret",
+       "SessionToken": "eyJ0eXAiOiJKV1QiLCJhbGc...",
+       "Expiration": "2024-01-01T12:00:00Z"
+     }
+   }
+   ↓
+8. User makes S3 requests with temporary credentials
+   - AWS SDK signs requests with temporary credentials
+   - SeaweedFS S3 gateway validates session token
+   - Gateway checks permissions via policy engine
+```
+
+### 2. Cross-Instance Token Validation
+
+```
+User Request → Load Balancer → Any S3 Gateway Instance
+                                      ↓
+                              Extract JWT Session Token
+                                      ↓
+                              Validate JWT Token
+                              (Self-contained - no external storage needed)
+                                      ↓
+                              Check Permissions
+                              (Shared policy engine)
+                                      ↓
+                              Allow/Deny Request
+```
+
+## Trust Policy Conditions
+
+Step 5 above evaluates the role's trust policy against context keys derived from the
+OIDC token's claims. The available keys are:
+
+| Condition key | Source |
+|---------------|--------|
+| `oidc:iss` | `iss` claim (issuer URL) |
+| `oidc:sub` | `sub` claim |
+| `oidc:aud` | `aud` claim |
+| `oidc:<claim>` | any other token claim, e.g. `oidc:roles`, `oidc:groups`, `oidc:email` |
+| `aws:FederatedProvider` | the provider `name` (e.g. `keycloak-oidc`) when its configured issuer matches the token, otherwise the raw issuer URL |
+| `aws:userid` | `sub` claim (same value as `oidc:sub` during trust-policy evaluation) |
+| `sts:DurationSeconds` | requested session duration, when supplied |
+
+During trust-policy evaluation `aws:userid` is the raw `sub` claim. Once the
+role has been assumed, the keys seen by request authorization differ: there
+`aws:userid` is a stable per-identity hash of `sub` and `iss` (see
+`ComputeParentUser`), so do not assume the two contexts carry the same value.
+
+Custom claims are always exposed under the `oidc:` prefix, so a trust policy must use
+`oidc:roles` (not a bare `roles`) to match a `roles` claim:
+
+```json
+"Condition": {
+  "StringEquals": {
+    "oidc:roles": "s3-admin"
+  }
+}
+```
+
+A multi-valued claim (such as a `roles` array) matches when any of its values equals
+the condition value. The same `oidc:` keys can be interpolated into policy resources,
+e.g. `arn:aws:s3:::bucket/${oidc:sub}/*`.
+
+### roleMapping vs. trust policy
+
+A provider's `roleMapping` and a role's trust policy apply to two different entry
+points and are not interchangeable:
+
+- **Direct OIDC** — an S3 request carrying `Authorization: Bearer <OIDC-JWT>`. The
+  gateway applies `roleMapping` to choose the caller's role from the token claims; the
+  first matching rule (or `defaultRole`) wins.
+- **STS `AssumeRoleWithWebIdentity`** — the caller names the role explicitly via
+  `RoleArn`, and that role's trust policy decides whether the assumption is allowed.
+  `roleMapping` does not select the role on this path; instead the token claims are
+  surfaced as the `oidc:` condition keys above for the trust policy to evaluate.
+
+## Configuration Management
+
+### Development Environment
+
+```json
+{
+  "sts": {
+    "tokenDuration": "1h",
+    "maxSessionLength": "12h",
+    "issuer": "seaweedfs-dev-sts",
+    "signingKey": "ZGV2LXNpZ25pbmcta2V5LTMyLWNoYXJhY3RlcnMtbG9uZw==",
+    "providers": [
+      {
+        "name": "dev-mock",
+        "type": "mock",
+        "enabled": true,
+        "config": {
+          "issuer": "http://localhost:9999",
+          "clientId": "dev-mock-client"
+        }
+      }
+    ]
+  }
+}
+```
+
+### Production Environment
+
+```json
+{
+  "sts": {
+    "tokenDuration": "1h",
+    "maxSessionLength": "12h",
+    "issuer": "seaweedfs-prod-sts",
+    "signingKey": "cHJvZC1zaWduaW5nLWtleS0zMi1jaGFyYWN0ZXJzLWxvbmctcmFuZG9t",
+    "providers": [
+      {
+        "name": "corporate-sso",
+        "type": "oidc",
+        "enabled": true,
+        "config": {
+          "issuer": "https://sso.company.com/realms/production",
+          "clientId": "seaweedfs-prod",
+          "clientSecret": "${SSO_CLIENT_SECRET}",
+          "scopes": ["openid", "profile", "email", "groups"],
+          "claimsMapping": {
+            "usernameClaim": "preferred_username",
+            "groupsClaim": "groups"
+          }
+        }
+      },
+      {
+        "name": "backup-auth",
+        "type": "oidc", 
+        "enabled": false,
+        "config": {
+          "issuer": "https://backup-sso.company.com",
+          "clientId": "seaweedfs-backup"
+        }
+      }
+    ]
+  }
+}
+```
+
+## Operational Best Practices
+
+### 1. Configuration Management
+
+- **Version Control**: Store configurations in Git with proper versioning
+- **Environment Separation**: Use separate configs for dev/staging/production
+- **Secret Management**: Use environment variable substitution for secrets
+- **Configuration Validation**: Test configurations before deployment
+
+### 2. Security Considerations
+
+- **Signing Key Security**: Use strong, randomly generated signing keys (32+ bytes)
+- **Key Rotation**: Implement signing key rotation procedures
+- **Secret Storage**: Store client secrets in secure secret management systems
+- **TLS Encryption**: Always use HTTPS for OIDC providers in production
+
+### 3. Monitoring and Troubleshooting
+
+- **Provider Health**: Monitor OIDC provider availability and response times
+- **Session Metrics**: Track active sessions, token validation errors
+- **Configuration Drift**: Alert on configuration inconsistencies between instances
+- **Authentication Logs**: Log authentication attempts for security auditing
+
+### 4. Capacity Planning
+
+- **Provider Performance**: Monitor OIDC provider response times and rate limits
+- **Token Validation**: Monitor JWT validation performance and caching
+- **Memory Usage**: Monitor JWT token validation caching and provider metadata
+
+## Migration Guide
+
+### From Manual Provider Registration
+
+**Before (Manual Registration):**
+```go
+// Each instance needs this code
+keycloakProvider := oidc.NewOIDCProvider("keycloak-oidc")
+keycloakProvider.Initialize(keycloakConfig)
+stsService.RegisterProvider(keycloakProvider)
+```
+
+**After (Configuration-Driven):**
+```json
+{
+  "sts": {
+    "providers": [
+      {
+        "name": "keycloak-oidc",
+        "type": "oidc",
+        "enabled": true,
+        "config": {
+          "issuer": "https://keycloak.company.com/realms/seaweedfs",
+          "clientId": "seaweedfs-s3"
+        }
+      }
+    ]
+  }
+}
+```
+
+### Migration Steps
+
+1. **Create Configuration File**: Convert manual provider registrations to JSON config
+2. **Test Single Instance**: Deploy config to one instance and verify functionality
+3. **Validate Consistency**: Ensure all instances load identical providers
+4. **Rolling Deployment**: Update instances one by one with new configuration
+5. **Remove Manual Code**: Clean up manual provider registration code
+
+## Troubleshooting
+
+### Common Issues
+
+#### 1. Provider Inconsistency
+
+**Symptoms**: Authentication works on some instances but not others
+**Diagnosis**: 
+```bash
+# Check provider counts on each instance
+curl http://instance1:8333/sts/providers | jq '.providers | length'
+curl http://instance2:8334/sts/providers | jq '.providers | length'
+```
+**Solution**: Ensure all instances use identical configuration files
+
+#### 2. Token Validation Failures
+
+**Symptoms**: "Invalid signature" or "Invalid issuer" errors
+**Diagnosis**: Check signing key and issuer consistency
+**Solution**: Verify `signingKey` and `issuer` are identical across all instances
+
+#### 3. Provider Loading Failures
+
+**Symptoms**: Providers not loaded at startup
+**Diagnosis**: Check logs for provider initialization errors
+**Solution**: Validate provider configuration against schema
+
+#### 4. OIDC Provider Connectivity
+
+**Symptoms**: "Failed to fetch JWKS" errors
+**Diagnosis**: Test OIDC provider connectivity from all instances
+**Solution**: Check network connectivity, DNS resolution, certificates
+
+### Debug Commands
+
+```bash
+# Test configuration loading
+weed s3 -iam.config=/path/to/config.json -test.config
+
+# Validate JWT tokens
+curl -X POST http://localhost:8333/sts/validate-token \
+  -H "Content-Type: application/json" \
+  -d '{"sessionToken": "eyJ0eXAiOiJKV1QiLCJhbGc..."}'
+
+# List loaded providers
+curl http://localhost:8333/sts/providers
+
+# Check session store
+curl http://localhost:8333/sts/sessions/count
+```
+
+## Performance Considerations
+
+### Token Validation Performance
+
+- **JWT Validation**: ~1-5ms per token validation
+- **JWKS Caching**: Cache JWKS responses to reduce OIDC provider load
+- **Session Lookup**: Filer session lookup adds ~10-20ms latency
+- **Concurrent Requests**: Each instance can handle 1000+ concurrent validations
+
+### Scaling Recommendations
+
+- **Horizontal Scaling**: Add more S3 gateway instances behind load balancer
+- **Session Store Optimization**: Use SSD storage for filer session store
+- **Provider Caching**: Implement JWKS caching to reduce provider load
+- **Connection Pooling**: Use connection pooling for filer communication
+
+## Summary
+
+The configuration-driven provider system solves critical distributed deployment issues:
+
+- ✅ **Automatic Provider Loading**: No manual registration code required
+- ✅ **Configuration Consistency**: All instances load identical providers from config
+- ✅ **Easy Management**: Update config file, restart services
+- ✅ **Production Ready**: Supports OIDC, proper session management, distributed storage
+- ✅ **Backwards Compatible**: Existing manual registration still works
+
+This enables SeaweedFS S3 Gateway to **scale horizontally** with **consistent authentication** across all instances, making it truly **production-ready for enterprise deployments**.