snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

weed/storage/erasure_coding/distribution/README.md

@@ -0,0 +1,209 @@
+# EC Distribution Package
+
+This package provides erasure coding (EC) shard distribution algorithms that are:
+
+- **Configurable**: Works with any EC ratio (e.g., 10+4, 8+4, 6+3)
+- **Reusable**: Used by shell commands, worker tasks, and seaweed-enterprise
+- **Topology-aware**: Distributes shards across data centers, racks, and nodes proportionally
+
+## Usage
+
+### Basic Usage with Default 10+4 EC
+
+```go
+import (
+    "github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding/distribution"
+)
+
+// Parse replication policy
+rep, _ := distribution.NewReplicationConfigFromString("110")
+
+// Use default 10+4 EC configuration
+ec := distribution.DefaultECConfig()
+
+// Calculate distribution plan
+dist := distribution.CalculateDistribution(ec, rep)
+
+fmt.Println(dist.Summary())
+// Output:
+// EC Configuration: 10+4 (total: 14, can lose: 4)
+// Replication: replication=110 (DCs:2, Racks/DC:2, Nodes/Rack:1)
+// Distribution Plan:
+//   Data Centers: 2 (target 7 shards each, max 9)
+//   Racks per DC: 2 (target 4 shards each, max 6)
+//   Nodes per Rack: 1 (target 4 shards each, max 6)
+```
+
+### Custom EC Ratios (seaweed-enterprise)
+
+```go
+// Create custom 8+4 EC configuration
+ec, err := distribution.NewECConfig(8, 4)
+if err != nil {
+    log.Fatal(err)
+}
+
+rep, _ := distribution.NewReplicationConfigFromString("200")
+dist := distribution.CalculateDistribution(ec, rep)
+
+// Check fault tolerance
+fmt.Println(dist.FaultToleranceAnalysis())
+// Output:
+// Fault Tolerance Analysis for 8+4:
+//   DC Failure: SURVIVABLE ✓
+//     - Losing one DC loses ~4 shards
+//     - Remaining: 8 shards (need 8)
+```
+
+### Planning Shard Moves
+
+```go
+// Build topology analysis
+analysis := distribution.NewTopologyAnalysis()
+
+// Add nodes and their shard locations
+for _, node := range nodes {
+    analysis.AddNode(&distribution.TopologyNode{
+        NodeID:     node.ID,
+        DataCenter: node.DC,
+        Rack:       node.Rack,
+        FreeSlots:  node.FreeSlots,
+    })
+    for _, shardID := range node.ShardIDs {
+        analysis.AddShardLocation(distribution.ShardLocation{
+            ShardID:    shardID,
+            NodeID:     node.ID,
+            DataCenter: node.DC,
+            Rack:       node.Rack,
+        })
+    }
+}
+analysis.Finalize()
+
+// Create rebalancer and plan moves
+rebalancer := distribution.NewRebalancer(ec, rep)
+plan, err := rebalancer.PlanRebalance(analysis)
+
+for _, move := range plan.Moves {
+    fmt.Printf("Move shard %d from %s to %s\n", 
+        move.ShardID, move.SourceNode.NodeID, move.DestNode.NodeID)
+}
+```
+
+## Algorithm
+
+### Proportional Distribution
+
+The replication policy `XYZ` is interpreted as a ratio:
+
+| Replication | DCs | Racks/DC | Nodes/Rack | 14 Shards Distribution |
+|-------------|-----|----------|------------|------------------------|
+| `000` | 1 | 1 | 1 | All in one place |
+| `001` | 1 | 1 | 2 | 7 per node |
+| `010` | 1 | 2 | 1 | 7 per rack |
+| `100` | 2 | 1 | 1 | 7 per DC |
+| `110` | 2 | 2 | 1 | 7/DC, 4/rack |
+| `200` | 3 | 1 | 1 | 5 per DC |
+
+### Rebalancing Process
+
+1. **DC-level balancing**: Move shards to achieve target shards per DC
+2. **Rack-level balancing**: Within each DC, balance across racks
+3. **Node-level balancing**: Within each rack, balance across nodes
+
+### Shard Priority: Data First, Parity Moves First
+
+When rebalancing, the algorithm prioritizes keeping data shards spread out:
+
+- **Data shards (0 to DataShards-1)**: Serve read requests directly
+- **Parity shards (DataShards to TotalShards-1)**: Only used for reconstruction
+
+**Rebalancing Strategy**:
+- When moving shards FROM an overloaded node, **parity shards are moved first**
+- This keeps data shards in place on well-distributed nodes
+- Result: Data shards remain spread out for optimal read performance
+
+```go
+// Check shard type
+if ec.IsDataShard(shardID) {
+    // Shard serves read requests
+}
+if ec.IsParityShard(shardID) {
+    // Shard only used for reconstruction
+}
+
+// Sort shards for placement (data first for initial distribution)
+sorted := ec.SortShardsDataFirst(shards)
+
+// Sort shards for rebalancing (parity first to move them away)
+sorted := ec.SortShardsParityFirst(shards)
+```
+
+### Fault Tolerance
+
+The package provides fault tolerance analysis:
+
+- **DC Failure**: Can the data survive complete DC loss?
+- **Rack Failure**: Can the data survive complete rack loss?
+- **Node Failure**: Can the data survive single node loss?
+
+For example, with 10+4 EC (can lose 4 shards):
+- Need 4+ DCs for DC-level fault tolerance
+- Need 4+ racks for rack-level fault tolerance
+- Usually survivable at node level
+
+## API Reference
+
+### Types
+
+- `ECConfig`: EC configuration (data shards, parity shards)
+- `ReplicationConfig`: Parsed replication policy
+- `ECDistribution`: Calculated distribution plan
+- `TopologyAnalysis`: Current shard distribution analysis
+- `Rebalancer`: Plans shard moves
+- `RebalancePlan`: List of planned moves
+- `ShardMove`: Single shard move operation
+
+### Key Functions
+
+- `NewECConfig(data, parity int)`: Create EC configuration
+- `DefaultECConfig()`: Returns 10+4 configuration
+- `CalculateDistribution(ec, rep)`: Calculate distribution plan
+- `NewRebalancer(ec, rep)`: Create rebalancer
+- `PlanRebalance(analysis)`: Generate rebalancing plan
+
+## Integration
+
+### Shell Commands
+
+The shell package wraps this distribution package for `ec.balance`:
+
+```go
+import "github.com/seaweedfs/seaweedfs/weed/shell"
+
+rebalancer := shell.NewProportionalECRebalancer(nodes, rp, diskType)
+moves, _ := rebalancer.PlanMoves(volumeId, locations)
+```
+
+### Worker Tasks
+
+Worker tasks can use the distribution package directly:
+
+```go
+import "github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding/distribution"
+
+ec := distribution.ECConfig{DataShards: 8, ParityShards: 4}
+rep := distribution.NewReplicationConfig(rp)
+dist := distribution.CalculateDistribution(ec, rep)
+```
+
+### seaweed-enterprise
+
+Enterprise features can provide custom EC configurations:
+
+```go
+// Custom EC ratio from license/config
+ec, _ := distribution.NewECConfig(customData, customParity)
+rebalancer := distribution.NewRebalancer(ec, rep)
+```
+