snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

test/erasure_coding/README.md

@@ -0,0 +1,123 @@
+# Erasure Coding Integration Tests
+
+This directory contains integration tests for the EC (Erasure Coding) encoding volume location timing bug fix.
+
+## The Bug
+
+The bug caused **double storage usage** during EC encoding because:
+
+1. **Silent failure**: Functions returned `nil` instead of proper error messages
+2. **Timing race condition**: Volume locations were collected **AFTER** EC encoding when master metadata was already updated
+3. **Missing cleanup**: Original volumes weren't being deleted after EC encoding
+
+This resulted in both original `.dat` files AND EC `.ec00-.ec13` files coexisting, effectively **doubling storage usage**.
+
+## The Fix
+
+The fix addresses all three issues:
+
+1. **Fixed silent failures**: Updated `doDeleteVolumes()` and `doEcEncode()` to return proper errors
+2. **Fixed timing race condition**: Created `doDeleteVolumesWithLocations()` that uses pre-collected volume locations
+3. **Enhanced cleanup**: Volume locations are now collected **BEFORE** EC encoding, preventing the race condition
+
+## Integration Tests
+
+### TestECEncodingVolumeLocationTimingBug
+The main integration test that:
+- **Simulates master timing race condition**: Tests what happens when volume locations are read from master AFTER EC encoding has updated the metadata
+- **Verifies fix effectiveness**: Checks for the "Collecting volume locations...before EC encoding" message that proves the fix is working
+- **Tests multi-server distribution**: Runs EC encoding with 6 volume servers to test shard distribution
+- **Validates cleanup**: Ensures original volumes are properly cleaned up after EC encoding
+
+### TestECEncodingMasterTimingRaceCondition
+A focused test that specifically targets the **master metadata timing race condition**:
+- **Simulates the exact race condition**: Tests volume location collection timing relative to master metadata updates
+- **Detects timing fix**: Verifies that volume locations are collected BEFORE EC encoding starts
+- **Demonstrates bug impact**: Shows what happens when volume locations are unavailable after master metadata update
+
+### TestECEncodingRegressionPrevention
+Regression tests that ensure:
+- **Function signatures**: Fixed functions still exist and return proper errors
+- **Timing patterns**: Volume location collection happens in the correct order
+
+## Test Architecture
+
+The tests use:
+- **Real SeaweedFS cluster**: 1 master server + 6 volume servers
+- **Multi-server setup**: Tests realistic EC shard distribution across multiple servers
+- **Timing simulation**: Goroutines and delays to simulate race conditions
+- **Output validation**: Checks for specific log messages that prove the fix is working
+
+## Why Integration Tests Were Necessary
+
+Unit tests could not catch this bug because:
+1. **Race condition**: The bug only occurred in real-world timing scenarios
+2. **Master-volume server interaction**: Required actual master metadata updates
+3. **File system operations**: Needed real volume creation and EC shard generation
+4. **Cleanup timing**: Required testing the sequence of operations in correct order
+
+The integration tests successfully catch the timing bug by:
+- **Testing real command execution**: Uses actual `ec.encode` shell command
+- **Simulating race conditions**: Creates timing scenarios that expose the bug
+- **Validating output messages**: Checks for the key "Collecting volume locations...before EC encoding" message
+- **Monitoring cleanup behavior**: Ensures original volumes are properly deleted
+
+## Running the Tests
+
+```bash
+# Run all integration tests
+go test -v
+
+# Run only the main timing test
+go test -v -run TestECEncodingVolumeLocationTimingBug
+
+# Run only the race condition test
+go test -v -run TestECEncodingMasterTimingRaceCondition
+
+# Skip integration tests (short mode)
+go test -v -short
+```
+
+## Manual Testing with Makefile
+
+A Makefile is provided for manual EC testing.
+
+**Requirements:** `curl`, `jq` (command-line JSON processor)
+
+```bash
+# Quick start: start cluster and populate data
+make setup
+
+# Open weed shell to run EC commands
+make shell
+
+# Individual targets
+make start      # Start test cluster (master + 6 volume servers + filer)
+make stop       # Stop test cluster
+make populate   # Populate ~300MB of test data
+make status     # Show cluster and EC shard status
+make clean      # Stop cluster and remove all test data
+make help       # Show all targets
+```
+
+### EC Rebalance Limited Slots (Unit Test)
+
+The "no free ec shard slots" issue is tested with a **unit test** that works directly on
+topology data structures without requiring a running cluster.
+
+**Location**: `weed/shell/ec_rebalance_slots_test.go`
+
+Tests included:
+- `TestECRebalanceWithLimitedSlots`: Tests a topology with 6 servers, 7 EC volumes (98 shards)
+- `TestECRebalanceZeroFreeSlots`: Reproduces the exact 0 free slots scenario
+
+**Known Issue**: When volume servers are at capacity (`volumeCount == maxVolumeCount`),
+the rebalance step fails with "no free ec shard slots" instead of recognizing that
+moving shards frees slots on source servers.
+
+## Test Results
+
+**With the fix**: Shows "Collecting volume locations for N volumes before EC encoding..." message
+**Without the fix**: No collection message, potential timing race condition
+
+The tests demonstrate that the fix prevents the volume location timing bug that caused double storage usage in EC encoding operations.