upstream-seaweedfs/VOLUME_SERVER_RUST_PLAN.md

# Execution Plan: SeaweedFS Volume Server — Go to Rust Port

## Scope Summary

| Component | Go Source | Lines (non-test) | Description |
|---|---|---|---|
| CLI & startup | `weed/command/volume.go` | 476 | ~40 CLI flags, server bootstrap |
| HTTP server + handlers | `weed/server/volume_server*.go` | 1,517 | Struct, routes, read/write/delete handlers |
| gRPC handlers | `weed/server/volume_grpc_*.go` | 3,073 | 40 RPC method implementations |
| Storage engine | `weed/storage/` | 15,271 | Volumes, needles, index, compaction, EC, backend |
| Protobuf definitions | `weed/pb/volume_server.proto` | 759 | Service + message definitions |
| Shared utilities | `weed/security/`, `weed/stats/`, `weed/util/` | ~2,000+ | JWT, TLS, metrics, helpers |
| **Total** | | **~23,000+** | |

## Rust Crate & Dependency Strategy

```
seaweed-volume/
├── Cargo.toml
├── build.rs                    # protobuf codegen
├── proto/
│   ├── volume_server.proto     # copied from Go, adapted
│   └── remote.proto
├── src/
│   ├── main.rs                 # CLI entry point
│   ├── config.rs               # CLI flags + config
│   ├── server/
│   │   ├── mod.rs
│   │   ├── volume_server.rs    # VolumeServer struct + lifecycle
│   │   ├── http_handlers.rs    # HTTP route dispatch
│   │   ├── http_read.rs        # GET/HEAD handlers
│   │   ├── http_write.rs       # POST/PUT handlers
│   │   ├── http_delete.rs      # DELETE handler
│   │   ├── http_admin.rs       # /status, /healthz, /ui
│   │   ├── grpc_service.rs     # gRPC trait impl dispatch
│   │   ├── grpc_vacuum.rs
│   │   ├── grpc_copy.rs
│   │   ├── grpc_erasure_coding.rs
│   │   ├── grpc_tail.rs
│   │   ├── grpc_admin.rs
│   │   ├── grpc_read_write.rs
│   │   ├── grpc_batch_delete.rs
│   │   ├── grpc_scrub.rs
│   │   ├── grpc_tier.rs
│   │   ├── grpc_remote.rs
│   │   ├── grpc_query.rs
│   │   ├── grpc_state.rs
│   │   └── grpc_client_to_master.rs  # heartbeat
│   ├── storage/
│   │   ├── mod.rs
│   │   ├── store.rs            # Store (multi-disk manager)
│   │   ├── volume.rs           # Volume struct + lifecycle
│   │   ├── volume_read.rs
│   │   ├── volume_write.rs
│   │   ├── volume_compact.rs
│   │   ├── volume_info.rs
│   │   ├── needle/
│   │   │   ├── mod.rs
│   │   │   ├── needle.rs       # Needle struct + serialization
│   │   │   ├── needle_read.rs
│   │   │   ├── needle_write.rs
│   │   │   ├── needle_map.rs   # in-memory NeedleMap
│   │   │   ├── needle_value.rs
│   │   │   └── crc.rs
│   │   ├── super_block.rs
│   │   ├── idx/
│   │   │   ├── mod.rs
│   │   │   └── idx.rs          # .idx file format read/write
│   │   ├── needle_map_leveldb.rs
│   │   ├── types.rs            # NeedleId, Offset, Size, DiskType
│   │   ├── disk_location.rs    # DiskLocation per-directory
│   │   ├── erasure_coding/
│   │   │   ├── mod.rs
│   │   │   ├── ec_volume.rs
│   │   │   ├── ec_shard.rs
│   │   │   ├── ec_encoder.rs   # Reed-Solomon encoding
│   │   │   └── ec_decoder.rs
│   │   └── backend/
│   │       ├── mod.rs
│   │       ├── disk.rs
│   │       └── s3_backend.rs   # tiered storage to S3
│   ├── topology/
│   │   └── volume_layout.rs    # replication placement
│   ├── security/
│   │   ├── mod.rs
│   │   ├── guard.rs            # whitelist + JWT gate
│   │   ├── jwt.rs
│   │   └── tls.rs
│   ├── stats/
│   │   ├── mod.rs
│   │   └── metrics.rs          # Prometheus counters/gauges
│   └── util/
│       ├── mod.rs
│       ├── grpc.rs
│       ├── http.rs
│       └── file.rs
└── tests/
    ├── integration/
    │   ├── http_read_test.rs
    │   ├── http_write_test.rs
    │   ├── grpc_test.rs
    │   └── storage_test.rs
    └── unit/
        ├── needle_test.rs
        ├── idx_test.rs
        ├── super_block_test.rs
        └── ec_test.rs
```

### Key Rust dependencies

| Purpose | Crate |
|---|---|
| Async runtime | `tokio` |
| gRPC | `tonic` + `prost` |
| HTTP server | `hyper` + `axum` |
| CLI parsing | `clap` (derive) |
| Prometheus metrics | `prometheus` |
| JWT | `jsonwebtoken` |
| TLS | `rustls` + `tokio-rustls` |
| LevelDB | `rusty-leveldb` or `rocksdb` |
| Reed-Solomon EC | `reed-solomon-erasure` |
| Logging | `tracing` + `tracing-subscriber` |
| Config (security.toml) | `toml` + `serde` |
| CRC32 | `crc32fast` |
| Memory-mapped files | `memmap2` |

---

## Phased Execution Plan

### Phase 1: Project Skeleton & Protobuf Codegen
**Goal:** Cargo project compiles, proto codegen works, CLI parses all flags.

**Steps:**

1.1. Create `seaweed-volume/Cargo.toml` with all dependencies listed above.

1.2. Copy `volume_server.proto` and `remote.proto` into `proto/`. Adjust package paths for Rust codegen.

1.3. Create `build.rs` using `tonic-build` to compile `.proto` files into Rust types.

1.4. Create `src/main.rs` with `clap` derive structs mirroring all 40 CLI flags from `weed/command/volume.go`:
   - `--port` (default 8080)
   - `--port.grpc` (default 0 → 10000+port)
   - `--port.public` (default 0 → same as port)
   - `--ip` (auto-detect)
   - `--id` (default empty → ip:port)
   - `--publicUrl`
   - `--ip.bind`
   - `--master` (default "localhost:9333")
   - `--mserver` (deprecated compat)
   - `--preStopSeconds` (default 10)
   - `--idleTimeout` (default 30)
   - `--dataCenter`
   - `--rack`
   - `--index` [memory|leveldb|leveldbMedium|leveldbLarge]
   - `--disk` [hdd|ssd|<tag>]
   - `--tags`
   - `--dir` (default temp dir)
   - `--dir.idx`
   - `--max` (default "8")
   - `--whiteList`
   - `--minFreeSpacePercent` (default "1")
   - `--minFreeSpace`
   - `--images.fix.orientation` (default false)
   - `--readMode` [local|proxy|redirect] (default "proxy")
   - `--cpuprofile`
   - `--memprofile`
   - `--compactionMBps` (default 0)
   - `--maintenanceMBps` (default 0)
   - `--fileSizeLimitMB` (default 256)
   - `--concurrentUploadLimitMB` (default 0)
   - `--concurrentDownloadLimitMB` (default 0)
   - `--pprof` (default false)
   - `--metricsPort` (default 0)
   - `--metricsIp`
   - `--inflightUploadDataTimeout` (default 60s)
   - `--inflightDownloadDataTimeout` (default 60s)
   - `--hasSlowRead` (default true)
   - `--readBufferSizeMB` (default 4)
   - `--index.leveldbTimeout` (default 0)
   - `--debug` (default false)
   - `--debug.port` (default 6060)

1.5. Implement the same flag validation logic from `startVolumeServer()`:
   - Parse comma-separated `--dir`, `--max`, `--minFreeSpace`, `--disk`, `--tags`
   - Replicate single-value-to-all-dirs expansion
   - Validate count matches between dirs and limits
   - `--mserver` backward compat

1.6. **Test:** `cargo build` succeeds. `cargo run -- --help` shows all flags. Proto types generated.

**Verification:** Run with `--port 8080 --dir /tmp --master localhost:9333` — should parse without error and print config.

---

### Phase 2: Core Storage Types & On-Disk Format
**Goal:** Read and write the SeaweedFS needle/volume binary format bit-for-bit compatible with Go.

**Source files to port:**
- `weed/storage/types/needle_types.go` → `src/storage/types.rs`
- `weed/storage/needle/needle.go` → `src/storage/needle/needle.rs`
- `weed/storage/needle/needle_read.go` → `src/storage/needle/needle_read.rs`
- `weed/storage/needle/needle_write.go` (partial) → `src/storage/needle/needle_write.rs`
- `weed/storage/needle/crc.go` → `src/storage/needle/crc.rs`
- `weed/storage/needle/needle_value_map.go` → `src/storage/needle/needle_value.rs`
- `weed/storage/super_block/super_block.go` → `src/storage/super_block.rs`
- `weed/storage/idx/` → `src/storage/idx/`

**Steps:**

2.1. **Fundamental types** (`types.rs`):
   - `NeedleId` (u64), `Offset` (u32 or u64 depending on version), `Size` (i32, negative = deleted)
   - `Cookie` (u32)
   - `DiskType` enum (HDD, SSD, Custom)
   - Version constants (Version1=1, Version2=2, Version3=3, CurrentVersion=3)
   - Byte serialization matching Go's `binary.BigEndian` encoding

2.2. **SuperBlock** (`super_block.rs`):
   - 8-byte header: Version(1) + ReplicaPlacement(1) + TTL(2) + CompactRevision(2) + Reserved(2)
   - `ReplicaPlacement` struct with same/diff rack/dc counts
   - `TTL` struct with count + unit
   - Read/write from first 8 bytes of `.dat` file
   - Match exact byte layout from `super_block.go`

2.3. **Needle binary format** (`needle.rs`, `needle_read.rs`):
   - Version 2/3 header: Cookie(4) + NeedleId(8) + Size(4)
   - Body: Data, Flags, Name, Mime, PairsSize, Pairs, LastModified, TTL, Checksum, AppendAtNs, Padding
   - CRC32 checksum (matching Go's `crc32.ChecksumIEEE`)
   - Padding to 8-byte alignment
   - Read path: read header → compute body length → read body → verify CRC

2.4. **Idx file format** (`idx/`):
   - Fixed 16-byte records: NeedleId(8) + Offset(4) + Size(4)
   - Sequential append-only file
   - Walk/iterate all entries
   - Binary search not used (loaded into memory map)

2.5. **NeedleMap (in-memory)** (`needle_map.rs`):
   - HashMap<NeedleId, NeedleValue> where NeedleValue = {Offset, Size}
   - Load from `.idx` file on volume mount
   - Support Get, Set, Delete operations
   - Track file count, deleted count, deleted byte count

2.6. **Tests:**
   - Unit test: write a needle to bytes → read it back → verify fields match
   - Unit test: write/read SuperBlock round-trip
   - Unit test: write/read idx entries round-trip
   - **Cross-compat test:** Use Go volume server to create a small volume with known data. Read it from Rust and verify all needles decoded correctly. (Keep test fixture `.dat`/`.idx` files in `tests/fixtures/`)

---

### Phase 3: Volume Struct & Lifecycle
**Goal:** Mount, read from, write to, and unmount a volume.

**Source files to port:**
- `weed/storage/volume.go` → `src/storage/volume.rs`
- `weed/storage/volume_read.go` → `src/storage/volume_read.rs`
- `weed/storage/volume_write.go` → `src/storage/volume_write.rs`
- `weed/storage/volume_loading.go`
- `weed/storage/volume_vacuum.go` → `src/storage/volume_compact.rs`
- `weed/storage/volume_info/volume_info.go` → `src/storage/volume_info.rs`
- `weed/storage/volume_super_block.go`

**Steps:**

3.1. **Volume struct** (`volume.rs`):
   - Fields: Id, dir, dataFile, nm (NeedleMap), SuperBlock, readOnly, lastModifiedTs, lastCompactIndexOffset, lastCompactRevision
   - `noWriteOrDelete` / `noWriteCanDelete` / `readOnly` state flags
   - File handles for `.dat` file (read + append)
   - Lock strategy: `RwLock` for concurrent reads, exclusive writes

3.2. **Volume loading** — exact logic from `volume_loading.go`:
   - Open `.dat` file, read SuperBlock from first 8 bytes
   - Load `.idx` file into NeedleMap
   - Handle `.vif` (VolumeInfo) JSON sidecar file
   - Set volume state based on SuperBlock + VolumeInfo

3.3. **Volume read** (`volume_read.rs`) — from `volume_read.go`:
   - `ReadNeedle(needleId, cookie)`: lookup in NeedleMap → seek in .dat → read needle bytes → verify cookie + CRC → return data
   - Handle deleted needles (Size < 0)
   - `ReadNeedleBlob(offset, size)`: raw blob read
   - `ReadNeedleMeta(needleId, offset, size)`: read metadata only

3.4. **Volume write** (`volume_write.rs`) — from `volume_write.go`:
   - `WriteNeedle(needle)`: serialize needle → append to .dat → update .idx → update NeedleMap
   - `DeleteNeedle(needleId)`: mark as deleted in NeedleMap + append tombstone to .idx
   - File size limit check
   - Concurrent write serialization (mutex on write path)

3.5. **Volume compaction** (`volume_compact.rs`) — from `volume_vacuum.go`:
   - `CheckCompact()`: compute garbage ratio
   - `Compact()`: create new .dat/.idx, copy only live needles, update compact revision
   - `CommitCompact()`: rename compacted files over originals
   - `CleanupCompact()`: remove temp files
   - Throttle by `compactionBytePerSecond`

3.6. **Volume info** (`volume_info.rs`):
   - Read/write `.vif` JSON sidecar
   - VolumeInfo protobuf struct mapping
   - Remote file references for tiered storage

3.7. **Tests:**
   - Mount a volume, write 100 needles, read them all back, verify content
   - Delete 50 needles, verify they return "deleted"
   - Compact, verify only 50 remain, verify content
   - Read Go-created volume fixtures

---

### Phase 4: Store (Multi-Volume, Multi-Disk Manager)
**Goal:** Manage multiple volumes across multiple disk directories.

**Source files to port:**
- `weed/storage/store.go` → `src/storage/store.rs`
- `weed/storage/disk_location.go` → `src/storage/disk_location.rs`
- `weed/storage/store_ec.go`
- `weed/storage/store_state.go`

**Steps:**

4.1. **DiskLocation** (`disk_location.rs`):
   - Directory path, max volume count, min free space, disk type, tags
   - Load all volumes from directory on startup
   - Track free space, check writable

4.2. **Store** (`store.rs`):
   - Vector of `DiskLocation`s
   - `GetVolume(volumeId)` → lookup across all locations
   - `HasVolume(volumeId)` check
   - `AllocateVolume(...)` — create new volume in appropriate location
   - `DeleteVolume(...)`, `MountVolume(...)`, `UnmountVolume(...)`
   - `DeleteCollection(collection)` — delete all volumes of a collection
   - Collect volume status for heartbeat
   - `SetStopping()`, `Close()`
   - Persistent state (maintenance mode) via `store_state.go`

4.3. **Store state** — `VolumeServerState` protobuf with maintenance flag, persisted to disk.

4.4. **Tests:**
   - Create store with 2 dirs, allocate volumes in each, verify load balancing
   - Mount/unmount/delete lifecycle
   - State persistence across restart

---

### Phase 5: Erasure Coding
**Goal:** Full EC shard encode/decode/read/write/rebuild.

**Source files to port:**
- `weed/storage/erasure_coding/` (3,599 lines)

**Steps:**

5.1. **EC volume + shard structs** — `EcVolume`, `EcShard` with file handles for `.ec00`–`.ec13` shard files + `.ecx` index + `.ecj` journal.

5.2. **EC encoder** — Reed-Solomon 10+4 (configurable) encoding using `reed-solomon-erasure` crate:
   - `VolumeEcShardsGenerate`: read .dat → split into data shards → compute parity → write .ec00-.ec13 + .ecx

5.3. **EC decoder/reader** — reconstruct data from any 10 of 14 shards:
   - `EcShardRead`: read range from a specific shard
   - Locate needle in EC volume via .ecx index
   - Handle cross-shard needle reads

5.4. **EC shard operations:**
   - Copy, delete, mount, unmount shards
   - `VolumeEcShardsRebuild`: rebuild missing shards from remaining
   - `VolumeEcShardsToVolume`: reconstruct .dat from EC shards
   - `VolumeEcBlobDelete`: mark deleted in EC journal
   - `VolumeEcShardsInfo`: report shard metadata

5.5. **Tests:**
   - Encode a volume → verify 14 shards created
   - Delete 4 shards → rebuild → verify data intact
   - Read individual needles from EC volume
   - Cross-compat with Go-generated EC shards

---

### Phase 6: Backend / Tiered Storage
**Goal:** Support tiered storage to remote backends (S3, etc).

**Source files to port:**
- `weed/storage/backend/` (1,850 lines)

**Steps:**

6.1. **Backend trait** — abstract `BackendStorage` trait with `ReadAt`, `WriteAt`, `Truncate`, `Close`, `Name`.

6.2. **Disk backend** — default local disk implementation.

6.3. **S3 backend** — upload .dat to S3, read ranges via S3 range requests.

6.4. **Tier move operations:**
   - `VolumeTierMoveDatToRemote`: upload .dat to remote, optionally delete local
   - `VolumeTierMoveDatFromRemote`: download .dat from remote

6.5. **Tests:**
   - Disk backend read/write round-trip
   - S3 backend with mock/localstack

---

### Phase 7: Security Layer
**Goal:** JWT authentication, whitelist guard, TLS configuration.

**Source files to port:**
- `weed/security/guard.go` → `src/security/guard.rs`
- `weed/security/jwt.go` → `src/security/jwt.rs`
- `weed/security/tls.go` → `src/security/tls.rs`

**Steps:**

7.1. **Guard** (`guard.rs`):
   - Whitelist IP check (exact match on `r.RemoteAddr`)
   - Wrap handlers with whitelist enforcement
   - `UpdateWhiteList()` for live reload

7.2. **JWT** (`jwt.rs`):
   - `SeaweedFileIdClaims` with `fid` field
   - Sign with HMAC-SHA256
   - Verify + decode with expiry check
   - Separate signing keys for read vs write
   - `GetJwt(request)` — extract from `Authorization: Bearer` header or `jwt` query param

7.3. **TLS** (`tls.rs`):
   - Load server TLS cert/key for gRPC and HTTPS
   - Load client TLS for mutual TLS
   - Read from `security.toml` config (same format as Go's viper config)

7.4. **Tests:**
   - JWT sign → verify round-trip
   - JWT with wrong key → reject
   - JWT with expired token → reject
   - JWT fid mismatch → reject
   - Whitelist allow/deny

---

### Phase 8: Prometheus Metrics
**Goal:** Export same metric names as Go for dashboard compatibility.

**Source files to port:**
- `weed/stats/metrics.go` (volume server counters/gauges/histograms)

**Steps:**

8.1. Define all Prometheus metrics matching Go names:
   - `VolumeServerRequestCounter` (labels: method, status)
   - `VolumeServerRequestHistogram` (labels: method)
   - `VolumeServerInFlightRequestsGauge` (labels: method)
   - `VolumeServerInFlightUploadSize`
   - `VolumeServerInFlightDownloadSize`
   - `VolumeServerConcurrentUploadLimit`
   - `VolumeServerConcurrentDownloadLimit`
   - `VolumeServerHandlerCounter` (labels: type — UploadLimitCond, DownloadLimitCond)
   - Read/Write/Delete request counters

8.2. Metrics HTTP endpoint on `--metricsPort`.

8.3. Optional push-based metrics loop (`LoopPushingMetric`).

8.4. **Test:** Verify metric names and labels match Go output.

---

### Phase 9: HTTP Server & Handlers
**Goal:** All HTTP endpoints with exact same behavior as Go.

**Source files to port:**
- `weed/server/volume_server.go` → `src/server/volume_server.rs`
- `weed/server/volume_server_handlers.go` → `src/server/http_handlers.rs`
- `weed/server/volume_server_handlers_read.go` → `src/server/http_read.rs`
- `weed/server/volume_server_handlers_write.go` → `src/server/http_write.rs`
- `weed/server/volume_server_handlers_admin.go` → `src/server/http_admin.rs`
- `weed/server/volume_server_handlers_helper.go` (URL parsing, proxy, JSON responses)
- `weed/server/volume_server_handlers_ui.go` → `src/server/http_admin.rs`

**Steps:**

9.1. **URL path parsing** — from `handlers_helper.go`:
   - Parse `/<vid>,<fid>` and `/<vid>/<fid>` patterns
   - Extract volume ID, file ID, filename, ext

9.2. **Route dispatch** — from `privateStoreHandler` and `publicReadOnlyHandler`:
   - `GET /` → `GetOrHeadHandler`
   - `HEAD /` → `GetOrHeadHandler`
   - `POST /` → `PostHandler` (whitelist gated)
   - `PUT /` → `PostHandler` (whitelist gated)
   - `DELETE /` → `DeleteHandler` (whitelist gated)
   - `OPTIONS /` → CORS preflight
   - `GET /status` → JSON status
   - `GET /healthz` → health check
   - `GET /ui/index.html` → HTML UI page
   - Static resources (CSS/JS for UI)

9.3. **GET/HEAD handler** (`http_read.rs`) — from `handlers_read.go` (468 lines):
   - JWT read authorization check
   - Lookup needle by volume ID + needle ID + cookie
   - ETag / If-None-Match / If-Modified-Since conditional responses
   - Content-Type from stored MIME or filename extension
   - Content-Disposition header
   - Content-Encoding (gzip/zstd stored data)
   - Range request support (HTTP 206 Partial Content)
   - JPEG orientation fix (if configured)
   - Proxy to replica on local miss (readMode=proxy)
   - Redirect to replica (readMode=redirect)
   - Download tracking (in-flight size accounting)

9.4. **POST/PUT handler** (`http_write.rs`) — from `handlers_write.go` (170 lines):
   - JWT write authorization check
   - Multipart form parsing
   - Extract file data, filename, content type, TTL, last-modified
   - Optional gzip/zstd compression
   - Write needle to volume
   - Replicate to peers (same logic as Go's `DistributedOperation`)
   - Return JSON: {name, size, eTag, error}

9.5. **DELETE handler** — already in handlers.go:
   - JWT authorization
   - Delete from local volume
   - Replicate delete to peers
   - Return JSON result

9.6. **Admin handlers** (`http_admin.rs`):
   - `/status` → JSON with volumes, version, disk status
   - `/healthz` → 200 OK if serving
   - `/ui/index.html` → HTML dashboard

9.7. **Concurrency limiting** — from `handlers.go`:
   - Upload concurrency limit with `sync::Condvar` + timeout
   - Download concurrency limit with proxy fallback to replicas
   - HTTP 429 on timeout, 499 on client cancel
   - Replication traffic bypasses upload limits

9.8. **Public port** — if configured, separate listener with read-only routes (GET/HEAD/OPTIONS only).

9.9. **Request ID middleware** — generate unique request ID per request.

9.10. **Tests:**
   - Integration: start server → upload file via POST → GET it back → verify content
   - Integration: upload → DELETE → GET returns 404
   - Integration: conditional GET with ETag → 304
   - Integration: range request → 206 with correct bytes
   - Integration: exceed upload limit → 429
   - Integration: whitelist enforcement
   - Integration: JWT enforcement

---

### Phase 10: gRPC Service Implementation
**Goal:** All 40 gRPC methods with exact logic.

**Source files to port:**
- `weed/server/volume_grpc_admin.go` (380 lines)
- `weed/server/volume_grpc_vacuum.go` (124 lines)
- `weed/server/volume_grpc_copy.go` (636 lines)
- `weed/server/volume_grpc_copy_incremental.go` (66 lines)
- `weed/server/volume_grpc_read_write.go` (74 lines)
- `weed/server/volume_grpc_batch_delete.go` (124 lines)
- `weed/server/volume_grpc_tail.go` (140 lines)
- `weed/server/volume_grpc_erasure_coding.go` (619 lines)
- `weed/server/volume_grpc_scrub.go` (121 lines)
- `weed/server/volume_grpc_tier_upload.go` (98 lines)
- `weed/server/volume_grpc_tier_download.go` (85 lines)
- `weed/server/volume_grpc_remote.go` (95 lines)
- `weed/server/volume_grpc_query.go` (69 lines)
- `weed/server/volume_grpc_state.go` (26 lines)
- `weed/server/volume_grpc_read_all.go` (35 lines)
- `weed/server/volume_grpc_client_to_master.go` (325 lines)

**Steps (grouped by functional area):**

10.1. **Implement `tonic::Service` for `VolumeServer`** — the generated trait from proto.

10.2. **Admin RPCs** (`grpc_admin.rs`):
   - `AllocateVolume` — create volume on appropriate disk location
   - `VolumeMount` / `VolumeUnmount` / `VolumeDelete`
   - `VolumeMarkReadonly` / `VolumeMarkWritable`
   - `VolumeConfigure` — change replication
   - `VolumeStatus` — return read-only, size, file counts
   - `VolumeServerStatus` — disk statuses, memory, version, DC, rack
   - `VolumeServerLeave` — deregister from master
   - `DeleteCollection`
   - `VolumeNeedleStatus` — get needle metadata by ID
   - `Ping` — latency measurement
   - `GetState` / `SetState` — maintenance mode

10.3. **Vacuum RPCs** (`grpc_vacuum.rs`):
   - `VacuumVolumeCheck` — return garbage ratio
   - `VacuumVolumeCompact` — stream progress (streaming response)
   - `VacuumVolumeCommit` — finalize compaction
   - `VacuumVolumeCleanup` — remove temp files

10.4. **Copy RPCs** (`grpc_copy.rs`):
   - `VolumeCopy` — stream .dat/.idx from source to create local copy
   - `VolumeSyncStatus` — return sync metadata
   - `VolumeIncrementalCopy` — stream .dat delta since timestamp (streaming)
   - `CopyFile` — generic file copy by extension (streaming)
   - `ReceiveFile` — receive streamed file (client streaming)
   - `ReadVolumeFileStatus` — return file timestamps and sizes

10.5. **Read/Write RPCs** (`grpc_read_write.rs`):
   - `ReadNeedleBlob` — raw needle blob read
   - `ReadNeedleMeta` — needle metadata
   - `WriteNeedleBlob` — raw needle blob write
   - `ReadAllNeedles` — stream all needles from volume(s) (streaming)

10.6. **Batch delete** (`grpc_batch_delete.rs`):
   - `BatchDelete` — delete multiple file IDs, return per-ID results

10.7. **Tail RPCs** (`grpc_tail.rs`):
   - `VolumeTailSender` — stream new needles since timestamp (streaming)
   - `VolumeTailReceiver` — connect to another volume server and tail its changes

10.8. **Erasure coding RPCs** (`grpc_erasure_coding.rs`):
   - `VolumeEcShardsGenerate` — generate EC shards from volume
   - `VolumeEcShardsRebuild` — rebuild missing shards
   - `VolumeEcShardsCopy` — copy shards from another server
   - `VolumeEcShardsDelete` — delete EC shards
   - `VolumeEcShardsMount` / `VolumeEcShardsUnmount`
   - `VolumeEcShardRead` — read from EC shard (streaming)
   - `VolumeEcBlobDelete` — mark blob deleted in EC volume
   - `VolumeEcShardsToVolume` — reconstruct volume from EC shards
   - `VolumeEcShardsInfo` — return shard metadata

10.9. **Scrub RPCs** (`grpc_scrub.rs`):
   - `ScrubVolume` — integrity check volumes (INDEX / FULL / LOCAL modes)
   - `ScrubEcVolume` — integrity check EC volumes

10.10. **Tier RPCs** (`grpc_tier.rs`):
   - `VolumeTierMoveDatToRemote` — upload to remote backend (streaming progress)
   - `VolumeTierMoveDatFromRemote` — download from remote (streaming progress)

10.11. **Remote storage** (`grpc_remote.rs`):
   - `FetchAndWriteNeedle` — fetch from remote storage, write locally, replicate

10.12. **Query** (`grpc_query.rs`):
   - `Query` — experimental CSV/JSON/Parquet select on stored data (streaming)

10.13. **Master heartbeat** (`grpc_client_to_master.rs`):
   - `heartbeat()` background task — periodic gRPC stream to master
   - Send: volume info, EC shard info, disk stats, has-no-space flags, deleted volumes
   - Receive: volume size limit, leader address, metrics config
   - Reconnect on failure with backoff
   - `StopHeartbeat()` for graceful shutdown

10.14. **Tests:**
   - Integration test per RPC: call via tonic client → verify response
   - Streaming RPCs: verify all chunks received
   - Error cases: invalid volume ID, non-existent volume, etc.
   - Heartbeat: mock master gRPC server, verify registration

---

### Phase 11: Startup, Lifecycle & Graceful Shutdown
**Goal:** Full server startup matching Go's `runVolume()` and `startVolumeServer()`.

**Steps:**

11.1. **Startup sequence** (match `volume.go` exactly):
   1. Load security configuration from `security.toml`
   2. Start metrics server on metrics port
   3. Parse folder/max/minFreeSpace/diskType/tags
   4. Validate all directory writable
   5. Resolve IP, bind IP, public URL, gRPC port
   6. Create `VolumeServer` struct
   7. Check with master (initial handshake)
   8. Create `Store` (loads all existing volumes from disk)
   9. Create security `Guard`
   10. Register HTTP routes on admin mux
   11. Optionally register public mux
   12. Start gRPC server on gRPC port
   13. Start public HTTP server (if separated)
   14. Start cluster HTTP server (with optional TLS)
   15. Start heartbeat background task
   16. Start metrics push loop
   17. Register SIGHUP handler for config reload + new volume loading

11.2. **Graceful shutdown** (match Go exactly):
   1. On SIGINT/SIGTERM:
   2. Stop heartbeat (notify master we're leaving)
   3. Wait `preStopSeconds`
   4. Stop public HTTP server
   5. Stop cluster HTTP server
   6. Graceful stop gRPC server
   7. `volumeServer.Shutdown()` → `store.Close()` (flush all volumes)

11.3. **Reload** (SIGHUP):
   - Reload security config
   - Update whitelist
   - Load newly appeared volumes from disk

11.4. **Tests:**
   - Start server → send SIGTERM → verify clean shutdown
   - Start server → SIGHUP → verify config reloaded

---

### Phase 12: Integration & Cross-Compatibility Testing
**Goal:** Rust volume server is a drop-in replacement for Go volume server.

**Steps:**

12.1. **Binary compatibility tests:**
   - Create volumes with Go volume server
   - Start Rust volume server on same data directory
   - Read all data → verify identical
   - Write new data with Rust → read with Go → verify

12.2. **API compatibility tests:**
   - Run same HTTP requests against both Go and Rust servers
   - Compare response bodies, headers, status codes
   - Test all gRPC RPCs against both

12.3. **Master interop test:**
   - Start Go master server
   - Register Rust volume server
   - Verify heartbeat works
   - Verify volume assignment works
   - Upload via filer → stored on Rust volume server → read back

12.4. **Performance benchmarks:**
   - Throughput: sequential writes, sequential reads
   - Latency: p50/p99 for read/write
   - Concurrency: parallel reads/writes
   - Compare Rust vs Go numbers

12.5. **Edge cases:**
   - Volume at max size
   - Disk full handling
   - Corrupt .dat file recovery
   - Network partition during replication
   - EC shard loss + rebuild

---

## Execution Order & Dependencies

```
Phase 1  (Skeleton + CLI)        ← no deps, start here
   ↓
Phase 2  (Storage types)         ← needs Phase 1 (types used everywhere)
   ↓
Phase 3  (Volume struct)         ← needs Phase 2
   ↓
Phase 4  (Store manager)         ← needs Phase 3
   ↓
Phase 7  (Security)              ← independent, can parallel with 3-4
Phase 8  (Metrics)               ← independent, can parallel with 3-4
   ↓
Phase 9  (HTTP server)           ← needs Phase 4 + 7 + 8
Phase 10 (gRPC server)           ← needs Phase 4 + 7 + 8
   ↓
Phase 5  (Erasure coding)        ← needs Phase 4, wire into Phase 10
Phase 6  (Tiered storage)        ← needs Phase 4, wire into Phase 10
   ↓
Phase 11 (Startup + shutdown)    ← needs Phase 9 + 10
   ↓
Phase 12 (Integration tests)     ← needs all above
```

## Estimated Scope

| Phase | Estimated Rust Lines | Complexity |
|---|---|---|
| 1. Skeleton + CLI | ~400 | Low |
| 2. Storage types | ~2,000 | High (binary compat critical) |
| 3. Volume struct | ~2,500 | High |
| 4. Store manager | ~1,000 | Medium |
| 5. Erasure coding | ~3,000 | High |
| 6. Tiered storage | ~1,500 | Medium |
| 7. Security | ~500 | Medium |
| 8. Metrics | ~300 | Low |
| 9. HTTP server | ~2,000 | High |
| 10. gRPC server | ~3,500 | High |
| 11. Startup/shutdown | ~500 | Medium |
| 12. Integration tests | ~2,000 | Medium |
| **Total** | **~19,000** | |

## Critical Invariants to Preserve

1. **Binary format compatibility** — Rust must read/write `.dat`, `.idx`, `.vif`, `.ecX` files identically to Go. A single byte off = data loss.
2. **gRPC wire compatibility** — Same proto, same field semantics. Go master must talk to Rust volume server seamlessly.
3. **HTTP API compatibility** — Same URL patterns, same JSON response shapes, same headers, same status codes.
4. **Replication protocol** — Write replication between Go and Rust volume servers must work bidirectionally.
5. **Heartbeat protocol** — Rust volume server must register with Go master and maintain heartbeat.
6. **CRC32 algorithm** — Must use IEEE polynomial (same as Go's `crc32.ChecksumIEEE`).
7. **JWT compatibility** — Tokens signed by Go filer/master must be verifiable by Rust volume server and vice versa.