snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

weed/s3api/s3lifecycle/DESIGN.md

@@ -0,0 +1,270 @@
+# S3 Lifecycle (Design)
+
+The S3 lifecycle worker replaces the streaming + heap design with a daily meta-log replay. The worker runs as a scheduled job: "start, do today's work, stop" — no long-running per-shard goroutines, no future-buffered match heap.
+
+This document is the as-built reference. For operator-facing guides, see the [SeaweedFS wiki](https://github.com/seaweedfs/seaweedfs/wiki).
+
+## Goal
+
+For each bucket lifecycle rule with TTL `D` days, the worker processes every object whose age exceeds `D` during each scheduled run, dispatching deletes via `LifecycleDelete`. Events whose dispatch returned a retryable / blocked outcome are reprocessed from the persisted cursor on later runs — head-of-line blocking is intentional rather than a per-key retry queue. No future-buffered Matches in memory. Cluster-wide delete rate cap allocated per worker. The worker exits when the pass completes.
+
+## Algorithm
+
+One pass per scheduled invocation. The pass spawns one filer meta-log subscription, fans events out to per-shard processors by `ShardID(bucket, key)`, and drains each shard's events through the router + dispatcher:
+
+```text
+dailyrun.Run(ctx, cfg):
+    runNow             = cfg.Now()                                  // frozen for the whole pass
+    snap               = cfg.Engine.Snapshot()
+    rsh                = engine.ReplayContentHash(snap)
+    maxTTL             = engine.MaxEffectiveTTL(snap)
+
+    if rsh != [32]byte{}:                                           // replay-eligible rules present
+        globalStartTsNs = min over cfg.Shards of (persisted cursor or runNow - maxTTL)
+        reader          = subscribeMeta(ShardPredicate ∈ cfg.Shards, StartTsNs = globalStartTsNs)
+        fanOut(reader.Events → shardEvents[shardID]) until ev.TsNs > runNow
+
+    spawn one goroutine per shard:
+        runShard(ctx, cfg, snap, runNow, shardID, shardEvents[shardID])
+
+    wait all
+    teardown reader + fan-out
+
+runShard(ctx, cfg, snap, runNow, shardID, events):
+    persisted, found   = cfg.Persister.Load(shardID)
+    retentionWindow    = cfg.RetentionWindow or maxTTL              // see "retention" below
+    promoted           = engine.PromotedHash(snap, retentionWindow)
+
+    if rsh == [32]byte{}:                                           // pure walker bucket
+        if walkerDue and Walker:
+            cfg.Walker(walkView)                                    // RulesForShard.walk
+            lastWalkedNs = runNow.UnixNano()
+        save cursor (TsNs=0, rsh, promoted, lastWalkedNs)
+        return
+
+    mustWalkRecovery   = found && (persisted.RuleSetHash != rsh || persisted.PromotedHash != promoted)
+    mustWalkColdStart  = !found
+
+    if mustWalkRecovery or mustWalkColdStart:
+        cfg.Walker(engine.RecoveryView(snap))                       // every rule, force-active
+        walkedThisPass = true
+        lastWalkedNs   = runNow.UnixNano()
+        if mustWalkRecovery:
+            save cursor (TsNs = runNow - maxTTL, rsh, promoted, lastWalkedNs)  // rewind
+            return
+
+    if Walker and !walkedThisPass and walkerDue:
+        cfg.Walker(walkView)                                        // steady-state walker
+        lastWalkedNs = runNow.UnixNano()
+
+    startTsNs = !found ? runNow - maxTTL : persisted.TsNs           // steady state honors cursor
+    lastOK, _, drainErr = drainShardEvents(ctx, cfg, runNow, shardID, snap, startTsNs, events)
+    save cursor (TsNs = lastOK, rsh, promoted, lastWalkedNs)
+```
+
+Walker invocations — three call sites, distinct conditions:
+
+| Branch | View | Trigger | Throttle |
+|---|---|---|---|
+| Recovery | `engine.RecoveryView(snap)` | `mustWalkColdStart` or `mustWalkRecovery` | Unconditional |
+| Steady state | `RulesForShard.walk` | `!walkedThisPass && walkerDue` | `WalkerInterval` |
+| Empty replay | `RulesForShard.walk` | `rsh == [32]byte{}` and `walkerDue` | `WalkerInterval` |
+
+`walkerDue` returns true when `WalkerInterval == 0`, or when no walk has happened yet (`LastWalkedNs == 0`), or when `runNow - LastWalkedNs >= WalkerInterval`. Within-pass double-fire suppression lives in `runShard`'s `walkedThisPass` flag, not in `walkerDue` — the recovery branch fires the walker with `RecoveryView` (a superset of every per-shard partition), so the steady-state branch must not re-walk in the same pass.
+
+## Engine surface
+
+```go
+// In engine:
+func (e *Engine) Snapshot() *Snapshot
+func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot)
+func RecoveryView(s *Snapshot) *Snapshot
+func ReplayContentHash(s *Snapshot) [32]byte
+func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte
+func MaxEffectiveTTL(s *Snapshot) time.Duration
+```
+
+`RulesForShard` and `RecoveryView` return new `*Snapshot` instances with cloned `*CompiledAction` objects and shared (by pointer) rule definitions. Fields that differ from the base:
+
+- **`active`** — per clone, set per view.
+- **`Mode`** — rewritten to `ModeEventDriven` on `replay` clones; preserved on `walk` and `recovery` clones. The rewrite is required because `router.Route` gates on `Mode == ModeEventDriven`, and today's compile preserves a persistent `prior.Mode = ModeScanOnly` that would otherwise lock a rule out of replay even after retention rehabilitates it.
+- **Action-map membership** — `replay` contains only replay-eligible clones; `walk` contains only walker-bound clones; `recovery` contains every action.
+
+Shared-by-pointer with the base: `Rule` definitions, predicate maps, `RuleHash` table.
+
+## `router.Route` integration
+
+`router.Route(ctx, snap, ev, now, lister)` iterates every action with `IsActive() == true` in the snapshot. Two snapshots can't disagree on activation if they share the same `*CompiledAction` pointers, which is why `RulesForShard` clones.
+
+| View | Clone settings | Why |
+|---|---|---|
+| `replay` | `active = true`, `Mode = ModeEventDriven` | `router.Route` requires `ModeEventDriven`. Forced regardless of `prior.Mode`. |
+| `walk` | `active = true`, `Mode` preserved | Walker accepts any non-`ModeDisabled` Mode. |
+| `recovery` | `active = true`, `Mode` preserved | Walker iterates all action clones. |
+
+## Subscription model
+
+One filer `SubscribeMetadata` stream per `dailyrun.Run()` call, covering every shard in `cfg.Shards`. The `Reader` carries a `ShardPredicate func(int) bool` that accepts the shard set; a fan-out goroutine routes events to per-shard channels by `ev.ShardID`. This replaces the earlier model (16 separate per-shard subscriptions per pass).
+
+`globalStartTsNs = min(per-shard cursor, runNow - maxTTL)`. Pre-loaded once at pass start so the subscription's `StartTsNs` covers every shard's needed range; per-shard drains then filter `ev.TsNs <= shard.startTsNs` locally.
+
+The subscription is bounded: `UntilNs = runNow`, so the filer ends the stream once it has delivered everything up to the pass boundary and the pass ends on its own. Fan-out also cancels the reader on the first `ev.TsNs > runNow` as a backstop (meta-log events arrive in TsNs order; everything after is past the boundary) — that used to be the *only* way a pass ended, which wedged the job for as long as the cluster stayed quiet. Per-shard channels are buffered to 256 events — large enough to absorb bursts without back-pressuring the fan-out.
+
+## Action kinds and dispatch paths
+
+Table uses the S3-spec rule names (what operators type in lifecycle XML). The corresponding engine constants in `weed/s3api/s3lifecycle/action_kind.go` are `ActionKindExpirationDays`, `ActionKindNoncurrentDays`, `ActionKindAbortMPU`, `ActionKindExpirationDate`, `ActionKindExpiredDeleteMarker`, `ActionKindNewerNoncurrent` — same one-to-one mapping, shorter spelling.
+
+| ActionKind | Trigger | Due time | Path | Early-stop in replay? |
+|---|---|---|---|---|
+| `ExpirationDays` | Latest-version PUT | `ev.TsNs + r.ExpirationDays` | Replay | Yes |
+| `NoncurrentDays` | Demotion (next PUT for same key) | `entry.NoncurrentSince + r.NoncurrentDays` | Replay | Yes |
+| `AbortIncompleteMultipartUpload` | MPU init | `mpu_init.TsNs + r.AbortMPUDaysAfterInitiation` | Replay | Yes |
+| `ExpirationDate` | Latest-version PUT, fires on `now >= r.ExpirationDate` | `r.ExpirationDate` (constant) | Walker | n/a |
+| `ExpiredObjectDeleteMarker` | Delete marker with `NumVersions == 1` | "now if orphaned, else never" | Walker | n/a |
+| `NewerNoncurrentVersions` | Version becomes noncurrent AND total noncurrents > `r.NewerNoncurrentVersions` | "now if over the cap, else never" | Walker | n/a |
+
+`ExpiredObjectDeleteMarker` and `NewerNoncurrentVersions` are walker-only because their due-time depends on current sibling state, not on any event's TsNs. The `done` early-stop in replay can't engage — there's nothing event-time-monotonic to early-stop on.
+
+## Cursor
+
+Persisted per shard at `/etc/s3/lifecycle/daily-cursors/shard-NN.json`. Shape (`weed/s3api/s3lifecycle/dailyrun/cursor.go`):
+
+```go
+type Cursor struct {
+    TsNs         int64       // last meta-log event whose matches all dispatched
+    RuleSetHash  [32]byte    // ReplayContentHash of the rule set that wrote this
+    PromotedHash [32]byte    // PromotedHash with retentionWindow at write time
+    LastWalkedNs int64       // wall-clock of the last successful walker fire
+}
+```
+
+`LastWalkedNs` is JSON-omitempty, so cursor files written before that field existed decode cleanly as zero (treated as "never walked steady-state" → next pass seeds the anchor).
+
+Cursor save uses a fresh `context.Background()` with a 5s timeout because a caller-imposed wall-clock cap on the pass (the shell driver's `-runtime`) cancels the drain's context. Saving with the canceled context would silently drop the cursor and the next pass would re-replay from the same floor.
+
+In steady state the start position honors the cursor verbatim — the floor `runNow - maxTTL` is applied only on cold start (`!found`). The drain freezes the cursor at the last pre-skip event so pending matches with `DueTime == TsNs + maxTTL` stay in scope across passes; bumping forward in steady state would orphan exactly those events.
+
+## Cursor hashes
+
+The cursor stores two hashes that together detect every situation invalidating "everything before persisted.TsNs has been processed under the same rules":
+
+**`RuleSetHash = engine.ReplayContentHash(snap)`** — content over the rule definitions (action kind, predicate, TTL value) of replay-eligible action kinds. Partition-independent.
+
+**`PromotedHash = engine.PromotedHash(snap, retentionWindow)`** — hash of replay-eligible rules currently classified as `walk` due to `scan_only` promotion (their TTL exceeds `retentionWindow`).
+
+Recovery triggers, complete list:
+
+| Trigger | Detection | Why |
+|---|---|---|
+| Cold start | No persisted cursor | First run for this shard |
+| Replay-rule edit | `RuleSetHash` mismatch | Replay-eligible rule content changed |
+| Partition flip | `PromotedHash` mismatch | A replay-eligible rule moved between `replay` and `walk` |
+
+Retention loss as a recovery trigger is a known gap: in stock SeaweedFS the filer's meta-log is effectively never GC'd (no on-disk retention policy for `/topics/.system/log`), so `cfg.RetentionWindow` defaults to `maxTTL` and PromotedHash stays empty. When operators add explicit meta-log retention, the cursor-vs-earliest-available check becomes load-bearing again.
+
+## Walker throttle
+
+`cfg.WalkerInterval` decouples the walker's cadence from `Run()` invocation cadence. The s3tests CI workflow invokes the worker every 2s; a single daily admin schedule invokes it once per day. The walker fire rate should be set by walk cost, not by the invocation interval — so:
+
+- Steady-state and empty-replay walker fires gate on `walkerDue(persisted.LastWalkedNs, runNow, WalkerInterval)`.
+- `0` keeps the prior "fire every pass" behavior (back-compat for tests).
+- Production: pick the walk cost budget per shard per cluster. Small cluster: 1h. Large cluster: 6h+.
+- Recovery walker fires (cold-start, hash mismatch) are unconditional — these are bounded events that must run once.
+
+Walker fires update `Cursor.LastWalkedNs` so the next pass's throttle has a fresh anchor. The recovery walker also updates it, so a steady-state branch in the same pass doesn't double-walk over the same superset.
+
+## Delete failure handling
+
+Cursor advance is gated on success. The cursor only moves past events whose matches all returned `DONE`, `NOOP_RESOLVED`, or `SKIPPED_OBJECT_LOCK`. Any other outcome (`RETRY_LATER`, `BLOCKED`, transport error after in-run retries) halts the run and persists the cursor at the last fully-processed event.
+
+- **Head-of-line blocking is intentional.** A transient filer error stalls today's pass; tomorrow's run resumes at the same cursor. Loud (operator sees stuck cursor in metrics) and idempotent (identity-CAS makes redundant deletes no-ops).
+- **In-run retry with backoff** for transport errors only — default 3 attempts, exponential backoff capped at 5s. Server-side outcomes are not retried in-run.
+- **No retry queue.** Removing the per-key freeze state was the whole point; adding it back would re-introduce the state machine the redesign replaces.
+
+## Rate limiting
+
+Cluster-wide deletes-per-second cap, set in admin config. The admin allocator:
+
+1. Counts workers capable of `s3_lifecycle` from the registry.
+2. Divides `cluster_deletes_per_second` by the count.
+3. Writes the per-worker share into `ExecuteJobRequest.ClusterContext.Metadata["s3_lifecycle.deletes_per_second"]`.
+
+The worker reads the share and constructs one `golang.org/x/time/rate.Limiter` shared across all shard goroutines. `dispatchWithRetry` calls `limiter.Wait(ctx)` before each `LifecycleDelete` RPC.
+
+## Observability
+
+Per-shard Prometheus gauges (`weed/stats/metrics.go`):
+
+| Metric | What it tells you |
+|---|---|
+| `s3_lifecycle_cursor_min_ts_ns{shard}` | `now - this` is the per-shard replay lag |
+| `s3_lifecycle_daily_run_last_walked_ns{shard}` | `now - this` is walker freshness; stuck = throttle misconfigured or walker failing |
+| `s3_lifecycle_daily_run_shard_duration_seconds{shard}` | Wall-clock per shard pass |
+| `s3_lifecycle_daily_run_events_scanned_total{shard}` | Counter of meta-log events drainShardEvents processed |
+| `s3_lifecycle_dispatch_limiter_wait_seconds` | Per-dispatch wait time on the cluster rate limiter |
+| `s3_lifecycle_dispatch_total{bucket,kind,outcome}` | Per-bucket dispatch counter |
+
+Heartbeat log line, emitted once per `Run()`:
+
+```text
+daily_run: status=ok shards=16 errors=0 duration=7s cursor_lag_max=2h walked_max_age=3m
+```
+
+Tokens `status`, `shards`, `errors`, `duration` are stable for grep. `cursor_lag_max=cold` and `walked_max_age=cold` distinguish "not started yet" from "0s caught up".
+
+## Data model
+
+### `noncurrent_since` on version entries
+
+A non-current version's TTL clock starts when the next version was written, not at its own mtime. The demoting PUT writes `NoncurrentSinceNs` on the demoted entry, set to the TsNs of the demoting meta-log event. Using the meta-log TsNs keeps `noncurrent_since` strictly monotonic in meta-log order across all replicas, immune to wall-clock skew.
+
+The lifecycle evaluator uses `ev.TsNs` for current-version rules and `entry.NoncurrentSinceNs` for noncurrent rules — both monotonic in iteration order. Legacy entries with `NoncurrentSinceNs == 0` fall back to entry mtime.
+
+`expected_mtime` passed to `LifecycleDelete` (for identity CAS) is always the entry's own mtime. CAS identity and TTL clock are separate concerns.
+
+## Components
+
+| Path | Role |
+|---|---|
+| `engine/` | Rule compilation, snapshot, partition views |
+| `evaluate.go`, `due_at.go`, `rule_hash.go`, `tags.go` | Engine-side rule evaluation |
+| `reader/` | Meta-log subscribe; one subscription per dailyrun.Run pass |
+| `router/router.go` | Per-event rule evaluation |
+| `bootstrap/walker.go` | Bucket walker with `RunForShard(view, shardID)` filter |
+| `dispatcher/filer_persister.go` | Filer-backed cursor I/O |
+| `dailyrun/run.go` | Main pass orchestrator: subscription, fan-out, per-shard `runShard` |
+| `dailyrun/cursor.go` | Cursor type + filer JSON serializer |
+| `dailyrun/walker_dispatcher.go` | Adapter from walker to `LifecycleDelete` RPC |
+
+## Configuration
+
+Admin config (`weed/worker/tasks/s3_lifecycle/`):
+
+| Key | Type | Default | What |
+|---|---|---|---|
+| `cluster_deletes_per_second` | int64 | 0 (unlimited) | Cluster-wide ceiling on lifecycle delete RPCs/s. Allocated per worker. |
+| `cluster_deletes_burst` | int64 | 0 (= 2× rate) | Token-bucket burst across cluster. |
+| `meta_log_retention_days` | int64 | 0 (unbounded) | How far back the filer's meta-log can reach. Rules with TTL > retention promote to walker. |
+| `walker_interval_minutes` | int64 | 0 (fire every pass) | Minimum time between steady-state walker fires per shard. Set positive when worker runs at tighter cadence than the desired walk frequency. |
+
+Worker config:
+
+| Key | Default | What |
+|---|---|---|
+| `max_runtime_minutes` | 60 | Wall-clock cap per `dailyrun.Run` call |
+
+## Failure & recovery
+
+- **Worker crashes mid-run.** Cursor advances only past successfully-deleted events. On restart, the next pass resumes at the same cursor and re-attempts. Identity-CAS makes redundant deletes no-ops.
+- **Transient delete failure.** Pass halts at the failing event, cursor stays. Tomorrow's pass retries from the same point. Stuck cursor is visible in `s3_lifecycle_cursor_min_ts_ns`; operators see head-of-line blocking and address the root cause.
+- **Identity drift** (object overwritten between event and delete). Handled by `LifecycleDelete` RPC's identity-CAS, which returns `NOOP_RESOLVED` for stale events. The algorithm dispatches optimistically and lets the server filter.
+- **Cold start, rule edit, partition flip.** All route into the recovery branch. The walker over `engine.RecoveryView(snap)` catches already-due objects across the full rule set, then the cursor rewinds (rule edit) or stays at the cold-start floor.
+
+## Future work
+
+Tracked as optimizations rather than blockers:
+
+1. **Long-lived subscription across passes.** Today the subscription is rebuilt per `Run()`. Keeping it alive across passes would eliminate the 7s ctx-timeout per pass and the start/teardown overhead. Requires per-shard pending heap (events whose `DueTime > runNow` would be parked in-memory instead of replayed) and a hot-swappable snapshot for mid-pass config changes. Multi-day refactor; current model works.
+2. **Bucket-coordinated walker.** Phase 4 has each shard walk the full bucket and filter by `ShardID(bucket, key)` — simple but 16× the listing cost. A per-bucket coordinator (the worker owning shard 0 for that bucket lists once, routes matches to other shards) would cut listing cost. Worth doing if listing becomes the bottleneck for very large buckets.
+3. **Per-bucket dispatch lag metric.** Currently only per-shard lag is exposed. Per-bucket would require a per-bucket cursor or a derived metric from `s3_lifecycle_dispatch_total{bucket,kind,outcome}`. Punted on cardinality concerns; revisit when an operator asks for it.
+4. **Meta-log retention plumbing.** If the filer adds GC for `/topics/.system/log`, the `PromotedHash` partition flip needs to consume the filer's actual retention horizon (currently dormant because retention is effectively infinite).