snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

weed/s3api/s3lifecycle/DESIGN.md

@@ -0,0 +1,270 @@
+# S3 Lifecycle (Design)
+
+The S3 lifecycle worker replaces the streaming + heap design with a daily meta-log replay. The worker runs as a scheduled job: "start, do today's work, stop" — no long-running per-shard goroutines, no future-buffered match heap.
+
+This document is the as-built reference. For operator-facing guides, see the [SeaweedFS wiki](https://github.com/seaweedfs/seaweedfs/wiki).
+
+## Goal
+
+For each bucket lifecycle rule with TTL `D` days, the worker processes every object whose age exceeds `D` during each scheduled run, dispatching deletes via `LifecycleDelete`. Events whose dispatch returned a retryable / blocked outcome are reprocessed from the persisted cursor on later runs — head-of-line blocking is intentional rather than a per-key retry queue. No future-buffered Matches in memory. Cluster-wide delete rate cap allocated per worker. The worker exits when the pass completes.
+
+## Algorithm
+
+One pass per scheduled invocation. The pass spawns one filer meta-log subscription, fans events out to per-shard processors by `ShardID(bucket, key)`, and drains each shard's events through the router + dispatcher:
+
+```text
+dailyrun.Run(ctx, cfg):
+    runNow             = cfg.Now()                                  // frozen for the whole pass
+    snap               = cfg.Engine.Snapshot()
+    rsh                = engine.ReplayContentHash(snap)
+    maxTTL             = engine.MaxEffectiveTTL(snap)
+
+    if rsh != [32]byte{}:                                           // replay-eligible rules present
+        globalStartTsNs = min over cfg.Shards of (persisted cursor or runNow - maxTTL)
+        reader          = subscribeMeta(ShardPredicate ∈ cfg.Shards, StartTsNs = globalStartTsNs)
+        fanOut(reader.Events → shardEvents[shardID]) until ev.TsNs > runNow
+
+    spawn one goroutine per shard:
+        runShard(ctx, cfg, snap, runNow, shardID, shardEvents[shardID])
+
+    wait all
+    teardown reader + fan-out
+
+runShard(ctx, cfg, snap, runNow, shardID, events):
+    persisted, found   = cfg.Persister.Load(shardID)
+    retentionWindow    = cfg.RetentionWindow or maxTTL              // see "retention" below
+    promoted           = engine.PromotedHash(snap, retentionWindow)
+
+    if rsh == [32]byte{}:                                           // pure walker bucket
+        if walkerDue and Walker:
+            cfg.Walker(walkView)                                    // RulesForShard.walk
+            lastWalkedNs = runNow.UnixNano()
+        save cursor (TsNs=0, rsh, promoted, lastWalkedNs)
+        return
+
+    mustWalkRecovery   = found && (persisted.RuleSetHash != rsh || persisted.PromotedHash != promoted)
+    mustWalkColdStart  = !found
+
+    if mustWalkRecovery or mustWalkColdStart:
+        cfg.Walker(engine.RecoveryView(snap))                       // every rule, force-active
+        walkedThisPass = true
+        lastWalkedNs   = runNow.UnixNano()
+        if mustWalkRecovery:
+            save cursor (TsNs = runNow - maxTTL, rsh, promoted, lastWalkedNs)  // rewind
+            return
+
+    if Walker and !walkedThisPass and walkerDue:
+        cfg.Walker(walkView)                                        // steady-state walker
+        lastWalkedNs = runNow.UnixNano()
+
+    startTsNs = !found ? runNow - maxTTL : persisted.TsNs           // steady state honors cursor
+    lastOK, _, drainErr = drainShardEvents(ctx, cfg, runNow, shardID, snap, startTsNs, events)
+    save cursor (TsNs = lastOK, rsh, promoted, lastWalkedNs)
+```
+
+Walker invocations — three call sites, distinct conditions:
+
+| Branch | View | Trigger | Throttle |
+|---|---|---|---|
+| Recovery | `engine.RecoveryView(snap)` | `mustWalkColdStart` or `mustWalkRecovery` | Unconditional |
+| Steady state | `RulesForShard.walk` | `!walkedThisPass && walkerDue` | `WalkerInterval` |
+| Empty replay | `RulesForShard.walk` | `rsh == [32]byte{}` and `walkerDue` | `WalkerInterval` |
+
+`walkerDue` returns true when `WalkerInterval == 0`, or when no walk has happened yet (`LastWalkedNs == 0`), or when `runNow - LastWalkedNs >= WalkerInterval`. Within-pass double-fire suppression lives in `runShard`'s `walkedThisPass` flag, not in `walkerDue` — the recovery branch fires the walker with `RecoveryView` (a superset of every per-shard partition), so the steady-state branch must not re-walk in the same pass.
+
+## Engine surface
+
+```go
+// In engine:
+func (e *Engine) Snapshot() *Snapshot
+func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot)
+func RecoveryView(s *Snapshot) *Snapshot
+func ReplayContentHash(s *Snapshot) [32]byte
+func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte
+func MaxEffectiveTTL(s *Snapshot) time.Duration
+```
+
+`RulesForShard` and `RecoveryView` return new `*Snapshot` instances with cloned `*CompiledAction` objects and shared (by pointer) rule definitions. Fields that differ from the base:
+
+- **`active`** — per clone, set per view.
+- **`Mode`** — rewritten to `ModeEventDriven` on `replay` clones; preserved on `walk` and `recovery` clones. The rewrite is required because `router.Route` gates on `Mode == ModeEventDriven`, and today's compile preserves a persistent `prior.Mode = ModeScanOnly` that would otherwise lock a rule out of replay even after retention rehabilitates it.
+- **Action-map membership** — `replay` contains only replay-eligible clones; `walk` contains only walker-bound clones; `recovery` contains every action.
+
+Shared-by-pointer with the base: `Rule` definitions, predicate maps, `RuleHash` table.
+
+## `router.Route` integration
+
+`router.Route(ctx, snap, ev, now, lister)` iterates every action with `IsActive() == true` in the snapshot. Two snapshots can't disagree on activation if they share the same `*CompiledAction` pointers, which is why `RulesForShard` clones.
+
+| View | Clone settings | Why |
+|---|---|---|
+| `replay` | `active = true`, `Mode = ModeEventDriven` | `router.Route` requires `ModeEventDriven`. Forced regardless of `prior.Mode`. |
+| `walk` | `active = true`, `Mode` preserved | Walker accepts any non-`ModeDisabled` Mode. |
+| `recovery` | `active = true`, `Mode` preserved | Walker iterates all action clones. |
+
+## Subscription model
+
+One filer `SubscribeMetadata` stream per `dailyrun.Run()` call, covering every shard in `cfg.Shards`. The `Reader` carries a `ShardPredicate func(int) bool` that accepts the shard set; a fan-out goroutine routes events to per-shard channels by `ev.ShardID`. This replaces the earlier model (16 separate per-shard subscriptions per pass).
+
+`globalStartTsNs = min(per-shard cursor, runNow - maxTTL)`. Pre-loaded once at pass start so the subscription's `StartTsNs` covers every shard's needed range; per-shard drains then filter `ev.TsNs <= shard.startTsNs` locally.
+
+The subscription is bounded: `UntilNs = runNow`, so the filer ends the stream once it has delivered everything up to the pass boundary and the pass ends on its own. Fan-out also cancels the reader on the first `ev.TsNs > runNow` as a backstop (meta-log events arrive in TsNs order; everything after is past the boundary) — that used to be the *only* way a pass ended, which wedged the job for as long as the cluster stayed quiet. Per-shard channels are buffered to 256 events — large enough to absorb bursts without back-pressuring the fan-out.
+
+## Action kinds and dispatch paths
+
+Table uses the S3-spec rule names (what operators type in lifecycle XML). The corresponding engine constants in `weed/s3api/s3lifecycle/action_kind.go` are `ActionKindExpirationDays`, `ActionKindNoncurrentDays`, `ActionKindAbortMPU`, `ActionKindExpirationDate`, `ActionKindExpiredDeleteMarker`, `ActionKindNewerNoncurrent` — same one-to-one mapping, shorter spelling.
+
+| ActionKind | Trigger | Due time | Path | Early-stop in replay? |
+|---|---|---|---|---|
+| `ExpirationDays` | Latest-version PUT | `ev.TsNs + r.ExpirationDays` | Replay | Yes |
+| `NoncurrentDays` | Demotion (next PUT for same key) | `entry.NoncurrentSince + r.NoncurrentDays` | Replay | Yes |
+| `AbortIncompleteMultipartUpload` | MPU init | `mpu_init.TsNs + r.AbortMPUDaysAfterInitiation` | Replay | Yes |
+| `ExpirationDate` | Latest-version PUT, fires on `now >= r.ExpirationDate` | `r.ExpirationDate` (constant) | Walker | n/a |
+| `ExpiredObjectDeleteMarker` | Delete marker with `NumVersions == 1` | "now if orphaned, else never" | Walker | n/a |
+| `NewerNoncurrentVersions` | Version becomes noncurrent AND total noncurrents > `r.NewerNoncurrentVersions` | "now if over the cap, else never" | Walker | n/a |
+
+`ExpiredObjectDeleteMarker` and `NewerNoncurrentVersions` are walker-only because their due-time depends on current sibling state, not on any event's TsNs. The `done` early-stop in replay can't engage — there's nothing event-time-monotonic to early-stop on.
+
+## Cursor
+
+Persisted per shard at `/etc/s3/lifecycle/daily-cursors/shard-NN.json`. Shape (`weed/s3api/s3lifecycle/dailyrun/cursor.go`):
+
+```go
+type Cursor struct {
+    TsNs         int64       // last meta-log event whose matches all dispatched
+    RuleSetHash  [32]byte    // ReplayContentHash of the rule set that wrote this
+    PromotedHash [32]byte    // PromotedHash with retentionWindow at write time
+    LastWalkedNs int64       // wall-clock of the last successful walker fire
+}
+```
+
+`LastWalkedNs` is JSON-omitempty, so cursor files written before that field existed decode cleanly as zero (treated as "never walked steady-state" → next pass seeds the anchor).
+
+Cursor save uses a fresh `context.Background()` with a 5s timeout because a caller-imposed wall-clock cap on the pass (the shell driver's `-runtime`) cancels the drain's context. Saving with the canceled context would silently drop the cursor and the next pass would re-replay from the same floor.
+
+In steady state the start position honors the cursor verbatim — the floor `runNow - maxTTL` is applied only on cold start (`!found`). The drain freezes the cursor at the last pre-skip event so pending matches with `DueTime == TsNs + maxTTL` stay in scope across passes; bumping forward in steady state would orphan exactly those events.
+
+## Cursor hashes
+
+The cursor stores two hashes that together detect every situation invalidating "everything before persisted.TsNs has been processed under the same rules":
+
+**`RuleSetHash = engine.ReplayContentHash(snap)`** — content over the rule definitions (action kind, predicate, TTL value) of replay-eligible action kinds. Partition-independent.
+
+**`PromotedHash = engine.PromotedHash(snap, retentionWindow)`** — hash of replay-eligible rules currently classified as `walk` due to `scan_only` promotion (their TTL exceeds `retentionWindow`).
+
+Recovery triggers, complete list:
+
+| Trigger | Detection | Why |
+|---|---|---|
+| Cold start | No persisted cursor | First run for this shard |
+| Replay-rule edit | `RuleSetHash` mismatch | Replay-eligible rule content changed |
+| Partition flip | `PromotedHash` mismatch | A replay-eligible rule moved between `replay` and `walk` |
+
+Retention loss as a recovery trigger is a known gap: in stock SeaweedFS the filer's meta-log is effectively never GC'd (no on-disk retention policy for `/topics/.system/log`), so `cfg.RetentionWindow` defaults to `maxTTL` and PromotedHash stays empty. When operators add explicit meta-log retention, the cursor-vs-earliest-available check becomes load-bearing again.
+
+## Walker throttle
+
+`cfg.WalkerInterval` decouples the walker's cadence from `Run()` invocation cadence. The s3tests CI workflow invokes the worker every 2s; a single daily admin schedule invokes it once per day. The walker fire rate should be set by walk cost, not by the invocation interval — so:
+
+- Steady-state and empty-replay walker fires gate on `walkerDue(persisted.LastWalkedNs, runNow, WalkerInterval)`.
+- `0` keeps the prior "fire every pass" behavior (back-compat for tests).
+- Production: pick the walk cost budget per shard per cluster. Small cluster: 1h. Large cluster: 6h+.
+- Recovery walker fires (cold-start, hash mismatch) are unconditional — these are bounded events that must run once.
+
+Walker fires update `Cursor.LastWalkedNs` so the next pass's throttle has a fresh anchor. The recovery walker also updates it, so a steady-state branch in the same pass doesn't double-walk over the same superset.
+
+## Delete failure handling
+
+Cursor advance is gated on success. The cursor only moves past events whose matches all returned `DONE`, `NOOP_RESOLVED`, or `SKIPPED_OBJECT_LOCK`. Any other outcome (`RETRY_LATER`, `BLOCKED`, transport error after in-run retries) halts the run and persists the cursor at the last fully-processed event.
+
+- **Head-of-line blocking is intentional.** A transient filer error stalls today's pass; tomorrow's run resumes at the same cursor. Loud (operator sees stuck cursor in metrics) and idempotent (identity-CAS makes redundant deletes no-ops).
+- **In-run retry with backoff** for transport errors only — default 3 attempts, exponential backoff capped at 5s. Server-side outcomes are not retried in-run.
+- **No retry queue.** Removing the per-key freeze state was the whole point; adding it back would re-introduce the state machine the redesign replaces.
+
+## Rate limiting
+
+Cluster-wide deletes-per-second cap, set in admin config. The admin allocator:
+
+1. Counts workers capable of `s3_lifecycle` from the registry.
+2. Divides `cluster_deletes_per_second` by the count.
+3. Writes the per-worker share into `ExecuteJobRequest.ClusterContext.Metadata["s3_lifecycle.deletes_per_second"]`.
+
+The worker reads the share and constructs one `golang.org/x/time/rate.Limiter` shared across all shard goroutines. `dispatchWithRetry` calls `limiter.Wait(ctx)` before each `LifecycleDelete` RPC.
+
+## Observability
+
+Per-shard Prometheus gauges (`weed/stats/metrics.go`):
+
+| Metric | What it tells you |
+|---|---|
+| `s3_lifecycle_cursor_min_ts_ns{shard}` | `now - this` is the per-shard replay lag |
+| `s3_lifecycle_daily_run_last_walked_ns{shard}` | `now - this` is walker freshness; stuck = throttle misconfigured or walker failing |
+| `s3_lifecycle_daily_run_shard_duration_seconds{shard}` | Wall-clock per shard pass |
+| `s3_lifecycle_daily_run_events_scanned_total{shard}` | Counter of meta-log events drainShardEvents processed |
+| `s3_lifecycle_dispatch_limiter_wait_seconds` | Per-dispatch wait time on the cluster rate limiter |
+| `s3_lifecycle_dispatch_total{bucket,kind,outcome}` | Per-bucket dispatch counter |
+
+Heartbeat log line, emitted once per `Run()`:
+
+```text
+daily_run: status=ok shards=16 errors=0 duration=7s cursor_lag_max=2h walked_max_age=3m
+```
+
+Tokens `status`, `shards`, `errors`, `duration` are stable for grep. `cursor_lag_max=cold` and `walked_max_age=cold` distinguish "not started yet" from "0s caught up".
+
+## Data model
+
+### `noncurrent_since` on version entries
+
+A non-current version's TTL clock starts when the next version was written, not at its own mtime. The demoting PUT writes `NoncurrentSinceNs` on the demoted entry, set to the TsNs of the demoting meta-log event. Using the meta-log TsNs keeps `noncurrent_since` strictly monotonic in meta-log order across all replicas, immune to wall-clock skew.
+
+The lifecycle evaluator uses `ev.TsNs` for current-version rules and `entry.NoncurrentSinceNs` for noncurrent rules — both monotonic in iteration order. Legacy entries with `NoncurrentSinceNs == 0` fall back to entry mtime.
+
+`expected_mtime` passed to `LifecycleDelete` (for identity CAS) is always the entry's own mtime. CAS identity and TTL clock are separate concerns.
+
+## Components
+
+| Path | Role |
+|---|---|
+| `engine/` | Rule compilation, snapshot, partition views |
+| `evaluate.go`, `due_at.go`, `rule_hash.go`, `tags.go` | Engine-side rule evaluation |
+| `reader/` | Meta-log subscribe; one subscription per dailyrun.Run pass |
+| `router/router.go` | Per-event rule evaluation |
+| `bootstrap/walker.go` | Bucket walker with `RunForShard(view, shardID)` filter |
+| `dispatcher/filer_persister.go` | Filer-backed cursor I/O |
+| `dailyrun/run.go` | Main pass orchestrator: subscription, fan-out, per-shard `runShard` |
+| `dailyrun/cursor.go` | Cursor type + filer JSON serializer |
+| `dailyrun/walker_dispatcher.go` | Adapter from walker to `LifecycleDelete` RPC |
+
+## Configuration
+
+Admin config (`weed/worker/tasks/s3_lifecycle/`):
+
+| Key | Type | Default | What |
+|---|---|---|---|
+| `cluster_deletes_per_second` | int64 | 0 (unlimited) | Cluster-wide ceiling on lifecycle delete RPCs/s. Allocated per worker. |
+| `cluster_deletes_burst` | int64 | 0 (= 2× rate) | Token-bucket burst across cluster. |
+| `meta_log_retention_days` | int64 | 0 (unbounded) | How far back the filer's meta-log can reach. Rules with TTL > retention promote to walker. |
+| `walker_interval_minutes` | int64 | 0 (fire every pass) | Minimum time between steady-state walker fires per shard. Set positive when worker runs at tighter cadence than the desired walk frequency. |
+
+Worker config:
+
+| Key | Default | What |
+|---|---|---|
+| `max_runtime_minutes` | 60 | Wall-clock cap per `dailyrun.Run` call |
+
+## Failure & recovery
+
+- **Worker crashes mid-run.** Cursor advances only past successfully-deleted events. On restart, the next pass resumes at the same cursor and re-attempts. Identity-CAS makes redundant deletes no-ops.
+- **Transient delete failure.** Pass halts at the failing event, cursor stays. Tomorrow's pass retries from the same point. Stuck cursor is visible in `s3_lifecycle_cursor_min_ts_ns`; operators see head-of-line blocking and address the root cause.
+- **Identity drift** (object overwritten between event and delete). Handled by `LifecycleDelete` RPC's identity-CAS, which returns `NOOP_RESOLVED` for stale events. The algorithm dispatches optimistically and lets the server filter.
+- **Cold start, rule edit, partition flip.** All route into the recovery branch. The walker over `engine.RecoveryView(snap)` catches already-due objects across the full rule set, then the cursor rewinds (rule edit) or stays at the cold-start floor.
+
+## Future work
+
+Tracked as optimizations rather than blockers:
+
+1. **Long-lived subscription across passes.** Today the subscription is rebuilt per `Run()`. Keeping it alive across passes would eliminate the 7s ctx-timeout per pass and the start/teardown overhead. Requires per-shard pending heap (events whose `DueTime > runNow` would be parked in-memory instead of replayed) and a hot-swappable snapshot for mid-pass config changes. Multi-day refactor; current model works.
+2. **Bucket-coordinated walker.** Phase 4 has each shard walk the full bucket and filter by `ShardID(bucket, key)` — simple but 16× the listing cost. A per-bucket coordinator (the worker owning shard 0 for that bucket lists once, routes matches to other shards) would cut listing cost. Worth doing if listing becomes the bottleneck for very large buckets.
+3. **Per-bucket dispatch lag metric.** Currently only per-shard lag is exposed. Per-bucket would require a per-bucket cursor or a derived metric from `s3_lifecycle_dispatch_total{bucket,kind,outcome}`. Punted on cardinality concerns; revisit when an operator asks for it.
+4. **Meta-log retention plumbing.** If the filer adds GC for `/topics/.system/log`, the `PromotedHash` partition flip needs to consume the filer's actual retention horizon (currently dormant because retention is effectively infinite).

weed/s3api/s3lifecycle/action_kind.go

@@ -0,0 +1,75 @@
+package s3lifecycle
+
+// ActionKey is the engine-wide identity of one compiled lifecycle action.
+// Bucket is part of the key because two buckets may carry rules with
+// identical RuleHash; without scoping they'd collide in any keyed map.
+// The on-disk path /etc/s3/lifecycle/<bucket>/<rule_hash>/<action_kind>/
+// mirrors this shape.
+type ActionKey struct {
+	Bucket     string
+	RuleHash   [8]byte
+	ActionKind ActionKind
+}
+
+// ActionKind values mirror the wire-form enum in s3_lifecycle.proto.
+type ActionKind int
+
+const (
+	ActionKindUnspecified ActionKind = iota
+	ActionKindExpirationDays
+	ActionKindExpirationDate
+	ActionKindNoncurrentDays
+	ActionKindNewerNoncurrent
+	ActionKindAbortMPU
+	ActionKindExpiredDeleteMarker
+)
+
+// String returns the leaf-directory name used in
+// /etc/s3/lifecycle/<bucket>/<rule_hash>/<action_kind>/.
+func (k ActionKind) String() string {
+	switch k {
+	case ActionKindExpirationDays:
+		return "expiration_days"
+	case ActionKindExpirationDate:
+		return "expiration_date"
+	case ActionKindNoncurrentDays:
+		return "noncurrent_days"
+	case ActionKindNewerNoncurrent:
+		return "newer_noncurrent"
+	case ActionKindAbortMPU:
+		return "abort_mpu"
+	case ActionKindExpiredDeleteMarker:
+		return "expired_delete_marker"
+	default:
+		return "unspecified"
+	}
+}
+
+// RuleActionKinds returns the compiled actions a single XML rule expands to,
+// in deterministic order. NewerNoncurrentVersions paired with NoncurrentDays
+// is subsumed into NONCURRENT_DAYS; only stand-alone NewerNoncurrent
+// produces a NEWER_NONCURRENT action.
+func RuleActionKinds(rule *Rule) []ActionKind {
+	if rule == nil {
+		return nil
+	}
+	var kinds []ActionKind
+	if rule.ExpirationDays > 0 {
+		kinds = append(kinds, ActionKindExpirationDays)
+	}
+	if !rule.ExpirationDate.IsZero() {
+		kinds = append(kinds, ActionKindExpirationDate)
+	}
+	if rule.ExpiredObjectDeleteMarker {
+		kinds = append(kinds, ActionKindExpiredDeleteMarker)
+	}
+	if rule.NoncurrentVersionExpirationDays > 0 {
+		kinds = append(kinds, ActionKindNoncurrentDays)
+	} else if rule.NewerNoncurrentVersions > 0 {
+		kinds = append(kinds, ActionKindNewerNoncurrent)
+	}
+	if rule.AbortMPUDaysAfterInitiation > 0 {
+		kinds = append(kinds, ActionKindAbortMPU)
+	}
+	return kinds
+}

weed/s3api/s3lifecycle/action_kind_test.go

@@ -0,0 +1,86 @@
+package s3lifecycle
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestRuleActionKinds_SingleAction(t *testing.T) {
+	cases := []struct {
+		name string
+		rule *Rule
+		want []ActionKind
+	}{
+		{"expiration_days", &Rule{ExpirationDays: 30}, []ActionKind{ActionKindExpirationDays}},
+		{"expiration_date", &Rule{ExpirationDate: mustTime(t, "2025-06-15T00:00:00Z")}, []ActionKind{ActionKindExpirationDate}},
+		{"expired_delete_marker", &Rule{ExpiredObjectDeleteMarker: true}, []ActionKind{ActionKindExpiredDeleteMarker}},
+		{"noncurrent_days", &Rule{NoncurrentVersionExpirationDays: 30}, []ActionKind{ActionKindNoncurrentDays}},
+		{"newer_noncurrent_alone", &Rule{NewerNoncurrentVersions: 3}, []ActionKind{ActionKindNewerNoncurrent}},
+		{"abort_mpu", &Rule{AbortMPUDaysAfterInitiation: 7}, []ActionKind{ActionKindAbortMPU}},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			if got := RuleActionKinds(c.rule); !reflect.DeepEqual(got, c.want) {
+				t.Fatalf("want %v, got %v", c.want, got)
+			}
+		})
+	}
+}
+
+func TestRuleActionKinds_MultiAction(t *testing.T) {
+	// The bug this fixes: a single XML <Rule> can declare ExpirationDays,
+	// AbortMPU, and NoncurrentDays in parallel. Each must compile to its
+	// own action with its own delay group / mode / pending — not be
+	// collapsed into one entry whose delay is the smallest.
+	rule := &Rule{
+		ExpirationDays:                  90,
+		AbortMPUDaysAfterInitiation:     7,
+		NoncurrentVersionExpirationDays: 30,
+	}
+	want := []ActionKind{
+		ActionKindExpirationDays,
+		ActionKindNoncurrentDays,
+		ActionKindAbortMPU,
+	}
+	if got := RuleActionKinds(rule); !reflect.DeepEqual(got, want) {
+		t.Fatalf("want %v, got %v", want, got)
+	}
+}
+
+func TestRuleActionKinds_NoncurrentDaysSubsumesNewerNoncurrent(t *testing.T) {
+	// When both NoncurrentDays and NewerNoncurrent are set on the same rule,
+	// they are AWS-paired conditions for ONE noncurrent expiration action,
+	// not two; only NoncurrentDays is emitted.
+	rule := &Rule{NoncurrentVersionExpirationDays: 30, NewerNoncurrentVersions: 3}
+	want := []ActionKind{ActionKindNoncurrentDays}
+	if got := RuleActionKinds(rule); !reflect.DeepEqual(got, want) {
+		t.Fatalf("want %v, got %v", want, got)
+	}
+}
+
+func TestRuleActionKinds_Empty(t *testing.T) {
+	if got := RuleActionKinds(&Rule{}); len(got) != 0 {
+		t.Fatalf("want empty, got %v", got)
+	}
+	if got := RuleActionKinds(nil); got != nil {
+		t.Fatalf("want nil, got %v", got)
+	}
+}
+
+func TestActionKind_StringIsStable(t *testing.T) {
+	// The string is the leaf-directory name on disk; renaming would
+	// orphan existing state, so pin the values.
+	cases := map[ActionKind]string{
+		ActionKindExpirationDays:      "expiration_days",
+		ActionKindExpirationDate:      "expiration_date",
+		ActionKindNoncurrentDays:      "noncurrent_days",
+		ActionKindNewerNoncurrent:     "newer_noncurrent",
+		ActionKindAbortMPU:            "abort_mpu",
+		ActionKindExpiredDeleteMarker: "expired_delete_marker",
+	}
+	for k, want := range cases {
+		if got := k.String(); got != want {
+			t.Fatalf("ActionKind(%d).String() = %q, want %q", k, got, want)
+		}
+	}
+}

weed/s3api/s3lifecycle/bootstrap/has_prefix_test.go

@@ -0,0 +1,33 @@
+package bootstrap
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+// HasPrefix is a thin wrapper around strings.HasPrefix exported by the
+// bootstrap package so call sites can avoid pulling strings just to do
+// a prefix check. Trivial but at 0% coverage — pin it.
+
+func TestHasPrefix(t *testing.T) {
+	cases := []struct {
+		name   string
+		path   string
+		prefix string
+		want   bool
+	}{
+		{"matching prefix", "logs/2026/01/01", "logs/", true},
+		{"exact match", "logs/", "logs/", true},
+		{"non-matching prefix", "metrics/x", "logs/", false},
+		{"shorter than prefix", "logs", "logs/", false},
+		{"empty prefix matches all", "anything", "", true},
+		{"both empty", "", "", true},
+		{"empty input non-empty prefix", "", "x", false},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			assert.Equal(t, c.want, HasPrefix(c.path, c.prefix))
+		})
+	}
+}

weed/s3api/s3lifecycle/bootstrap/walker.go

@@ -0,0 +1,197 @@
+// Package bootstrap is the bucket-level lifecycle walker. The walker
+// iterates every entry under a bucket, evaluates every active ActionKey
+// against it, and dispatches inline-delete RPCs for currently-due actions;
+// not-yet-due entries are left for the meta-log reader to pick up later.
+//
+// Callback-driven so the listing source and the LifecycleDelete dispatcher
+// can be supplied separately (real client or test fake).
+package bootstrap
+
+import (
+	"context"
+	"strings"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/seaweedfs/seaweedfs/weed/stats"
+)
+
+// Entry is the routing-relevant slice of a filer entry. SuccessorModTime
+// and NoncurrentIndex are populated only on versioned-bucket walks; the
+// retention path bails out conservatively when they're zero / nil.
+//
+// MPU init directories at .uploads/<id> populate DestKey with the
+// destination object key (from entry.Extended[ExtMultipartObjectKey]) so
+// rule-prefix matching works on the user's intended path while Path
+// stays as the upload directory the dispatcher must rm.
+type Entry struct {
+	Path           string
+	DestKey        string
+	ModTime        time.Time
+	Size           int64
+	IsDirectory    bool
+	IsLatest       bool
+	IsDeleteMarker bool
+	IsMPUInit      bool
+	NumVersions    int
+	Tags           map[string]string
+
+	SuccessorModTime time.Time
+	NoncurrentIndex  *int
+
+	// VersionID is the S3 version id of this entry, empty for
+	// non-versioned buckets. Populated by the ListFunc adapter when
+	// the entry came from a `.versions/` directory; the walker itself
+	// doesn't use it, but the Dispatcher needs it to address the
+	// right version on LifecycleDelete.
+	VersionID string
+}
+
+// ListFunc must skip entries with Path <= start so kill-resume picks up
+// where the previous run stopped.
+type ListFunc func(ctx context.Context, bucket, start string, cb func(*Entry) error) error
+
+// Dispatcher executes one (action, entry) verdict. An error halts the walk;
+// the caller decides whether to retry from the recorded last_scanned_path.
+type Dispatcher interface {
+	Delete(ctx context.Context, action *engine.CompiledAction, entry *Entry) error
+}
+
+// Checkpoint is the resume state. Caller persists it under
+// /etc/s3/lifecycle/<bucket>/_bootstrap.
+type Checkpoint struct {
+	LastScannedPath string
+	Completed       bool
+}
+
+type WalkOptions struct {
+	Resume string
+	Now    time.Time
+}
+
+// Walk iterates entries via list, evaluates each active ActionKey via
+// MatchPath + EvaluateAction, and calls Dispatcher.Delete for currently-due
+// actions. SCAN_AT_DATE actions are skipped (their bootstrap is scheduled
+// separately).
+func Walk(ctx context.Context, snap *engine.Snapshot, bucket string, list ListFunc, dispatch Dispatcher, opts WalkOptions) (Checkpoint, error) {
+	now := opts.Now
+	if now.IsZero() {
+		now = time.Now().UTC()
+	}
+	cp := Checkpoint{LastScannedPath: opts.Resume}
+
+	// Reuse one ObjectInfo across the walk; EvaluateAction reads it
+	// synchronously without retaining.
+	var info s3lifecycle.ObjectInfo
+
+	err := list(ctx, bucket, opts.Resume, func(entry *Entry) error {
+		if entry == nil || entry.Path == "" {
+			return nil
+		}
+		// MPU init directories at .uploads/<id> are the one directory
+		// shape lifecycle cares about; everything else stays out.
+		if entry.IsDirectory && !entry.IsMPUInit {
+			cp.LastScannedPath = entry.Path
+			return nil
+		}
+		if err := walkEntry(ctx, snap, bucket, entry, dispatch, now, &info); err != nil {
+			return err
+		}
+		cp.LastScannedPath = entry.Path
+		return nil
+	})
+	if err != nil {
+		return cp, err
+	}
+	cp.Completed = true
+	return cp, nil
+}
+
+func walkEntry(ctx context.Context, snap *engine.Snapshot, bucket string, entry *Entry, dispatch Dispatcher, now time.Time, info *s3lifecycle.ObjectInfo) error {
+	// MPU init: rule-prefix matching uses the destination key, not the
+	// .uploads/<id> directory path. A bare directory with no DestKey is
+	// either a stray dir or an init mid-write before metadata landed —
+	// skip rather than guess.
+	matchKey := entry.Path
+	if entry.IsMPUInit {
+		if entry.DestKey == "" {
+			return nil
+		}
+		matchKey = entry.DestKey
+	}
+	keys := snap.MatchPath(bucket, matchKey, nil)
+	if len(keys) == 0 {
+		return nil
+	}
+	*info = s3lifecycle.ObjectInfo{
+		Key:              matchKey,
+		ModTime:          entry.ModTime,
+		Size:             entry.Size,
+		IsLatest:         entry.IsLatest,
+		IsDeleteMarker:   entry.IsDeleteMarker,
+		IsMPUInit:        entry.IsMPUInit,
+		NumVersions:      entry.NumVersions,
+		SuccessorModTime: entry.SuccessorModTime,
+		NoncurrentIndex:  entry.NoncurrentIndex,
+		Tags:             entry.Tags,
+	}
+	for _, key := range keys {
+		action := snap.Action(key)
+		if action == nil {
+			continue
+		}
+		// DISABLED can be flipped at runtime independent of XML Status,
+		// so skip it even though EvaluateAction would also reject.
+		// SCAN_AT_DATE actions are processed here too — the date check
+		// in EvaluateAction (now.Before(rule.ExpirationDate)) gates the
+		// dispatch, so pre-date walks are no-ops and post-date walks
+		// expire eligible objects. The earlier "scan-at-date runs its
+		// own bootstrap" plan was never wired; until that lands, the
+		// regular bootstrap walk is the only path that fires
+		// ExpirationDate rules.
+		if action.Mode == engine.ModeDisabled {
+			continue
+		}
+		// (kind, info) shape gate: ABORT_MPU only on MPU init records,
+		// every other kind only on regular objects/versions. Mismatched
+		// pairs would either dispatch a noncurrent action with empty
+		// version_id (server BLOCKs, cursor freezes) or dispatch
+		// ABORT_MPU against a regular object path.
+		if entry.IsMPUInit && key.ActionKind != s3lifecycle.ActionKindAbortMPU {
+			continue
+		}
+		if !entry.IsMPUInit && key.ActionKind == s3lifecycle.ActionKindAbortMPU {
+			continue
+		}
+		res := s3lifecycle.EvaluateAction(action.Rule, key.ActionKind, info, now)
+		if res.Action == s3lifecycle.ActionNone {
+			continue
+		}
+		if err := dispatch.Delete(ctx, action, entry); err != nil {
+			glog.Warningf("lifecycle bootstrap: dispatch %s/%s kind=%s: %v",
+				bucket, entry.Path, key.ActionKind, err)
+			return err
+		}
+		stats.S3LifecycleBootstrapDispatchCounter.WithLabelValues(bucket, key.ActionKind.String()).Inc()
+	}
+	return nil
+}
+
+// EntryCallback wraps an in-memory slice as a ListFunc; useful for tests.
+func EntryCallback(entries []*Entry) ListFunc {
+	return func(ctx context.Context, bucket, start string, cb func(*Entry) error) error {
+		for _, e := range entries {
+			if start != "" && e.Path <= start {
+				continue
+			}
+			if err := cb(e); err != nil {
+				return err
+			}
+		}
+		return nil
+	}
+}
+
+func HasPrefix(path, prefix string) bool { return strings.HasPrefix(path, prefix) }

weed/s3api/s3lifecycle/bootstrap/walker_test.go

@@ -0,0 +1,431 @@
+package bootstrap
+
+import (
+	"context"
+	"errors"
+	"reflect"
+	"sort"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+)
+
+// recorder captures dispatched (action, entry) pairs for assertion.
+type recorder struct {
+	calls []dispatchCall
+	err   error // when set, every Delete returns this error
+}
+
+type dispatchCall struct {
+	kind s3lifecycle.ActionKind
+	path string
+}
+
+func (r *recorder) Delete(ctx context.Context, action *engine.CompiledAction, entry *Entry) error {
+	if r.err != nil {
+		return r.err
+	}
+	r.calls = append(r.calls, dispatchCall{kind: action.Key.ActionKind, path: entry.Path})
+	return nil
+}
+
+func mustTime(t *testing.T, s string) time.Time {
+	t.Helper()
+	tm, err := time.Parse(time.RFC3339, s)
+	if err != nil {
+		t.Fatalf("parse %s: %v", s, err)
+	}
+	return tm
+}
+
+func compileEvDriven(t *testing.T, bucket string, rules ...*s3lifecycle.Rule) *engine.Snapshot {
+	t.Helper()
+	prior := map[s3lifecycle.ActionKey]engine.PriorState{}
+	for _, r := range rules {
+		rh := s3lifecycle.RuleHash(r)
+		for _, k := range s3lifecycle.RuleActionKinds(r) {
+			prior[s3lifecycle.ActionKey{Bucket: bucket, RuleHash: rh, ActionKind: k}] = engine.PriorState{BootstrapComplete: true}
+		}
+	}
+	e := engine.New()
+	return e.Compile([]engine.CompileInput{{Bucket: bucket, Rules: rules}}, engine.CompileOptions{PriorStates: prior})
+}
+
+func TestWalk_DispatchesDueActions(t *testing.T) {
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+		Prefix:         "logs/",
+	}
+	snap := compileEvDriven(t, "bk", rule)
+
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 60) // past the 30d threshold
+	entries := []*Entry{
+		{Path: "data/x", IsLatest: true, ModTime: mod}, // wrong prefix
+		{Path: "logs/a", IsLatest: true, ModTime: mod}, // due
+		{Path: "logs/b", IsLatest: true, ModTime: now}, // not yet due (mod=now)
+	}
+
+	rec := &recorder{}
+	cp, err := Walk(context.Background(), snap, "bk", EntryCallback(entries), rec, WalkOptions{Now: now})
+	if err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if !cp.Completed {
+		t.Fatalf("walk should complete")
+	}
+	if cp.LastScannedPath != "logs/b" {
+		t.Fatalf("checkpoint last scanned want logs/b, got %q", cp.LastScannedPath)
+	}
+	if len(rec.calls) != 1 || rec.calls[0].path != "logs/a" {
+		t.Fatalf("dispatched calls want [logs/a], got %v", rec.calls)
+	}
+}
+
+func TestWalk_MultiActionRule_AllDueDispatched(t *testing.T) {
+	// One rule with three actions; all currently-due for the entry. The
+	// walker must dispatch one Delete per action — this is the
+	// regression that the per-action keying fixes.
+	rule := &s3lifecycle.Rule{
+		ID:                              "multi",
+		Status:                          s3lifecycle.StatusEnabled,
+		ExpirationDays:                  30,
+		NoncurrentVersionExpirationDays: 7,
+		AbortMPUDaysAfterInitiation:     5,
+	}
+	snap := compileEvDriven(t, "bk", rule)
+
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 100) // past every threshold
+	// Use a single entry that satisfies all three action shapes is
+	// unrealistic; in practice each shape is a different entry. Cover
+	// each shape independently.
+	entries := []*Entry{
+		// Current version under ExpirationDays.
+		{Path: "obj/a", IsLatest: true, ModTime: mod},
+		// Non-current version under NoncurrentDays.
+		{Path: "obj/a/.versions/v1", IsLatest: false, ModTime: mod, SuccessorModTime: mod},
+		// MPU init under AbortMPU. Real init is a directory; DestKey
+		// carries the eventual object key for prefix matching.
+		{Path: ".uploads/u1", IsDirectory: true, IsMPUInit: true, DestKey: "obj/a", ModTime: mod},
+	}
+
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback(entries), rec, WalkOptions{Now: now}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+
+	// Exact-shape assertion: each entry dispatches exactly one action,
+	// and ABORT_MPU only fires on the .uploads/<id> entry. A weaker
+	// "kinds-as-set" check would have missed the (kind, info) gating
+	// regression where an MPU init also fired NONCURRENT_DAYS.
+	want := []dispatchCall{
+		{kind: s3lifecycle.ActionKindAbortMPU, path: ".uploads/u1"},
+		{kind: s3lifecycle.ActionKindExpirationDays, path: "obj/a"},
+		{kind: s3lifecycle.ActionKindNoncurrentDays, path: "obj/a/.versions/v1"},
+	}
+	got := append([]dispatchCall(nil), rec.calls...)
+	sort.Slice(got, func(i, j int) bool {
+		if got[i].path != got[j].path {
+			return got[i].path < got[j].path
+		}
+		return got[i].kind < got[j].kind
+	})
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("dispatch calls mismatch:\n got %+v\nwant %+v", got, want)
+	}
+}
+
+func TestWalk_NotYetDueSkipped(t *testing.T) {
+	// The reader (Phase 3) is responsible for not-yet-due entries; the
+	// walker dispatches only currently-due ones, so the meta-log path
+	// stays the steady-state route.
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	snap := compileEvDriven(t, "bk", rule)
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.Add(s3lifecycle.DaysToDuration(10)) // before the 30d threshold
+
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback([]*Entry{
+		{Path: "x/a", IsLatest: true, ModTime: mod},
+	}), rec, WalkOptions{Now: now}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 0 {
+		t.Fatalf("not-yet-due entry should not dispatch, got %v", rec.calls)
+	}
+}
+
+func TestWalk_DateActionFiresAfterDate(t *testing.T) {
+	// The dedicated SCAN_AT_DATE bootstrap was never wired; until it
+	// lands, the regular bootstrap walker is the only path that fires
+	// ExpirationDate rules. Entries past the rule's date must dispatch.
+	date := mustTime(t, "2025-06-15T00:00:00Z")
+	rule := &s3lifecycle.Rule{
+		ID:             "d",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDate: date,
+	}
+	snap := compileEvDriven(t, "bk", rule)
+
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback([]*Entry{
+		{Path: "x/a", IsLatest: true, ModTime: mustTime(t, "2024-01-01T00:00:00Z")},
+	}), rec, WalkOptions{Now: date.AddDate(0, 1, 0)}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 1 || rec.calls[0].path != "x/a" {
+		t.Fatalf("date kind past the rule date must dispatch, got %v", rec.calls)
+	}
+}
+
+func TestWalk_DateActionSkippedBeforeDate(t *testing.T) {
+	// Pre-date walks are no-ops: EvaluateAction returns ActionNone when
+	// now is before rule.ExpirationDate.
+	date := mustTime(t, "2025-06-15T00:00:00Z")
+	rule := &s3lifecycle.Rule{
+		ID:             "d",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDate: date,
+	}
+	snap := compileEvDriven(t, "bk", rule)
+
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback([]*Entry{
+		{Path: "x/a", IsLatest: true, ModTime: mustTime(t, "2024-01-01T00:00:00Z")},
+	}), rec, WalkOptions{Now: date.AddDate(0, -1, 0)}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 0 {
+		t.Fatalf("pre-date walk must not dispatch, got %v", rec.calls)
+	}
+}
+
+func TestWalk_DirectoryEntriesSkipped(t *testing.T) {
+	// SeaweedFS directory entries can co-exist in the listing; the walker
+	// must not dispatch deletes against them even when their path matches.
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 1}
+	snap := compileEvDriven(t, "bk", rule)
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 10)
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback([]*Entry{
+		{Path: "x", IsDirectory: true, ModTime: mod}, // directory; must skip
+		{Path: "x/file", IsLatest: true, ModTime: mod},
+	}), rec, WalkOptions{Now: now}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 1 || rec.calls[0].path != "x/file" {
+		t.Fatalf("only the file should dispatch, got %v", rec.calls)
+	}
+}
+
+func TestWalk_DisabledModeSkipped(t *testing.T) {
+	// An operator-flipped ModeDisabled must short-circuit the walker even
+	// when the XML rule status is "Enabled" and EvaluateAction would
+	// otherwise fire.
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 1}
+	rh := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]engine.PriorState{
+		{Bucket: "bk", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}: {
+			BootstrapComplete: true, Mode: engine.ModeDisabled,
+		},
+	}
+	e := engine.New()
+	snap := e.Compile([]engine.CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}}, engine.CompileOptions{PriorStates: prior})
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 10)
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback([]*Entry{
+		{Path: "x/a", IsLatest: true, ModTime: mod},
+	}), rec, WalkOptions{Now: now}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 0 {
+		t.Fatalf("disabled action must not dispatch, got %v", rec.calls)
+	}
+}
+
+func TestWalk_PendingBootstrapNotDispatched(t *testing.T) {
+	// Without bootstrap_complete=true in PriorStates, the engine compiles
+	// the action as inactive. MatchPath filters on IsActive, so the
+	// walker won't dispatch.
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 1,
+	}
+	e := engine.New()
+	snap := e.Compile([]engine.CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}}, engine.CompileOptions{})
+
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 10)
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback([]*Entry{
+		{Path: "x/a", IsLatest: true, ModTime: mod},
+	}), rec, WalkOptions{Now: now}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 0 {
+		t.Fatalf("inactive action should not dispatch, got %v", rec.calls)
+	}
+}
+
+func TestWalk_DispatchErrorHaltsAtCheckpoint(t *testing.T) {
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 1,
+	}
+	snap := compileEvDriven(t, "bk", rule)
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 10)
+	entries := []*Entry{
+		{Path: "a", IsLatest: true, ModTime: mod},
+		{Path: "b", IsLatest: true, ModTime: mod},
+		{Path: "c", IsLatest: true, ModTime: mod},
+	}
+
+	wantErr := errors.New("dispatch boom")
+	rec := &recorder{err: wantErr}
+	cp, err := Walk(context.Background(), snap, "bk", EntryCallback(entries), rec, WalkOptions{Now: now})
+	if !errors.Is(err, wantErr) {
+		t.Fatalf("want dispatch error, got %v", err)
+	}
+	if cp.Completed {
+		t.Fatalf("walk should not be Completed on dispatch failure")
+	}
+	// Walker stops on first failure; checkpoint stays at whatever was
+	// recorded BEFORE the failed entry. Path "a" is the failing entry,
+	// so LastScannedPath stays at the resume point (empty here).
+	if cp.LastScannedPath != "" {
+		t.Fatalf("checkpoint should not advance past failing entry, got %q", cp.LastScannedPath)
+	}
+}
+
+func TestWalk_ResumeFromCheckpoint(t *testing.T) {
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 1,
+	}
+	snap := compileEvDriven(t, "bk", rule)
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 10)
+	entries := []*Entry{
+		{Path: "a", IsLatest: true, ModTime: mod},
+		{Path: "b", IsLatest: true, ModTime: mod},
+		{Path: "c", IsLatest: true, ModTime: mod},
+	}
+
+	rec := &recorder{}
+	cp, err := Walk(context.Background(), snap, "bk", EntryCallback(entries), rec, WalkOptions{Now: now, Resume: "b"})
+	if err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if !cp.Completed {
+		t.Fatalf("walk should complete")
+	}
+	// Only "c" is processed (entries with Path <= "b" are skipped).
+	if len(rec.calls) != 1 || rec.calls[0].path != "c" {
+		t.Fatalf("Resume should only process c, got %v", rec.calls)
+	}
+	if cp.LastScannedPath != "c" {
+		t.Fatalf("checkpoint want c, got %q", cp.LastScannedPath)
+	}
+}
+
+func TestWalk_MPUInitDirMatchesByDestKey(t *testing.T) {
+	// Existing in-flight MPUs predate the meta-log subscription, so they
+	// only get cleaned up via the bootstrap walk. The init record is a
+	// directory whose path is .uploads/<id>; the rule's Filter.Prefix
+	// applies to the destination object key, not the upload directory.
+	rule := &s3lifecycle.Rule{
+		ID:                          "r-mpu",
+		Status:                      s3lifecycle.StatusEnabled,
+		Prefix:                      "logs/",
+		AbortMPUDaysAfterInitiation: 7,
+	}
+	snap := compileEvDriven(t, "bk", rule)
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 8) // past the 7d threshold
+
+	entries := []*Entry{
+		// Matches: dest key under logs/.
+		{Path: ".uploads/u-match", IsDirectory: true, IsMPUInit: true, DestKey: "logs/foo.txt", ModTime: mod},
+		// Filtered out: dest key under data/.
+		{Path: ".uploads/u-skip", IsDirectory: true, IsMPUInit: true, DestKey: "data/foo.txt", ModTime: mod},
+		// No DestKey: malformed init mid-write; skip rather than guess.
+		{Path: ".uploads/u-bare", IsDirectory: true, IsMPUInit: true, ModTime: mod},
+	}
+
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback(entries), rec, WalkOptions{Now: now}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 1 {
+		t.Fatalf("expected 1 dispatch (u-match only), got %v", rec.calls)
+	}
+	if rec.calls[0].path != ".uploads/u-match" {
+		t.Fatalf("dispatch path=%q, want .uploads/u-match (the rm target)", rec.calls[0].path)
+	}
+	if rec.calls[0].kind != s3lifecycle.ActionKindAbortMPU {
+		t.Fatalf("dispatch kind=%v, want AbortMPU", rec.calls[0].kind)
+	}
+}
+
+func TestWalk_NonMPUDirectorySkipped(t *testing.T) {
+	// Non-MPU directories must still be skipped — the relaxed
+	// IsDirectory check is gated on IsMPUInit.
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 1}
+	snap := compileEvDriven(t, "bk", rule)
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 100)
+
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback([]*Entry{
+		{Path: "a/", IsDirectory: true, IsLatest: true, ModTime: mod},
+	}), rec, WalkOptions{Now: now}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 0 {
+		t.Fatalf("plain directory should not dispatch, got %v", rec.calls)
+	}
+}
+
+func TestWalk_MPUInitDoesNotFireNoncurrent(t *testing.T) {
+	// Same rule covers both AbortMPU and NoncurrentVersionExpiration; the
+	// MPU init record must dispatch only the AbortMPU action. Without the
+	// engine guard, NONCURRENT_DAYS would fire (IsLatest=false) and the
+	// server would BLOCK on empty version_id, freezing the cursor.
+	rule := &s3lifecycle.Rule{
+		ID:                              "r",
+		Status:                          s3lifecycle.StatusEnabled,
+		AbortMPUDaysAfterInitiation:     7,
+		NoncurrentVersionExpirationDays: 7,
+	}
+	snap := compileEvDriven(t, "bk", rule)
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.AddDate(0, 0, 30)
+
+	rec := &recorder{}
+	if _, err := Walk(context.Background(), snap, "bk", EntryCallback([]*Entry{
+		{Path: ".uploads/u1", IsDirectory: true, IsMPUInit: true, DestKey: "obj/a", ModTime: mod},
+	}), rec, WalkOptions{Now: now}); err != nil {
+		t.Fatalf("Walk: %v", err)
+	}
+	if len(rec.calls) != 1 {
+		t.Fatalf("expected 1 dispatch (AbortMPU only), got %v", rec.calls)
+	}
+	if rec.calls[0].kind != s3lifecycle.ActionKindAbortMPU {
+		t.Fatalf("kind=%v, want AbortMPU", rec.calls[0].kind)
+	}
+}

weed/s3api/s3lifecycle/dailyrun/cursor.go

@@ -0,0 +1,163 @@
+// Package dailyrun implements the daily-replay s3 lifecycle worker
+// described in weed/s3api/s3lifecycle/DESIGN.md. One pass per day per
+// shard reads the meta-log forward from the persisted cursor, drains
+// every event whose due_time is past now, and exits — replacing the
+// streaming + heap pipeline with a bounded, idempotent scan.
+//
+// Phase 2 (this file's first version): replay-only. Buckets whose
+// compiled rules include walker-bound action kinds (ExpirationDate,
+// ExpiredDeleteMarker, NewerNoncurrent) or any scan_only promotion are
+// refused with a typed error so flipping the algorithm flag is a loud
+// failure rather than silent data loss. Phase 4 wires the walker and
+// the recovery branch.
+package dailyrun
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dispatcher"
+)
+
+// CursorDir is the filer directory holding per-shard daily-replay
+// cursor files. Kept distinct from dispatcher.CursorDir so a deployment
+// running both algorithms during cutover doesn't have one trample the
+// other's persisted state.
+const CursorDir = "/etc/s3/lifecycle/daily-cursors"
+
+// cursorFileVersion bumps when the on-disk shape changes. Phase 2
+// writes version 1; Phase 4 will continue writing version 1 since the
+// schema (TsNs + RuleSetHash + PromotedHash) is already final.
+const cursorFileVersion = 1
+
+// Cursor captures everything daily_run needs to decide whether to
+// recover (Phase 4) or continue steady-state. TsNs is the latest
+// meta-log event whose Matches all dispatched successfully (or as
+// NOOP_RESOLVED); RuleSetHash is the content hash of the replay-eligible
+// rules from the run that wrote this cursor; PromotedHash is the hash
+// of replay-eligible rules currently in scan_only. LastWalkedNs is the
+// runNow of the most recent successful steady-state / empty-replay
+// walker fire on this shard — runShard uses it together with
+// cfg.WalkerInterval to throttle the walker independently of Run()
+// invocation frequency (so a worker driven by a 2s test ticker doesn't
+// crush filer with a full bucket walk per tick).
+//
+// In Phase 2 PromotedHash is always the empty hash because the walker
+// path isn't wired yet — any rule that would land in walk causes the
+// run to refuse. Phase 4 starts writing a real value.
+type Cursor struct {
+	TsNs         int64
+	RuleSetHash  [32]byte
+	PromotedHash [32]byte
+	LastWalkedNs int64
+}
+
+// cursorFile is the on-disk JSON shape. Bytes are base64-encoded by
+// encoding/json automatically.
+//
+// last_walked_ns is omitempty so cursor files written before the field
+// existed still decode cleanly: an older file has no last_walked_ns,
+// which marshals back into LastWalkedNs=0, which runShard treats as
+// "never walked steady-state" and so the next pass fires the walker
+// (the same path a cold-start cursor takes). No version bump needed.
+type cursorFile struct {
+	Version      int    `json:"version"`
+	ShardID      int    `json:"shard_id"`
+	TsNs         int64  `json:"ts_ns"`
+	RuleSetHash  []byte `json:"rule_set_hash"`
+	PromotedHash []byte `json:"promoted_hash"`
+	LastWalkedNs int64  `json:"last_walked_ns,omitempty"`
+}
+
+// CursorPersister loads and saves daily-replay cursors. The Phase 2
+// production implementation is FilerCursorPersister; tests inject a
+// fake.
+type CursorPersister interface {
+	Load(ctx context.Context, shardID int) (Cursor, bool, error) // (cursor, found, err)
+	Save(ctx context.Context, shardID int, c Cursor) error
+}
+
+// FilerCursorPersister writes cursors to CursorDir as one JSON file per
+// shard. Reuses dispatcher.FilerStore for the actual filer I/O so both
+// algorithms share the same minimal storage abstraction.
+type FilerCursorPersister struct {
+	Store dispatcher.FilerStore
+}
+
+func cursorFileName(shardID int) string {
+	return fmt.Sprintf("shard-%02d.json", shardID)
+}
+
+// Load returns (zero-Cursor, false, nil) only when the cursor file
+// does not exist yet (cold start). Every other failure mode — empty
+// file, malformed JSON, wrong version, wrong shard, hash slices not
+// exactly 32 bytes — returns an error so the daily run halts and is
+// fixed by an operator rather than silently re-scanning from time zero.
+//
+// Strict shape validation is load-bearing because the cursor is
+// load → mutate → save in a single run: a partial-truncate that
+// silently zero-padded the rule_set_hash would be persisted back as a
+// real-looking hash, masking the corruption forever.
+func (p *FilerCursorPersister) Load(ctx context.Context, shardID int) (Cursor, bool, error) {
+	if p.Store == nil {
+		return Cursor{}, false, errors.New("FilerCursorPersister: nil Store")
+	}
+	content, err := p.Store.Read(ctx, CursorDir, cursorFileName(shardID))
+	if err != nil {
+		if errors.Is(err, filer_pb.ErrNotFound) {
+			return Cursor{}, false, nil
+		}
+		return Cursor{}, false, fmt.Errorf("cursor read shard=%d: %w", shardID, err)
+	}
+	if len(content) == 0 {
+		return Cursor{}, false, fmt.Errorf("cursor shard=%d: file exists but is empty (partial write or external truncation)", shardID)
+	}
+	var cf cursorFile
+	if err := json.Unmarshal(content, &cf); err != nil {
+		return Cursor{}, false, fmt.Errorf("cursor decode shard=%d: %w", shardID, err)
+	}
+	if cf.Version != cursorFileVersion {
+		return Cursor{}, false, fmt.Errorf("cursor version shard=%d: got %d, want %d", shardID, cf.Version, cursorFileVersion)
+	}
+	if cf.ShardID != shardID {
+		return Cursor{}, false, fmt.Errorf("cursor shard mismatch: file declares shard=%d, requested shard=%d", cf.ShardID, shardID)
+	}
+	if len(cf.RuleSetHash) != 32 {
+		return Cursor{}, false, fmt.Errorf("cursor rule_set_hash shard=%d: got %d bytes, want 32", shardID, len(cf.RuleSetHash))
+	}
+	if len(cf.PromotedHash) != 32 {
+		return Cursor{}, false, fmt.Errorf("cursor promoted_hash shard=%d: got %d bytes, want 32", shardID, len(cf.PromotedHash))
+	}
+	c := Cursor{TsNs: cf.TsNs, LastWalkedNs: cf.LastWalkedNs}
+	copy(c.RuleSetHash[:], cf.RuleSetHash)
+	copy(c.PromotedHash[:], cf.PromotedHash)
+	return c, true, nil
+}
+
+func (p *FilerCursorPersister) Save(ctx context.Context, shardID int, c Cursor) error {
+	if p.Store == nil {
+		return errors.New("FilerCursorPersister: nil Store")
+	}
+	cf := cursorFile{
+		Version:      cursorFileVersion,
+		ShardID:      shardID,
+		TsNs:         c.TsNs,
+		RuleSetHash:  c.RuleSetHash[:],
+		PromotedHash: c.PromotedHash[:],
+		LastWalkedNs: c.LastWalkedNs,
+	}
+	var buf bytes.Buffer
+	enc := json.NewEncoder(&buf)
+	enc.SetIndent("", "  ")
+	if err := enc.Encode(cf); err != nil {
+		return fmt.Errorf("cursor encode shard=%d: %w", shardID, err)
+	}
+	if err := p.Store.Save(ctx, CursorDir, cursorFileName(shardID), buf.Bytes()); err != nil {
+		return fmt.Errorf("cursor save shard=%d: %w", shardID, err)
+	}
+	return nil
+}

weed/s3api/s3lifecycle/dailyrun/cursor_summary_test.go

@@ -0,0 +1,74 @@
+package dailyrun
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// TestSummarizeShardCursorLag_AllColdStart pins the marker tokens for
+// the worker's very first heartbeat — no cursor file exists yet, no
+// walker has fired, the line must say "cold" rather than "0s" so
+// operators can tell the difference between "I'm caught up" and
+// "I haven't started." Both states would be 0s otherwise.
+func TestSummarizeShardCursorLag_AllColdStart(t *testing.T) {
+	p := newMemPersister()
+	cfg := Config{Persister: p, Shards: []int{0, 1, 2}}
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+
+	got := summarizeShardCursorLag(context.Background(), cfg, runNow)
+	assert.Equal(t, "cursor_lag_max=cold walked_max_age=cold", got)
+}
+
+// TestSummarizeShardCursorLag_PicksMaxAcrossShards confirms the worst-
+// case lag wins, not the average. Operators alert on the worst shard;
+// the cluster is only as healthy as its slowest one.
+func TestSummarizeShardCursorLag_PicksMaxAcrossShards(t *testing.T) {
+	p := newMemPersister()
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+	require.NoError(t, p.Save(context.Background(), 0, Cursor{
+		TsNs:         runNow.Add(-5 * time.Minute).UnixNano(),
+		LastWalkedNs: runNow.Add(-30 * time.Minute).UnixNano(),
+	}))
+	require.NoError(t, p.Save(context.Background(), 1, Cursor{
+		TsNs:         runNow.Add(-3 * time.Hour).UnixNano(), // worst cursor
+		LastWalkedNs: runNow.Add(-10 * time.Minute).UnixNano(),
+	}))
+	require.NoError(t, p.Save(context.Background(), 2, Cursor{
+		TsNs:         runNow.Add(-30 * time.Minute).UnixNano(),
+		LastWalkedNs: runNow.Add(-2 * time.Hour).UnixNano(), // worst walker
+	}))
+	cfg := Config{Persister: p, Shards: []int{0, 1, 2}}
+
+	got := summarizeShardCursorLag(context.Background(), cfg, runNow)
+	assert.Contains(t, got, "cursor_lag_max=3h0m0s")
+	assert.Contains(t, got, "walked_max_age=2h0m0s")
+}
+
+// TestSummarizeShardCursorLag_PartialFill — some shards have cursors,
+// some don't. Don't let the unwalked-yet shards contaminate the worst-
+// case calculation; just compute the max over what we have.
+func TestSummarizeShardCursorLag_PartialFill(t *testing.T) {
+	p := newMemPersister()
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+	require.NoError(t, p.Save(context.Background(), 0, Cursor{
+		TsNs: runNow.Add(-15 * time.Minute).UnixNano(),
+		// LastWalkedNs not set — shard 0 hasn't walked yet
+	}))
+	require.NoError(t, p.Save(context.Background(), 1, Cursor{
+		// TsNs not set — shard 1 hasn't drained yet
+		LastWalkedNs: runNow.Add(-45 * time.Minute).UnixNano(),
+	}))
+	cfg := Config{Persister: p, Shards: []int{0, 1, 2}}
+
+	got := summarizeShardCursorLag(context.Background(), cfg, runNow)
+	// Only shard 0 contributes to cursor_lag_max; only shard 1 to walked_max_age.
+	assert.Contains(t, got, "cursor_lag_max=15m0s")
+	assert.Contains(t, got, "walked_max_age=45m0s")
+	// Shard 2 wasn't saved at all — it doesn't show up as either.
+	assert.False(t, strings.Contains(got, "cold"), "any-fill case should not emit cold markers")
+}

weed/s3api/s3lifecycle/dailyrun/cursor_test.go

@@ -0,0 +1,155 @@
+package dailyrun
+
+import (
+	"context"
+	"sync"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeStore is a minimal in-memory dispatcher.FilerStore for cursor tests.
+type fakeStore struct {
+	mu    sync.Mutex
+	files map[string][]byte
+}
+
+func newFakeStore() *fakeStore {
+	return &fakeStore{files: map[string][]byte{}}
+}
+
+func (s *fakeStore) key(dir, name string) string { return dir + "/" + name }
+
+func (s *fakeStore) Read(_ context.Context, dir, name string) ([]byte, error) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	b, ok := s.files[s.key(dir, name)]
+	if !ok {
+		return nil, filer_pb.ErrNotFound
+	}
+	return append([]byte(nil), b...), nil
+}
+
+func (s *fakeStore) Save(_ context.Context, dir, name string, content []byte) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.files[s.key(dir, name)] = append([]byte(nil), content...)
+	return nil
+}
+
+func TestFilerCursorPersister_EmptyLoadReturnsNotFound(t *testing.T) {
+	p := &FilerCursorPersister{Store: newFakeStore()}
+	c, found, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	assert.False(t, found)
+	assert.Equal(t, Cursor{}, c)
+}
+
+func TestFilerCursorPersister_SaveLoadRoundTrip(t *testing.T) {
+	p := &FilerCursorPersister{Store: newFakeStore()}
+	in := Cursor{
+		TsNs: 1700000000_000000123,
+	}
+	for i := range in.RuleSetHash {
+		in.RuleSetHash[i] = byte(i + 1)
+	}
+	for i := range in.PromotedHash {
+		in.PromotedHash[i] = byte(0xff - i)
+	}
+	require.NoError(t, p.Save(context.Background(), 3, in))
+
+	out, found, err := p.Load(context.Background(), 3)
+	require.NoError(t, err)
+	require.True(t, found)
+	assert.Equal(t, in, out)
+}
+
+func TestFilerCursorPersister_IsolatesShards(t *testing.T) {
+	p := &FilerCursorPersister{Store: newFakeStore()}
+	a := Cursor{TsNs: 100}
+	b := Cursor{TsNs: 200}
+	require.NoError(t, p.Save(context.Background(), 0, a))
+	require.NoError(t, p.Save(context.Background(), 1, b))
+
+	got0, _, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	got1, _, err := p.Load(context.Background(), 1)
+	require.NoError(t, err)
+	assert.Equal(t, int64(100), got0.TsNs)
+	assert.Equal(t, int64(200), got1.TsNs)
+}
+
+func TestFilerCursorPersister_CorruptDataReturnsError(t *testing.T) {
+	store := newFakeStore()
+	store.files[store.key(CursorDir, "shard-00.json")] = []byte("not json")
+	p := &FilerCursorPersister{Store: store}
+	_, _, err := p.Load(context.Background(), 0)
+	require.Error(t, err)
+}
+
+func TestFilerCursorPersister_EmptyFileReturnsError(t *testing.T) {
+	// A cursor file that exists but is empty signals a partial write
+	// or external truncation. Treating it as "cold start" would silently
+	// re-scan from time zero and burn through a meta-log window for no
+	// reason. Pin that we error instead.
+	store := newFakeStore()
+	store.files[store.key(CursorDir, "shard-00.json")] = []byte{}
+	p := &FilerCursorPersister{Store: store}
+	_, _, err := p.Load(context.Background(), 0)
+	require.Error(t, err)
+}
+
+func TestFilerCursorPersister_WrongVersionReturnsError(t *testing.T) {
+	// Future schema bumps must not silently overwrite a cursor written
+	// by a newer worker. Pin that contract here.
+	store := newFakeStore()
+	store.files[store.key(CursorDir, "shard-00.json")] = []byte(`{"version":42,"shard_id":0,"ts_ns":0,"rule_set_hash":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","promoted_hash":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="}`)
+	p := &FilerCursorPersister{Store: store}
+	_, _, err := p.Load(context.Background(), 0)
+	require.Error(t, err)
+}
+
+func TestFilerCursorPersister_ShardIDMismatchReturnsError(t *testing.T) {
+	// A file declaring shard=1 read by a shard=0 caller is corruption
+	// (or a misroute); applying that cursor would advance the wrong
+	// shard's state. Reject loudly.
+	store := newFakeStore()
+	// Build a valid v1 payload but with shard_id=1 in the body.
+	in := Cursor{TsNs: 42}
+	for i := range in.RuleSetHash {
+		in.RuleSetHash[i] = byte(i + 1)
+	}
+	for i := range in.PromotedHash {
+		in.PromotedHash[i] = byte(i)
+	}
+	p := &FilerCursorPersister{Store: store}
+	require.NoError(t, p.Save(context.Background(), 1, in))
+	// Reparent that file as if it lived at shard 0 — the test's fake
+	// store only keys by name, so we copy the bytes manually.
+	store.mu.Lock()
+	store.files[store.key(CursorDir, "shard-00.json")] = store.files[store.key(CursorDir, "shard-01.json")]
+	store.mu.Unlock()
+	_, _, err := p.Load(context.Background(), 0)
+	require.Error(t, err)
+}
+
+func TestFilerCursorPersister_HashLengthMismatchReturnsError(t *testing.T) {
+	// Hand-rolled JSON with hashes shorter than 32 bytes. copy() would
+	// silently zero-pad which makes the hash comparison match a value
+	// that doesn't actually exist on disk. Reject loudly so an operator
+	// fixes the persisted state.
+	store := newFakeStore()
+	store.files[store.key(CursorDir, "shard-00.json")] = []byte(`{"version":1,"shard_id":0,"ts_ns":0,"rule_set_hash":"AA==","promoted_hash":"AA=="}`)
+	p := &FilerCursorPersister{Store: store}
+	_, _, err := p.Load(context.Background(), 0)
+	require.Error(t, err)
+}
+
+func TestFilerCursorPersister_NilStoreErrors(t *testing.T) {
+	p := &FilerCursorPersister{}
+	_, _, err := p.Load(context.Background(), 0)
+	require.Error(t, err)
+	require.Error(t, p.Save(context.Background(), 0, Cursor{}))
+}

weed/s3api/s3lifecycle/dailyrun/dispatch.go

@@ -0,0 +1,125 @@
+package dailyrun
+
+import (
+	"context"
+	"errors"
+	"math/rand"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
+)
+
+// transportRetryAttempts is the in-run retry budget for transport
+// errors (RPC dial failures, stream resets). Server-side outcomes
+// (RETRY_LATER, BLOCKED) are NOT retried in-run — the server already
+// classified them as "wait until next run." The retry exists only to
+// paper over a single flake from the network.
+const transportRetryAttempts = 3
+
+// transportRetryInitial is the first sleep after a failed attempt.
+// Doubled per attempt up to transportRetryMax. Phase 2 keeps these
+// small because the daily run's own MaxRuntime is the larger cap.
+const (
+	transportRetryInitial = 200 * time.Millisecond
+	transportRetryMax     = 5 * time.Second
+)
+
+// dispatchWithRetry sends one LifecycleDelete request, retrying on
+// transport errors up to transportRetryAttempts times with exponential
+// backoff. Returns the server's outcome on success, or an error if all
+// retries exhausted. Server outcomes (RETRY_LATER / BLOCKED) bypass
+// the retry — the daily run's halt-on-failure semantics handles them.
+func dispatchWithRetry(ctx context.Context, client LifecycleClient, m router.Match) (s3_lifecycle_pb.LifecycleDeleteOutcome, error) {
+	req := buildDeleteRequest(m)
+	backoff := transportRetryInitial
+	var lastErr error
+	for attempt := 1; attempt <= transportRetryAttempts; attempt++ {
+		resp, err := client.LifecycleDelete(ctx, req)
+		if err == nil {
+			return resp.Outcome, nil
+		}
+		// Context cancellation is shutdown, not a transport flake.
+		// Surface it immediately so the caller halts and tomorrow
+		// resumes from the same cursor.
+		if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
+			return s3_lifecycle_pb.LifecycleDeleteOutcome_LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED, err
+		}
+		lastErr = err
+		if attempt == transportRetryAttempts {
+			break
+		}
+		select {
+		case <-ctx.Done():
+			return s3_lifecycle_pb.LifecycleDeleteOutcome_LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED, ctx.Err()
+		case <-time.After(jitter(backoff)):
+		}
+		backoff *= 2
+		if backoff > transportRetryMax {
+			backoff = transportRetryMax
+		}
+	}
+	return s3_lifecycle_pb.LifecycleDeleteOutcome_LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED, lastErr
+}
+
+// jitter returns a duration in the range [d/2, d) using equal jitter.
+// Prevents thundering herds when many daily-run workers retry simultaneously.
+func jitter(d time.Duration) time.Duration {
+	if d <= 0 {
+		return 0
+	}
+	half := d / 2
+	if half <= 0 {
+		return d
+	}
+	return half + time.Duration(rand.Int63n(int64(half)))
+}
+
+// buildDeleteRequest constructs the LifecycleDelete RPC payload for a
+// router Match. Mirrors dispatcher.dispatchOne's request shape — both
+// targets the same server-side handler and the proto encoding must
+// match exactly. Duplicated rather than shared because Phase 5
+// deletes the dispatcher's call site and the cross-package dependency
+// would just create a temporary import.
+func buildDeleteRequest(m router.Match) *s3_lifecycle_pb.LifecycleDeleteRequest {
+	rh := m.Key.RuleHash
+	return &s3_lifecycle_pb.LifecycleDeleteRequest{
+		Bucket:           m.Bucket,
+		ObjectPath:       m.ObjectKey,
+		VersionId:        m.VersionID,
+		RuleHash:         rh[:],
+		ActionKind:       toProtoActionKind(m.Key.ActionKind),
+		ExpectedIdentity: toProtoIdentity(m.Identity),
+	}
+}
+
+func toProtoActionKind(k s3lifecycle.ActionKind) s3_lifecycle_pb.ActionKind {
+	switch k {
+	case s3lifecycle.ActionKindExpirationDays:
+		return s3_lifecycle_pb.ActionKind_EXPIRATION_DAYS
+	case s3lifecycle.ActionKindExpirationDate:
+		return s3_lifecycle_pb.ActionKind_EXPIRATION_DATE
+	case s3lifecycle.ActionKindNoncurrentDays:
+		return s3_lifecycle_pb.ActionKind_NONCURRENT_DAYS
+	case s3lifecycle.ActionKindNewerNoncurrent:
+		return s3_lifecycle_pb.ActionKind_NEWER_NONCURRENT
+	case s3lifecycle.ActionKindAbortMPU:
+		return s3_lifecycle_pb.ActionKind_ABORT_MPU
+	case s3lifecycle.ActionKindExpiredDeleteMarker:
+		return s3_lifecycle_pb.ActionKind_EXPIRED_DELETE_MARKER
+	}
+	return s3_lifecycle_pb.ActionKind_ACTION_KIND_UNSPECIFIED
+}
+
+func toProtoIdentity(id *router.EntryIdentity) *s3_lifecycle_pb.EntryIdentity {
+	if id == nil {
+		return nil
+	}
+	return &s3_lifecycle_pb.EntryIdentity{
+		MtimeNs:      id.MtimeNs,
+		Size:         id.Size,
+		HeadFid:      id.HeadFid,
+		ExtendedHash: id.ExtendedHash,
+	}
+}

weed/s3api/s3lifecycle/dailyrun/dispatch_test.go

@@ -0,0 +1,164 @@
+package dailyrun
+
+import (
+	"context"
+	"errors"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// fakeLifecycleClient is the test double for LifecycleClient. It
+// returns a scripted sequence of (outcome, error) pairs and counts
+// invocations so retry behavior can be pinned exactly.
+type fakeLifecycleClient struct {
+	scripted []scriptedResp
+	calls    int32
+}
+
+type scriptedResp struct {
+	outcome s3_lifecycle_pb.LifecycleDeleteOutcome
+	err     error
+}
+
+func (c *fakeLifecycleClient) LifecycleDelete(_ context.Context, _ *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
+	idx := int(atomic.AddInt32(&c.calls, 1)) - 1
+	if idx >= len(c.scripted) {
+		// Default to the last scripted response if the test under-specifies.
+		idx = len(c.scripted) - 1
+	}
+	r := c.scripted[idx]
+	if r.err != nil {
+		return nil, r.err
+	}
+	return &s3_lifecycle_pb.LifecycleDeleteResponse{Outcome: r.outcome}, nil
+}
+
+func sampleMatch() router.Match {
+	return router.Match{
+		Key:       s3lifecycle.ActionKey{Bucket: "b", ActionKind: s3lifecycle.ActionKindExpirationDays},
+		Bucket:    "b",
+		ObjectKey: "obj",
+	}
+}
+
+// Speed up dispatch retries inside tests so the suite stays fast.
+// The defaults (200ms initial, 5s max) make exponential backoff cases
+// take seconds; tests shrink to microseconds via a separate helper.
+func dispatchWithRetryFast(t *testing.T, ctx context.Context, c LifecycleClient, m router.Match) (s3_lifecycle_pb.LifecycleDeleteOutcome, error) {
+	t.Helper()
+	// We can't override the package-private constants, but every test
+	// path here uses small attempt counts so even the production
+	// backoff is fine.
+	return dispatchWithRetry(ctx, c, m)
+}
+
+func TestDispatch_FirstAttemptSucceeds(t *testing.T) {
+	c := &fakeLifecycleClient{scripted: []scriptedResp{
+		{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE},
+	}}
+	out, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
+	require.NoError(t, err)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_DONE, out)
+	assert.Equal(t, int32(1), c.calls)
+}
+
+func TestDispatch_TransportRetryThenSucceed(t *testing.T) {
+	// Two transport flakes then a success. The retry loop tries 3 times
+	// by default; we expect 3 calls and the final outcome.
+	c := &fakeLifecycleClient{scripted: []scriptedResp{
+		{err: errors.New("transport boom")},
+		{err: errors.New("transport boom")},
+		{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED},
+	}}
+	out, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
+	require.NoError(t, err)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED, out)
+	assert.Equal(t, int32(3), c.calls)
+}
+
+func TestDispatch_ExhaustsRetriesReturnsError(t *testing.T) {
+	// All N attempts fail with transport errors. Caller must see an
+	// error and the call count must equal transportRetryAttempts.
+	failing := errors.New("transport persistent")
+	c := &fakeLifecycleClient{scripted: []scriptedResp{
+		{err: failing}, {err: failing}, {err: failing},
+	}}
+	_, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
+	require.Error(t, err)
+	assert.Equal(t, int32(transportRetryAttempts), c.calls)
+}
+
+func TestDispatch_ServerOutcomeRetryLaterIsNotRetried(t *testing.T) {
+	// RETRY_LATER is a SERVER outcome; the daily run's halt-on-failure
+	// caller handles it. dispatchWithRetry must surface it on the first
+	// successful RPC and NOT retry in-run.
+	c := &fakeLifecycleClient{scripted: []scriptedResp{
+		{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER},
+	}}
+	out, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
+	require.NoError(t, err)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER, out)
+	assert.Equal(t, int32(1), c.calls, "server outcome must not trigger transport retry")
+}
+
+func TestDispatch_ServerOutcomeBlockedNotRetried(t *testing.T) {
+	c := &fakeLifecycleClient{scripted: []scriptedResp{
+		{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED},
+	}}
+	out, err := dispatchWithRetryFast(t, context.Background(), c, sampleMatch())
+	require.NoError(t, err)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED, out)
+	assert.Equal(t, int32(1), c.calls)
+}
+
+func TestDispatch_ContextCancelledShortCircuits(t *testing.T) {
+	// Context cancellation must surface immediately, not consume the
+	// retry budget. Cursor-staying semantics in the caller depends on
+	// this distinction.
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	c := &fakeLifecycleClient{scripted: []scriptedResp{
+		{err: context.Canceled},
+	}}
+	_, err := dispatchWithRetryFast(t, ctx, c, sampleMatch())
+	require.ErrorIs(t, err, context.Canceled)
+	assert.Equal(t, int32(1), c.calls, "context cancellation must not consume the retry budget")
+}
+
+func TestBuildDeleteRequest_RuleHashAndIdentity(t *testing.T) {
+	// Pin the request shape so a future Match-struct refactor doesn't
+	// silently drop the identity-CAS witness or the rule hash slice.
+	m := router.Match{
+		Bucket:    "bucket",
+		ObjectKey: "obj.txt",
+		VersionID: "v_1",
+		Key: s3lifecycle.ActionKey{
+			Bucket:     "bucket",
+			RuleHash:   [8]byte{1, 2, 3, 4, 5, 6, 7, 8},
+			ActionKind: s3lifecycle.ActionKindNoncurrentDays,
+		},
+		Identity: &router.EntryIdentity{
+			MtimeNs:      time.Unix(1700000000, 12345).UnixNano(),
+			Size:         42,
+			HeadFid:      "1,abc",
+			ExtendedHash: []byte{0xaa, 0xbb},
+		},
+	}
+	req := buildDeleteRequest(m)
+	assert.Equal(t, "bucket", req.Bucket)
+	assert.Equal(t, "obj.txt", req.ObjectPath)
+	assert.Equal(t, "v_1", req.VersionId)
+	assert.Equal(t, []byte{1, 2, 3, 4, 5, 6, 7, 8}, req.RuleHash)
+	assert.Equal(t, s3_lifecycle_pb.ActionKind_NONCURRENT_DAYS, req.ActionKind)
+	require.NotNil(t, req.ExpectedIdentity)
+	assert.Equal(t, int64(42), req.ExpectedIdentity.Size)
+	assert.Equal(t, "1,abc", req.ExpectedIdentity.HeadFid)
+	assert.Equal(t, []byte{0xaa, 0xbb}, req.ExpectedIdentity.ExtendedHash)
+}

weed/s3api/s3lifecycle/dailyrun/filer_list_func.go

@@ -0,0 +1,345 @@
+package dailyrun
+
+import (
+	"context"
+	"fmt"
+	"sort"
+	"strings"
+	"sync/atomic"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/bootstrap"
+	"github.com/seaweedfs/seaweedfs/weed/util"
+)
+
+// listPageSize is the page size for paginated directory listings. The
+// filer caps SeaweedList(..., limit=0) at DirListingLimit (1000 by
+// default) per call, so a single-page list would silently truncate
+// large directories. Atomic so tests can shrink it without racing.
+var listPageSize atomic.Uint32
+
+func init() { listPageSize.Store(1024) }
+
+// FilerListFunc returns a bootstrap.ListFunc that streams entries
+// under <bucketsPath>/<bucket>. Versioned siblings are expanded with
+// IsLatest / NumVersions / NoncurrentIndex / SuccessorModTime so the
+// walker's NoncurrentDays evaluation has the same per-version state
+// the streaming bootstrap injects via reader.Event.BootstrapVersion.
+// MPU init records at .uploads/<id> with ExtMultipartObjectKey set
+// are emitted whole with IsMPUInit=true and DestKey carrying the
+// user's intended path.
+func FilerListFunc(client filer_pb.SeaweedFilerClient, bucketsPath string) bootstrap.ListFunc {
+	return func(ctx context.Context, bucket, start string, cb func(*bootstrap.Entry) error) error {
+		if client == nil {
+			return fmt.Errorf("FilerListFunc: nil client")
+		}
+		root := strings.TrimSuffix(bucketsPath, "/") + "/" + bucket
+		return walkBucketTree(ctx, client, root, root, start, cb)
+	}
+}
+
+// walkBucketTree recurses through dir in two passes per level. Pass 1
+// expands `.versions/` dirs (populating skipBare with the bare null-
+// version keys that pass 2 must suppress). Pass 2 emits regular files
+// and recurses into non-special subdirectories.
+//
+// The two-pass shape mirrors scheduler/bootstrap.go's walkBucketDir
+// (see that file's comment for why `.versions/` has to be processed
+// before its bare sibling).
+func walkBucketTree(ctx context.Context, client filer_pb.SeaweedFilerClient, dir, bucketRoot, start string, cb func(*bootstrap.Entry) error) error {
+	skipBare := map[string]bool{}
+
+	// Pass 1: .versions/ dirs only.
+	if err := listAll(ctx, client, dir, func(e *filer_pb.Entry) error {
+		if e == nil || e.Attributes == nil {
+			return nil
+		}
+		if !e.IsDirectory || !isVersionsDir(e) {
+			return nil
+		}
+		full := dir + "/" + e.Name
+		key := strings.TrimPrefix(full, bucketRoot+"/")
+		return expandVersionsDir(ctx, client, bucketRoot, key, e, start, skipBare, cb)
+	}); err != nil {
+		return err
+	}
+
+	// Pass 2: everything else.
+	return listAll(ctx, client, dir, func(e *filer_pb.Entry) error {
+		if e == nil || e.Attributes == nil {
+			return nil
+		}
+		if e.IsDirectory && isVersionsDir(e) {
+			return nil
+		}
+		full := dir + "/" + e.Name
+		key := strings.TrimPrefix(full, bucketRoot+"/")
+		if e.IsDirectory {
+			if isMPUInitDir(key, e) {
+				if start != "" && key <= start {
+					return nil
+				}
+				destKey := string(e.Extended[s3_constants.ExtMultipartObjectKey])
+				return cb(&bootstrap.Entry{
+					Path:      key,
+					DestKey:   destKey,
+					IsMPUInit: true,
+					ModTime:   time.Unix(e.Attributes.Mtime, int64(e.Attributes.MtimeNs)),
+					Size:      int64(e.Attributes.FileSize),
+				})
+			}
+			return walkBucketTree(ctx, client, full, bucketRoot, start, cb)
+		}
+		if skipBare[key] {
+			return nil
+		}
+		if start != "" && key <= start {
+			return nil
+		}
+		entry := &bootstrap.Entry{
+			Path:     key,
+			ModTime:  time.Unix(e.Attributes.Mtime, int64(e.Attributes.MtimeNs)),
+			Size:     int64(e.Attributes.FileSize),
+			IsLatest: true, // Non-versioned default.
+			Tags:     extractTags(e.Extended),
+		}
+		return cb(entry)
+	})
+}
+
+// versionItem captures one sibling of a `.versions/` expansion. bareKey
+// is the bucket-relative path of the bare null-version entry when the
+// item represents it; for real version files it stays empty.
+type versionItem struct {
+	entry          *filer_pb.Entry
+	versionID      string
+	bareKey        string
+	isExplicitNull bool
+}
+
+// expandVersionsDir handles the `.versions/<key>` directory. Lists
+// version files, optionally appends the bare null-version sibling,
+// sorts newest-first, resolves the latest, and emits one Entry per
+// version with the sibling state walkEntry needs to evaluate
+// NoncurrentDays / NewerNoncurrent / ExpirationDays correctly.
+//
+// Ported from scheduler/bootstrap.go's same-named helper; both must
+// agree on sort, latest resolution, and successor derivation so the
+// streaming and walker paths reach the same verdict for the same
+// objects. Phase 5 deletes the scheduler copy.
+//
+// Resume note: every emitted sibling shares Path = logical key, so a
+// resume after a mid-expansion failure rewalks the whole sibling
+// group. Acceptable today because Phase 4b doesn't persist a
+// Checkpoint between runs (start is always "" via runShard).
+func expandVersionsDir(ctx context.Context, client filer_pb.SeaweedFilerClient, bucketRoot, versionsKey string, versionsEntry *filer_pb.Entry, start string, skipBare map[string]bool, cb func(*bootstrap.Entry) error) error {
+	logical := strings.TrimSuffix(versionsKey, s3_constants.VersionsFolder)
+	if logical == "" {
+		return nil
+	}
+	versionsDir := bucketRoot + "/" + versionsKey
+	var children []*filer_pb.Entry
+	if err := listAll(ctx, client, versionsDir, func(e *filer_pb.Entry) error {
+		if e != nil && e.Attributes != nil && !e.IsDirectory {
+			children = append(children, e)
+		}
+		return nil
+	}); err != nil {
+		return fmt.Errorf("list %s: %w", versionsDir, err)
+	}
+	items := make([]versionItem, 0, len(children)+1)
+	for _, e := range children {
+		if id, ok := e.Extended[s3_constants.ExtVersionIdKey]; ok && len(id) > 0 {
+			items = append(items, versionItem{entry: e, versionID: string(id)})
+		}
+	}
+	if len(items) == 0 {
+		// Coincidentally-named user folder, or an empty `.versions`
+		// container. Treat it as a regular subdirectory so user-named
+		// files inside still surface.
+		return walkBucketTree(ctx, client, versionsDir, bucketRoot, start, cb)
+	}
+	if nullEntry, nullKey, explicit, ok := lookupNullVersion(ctx, client, bucketRoot, logical); ok {
+		items = append(items, versionItem{
+			entry:          nullEntry,
+			versionID:      "null",
+			bareKey:        nullKey,
+			isExplicitNull: explicit,
+		})
+	}
+
+	// Sort newest-first by mtime, ties broken by version_id (newer wins).
+	sort.SliceStable(items, func(i, j int) bool {
+		mi := items[i].entry.Attributes.Mtime*int64(1e9) + int64(items[i].entry.Attributes.MtimeNs)
+		mj := items[j].entry.Attributes.Mtime*int64(1e9) + int64(items[j].entry.Attributes.MtimeNs)
+		if mi != mj {
+			return mi > mj
+		}
+		return s3lifecycle.CompareVersionIds(items[i].versionID, items[j].versionID) < 0
+	})
+
+	// Resolve latest:
+	//   1. Pointer names a real id -> that wins.
+	//   2. Pointer absent (or stale: set but no sibling carries it)
+	//      + any EXPLICIT null bare exists -> null is latest.
+	//   3. Otherwise -> newest sibling (latestPos = 0 by default).
+	//
+	// A stale pointer falls through to the no-pointer fallback rather
+	// than silently leaving latestPos at 0 with no documented intent;
+	// the value happens to be the same today (newest sibling wins
+	// either way) but the explicit branching protects against future
+	// fallback refinements diverging by accident.
+	latestID := string(versionsEntry.Extended[s3_constants.ExtLatestVersionIdKey])
+	latestPos := 0
+	pointerResolved := false
+	if latestID != "" {
+		for i, it := range items {
+			if it.versionID == latestID {
+				latestPos = i
+				pointerResolved = true
+				break
+			}
+		}
+	}
+	if !pointerResolved {
+		for i, it := range items {
+			if it.versionID == "null" && it.isExplicitNull {
+				latestPos = i
+				break
+			}
+		}
+	}
+
+	if start != "" && logical <= start {
+		// All siblings share Path=logical, so the whole group is
+		// either above or below the resume marker.
+		return nil
+	}
+	for i, it := range items {
+		successor := s3lifecycle.SuccessorFromEntryStamp(it.entry)
+		if successor.IsZero() && i > 0 {
+			prev := items[i-1].entry.Attributes
+			successor = time.Unix(prev.Mtime, int64(prev.MtimeNs))
+		}
+		isLatest := i == latestPos
+		entry := &bootstrap.Entry{
+			Path:             logical,
+			VersionID:        it.versionID,
+			ModTime:          time.Unix(it.entry.Attributes.Mtime, int64(it.entry.Attributes.MtimeNs)),
+			Size:             int64(it.entry.Attributes.FileSize),
+			IsLatest:         isLatest,
+			IsDeleteMarker:   string(it.entry.Extended[s3_constants.ExtDeleteMarkerKey]) == "true",
+			NumVersions:      len(items),
+			SuccessorModTime: successor,
+			Tags:             extractTags(it.entry.Extended),
+		}
+		if !isLatest {
+			rank := i
+			if i > latestPos {
+				rank = i - 1
+			}
+			entry.NoncurrentIndex = &rank
+		}
+		if err := cb(entry); err != nil {
+			return err
+		}
+		if it.versionID == "null" && skipBare != nil {
+			skipBare[it.bareKey] = true
+		}
+	}
+	return nil
+}
+
+// lookupNullVersion returns the bare-key entry that represents the null
+// version of logical, if any. Both regular files and S3 directory-key
+// markers qualify. explicit is true when the entry carries
+// ExtVersionIdKey == "null" — the marker the suspended-versioning
+// write path applies; only an explicit-null bare can outrank a missing
+// `.versions/` pointer per the latest-resolution rules above.
+func lookupNullVersion(ctx context.Context, client filer_pb.SeaweedFilerClient, bucketRoot, logical string) (*filer_pb.Entry, string, bool, bool) {
+	parent, name := util.NewFullPath(bucketRoot, logical).DirAndName()
+	resp, err := filer_pb.LookupEntry(ctx, client, &filer_pb.LookupDirectoryEntryRequest{
+		Directory: parent,
+		Name:      name,
+	})
+	if err != nil || resp == nil || resp.Entry == nil {
+		return nil, "", false, false
+	}
+	e := resp.Entry
+	if e.IsDirectory && !e.IsDirectoryKeyObject() {
+		return nil, "", false, false
+	}
+	explicit := false
+	if id, hasID := e.Extended[s3_constants.ExtVersionIdKey]; hasID && string(id) == "null" {
+		explicit = true
+	}
+	return e, strings.TrimPrefix(parent+"/"+name, bucketRoot+"/"), explicit, true
+}
+
+// listAll issues paginated SeaweedList calls until exhausted. Ported
+// from scheduler/bootstrap.go's same-named helper; Phase 5 deletes
+// the scheduler copy when the streaming path is removed.
+func listAll(ctx context.Context, client filer_pb.SeaweedFilerClient, dir string, fn func(*filer_pb.Entry) error) error {
+	pageSize := listPageSize.Load()
+	startFrom := ""
+	for {
+		var pageCount uint32
+		var lastName string
+		if err := filer_pb.SeaweedList(ctx, client, dir, "", func(e *filer_pb.Entry, _ bool) error {
+			pageCount++
+			if e != nil {
+				lastName = e.Name
+			}
+			return fn(e)
+		}, startFrom, false, pageSize); err != nil {
+			return err
+		}
+		if pageCount < pageSize {
+			return nil
+		}
+		startFrom = lastName
+	}
+}
+
+func isVersionsDir(entry *filer_pb.Entry) bool {
+	return entry.IsDirectory && strings.HasSuffix(entry.Name, s3_constants.VersionsFolder)
+}
+
+// isMPUInitDir mirrors router.mpuInitInfo: a directory at
+// .uploads/<id> carrying the destination key in Extended is the MPU
+// init record. Verified shape + presence of ExtMultipartObjectKey;
+// directories at .uploads/<id> without the key are mid-write before
+// metadata landed and stay out of the dispatch path.
+func isMPUInitDir(key string, entry *filer_pb.Entry) bool {
+	uploadsPrefix := s3_constants.MultipartUploadsFolder + "/"
+	if !strings.HasPrefix(key, uploadsPrefix) {
+		return false
+	}
+	rest := key[len(uploadsPrefix):]
+	if rest == "" || strings.ContainsRune(rest, '/') {
+		return false
+	}
+	v, ok := entry.Extended[s3_constants.ExtMultipartObjectKey]
+	return ok && len(v) > 0
+}
+
+func extractTags(ext map[string][]byte) map[string]string {
+	if len(ext) == 0 {
+		return nil
+	}
+	prefix := s3_constants.AmzObjectTagging + "-"
+	var tags map[string]string
+	for k, v := range ext {
+		if !strings.HasPrefix(k, prefix) {
+			continue
+		}
+		if tags == nil {
+			tags = make(map[string]string)
+		}
+		tags[k[len(prefix):]] = string(v)
+	}
+	return tags
+}

weed/s3api/s3lifecycle/dailyrun/filer_list_func_test.go

@@ -0,0 +1,495 @@
+package dailyrun
+
+import (
+	"context"
+	"io"
+	"sort"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/bootstrap"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/metadata"
+)
+
+// fakeFilerStream implements the ListEntries server-streaming client.
+type fakeFilerStream struct {
+	responses []*filer_pb.ListEntriesResponse
+	idx       int
+	ctx       context.Context
+}
+
+func (s *fakeFilerStream) Recv() (*filer_pb.ListEntriesResponse, error) {
+	if s.ctx != nil {
+		if err := s.ctx.Err(); err != nil {
+			return nil, err
+		}
+	}
+	if s.idx >= len(s.responses) {
+		return nil, io.EOF
+	}
+	r := s.responses[s.idx]
+	s.idx++
+	return r, nil
+}
+func (s *fakeFilerStream) Header() (metadata.MD, error) { return metadata.MD{}, nil }
+func (s *fakeFilerStream) Trailer() metadata.MD         { return metadata.MD{} }
+func (s *fakeFilerStream) CloseSend() error             { return nil }
+func (s *fakeFilerStream) Context() context.Context {
+	if s.ctx != nil {
+		return s.ctx
+	}
+	return context.Background()
+}
+func (s *fakeFilerStream) SendMsg(any) error { return nil }
+func (s *fakeFilerStream) RecvMsg(any) error { return nil }
+
+// fakeFiler maps directory paths to their immediate children. Only
+// ListEntries is implemented; other methods of SeaweedFilerClient are
+// inherited from the embedded interface and panic if called.
+type fakeFiler struct {
+	filer_pb.SeaweedFilerClient
+
+	mu   sync.Mutex
+	tree map[string][]*filer_pb.Entry
+}
+
+func (c *fakeFiler) LookupDirectoryEntry(_ context.Context, in *filer_pb.LookupDirectoryEntryRequest, _ ...grpc.CallOption) (*filer_pb.LookupDirectoryEntryResponse, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	for _, e := range c.tree[in.Directory] {
+		if e != nil && e.Name == in.Name {
+			return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil
+		}
+	}
+	return nil, filer_pb.ErrNotFound
+}
+
+func (c *fakeFiler) ListEntries(ctx context.Context, in *filer_pb.ListEntriesRequest, _ ...grpc.CallOption) (grpc.ServerStreamingClient[filer_pb.ListEntriesResponse], error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	src := c.tree[in.Directory]
+	// Mirror the filer: sort by name, honor StartFromFileName exclusive,
+	// cap at Limit. listAll's pagination loop depends on these.
+	filtered := make([]*filer_pb.Entry, 0, len(src))
+	for _, e := range src {
+		if e == nil {
+			continue
+		}
+		if in.StartFromFileName != "" && !in.InclusiveStartFrom && e.Name <= in.StartFromFileName {
+			continue
+		}
+		filtered = append(filtered, e)
+	}
+	sort.SliceStable(filtered, func(i, j int) bool { return filtered[i].Name < filtered[j].Name })
+	if in.Limit > 0 && uint32(len(filtered)) > in.Limit {
+		filtered = filtered[:in.Limit]
+	}
+	resps := make([]*filer_pb.ListEntriesResponse, 0, len(filtered))
+	for _, e := range filtered {
+		resps = append(resps, &filer_pb.ListEntriesResponse{Entry: e})
+	}
+	return &fakeFilerStream{responses: resps, ctx: ctx}, nil
+}
+
+func file(name string, mtime time.Time, size int64) *filer_pb.Entry {
+	return &filer_pb.Entry{
+		Name: name,
+		Attributes: &filer_pb.FuseAttributes{
+			Mtime:    mtime.Unix(),
+			MtimeNs:  int32(mtime.Nanosecond()),
+			FileSize: uint64(size),
+		},
+	}
+}
+
+func dir(name string) *filer_pb.Entry {
+	return &filer_pb.Entry{Name: name, IsDirectory: true, Attributes: &filer_pb.FuseAttributes{}}
+}
+
+func TestFilerListFunc_EmitsFlatFiles(t *testing.T) {
+	mtime := time.Now().Add(-7 * 24 * time.Hour)
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {file("a.txt", mtime, 10), file("b.txt", mtime, 20)},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got []string
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = append(got, e.Path)
+		return nil
+	}))
+	assert.Equal(t, []string{"a.txt", "b.txt"}, got)
+}
+
+func TestFilerListFunc_PropagatesTagsOnFlatFiles(t *testing.T) {
+	mtime := time.Now().Add(-7 * 24 * time.Hour)
+	tagged := fileWithExt("tagged.txt", mtime, 10, map[string][]byte{
+		s3_constants.AmzObjectTagging + "-env":  []byte("temp"),
+		s3_constants.AmzObjectTagging + "-tier": []byte("cold"),
+	})
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {tagged},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got *bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = e
+		return nil
+	}))
+	require.NotNil(t, got)
+	assert.Equal(t, map[string]string{"env": "temp", "tier": "cold"}, got.Tags)
+}
+
+func TestFilerListFunc_RecursesIntoSubdirs(t *testing.T) {
+	mtime := time.Now()
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt":           {dir("logs"), file("root.txt", mtime, 1)},
+		"/buckets/bkt/logs":      {dir("2026"), file("a.log", mtime, 5)},
+		"/buckets/bkt/logs/2026": {file("b.log", mtime, 7)},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var paths []string
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		paths = append(paths, e.Path)
+		return nil
+	}))
+	sort.Strings(paths)
+	assert.Equal(t, []string{"logs/2026/b.log", "logs/a.log", "root.txt"}, paths)
+}
+
+func TestFilerListFunc_MPUInitEmitsWithDestKey(t *testing.T) {
+	// .uploads/<id>/ with ExtMultipartObjectKey is the MPU init record.
+	// One Entry per init, IsMPUInit=true, DestKey = the user's path.
+	mtime := time.Now()
+	mpuDir := dir("upload-id-1")
+	mpuDir.Extended = map[string][]byte{
+		s3_constants.ExtMultipartObjectKey: []byte("user/path/object"),
+	}
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {
+			file("regular.txt", mtime, 1),
+			dir(s3_constants.MultipartUploadsFolder),
+		},
+		"/buckets/bkt/" + s3_constants.MultipartUploadsFolder: {
+			mpuDir,
+		},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got []*bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = append(got, e)
+		return nil
+	}))
+	require.Len(t, got, 2)
+	byPath := map[string]*bootstrap.Entry{}
+	for _, e := range got {
+		byPath[e.Path] = e
+	}
+	require.NotNil(t, byPath["regular.txt"])
+	require.NotNil(t, byPath[s3_constants.MultipartUploadsFolder+"/upload-id-1"])
+	mpu := byPath[s3_constants.MultipartUploadsFolder+"/upload-id-1"]
+	assert.True(t, mpu.IsMPUInit)
+	assert.Equal(t, "user/path/object", mpu.DestKey)
+}
+
+func TestFilerListFunc_MPUInitWithoutDestKeyIsSkipped(t *testing.T) {
+	// A `.uploads/<id>` directory missing ExtMultipartObjectKey is
+	// mid-write before metadata landed; the dispatcher would error
+	// on empty DestKey. Skip it so the walk doesn't halt.
+	mtime := time.Now()
+	mpuDir := dir("upload-id-2") // no Extended
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {
+			file("regular.txt", mtime, 1),
+			dir(s3_constants.MultipartUploadsFolder),
+		},
+		"/buckets/bkt/" + s3_constants.MultipartUploadsFolder: {
+			mpuDir,
+		},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var paths []string
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		paths = append(paths, e.Path)
+		return nil
+	}))
+	assert.Equal(t, []string{"regular.txt"}, paths)
+}
+
+// fileWithExt is a versioned-file entry helper.
+func fileWithExt(name string, mtime time.Time, size int64, ext map[string][]byte) *filer_pb.Entry {
+	e := file(name, mtime, size)
+	e.Extended = ext
+	return e
+}
+
+func versionsDir(name string, latestID string) *filer_pb.Entry {
+	d := dir(name)
+	d.Extended = map[string][]byte{}
+	if latestID != "" {
+		d.Extended[s3_constants.ExtLatestVersionIdKey] = []byte(latestID)
+	}
+	return d
+}
+
+func TestFilerListFunc_VersionedExpansionMarksLatestByPointer(t *testing.T) {
+	// .versions/<key>/ with three real versions; parent's
+	// ExtLatestVersionIdKey points to v2 → IsLatest set on v2; the
+	// other two get NoncurrentIndex computed against the latest's
+	// position in the sorted (newest-first) list.
+	t1 := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	t2 := time.Date(2026, 2, 1, 0, 0, 0, 0, time.UTC)
+	t3 := time.Date(2026, 3, 1, 0, 0, 0, 0, time.UTC)
+	versions := []*filer_pb.Entry{
+		fileWithExt("v1", t1, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("v1")}),
+		fileWithExt("v2", t2, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("v2")}),
+		fileWithExt("v3", t3, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("v3")}),
+	}
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {versionsDir("foo"+s3_constants.VersionsFolder, "v2")},
+		"/buckets/bkt/foo" + s3_constants.VersionsFolder: versions,
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got []*bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = append(got, e)
+		return nil
+	}))
+	require.Len(t, got, 3)
+
+	byID := map[string]*bootstrap.Entry{}
+	for _, e := range got {
+		byID[e.VersionID] = e
+		assert.Equal(t, "foo", e.Path, "every sibling's Path is the logical key")
+		assert.Equal(t, 3, e.NumVersions)
+	}
+	assert.True(t, byID["v2"].IsLatest, "pointer wins regardless of mtime order")
+	assert.False(t, byID["v1"].IsLatest)
+	assert.False(t, byID["v3"].IsLatest)
+	require.NotNil(t, byID["v3"].NoncurrentIndex, "noncurrent siblings get a rank")
+	require.NotNil(t, byID["v1"].NoncurrentIndex, "noncurrent siblings get a rank")
+}
+
+func TestFilerListFunc_VersionedExpansionStalePointerFallsBackToNewestSibling(t *testing.T) {
+	// ExtLatestVersionIdKey names a version that no sibling carries
+	// (stale pointer left behind by a write whose update was lost or
+	// raced). The fallback must NOT silently treat items[0] as latest
+	// via the default; structurally it must drop into the no-pointer
+	// path so the explicit-null bare check still runs.
+	tOld := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	tNew := time.Date(2026, 3, 1, 0, 0, 0, 0, time.UTC)
+	versions := []*filer_pb.Entry{
+		fileWithExt("v_old", tOld, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("vold")}),
+		fileWithExt("v_new", tNew, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("vnew")}),
+	}
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {versionsDir("foo"+s3_constants.VersionsFolder, "v_GHOST")},
+		"/buckets/bkt/foo" + s3_constants.VersionsFolder: versions,
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got []*bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = append(got, e)
+		return nil
+	}))
+	require.Len(t, got, 2)
+	byID := map[string]*bootstrap.Entry{}
+	for _, e := range got {
+		byID[e.VersionID] = e
+	}
+	assert.True(t, byID["vnew"].IsLatest, "stale pointer must fall back to newest sibling")
+	assert.False(t, byID["vold"].IsLatest)
+}
+
+func TestFilerListFunc_VersionedExpansionNoPointerNewestSiblingWins(t *testing.T) {
+	// Parent has no ExtLatestVersionIdKey. With no explicit-null bare
+	// version, the newest-by-mtime sibling becomes latest.
+	tNew := time.Date(2026, 3, 1, 0, 0, 0, 0, time.UTC)
+	tOld := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	versions := []*filer_pb.Entry{
+		fileWithExt("v_old", tOld, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("vold")}),
+		fileWithExt("v_new", tNew, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("vnew")}),
+	}
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {versionsDir("foo"+s3_constants.VersionsFolder, "")},
+		"/buckets/bkt/foo" + s3_constants.VersionsFolder: versions,
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got []*bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = append(got, e)
+		return nil
+	}))
+	require.Len(t, got, 2)
+	byID := map[string]*bootstrap.Entry{}
+	for _, e := range got {
+		byID[e.VersionID] = e
+	}
+	assert.True(t, byID["vnew"].IsLatest, "newest sibling wins when pointer is absent")
+	assert.False(t, byID["vold"].IsLatest)
+}
+
+func TestFilerListFunc_VersionedExpansionExplicitNullIsLatestWhenPointerMissing(t *testing.T) {
+	// Suspended-versioning shape: bare object marked with
+	// ExtVersionIdKey="null", parent has no pointer. null is latest.
+	t1 := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	tNull := time.Date(2026, 3, 1, 0, 0, 0, 0, time.UTC) // newest
+	bareNull := fileWithExt("foo", tNull, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("null")})
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {
+			bareNull,
+			versionsDir("foo"+s3_constants.VersionsFolder, ""),
+		},
+		"/buckets/bkt/foo" + s3_constants.VersionsFolder: {
+			fileWithExt("v1", t1, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("v1")}),
+		},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got []*bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = append(got, e)
+		return nil
+	}))
+	// 2 sibling entries from expansion (null + v1). The bare "foo"
+	// MUST be suppressed in pass 2 via skipBare.
+	require.Len(t, got, 2)
+	byID := map[string]*bootstrap.Entry{}
+	for _, e := range got {
+		byID[e.VersionID] = e
+	}
+	require.NotNil(t, byID["null"])
+	require.NotNil(t, byID["v1"])
+	assert.True(t, byID["null"].IsLatest)
+	assert.False(t, byID["v1"].IsLatest)
+
+	// Walk again, verify no duplicate emission of the bare "foo".
+	count := 0
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		if e.Path == "foo" && e.VersionID == "" {
+			t.Errorf("bare foo should be suppressed by skipBare, got %+v", e)
+		}
+		count++
+		return nil
+	}))
+	assert.Equal(t, 2, count)
+}
+
+func TestFilerListFunc_VersionedExpansionExplicitNullBeatsNewerSiblingWhenPointerMissing(t *testing.T) {
+	// Suspended-versioning current-ness is carried by the explicit bare
+	// null, not by mtime ordering. Backdating the null to make it due
+	// must not demote it behind a newer noncurrent sibling when the
+	// .versions pointer is absent.
+	tNull := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) // older
+	tV1 := time.Date(2026, 3, 1, 0, 0, 0, 0, time.UTC)   // newer
+	bareNull := fileWithExt("foo", tNull, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("null")})
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {
+			bareNull,
+			versionsDir("foo"+s3_constants.VersionsFolder, ""),
+		},
+		"/buckets/bkt/foo" + s3_constants.VersionsFolder: {
+			fileWithExt("v1", tV1, 1, map[string][]byte{s3_constants.ExtVersionIdKey: []byte("v1")}),
+		},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got []*bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = append(got, e)
+		return nil
+	}))
+	require.Len(t, got, 2)
+	byID := map[string]*bootstrap.Entry{}
+	for _, e := range got {
+		byID[e.VersionID] = e
+	}
+	require.NotNil(t, byID["null"])
+	require.NotNil(t, byID["v1"])
+	assert.True(t, byID["null"].IsLatest, "explicit null must stay latest even when older than noncurrent siblings")
+	assert.False(t, byID["v1"].IsLatest)
+}
+
+func TestFilerListFunc_VersionsDirWithoutMarkersRecursesAsRegular(t *testing.T) {
+	// A `.versions`-named folder whose children have no
+	// ExtVersionIdKey is a coincidence (user folder). Recurse into
+	// it; the file inside should surface as a regular entry.
+	mtime := time.Now()
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {versionsDir("looksLikeUserFolder"+s3_constants.VersionsFolder, "")},
+		"/buckets/bkt/looksLikeUserFolder" + s3_constants.VersionsFolder: {
+			file("inner.txt", mtime, 1),
+		},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var paths []string
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		paths = append(paths, e.Path)
+		return nil
+	}))
+	assert.Equal(t, []string{"looksLikeUserFolder" + s3_constants.VersionsFolder + "/inner.txt"}, paths)
+}
+
+func TestFilerListFunc_VersionedDeleteMarkerPropagates(t *testing.T) {
+	mtime := time.Now()
+	versions := []*filer_pb.Entry{
+		fileWithExt("v1", mtime, 0, map[string][]byte{
+			s3_constants.ExtVersionIdKey:    []byte("v1"),
+			s3_constants.ExtDeleteMarkerKey: []byte("true"),
+		}),
+	}
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {versionsDir("foo"+s3_constants.VersionsFolder, "v1")},
+		"/buckets/bkt/foo" + s3_constants.VersionsFolder: versions,
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got *bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = e
+		return nil
+	}))
+	require.NotNil(t, got)
+	assert.True(t, got.IsDeleteMarker, "ExtDeleteMarkerKey='true' must surface as IsDeleteMarker")
+}
+
+func TestFilerListFunc_HonorsStart(t *testing.T) {
+	// The walker's kill-resume contract: skip entries whose Path <= start.
+	mtime := time.Now()
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {file("a", mtime, 1), file("b", mtime, 1), file("c", mtime, 1)},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got []string
+	require.NoError(t, listFn(context.Background(), "bkt", "a", func(e *bootstrap.Entry) error {
+		got = append(got, e.Path)
+		return nil
+	}))
+	assert.Equal(t, []string{"b", "c"}, got)
+}
+
+func TestFilerListFunc_NilClient(t *testing.T) {
+	listFn := FilerListFunc(nil, "/buckets")
+	require.Error(t, listFn(context.Background(), "bkt", "", func(*bootstrap.Entry) error { return nil }))
+}
+
+func TestFilerListFunc_AttributesPropagate(t *testing.T) {
+	mtime := time.Date(2026, 5, 11, 12, 0, 0, 1234, time.UTC)
+	client := &fakeFiler{tree: map[string][]*filer_pb.Entry{
+		"/buckets/bkt": {file("obj", mtime, 4096)},
+	}}
+	listFn := FilerListFunc(client, "/buckets")
+	var got *bootstrap.Entry
+	require.NoError(t, listFn(context.Background(), "bkt", "", func(e *bootstrap.Entry) error {
+		got = e
+		return nil
+	}))
+	require.NotNil(t, got)
+	assert.Equal(t, "obj", got.Path)
+	assert.Equal(t, mtime.Unix(), got.ModTime.Unix())
+	assert.Equal(t, int64(1234), got.ModTime.UnixNano()-mtime.Unix()*int64(time.Second))
+	assert.Equal(t, int64(4096), got.Size)
+	assert.True(t, got.IsLatest)
+}

weed/s3api/s3lifecycle/dailyrun/jitter_test.go

@@ -0,0 +1,44 @@
+package dailyrun
+
+import (
+	"testing"
+	"time"
+)
+
+func TestJitterBounds(t *testing.T) {
+	cases := []time.Duration{
+		200 * time.Millisecond,
+		1 * time.Second,
+		5 * time.Second,
+	}
+
+	for _, d := range cases {
+		for i := 0; i < 100; i++ {
+			j := jitter(d)
+			if j < d/2 {
+				t.Errorf("jitter(%v) = %v, below lower bound %v", d, j, d/2)
+			}
+			if j >= d {
+				t.Errorf("jitter(%v) = %v, at or above upper bound %v", d, j, d)
+			}
+		}
+	}
+}
+
+func TestJitterZeroAndNegative(t *testing.T) {
+	if j := jitter(0); j != 0 {
+		t.Errorf("jitter(0) = %v, want 0", j)
+	}
+	if j := jitter(-1 * time.Second); j != 0 {
+		t.Errorf("jitter(-1s) = %v, want 0", j)
+	}
+}
+
+func TestJitterTinyDuration(t *testing.T) {
+	// When d < 2, half == 0 and rand.Int63n(0) panics.
+	// We should return d unmodified in that case.
+	j := jitter(1 * time.Nanosecond)
+	if j != 1*time.Nanosecond {
+		t.Errorf("jitter(1ns) = %v, want 1ns", j)
+	}
+}

weed/s3api/s3lifecycle/dailyrun/process_matches_test.go

@@ -0,0 +1,197 @@
+package dailyrun
+
+import (
+	"context"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	dto "github.com/prometheus/client_model/go"
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
+	"github.com/seaweedfs/seaweedfs/weed/stats"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// recordingClient captures every LifecycleDelete request the
+// daily-run path emits. Default outcome is DONE; tests that need
+// other outcomes set responses by index, or outcomeByObject when
+// shards dispatch concurrently and call order isn't fixed.
+type recordingClient struct {
+	mu              sync.Mutex
+	requests        []*s3_lifecycle_pb.LifecycleDeleteRequest
+	responses       []s3_lifecycle_pb.LifecycleDeleteOutcome
+	outcomeByObject map[string]s3_lifecycle_pb.LifecycleDeleteOutcome
+	calls           atomic.Int32
+}
+
+func (c *recordingClient) LifecycleDelete(_ context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
+	c.mu.Lock()
+	c.requests = append(c.requests, req)
+	idx := int(c.calls.Add(1)) - 1
+	byObject, pinned := c.outcomeByObject[req.ObjectPath]
+	c.mu.Unlock()
+	if pinned {
+		return &s3_lifecycle_pb.LifecycleDeleteResponse{Outcome: byObject}, nil
+	}
+	out := s3_lifecycle_pb.LifecycleDeleteOutcome_DONE
+	if idx < len(c.responses) {
+		out = c.responses[idx]
+	}
+	return &s3_lifecycle_pb.LifecycleDeleteResponse{Outcome: out}, nil
+}
+
+func (c *recordingClient) seenObjects() []string {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	out := make([]string, 0, len(c.requests))
+	for _, r := range c.requests {
+		out = append(out, r.ObjectPath)
+	}
+	return out
+}
+
+// Phase 2's processMatches must NOT use a per-rule done flag.
+// routePointerTransitionExpand can return multiple matches for the
+// same ActionKey on different objects, each with its own
+// SuccessorModTime from a distinct demoting event. A not-yet-due
+// sibling must never gate a sibling that's already past its DueTime.
+// Pin that here.
+func TestProcessMatches_NotYetDueDoesNotSuppressDueOnSameRule(t *testing.T) {
+	runNow := time.Now()
+	rh := [8]byte{1, 2, 3, 4, 5, 6, 7, 8}
+	rule := s3lifecycle.ActionKey{
+		Bucket:     "b",
+		RuleHash:   rh,
+		ActionKind: s3lifecycle.ActionKindNoncurrentDays,
+	}
+	// Three matches for the same ActionKey, different objects, mixed
+	// due-times. Order intentionally puts the not-yet-due match first
+	// so a buggy per-rule done flag would suppress the others.
+	matches := []router.Match{
+		{Key: rule, Bucket: "b", ObjectKey: "future-sibling", DueTime: runNow.Add(48 * time.Hour)},
+		{Key: rule, Bucket: "b", ObjectKey: "due-sibling-a", DueTime: runNow.Add(-48 * time.Hour)},
+		{Key: rule, Bucket: "b", ObjectKey: "due-sibling-b", DueTime: runNow.Add(-24 * time.Hour)},
+	}
+	client := &recordingClient{}
+	cfg := Config{Client: client}
+	skipped, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{TsNs: runNow.UnixNano()}, matches)
+	require.NoError(t, err)
+	require.False(t, halted)
+	require.True(t, skipped, "at least one future-DueTime match must flag skippedAny")
+
+	// Both due siblings must have been dispatched.
+	seen := client.seenObjects()
+	assert.ElementsMatch(t, []string{"due-sibling-a", "due-sibling-b"}, seen,
+		"due matches must dispatch regardless of where the future-sibling sits in the slice")
+}
+
+func TestProcessMatches_OrderingDoesNotMatter(t *testing.T) {
+	runNow := time.Now()
+	rh := [8]byte{0xaa}
+	rule := s3lifecycle.ActionKey{Bucket: "b", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	matches := []router.Match{
+		{Key: rule, Bucket: "b", ObjectKey: "a", DueTime: runNow.Add(-2 * time.Hour)},
+		{Key: rule, Bucket: "b", ObjectKey: "b", DueTime: runNow.Add(-1 * time.Hour)},
+		{Key: rule, Bucket: "b", ObjectKey: "future", DueTime: runNow.Add(time.Hour)},
+	}
+	client := &recordingClient{}
+	cfg := Config{Client: client}
+	skipped, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, matches)
+	require.NoError(t, err)
+	require.False(t, halted)
+	require.True(t, skipped)
+	assert.ElementsMatch(t, []string{"a", "b"}, client.seenObjects())
+}
+
+func TestProcessMatches_HaltOnServerOutcomeStopsRemaining(t *testing.T) {
+	runNow := time.Now()
+	rh := [8]byte{0xbb}
+	rule := s3lifecycle.ActionKey{Bucket: "b", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	matches := []router.Match{
+		{Key: rule, Bucket: "b", ObjectKey: "ok", DueTime: runNow.Add(-time.Hour)},
+		{Key: rule, Bucket: "b", ObjectKey: "blocked", DueTime: runNow.Add(-time.Hour)},
+		{Key: rule, Bucket: "b", ObjectKey: "would-be-next", DueTime: runNow.Add(-time.Hour)},
+	}
+	client := &recordingClient{responses: []s3_lifecycle_pb.LifecycleDeleteOutcome{
+		s3_lifecycle_pb.LifecycleDeleteOutcome_DONE,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_DONE, // would be a bug if reached
+	}}
+	cfg := Config{Client: client}
+	_, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, matches)
+	require.NoError(t, err)
+	require.True(t, halted, "BLOCKED outcome must halt the event loop")
+	assert.Equal(t, []string{"ok", "blocked"}, client.seenObjects(),
+		"would-be-next must NOT be dispatched after the halt")
+}
+
+func TestProcessMatches_EmptyMatchesIsNoop(t *testing.T) {
+	runNow := time.Now()
+	client := &recordingClient{}
+	cfg := Config{Client: client}
+	skipped, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, nil)
+	require.NoError(t, err)
+	require.False(t, halted)
+	require.False(t, skipped)
+	assert.Empty(t, client.seenObjects())
+}
+
+func TestProcessMatches_AllDueNoSkippedFlag(t *testing.T) {
+	// All matches past their DueTime — skippedAny must be false so the
+	// caller can advance the cursor past this event.
+	runNow := time.Now()
+	rh := [8]byte{0xcc}
+	rule := s3lifecycle.ActionKey{Bucket: "b", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	matches := []router.Match{
+		{Key: rule, Bucket: "b", ObjectKey: "a", DueTime: runNow.Add(-time.Hour)},
+		{Key: rule, Bucket: "b", ObjectKey: "b", DueTime: runNow.Add(-30 * time.Minute)},
+	}
+	client := &recordingClient{}
+	cfg := Config{Client: client}
+	skipped, halted, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, matches)
+	require.NoError(t, err)
+	require.False(t, halted)
+	require.False(t, skipped, "every match was past DueTime; skippedAny must be false")
+}
+
+func TestProcessMatches_DispatchCounterIncrements(t *testing.T) {
+	// Pin that dispatched matches increment S3LifecycleDispatchCounter
+	// with the outcome label so a refactor doesn't silently drop the
+	// observability hook. Use a bucket/kind no other test touches to
+	// keep the read-after-write stable.
+	runNow := time.Now()
+	rh := [8]byte{0xfe}
+	rule := s3lifecycle.ActionKey{Bucket: "metrics-pin-bkt", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	matches := []router.Match{
+		{Key: rule, Bucket: "metrics-pin-bkt", ObjectKey: "obj", DueTime: runNow.Add(-time.Hour)},
+	}
+	client := &recordingClient{} // default DONE
+	before := dispatchCounterValue("metrics-pin-bkt", "expiration_days", "DONE")
+	// Delete the label row on exit so this test doesn't leak into the
+	// in-process Prometheus registry that other tests share.
+	defer stats.S3LifecycleDispatchCounter.DeleteLabelValues("metrics-pin-bkt", "expiration_days", "DONE")
+	cfg := Config{Client: client}
+	_, _, err := processMatches(context.Background(), cfg, runNow, &reader.Event{}, matches)
+	require.NoError(t, err)
+	after := dispatchCounterValue("metrics-pin-bkt", "expiration_days", "DONE")
+	assert.Equal(t, before+1, after, "DONE outcome must increment the dispatch counter")
+}
+
+// dispatchCounterValue reads the current value of the shared
+// S3LifecycleDispatchCounter for the given (bucket, kind, outcome).
+func dispatchCounterValue(bucket, kind, outcome string) float64 {
+	m := stats.S3LifecycleDispatchCounter.WithLabelValues(bucket, kind, outcome)
+	var pm dto.Metric
+	if err := m.Write(&pm); err != nil {
+		return 0
+	}
+	if pm.Counter == nil {
+		return 0
+	}
+	return pm.Counter.GetValue()
+}

weed/s3api/s3lifecycle/dailyrun/run.go

@@ -0,0 +1,772 @@
+package dailyrun
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strconv"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
+	"github.com/seaweedfs/seaweedfs/weed/stats"
+	"github.com/seaweedfs/seaweedfs/weed/util"
+	"golang.org/x/time/rate"
+)
+
+// LifecycleClient mirrors dispatcher's contract; duplicated to avoid an
+// import cycle.
+type LifecycleClient interface {
+	LifecycleDelete(ctx context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error)
+}
+
+// WalkerFunc handles the per-shard bucket walk for a given engine view.
+// Phase 4b uses it on the recovery branch (rule-content edit / partition
+// flip) so already-due objects across the rewritten rule set get caught
+// before the cursor rewinds.
+type WalkerFunc func(ctx context.Context, view *engine.Snapshot, shardID int) error
+
+type Config struct {
+	Shards      []int
+	BucketsPath string
+	Engine      *engine.Engine
+	FilerClient filer_pb.SeaweedFilerClient
+	Client      LifecycleClient
+	Persister   CursorPersister
+	Lister      router.SiblingLister
+
+	// Workers <= 0 -> 1 (serial).
+	Workers int
+
+	// nil -> no rate limit. Shared across all shard goroutines.
+	Limiter *rate.Limiter
+
+	// Meta-log retention boundary. Rules whose effective TTL exceeds
+	// this can't be serviced by replay alone and get partitioned into
+	// the walk view (engine.PromotedHash). 0 falls back to maxTTL,
+	// which keeps PromotedHash empty and the partition-flip recovery
+	// trigger dormant.
+	RetentionWindow time.Duration
+
+	// Walker is invoked on the recovery branch (rule-content edit or
+	// partition flip) before the cursor rewind. It receives the
+	// engine.RecoveryView so only the rules that need bulk re-evaluation
+	// are walked, and the per-shard ID so the implementation can filter
+	// entries. nil disables walker invocation entirely — the cursor
+	// still rewinds, matching Phase 4a behavior.
+	Walker WalkerFunc
+
+	// WalkerInterval throttles the steady-state and empty-replay walker
+	// invocations: the walker fires only when the time since the
+	// per-shard cursor's LastWalkedNs exceeds this value. Cold-start
+	// and recovery walker fires (RecoveryView) are unconditional —
+	// those are bounded events that must run once when the trigger
+	// condition is met. 0 means "fire on every pass" (the prior
+	// behavior, suitable for s3tests-style rapid-cadence drivers and
+	// for in-repo integration tests).
+	//
+	// Production deployments should set this to roughly the walk cost
+	// budget per shard per cluster: e.g., 1h for a small cluster, 6h
+	// for a large one. The walker reads the whole bucket subtree it
+	// covers, so a too-short interval crushes the filer; a too-long
+	// interval delays ExpirationDate and ExpiredObjectDeleteMarker
+	// dispatches.
+	WalkerInterval time.Duration
+
+	ClientName string
+	// 0 -> randomized per-run.
+	ClientID int32
+
+	// nil -> time.Now.
+	Now func() time.Time
+
+	// 0 -> unbounded.
+	EventBudget int
+
+	// 0 -> defaultSubscriptionReceiveTimeout.
+	SubscriptionReceiveTimeout time.Duration
+}
+
+// Above the filer's 15m maxGapStall, so a subscriber parked on a gap is
+// not mistaken for a stalled one.
+const defaultSubscriptionReceiveTimeout = 20 * time.Minute
+
+// Run executes the daily replay for every shard in cfg.Shards
+// concurrently. Returns the first shard error; the rest log and run to
+// completion so one shard's transient failure doesn't lose other shards'
+// progress.
+//
+// All cfg.Shards share one meta-log subscription. The Reader's
+// ShardPredicate accepts any shard in cfg.Shards, and a fan-out
+// goroutine routes events to per-shard channels by ev.ShardID. This
+// replaces the earlier per-shard Reader that opened 16 SubscribeMetadata
+// streams to filer per pass.
+func Run(ctx context.Context, cfg Config) error {
+	if err := validate(cfg); err != nil {
+		return err
+	}
+	now := cfg.Now
+	if now == nil {
+		now = func() time.Time { return time.Now().UTC() }
+	}
+	// Freeze "now" so shards agree on the boundary.
+	runNow := now()
+	startedAt := time.Now()
+
+	// Capture once so a mid-run Compile can't make shards disagree.
+	snap := cfg.Engine.Snapshot()
+	rsh := engine.ReplayContentHash(snap)
+	maxTTL := engine.MaxEffectiveTTL(snap)
+
+	// Since all shards share one subscription, every shard goroutine
+	// must be live to drain its channel — capping concurrency would
+	// stall the fan-out. cfg.Workers no longer gates shard concurrency
+	// (dispatch is throttled via cfg.Limiter); accepted for backwards
+	// compatibility but otherwise inert.
+	_ = cfg.Workers
+
+	// rsh==[32]byte{} means no replay-eligible rules — runShard's
+	// walker-only branch fires and saves; no subscription needed.
+	var (
+		shardEvents     map[int]chan *reader.Event
+		readerDone      chan error
+		fanoutDone      chan struct{}
+		cancelRead      context.CancelFunc
+		globalStartTsNs int64
+		// Set by the reader goroutine when it stops for a reason we asked
+		// for, rather than a broken stream.
+		stoppedOnPurpose atomic.Bool
+	)
+	if rsh != [32]byte{} {
+		// Pre-load all per-shard cursors so the shared subscription
+		// can start from min(per-shard startTsNs). Using the cold-start
+		// floor (runNow - maxTTL) globally would skip past pending
+		// events on shards whose cursor is older than the floor —
+		// exactly the case where a rule's TTL equals maxTTL and an
+		// older event still has DueTime <= runNow.
+		globalStartTsNs = computeGlobalStartTsNs(ctx, cfg, runNow, maxTTL)
+		shardEvents, readerDone, fanoutDone, cancelRead = startSharedSubscription(ctx, cfg, runNow, globalStartTsNs, &stoppedOnPurpose)
+		defer cancelRead()
+	}
+
+	var wg sync.WaitGroup
+	errCh := make(chan error, len(cfg.Shards))
+	for _, sh := range cfg.Shards {
+		sh := sh
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			var ch <-chan *reader.Event
+			if shardEvents != nil {
+				ch = shardEvents[sh]
+			}
+			if err := runShard(ctx, cfg, snap, runNow, sh, ch); err != nil {
+				errCh <- err
+			}
+			// A halted dispatch returns runShard mid-stream. Keep
+			// discarding, or the fan-out blocks on this shard's full
+			// buffer and starves every other drain.
+			if ch != nil {
+				for range ch {
+				}
+			}
+		}()
+	}
+	wg.Wait()
+	close(errCh)
+
+	// Tear down the shared subscription. cancelRead unblocks both the
+	// reader's gRPC stream and the fan-out's send loop; we wait on both
+	// so their goroutines don't outlive Run.
+	var readerErr error
+	if cancelRead != nil {
+		cancelRead()
+		<-fanoutDone
+		if rerr := <-readerDone; rerr != nil && !stoppedOnPurpose.Load() {
+			// The drains ended on a closed channel, so the pass is
+			// truncated. Cursors hold what was processed; what matters is
+			// that the job doesn't go green on a dead subscription.
+			readerErr = fmt.Errorf("shared meta-log subscription: %w", rerr)
+			glog.Warningf("daily_run: %v", readerErr)
+		}
+	}
+
+	var first error
+	errCount := 0
+	for err := range errCh {
+		errCount++
+		if first == nil {
+			first = err
+		} else {
+			glog.V(1).Infof("daily_run: additional shard error: %v", err)
+		}
+	}
+	if readerErr != nil {
+		errCount++
+		if first == nil {
+			first = readerErr
+		}
+	}
+	status := "ok"
+	if first != nil {
+		status = "error"
+	}
+	// Summary line is parsed by operator scripts and CI log greps; keep
+	// the existing key=value tokens stable and append new ones at the
+	// end. cursor_lag_max / walked_max_age are the per-pass observability
+	// floor — for finer detail, scrape the per-shard gauges.
+	lagSummary := summarizeShardCursorLag(ctx, cfg, runNow)
+	glog.V(0).Infof("daily_run: status=%s shards=%d errors=%d duration=%s %s",
+		status, len(cfg.Shards), errCount, time.Since(startedAt).Round(time.Millisecond), lagSummary)
+	return first
+}
+
+// summarizeShardCursorLag walks the persisted cursors one more time
+// after the run, computes the worst (largest) lag and walker-age across
+// the cfg.Shards set, and renders them as key=value tokens. Returns
+// a string suitable for appending to the heartbeat line. Load errors
+// short-circuit to a marker token so the heartbeat doesn't lie about
+// the state.
+func summarizeShardCursorLag(ctx context.Context, cfg Config, runNow time.Time) string {
+	var (
+		maxLag    time.Duration
+		maxAge    time.Duration
+		anyCursor bool
+		anyWalked bool
+	)
+	for _, sh := range cfg.Shards {
+		c, found, err := cfg.Persister.Load(ctx, sh)
+		if err != nil || !found {
+			continue
+		}
+		if c.TsNs > 0 {
+			lag := runNow.Sub(time.Unix(0, c.TsNs))
+			if !anyCursor || lag > maxLag {
+				maxLag = lag
+				anyCursor = true
+			}
+		}
+		if c.LastWalkedNs > 0 {
+			age := runNow.Sub(time.Unix(0, c.LastWalkedNs))
+			if !anyWalked || age > maxAge {
+				maxAge = age
+				anyWalked = true
+			}
+		}
+	}
+	lagStr := "cursor_lag_max=cold"
+	if anyCursor {
+		lagStr = fmt.Sprintf("cursor_lag_max=%s", maxLag.Round(time.Second))
+	}
+	ageStr := "walked_max_age=cold"
+	if anyWalked {
+		ageStr = fmt.Sprintf("walked_max_age=%s", maxAge.Round(time.Second))
+	}
+	return lagStr + " " + ageStr
+}
+
+// computeGlobalStartTsNs scans every shard's persisted cursor and
+// returns the minimum startTsNs the shared subscription must cover.
+// For each shard the start point is the persisted cursor (steady state)
+// or runNow - maxTTL (cold start, when no cursor exists). Load errors
+// downgrade to the cold-start floor for that shard — failing closed
+// would be worse than re-scanning maxTTL of events.
+func computeGlobalStartTsNs(ctx context.Context, cfg Config, runNow time.Time, maxTTL time.Duration) int64 {
+	floor := runNow.Add(-maxTTL).UnixNano()
+	min := int64(0)
+	first := true
+	for _, sh := range cfg.Shards {
+		var start int64
+		if persisted, found, err := cfg.Persister.Load(ctx, sh); err == nil && found {
+			start = persisted.TsNs
+		} else {
+			start = floor
+		}
+		if first || start < min {
+			min = start
+			first = false
+		}
+	}
+	if first {
+		// No shards (validate() should have caught this, but defensive).
+		return floor
+	}
+	return min
+}
+
+// startSharedSubscription opens one SubscribeMetadata stream covering
+// every shard in cfg.Shards and fans events out to per-shard channels.
+// Subscription floor is the caller-supplied globalStartTsNs (typically
+// min over per-shard cursors). Shards whose own startTsNs is fresher
+// filter out already-past events themselves inside drainShardEvents.
+// Events arriving with TsNs > runUpTo (the pass boundary) cause the
+// fan-out to cancel the reader and close all per-shard channels,
+// ending the pass.
+func startSharedSubscription(ctx context.Context, cfg Config, runNow time.Time, globalStartTsNs int64, stoppedOnPurpose *atomic.Bool) (map[int]chan *reader.Event, chan error, chan struct{}, context.CancelFunc) {
+	// Set by the fan-out when it ends the pass on an event past runUpTo.
+	var pastBoundary atomic.Bool
+	shardSet := make(map[int]bool, len(cfg.Shards))
+	shardEvents := make(map[int]chan *reader.Event, len(cfg.Shards))
+	for _, sh := range cfg.Shards {
+		shardSet[sh] = true
+		// Per-shard channel size has to absorb event bursts between
+		// drains' iterations without backpressuring the fan-out.
+		// 256 covers a busy test bucket; production tuning can lift
+		// this further.
+		shardEvents[sh] = make(chan *reader.Event, 256)
+	}
+
+	clientName := cfg.ClientName
+	if clientName == "" {
+		clientName = "worker-s3-lifecycle-daily"
+	}
+	clientID := cfg.ClientID
+	if clientID == 0 {
+		clientID = int32(util.RandomInt32())
+	}
+
+	runUpTo := runNow.UnixNano()
+	receiveTimeout := cfg.SubscriptionReceiveTimeout
+	if receiveTimeout == 0 {
+		receiveTimeout = defaultSubscriptionReceiveTimeout
+	}
+	events := make(chan *reader.Event, 4*len(cfg.Shards))
+	rd := &reader.Reader{
+		ShardPredicate: func(id int) bool { return shardSet[id] },
+		BucketsPath:    cfg.BucketsPath,
+		StartTsNs:      globalStartTsNs,
+		// The pass covers (globalStartTsNs, runUpTo]. Unbounded, it ends
+		// only when an unrelated write pushes an event past runUpTo — so
+		// a quiet cluster wedges it.
+		UntilTsNs:      runUpTo,
+		Events:         events,
+		EventBudget:    cfg.EventBudget,
+		ReceiveTimeout: receiveTimeout,
+	}
+
+	readerCtx, cancelReader := context.WithCancel(ctx)
+	readerDone := make(chan error, 1)
+	go func() {
+		rerr := rd.Run(readerCtx, cfg.FilerClient, clientName, clientID)
+		// Not decidable from the error: a stream we cancel and one the
+		// filer cancels both arrive as codes.Canceled. Snapshot intent
+		// here rather than after teardown, or a deadline expiring during
+		// the drains and cursor saves masks a real failure.
+		stoppedOnPurpose.Store(ctx.Err() != nil || pastBoundary.Load())
+		readerDone <- rerr
+		// Only sender. Closing is what ends the fan-out, and through it
+		// every drain, when the stream finishes on its own.
+		close(events)
+	}()
+
+	fanoutDone := make(chan struct{})
+	go func() {
+		defer close(fanoutDone)
+		defer func() {
+			for _, ch := range shardEvents {
+				close(ch)
+			}
+		}()
+		for {
+			select {
+			case <-readerCtx.Done():
+				return
+			case ev, ok := <-events:
+				if !ok {
+					return
+				}
+				if ev == nil {
+					continue
+				}
+				// Meta-log events arrive in TsNs order; the first
+				// event past runUpTo means everything after is past
+				// too. Cancel the reader so subsequent passes don't
+				// pay for stream tail we'd drop anyway.
+				if ev.TsNs > runUpTo {
+					pastBoundary.Store(true)
+					cancelReader()
+					return
+				}
+				ch := shardEvents[ev.ShardID]
+				if ch == nil {
+					continue
+				}
+				select {
+				case <-readerCtx.Done():
+					return
+				case ch <- ev:
+				}
+			}
+		}
+	}()
+
+	return shardEvents, readerDone, fanoutDone, cancelReader
+}
+
+func validate(cfg Config) error {
+	if cfg.Engine == nil {
+		return errors.New("daily_run: nil Engine")
+	}
+	if cfg.FilerClient == nil {
+		return errors.New("daily_run: nil FilerClient")
+	}
+	if cfg.Client == nil {
+		return errors.New("daily_run: nil Client (LifecycleDelete RPC)")
+	}
+	if cfg.Persister == nil {
+		return errors.New("daily_run: nil Persister")
+	}
+	if cfg.Lister == nil {
+		return errors.New("daily_run: nil Lister")
+	}
+	if cfg.BucketsPath == "" {
+		return errors.New("daily_run: empty BucketsPath")
+	}
+	// A negative WalkerInterval would silently fall through walkerDue's
+	// `interval <= 0` branch and re-enable "walk every pass", defeating
+	// the throttle whose whole point is bounding filer load. The
+	// admin-config parser already clamps negative input to zero
+	// (worker/tasks/s3_lifecycle/config.go), but callers using
+	// dailyrun.Config directly (tests, embedders, future drivers) get
+	// a loud failure instead.
+	if cfg.WalkerInterval < 0 {
+		return fmt.Errorf("daily_run: negative WalkerInterval %v (0 = unthrottled, positive values throttle)", cfg.WalkerInterval)
+	}
+	// Negative would reach the reader as "disabled" by accident.
+	if cfg.SubscriptionReceiveTimeout < 0 {
+		return fmt.Errorf("daily_run: negative SubscriptionReceiveTimeout %v (0 = default %v)", cfg.SubscriptionReceiveTimeout, defaultSubscriptionReceiveTimeout)
+	}
+	for _, sh := range cfg.Shards {
+		if sh < 0 || sh >= s3lifecycle.ShardCount {
+			return fmt.Errorf("daily_run: shard %d out of [0, %d)", sh, s3lifecycle.ShardCount)
+		}
+	}
+	return nil
+}
+
+// runShard executes one daily-replay pass; see DESIGN.md for algorithm.
+// Three walker invocation paths under cfg.Walker (when set):
+//   - cold start / recovery: RecoveryView. Unconditional — these are
+//     bounded events (first run for a shard, or rule-content edit / partition
+//     flip) and the walker must run once when the trigger condition is met.
+//   - steady state: RulesForShard's walk view, throttled by cfg.WalkerInterval
+//     against persisted.LastWalkedNs so the walker fires on its own cadence
+//     independently of how often Run() is invoked.
+//   - empty replay: same throttling — buckets with only walker-bound rules
+//     use the walk view, and a too-fast Run() driver would otherwise burn
+//     the filer with a full subtree scan per tick.
+func runShard(ctx context.Context, cfg Config, snap *engine.Snapshot, runNow time.Time, shardID int, events <-chan *reader.Event) error {
+	shardLabel := strconv.Itoa(shardID)
+	shardStart := time.Now()
+	defer func() {
+		stats.S3LifecycleDailyRunShardDurationSeconds.WithLabelValues(shardLabel).Observe(time.Since(shardStart).Seconds())
+	}()
+	persisted, found, err := cfg.Persister.Load(ctx, shardID)
+	if err != nil {
+		return fmt.Errorf("shard=%d: load cursor: %w", shardID, err)
+	}
+
+	// engine.ReplayContentHash has a different byte layout than the
+	// Phase-2 local helper. First post-upgrade run mismatches every
+	// cursor and drops into the rule-change branch below — bounded
+	// one-time rewind to runNow - maxTTL, self-healing on save.
+	rsh := engine.ReplayContentHash(snap)
+	maxTTL := engine.MaxEffectiveTTL(snap)
+	// Operator-supplied retention falls back to maxTTL. In steady
+	// state every active replay rule has TTL <= maxTTL by construction,
+	// so promoted is empty and the partition-flip trigger is dormant.
+	// During bootstrap (rules compiled but not yet active) maxTTL is
+	// 0, retentionWindow is 0, and every rule with TTL > 0 lands in
+	// the walk partition; the resulting non-empty promoted forces a
+	// recovery walk on the first run after rules activate, which is
+	// the intended bootstrap behavior. Once the handler plumbs the
+	// real meta-log retention here, PromotedHash starts catching
+	// retention-driven partition flips in addition.
+	retentionWindow := cfg.RetentionWindow
+	if retentionWindow <= 0 {
+		retentionWindow = maxTTL
+	}
+	promoted := engine.PromotedHash(snap, retentionWindow)
+
+	// lastWalkedNs threads the steady-state walker's clock through the
+	// cursor saves below. Start with the persisted value; only update
+	// it when a steady-state / empty-replay walker fire actually
+	// happens. Cold-start and recovery walker fires also bump it so
+	// the next pass's throttle has a fresh anchor.
+	//
+	// walkedThisPass is the in-pass guard: walkerDue answers the
+	// persisted-state question ("has enough time elapsed?"), but a
+	// cold-start or recovery branch above the steady-state check
+	// already fired the walker with RecoveryView, which is a superset
+	// of every per-shard partition. Keeping the within-pass guard out
+	// of walkerDue lets that function stay pure (interval/never-walked
+	// logic only) and side-steps the test-injectable-runNow ambiguity
+	// where two distinct passes happen to share a UnixNano.
+	lastWalkedNs := persisted.LastWalkedNs
+	walkedThisPass := false
+
+	if rsh == [32]byte{} {
+		// No replay-eligible rules. Walker-only rules
+		// (ExpirationDate / ExpiredDeleteMarker / NewerNoncurrent)
+		// or scan_only-promoted rules might still need a walk; run
+		// the walker (throttled by WalkerInterval) before persisting
+		// the empty cursor and returning. Without this, a bucket whose
+		// only rule is walker-bound would never have it dispatched —
+		// the bug TestLifecycleExpirationDateInThePast caught.
+		if cfg.Walker != nil && walkerDue(lastWalkedNs, runNow, cfg.WalkerInterval) {
+			if _, walkView := snap.RulesForShard(shardID, retentionWindow); walkView != nil && len(walkView.AllActions()) > 0 {
+				if werr := cfg.Walker(ctx, walkView, shardID); werr != nil {
+					return fmt.Errorf("shard=%d: steady walk (empty replay): %w", shardID, werr)
+				}
+				lastWalkedNs = runNow.UnixNano()
+				walkedThisPass = true
+			}
+		}
+		_ = walkedThisPass // empty-replay branch returns; no downstream walker.
+		return saveCursorAndPublish(ctx, cfg.Persister, shardID, Cursor{
+			TsNs:         0,
+			RuleSetHash:  rsh,
+			PromotedHash: promoted,
+			LastWalkedNs: lastWalkedNs,
+		})
+	}
+
+	// Recovery / cold-start walker:
+	//   - found && hashes mismatch: rule edit or partition flip — walk
+	//     the rewritten rule set so already-due objects fire before the
+	//     cursor rewinds, then rewind for meta-log replay.
+	//   - !found: first run for this shard. Pre-existing objects PUT
+	//     before the rule was added live OUTSIDE the meta-log scan
+	//     window (TsNs > runNow - maxTTL) and would never replay; the
+	//     walker has to discover them. The streaming worker did this
+	//     via BucketBootstrapper; daily-replay needs the same.
+	mustWalkRecovery := found && (persisted.RuleSetHash != rsh || persisted.PromotedHash != promoted)
+	mustWalkColdStart := !found
+	if mustWalkRecovery || mustWalkColdStart {
+		if cfg.Walker != nil {
+			if werr := cfg.Walker(ctx, engine.RecoveryView(snap), shardID); werr != nil {
+				return fmt.Errorf("shard=%d: recovery walk: %w", shardID, werr)
+			}
+			lastWalkedNs = runNow.UnixNano()
+			walkedThisPass = true
+		}
+		if mustWalkRecovery {
+			// Rule changed: rewind cursor so the sliding replay re-scans
+			// the new max-TTL window and the persisted hashes match the
+			// new rule set.
+			next := Cursor{
+				TsNs:         runNow.Add(-maxTTL).UnixNano(),
+				RuleSetHash:  rsh,
+				PromotedHash: promoted,
+				LastWalkedNs: lastWalkedNs,
+			}
+			return saveCursorAndPublish(ctx, cfg.Persister, shardID, next)
+		}
+		// Cold start: keep TsNs=0 so the drain below floors to
+		// runNow - maxTTL and the cursor is saved fresh after the run.
+	}
+
+	// Steady-state walker for walker-bound and scan_only-promoted rules.
+	// RulesForShard splits the snapshot using the same retentionWindow
+	// PromotedHash used, so the walk view is exactly the partition the
+	// hash already accounted for. Empty walk view (no rules need walking
+	// today) skips the call so non-versioned, replay-only deployments
+	// don't pay an O(N) bucket-walk per run.
+	//
+	// Two-stage gate: walkedThisPass suppresses a double walk when the
+	// cold-start branch above already fired (RecoveryView covered it).
+	// walkerDue applies the persisted-state throttle for the case
+	// where this is a normal pass that didn't trigger cold-start.
+	if cfg.Walker != nil && !walkedThisPass && walkerDue(lastWalkedNs, runNow, cfg.WalkerInterval) {
+		if _, walkView := snap.RulesForShard(shardID, retentionWindow); walkView != nil && len(walkView.AllActions()) > 0 {
+			if werr := cfg.Walker(ctx, walkView, shardID); werr != nil {
+				return fmt.Errorf("shard=%d: steady walk: %w", shardID, werr)
+			}
+			lastWalkedNs = runNow.UnixNano()
+		}
+	}
+
+	// Cold start: scan from now-maxTTL so already-due objects within
+	// meta-log retention still expire. In steady state honor the
+	// cursor as-is: the drain freezes the cursor at the last pre-skip
+	// event so pending matches with DueTime == TsNs+maxTTL stay in
+	// scope across passes. Bumping forward to runNow-maxTTL would
+	// orphan exactly those events (the test_lifecyclev2_expiration
+	// regression: cursor saved at the no-match event right before
+	// the not-yet-due expire3 PUT, then floor at runNow=PUT+maxTTL
+	// equals PUT — bumping past the expire3 event itself).
+	startTsNs := persisted.TsNs
+	if !found {
+		startTsNs = runNow.Add(-maxTTL).UnixNano()
+	}
+
+	lastOK, _, drainErr := drainShardEvents(ctx, cfg, runNow, shardID, snap, startTsNs, events)
+	// Fresh ctx: a wall-clock cap on the pass (the shell driver's
+	// -runtime) cancels the drain's, and saving with it would drop the
+	// cursor and re-replay from the same floor next pass.
+	saveCtx, saveCancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer saveCancel()
+	if drainErr != nil {
+		_ = saveCursorAndPublish(saveCtx, cfg.Persister, shardID, Cursor{TsNs: lastOK, RuleSetHash: rsh, PromotedHash: promoted, LastWalkedNs: lastWalkedNs})
+		// Cut short by its wall-clock cap: truncated, not failed — the
+		// cursor above carries progress forward.
+		if errors.Is(drainErr, context.DeadlineExceeded) || errors.Is(drainErr, context.Canceled) {
+			return nil
+		}
+		return fmt.Errorf("shard=%d: drain: %w", shardID, drainErr)
+	}
+
+	return saveCursorAndPublish(saveCtx, cfg.Persister, shardID, Cursor{
+		TsNs:         lastOK,
+		RuleSetHash:  rsh,
+		PromotedHash: promoted,
+		LastWalkedNs: lastWalkedNs,
+	})
+}
+
+// saveCursorAndPublish persists the cursor and, on success, updates the
+// per-shard Prometheus gauges so operators can read cursor lag
+// (now - cursor_min_ts_ns) and walker freshness (now - last_walked_ns)
+// without having to read the cursor file directly. On save failure the
+// gauges are left untouched: their last successful-save value is more
+// useful than a value that doesn't match what's on the filer.
+func saveCursorAndPublish(ctx context.Context, p CursorPersister, shardID int, c Cursor) error {
+	if err := p.Save(ctx, shardID, c); err != nil {
+		return err
+	}
+	label := strconv.Itoa(shardID)
+	stats.S3LifecycleCursorMinTsNs.WithLabelValues(label).Set(float64(c.TsNs))
+	stats.S3LifecycleDailyRunLastWalkedNs.WithLabelValues(label).Set(float64(c.LastWalkedNs))
+	return nil
+}
+
+// walkerDue answers the persisted-state throttle: has enough time
+// elapsed since the last walker fire on this shard? interval == 0
+// means "fire every pass" (the prior, unconditional behavior).
+// lastWalkedNs == 0 means "never walked steady-state" — fire to
+// establish the throttle anchor. Otherwise gate on
+// (runNow - lastWalkedNs) >= interval.
+//
+// Within-pass double-fire suppression (cold-start / recovery walker
+// already ran this pass; steady-state branch must not re-walk a
+// superset partition) lives in runShard's walkedThisPass local flag
+// rather than here. Keeping walkerDue pure means the function answers
+// one question and a test that injects two distinct passes with the
+// same runNow doesn't get falsely throttled.
+func walkerDue(lastWalkedNs int64, runNow time.Time, interval time.Duration) bool {
+	if interval <= 0 || lastWalkedNs == 0 {
+		return true
+	}
+	return runNow.UnixNano()-lastWalkedNs >= int64(interval)
+}
+
+// drainShardEvents reads pre-fanned-out events for this shard from the
+// shared meta-log subscription and dispatches matches whose due_time is
+// past runNow. cursorAdvanceTo stops growing at the first event with a
+// not-yet-due match so that event is re-scanned in a later run.
+// halted=true marks an unresolved dispatch outcome.
+//
+// The subscription itself lives at the Run() level — one filer stream
+// fans out to per-shard channels via shardEventDispatcher. Each shard
+// drains its own channel and advances its own cursor; future-TsNs
+// events (relative to this pass's runNow) are filtered out by the
+// dispatcher before they reach this channel.
+func drainShardEvents(ctx context.Context, cfg Config, runNow time.Time, shardID int, snap *engine.Snapshot, startTsNs int64, events <-chan *reader.Event) (int64, bool, error) {
+	cursorAdvanceTo := startTsNs
+	stuck := false
+	halted := false
+
+drain:
+	for {
+		select {
+		case <-ctx.Done():
+			return cursorAdvanceTo, true, ctx.Err()
+		case ev, ok := <-events:
+			if !ok {
+				break drain
+			}
+			if ev == nil {
+				continue
+			}
+			// The global subscription starts at min(per-shard startTsNs),
+			// so this shard may receive events that are already past its
+			// own cursor. Skip them rather than re-dispatching.
+			if ev.TsNs <= startTsNs {
+				continue
+			}
+			stats.S3LifecycleDailyRunEventsScanned.WithLabelValues(strconv.Itoa(shardID)).Inc()
+			matches := router.Route(ctx, snap, ev, runNow, cfg.Lister)
+			eventSkipped, eventHalted, eventErr := processMatches(ctx, cfg, runNow, ev, matches)
+			if eventErr != nil {
+				return cursorAdvanceTo, true, eventErr
+			}
+			if eventHalted {
+				halted = true
+				break drain
+			}
+			if eventSkipped {
+				stuck = true
+				continue
+			}
+			if !stuck {
+				cursorAdvanceTo = ev.TsNs
+			}
+		}
+	}
+
+	return cursorAdvanceTo, halted, nil
+}
+
+// processMatches dispatches matches from one event. A future-DueTime
+// match is silently skipped (per-match, not per-rule): one event can
+// emit multiple matches for the same ActionKey across noncurrent
+// siblings, so a not-yet-due sibling must not gate a due one. The
+// cursor-advance gate upstream re-scans skipped events tomorrow.
+func processMatches(ctx context.Context, cfg Config, runNow time.Time, ev *reader.Event, matches []router.Match) (skippedAny, halted bool, err error) {
+	for _, m := range matches {
+		if m.DueTime.After(runNow) {
+			skippedAny = true
+			continue
+		}
+		if cfg.Limiter != nil {
+			waitStart := time.Now()
+			if waitErr := cfg.Limiter.Wait(ctx); waitErr != nil {
+				return skippedAny, true, waitErr
+			}
+			stats.S3LifecycleDispatchLimiterWaitSeconds.Observe(time.Since(waitStart).Seconds())
+		}
+		outcome, dispatchErr := dispatchWithRetry(ctx, cfg.Client, m)
+		if dispatchErr != nil {
+			glog.V(1).Infof("daily_run: transport error on %s/%s %s: %v",
+				m.Bucket, m.ObjectKey, m.Key.ActionKind, dispatchErr)
+			// "RPC_ERROR" matches the streaming dispatcher's label
+			// (dispatcher/dispatcher.go) so transport failures
+			// aggregate under one outcome key across paths.
+			stats.S3LifecycleDispatchCounter.WithLabelValues(m.Bucket, m.Key.ActionKind.String(), "RPC_ERROR").Inc()
+			return skippedAny, true, nil
+		}
+		stats.S3LifecycleDispatchCounter.WithLabelValues(m.Bucket, m.Key.ActionKind.String(), outcome.String()).Inc()
+		switch outcome {
+		case s3_lifecycle_pb.LifecycleDeleteOutcome_DONE,
+			s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED,
+			s3_lifecycle_pb.LifecycleDeleteOutcome_SKIPPED_OBJECT_LOCK:
+		case s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER,
+			s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED:
+			glog.V(1).Infof("daily_run: %s on %s/%s %s",
+				outcome, m.Bucket, m.ObjectKey, m.Key.ActionKind)
+			return skippedAny, true, nil
+		default:
+			glog.V(1).Infof("daily_run: unknown outcome %v on %s/%s", outcome, m.Bucket, m.ObjectKey)
+			return skippedAny, true, nil
+		}
+		_ = ev
+	}
+	return skippedAny, false, nil
+}

weed/s3api/s3lifecycle/dailyrun/run_bounded_test.go

@@ -0,0 +1,297 @@
+package dailyrun
+
+import (
+	"context"
+	"io"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// idleSubscribeStream: nothing to deliver past the pass boundary. A
+// bounded subscription gets EOF; an unbounded one blocks forever.
+type idleSubscribeStream struct {
+	grpc.ClientStream
+	ctx     context.Context
+	untilNs int64
+}
+
+func (s *idleSubscribeStream) Recv() (*filer_pb.SubscribeMetadataResponse, error) {
+	if s.untilNs != 0 {
+		return nil, io.EOF
+	}
+	<-s.ctx.Done()
+	return nil, s.ctx.Err()
+}
+
+type idleSubscribeClient struct {
+	filer_pb.SeaweedFilerClient
+	gotReq chan *filer_pb.SubscribeMetadataRequest
+}
+
+func (c *idleSubscribeClient) SubscribeMetadata(ctx context.Context, req *filer_pb.SubscribeMetadataRequest, _ ...grpc.CallOption) (filer_pb.SeaweedFiler_SubscribeMetadataClient, error) {
+	select {
+	case c.gotReq <- req:
+	default:
+	}
+	return &idleSubscribeStream{ctx: ctx, untilNs: req.UntilNs}, nil
+}
+
+// The reported wedge: the pass ended only when an unrelated write pushed
+// an event past its boundary, so a quiet cluster starved every drain and
+// Run's WaitGroup never reached zero.
+func TestRun_EndsOnIdleSubscription(t *testing.T) {
+	e := engine.New()
+	e.Compile([]engine.CompileInput{
+		{Bucket: "b1", Rules: []*s3lifecycle.Rule{
+			{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 30},
+		}},
+	}, engine.CompileOptions{})
+	require.NotEqual(t, [32]byte{}, engine.ReplayContentHash(e.Snapshot()),
+		"rule must be replay-eligible or Run skips the subscription entirely")
+
+	client := &idleSubscribeClient{gotReq: make(chan *filer_pb.SubscribeMetadataRequest, 1)}
+	runNow := time.Now().UTC()
+	cfg := Config{
+		Shards:      []int{0, 1, 2},
+		BucketsPath: "/buckets",
+		Engine:      e,
+		FilerClient: client,
+		Client:      stubLifecycleClient{},
+		Persister:   newMemPersister(),
+		Lister:      stubSiblingLister{},
+		Now:         func() time.Time { return runNow },
+	}
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.NoError(t, err)
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run did not return on an idle subscription")
+	}
+
+	req := <-client.gotReq
+	assert.Equal(t, runNow.UnixNano(), req.UntilNs, "subscription must stop at the pass boundary")
+}
+
+// failingSubscribeClient fails the subscribe call outright.
+type failingSubscribeClient struct {
+	filer_pb.SeaweedFilerClient
+	err error
+}
+
+func (c *failingSubscribeClient) SubscribeMetadata(_ context.Context, _ *filer_pb.SubscribeMetadataRequest, _ ...grpc.CallOption) (filer_pb.SeaweedFiler_SubscribeMetadataClient, error) {
+	return nil, c.err
+}
+
+// Flip side of closing the event channel: the drains now end cleanly on
+// a dead subscription, so the pass would otherwise go green having
+// processed nothing.
+func TestRun_ReaderFailureFailsThePass(t *testing.T) {
+	e := engine.New()
+	e.Compile([]engine.CompileInput{
+		{Bucket: "b1", Rules: []*s3lifecycle.Rule{
+			{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 30},
+		}},
+	}, engine.CompileOptions{})
+
+	cfg := Config{
+		Shards:      []int{0, 1},
+		BucketsPath: "/buckets",
+		Engine:      e,
+		FilerClient: &failingSubscribeClient{err: status.Error(codes.Unavailable, "filer down")},
+		Client:      stubLifecycleClient{},
+		Persister:   newMemPersister(),
+		Lister:      stubSiblingLister{},
+	}
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.Error(t, err, "a dead subscription must not report a successful pass")
+		assert.Contains(t, err.Error(), "subscription")
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run did not return after the subscription failed")
+	}
+}
+
+// blockingSubscribeStream never delivers; only a cap ends the pass.
+type blockingSubscribeStream struct {
+	grpc.ClientStream
+	ctx context.Context
+}
+
+func (s *blockingSubscribeStream) Recv() (*filer_pb.SubscribeMetadataResponse, error) {
+	<-s.ctx.Done()
+	// A canceled gRPC stream returns a status code, not context.Canceled.
+	return nil, status.Error(codes.Canceled, "context canceled")
+}
+
+type blockingSubscribeClient struct {
+	filer_pb.SeaweedFilerClient
+}
+
+func (c *blockingSubscribeClient) SubscribeMetadata(ctx context.Context, _ *filer_pb.SubscribeMetadataRequest, _ ...grpc.CallOption) (filer_pb.SeaweedFiler_SubscribeMetadataClient, error) {
+	return &blockingSubscribeStream{ctx: ctx}, nil
+}
+
+// Counterweight to TestRun_ServerSideCancelFailsThePass: same
+// codes.Canceled from the reader, opposite verdict. Only intent
+// separates them, which is why the decision reads ctx not the error.
+func TestRun_CappedPassIsNotAFailure(t *testing.T) {
+	e := engine.New()
+	e.Compile([]engine.CompileInput{
+		{Bucket: "b1", Rules: []*s3lifecycle.Rule{
+			{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 30},
+		}},
+	}, engine.CompileOptions{})
+
+	cfg := Config{
+		Shards:      []int{0, 1},
+		BucketsPath: "/buckets",
+		Engine:      e,
+		FilerClient: &blockingSubscribeClient{},
+		Client:      stubLifecycleClient{},
+		Persister:   newMemPersister(),
+		Lister:      stubSiblingLister{},
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
+	defer cancel()
+	done := make(chan error, 1)
+	go func() { done <- Run(ctx, cfg) }()
+	select {
+	case err := <-done:
+		require.NoError(t, err, "a pass truncated by its own wall-clock cap is not a failure")
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run did not return after the pass was capped")
+	}
+}
+
+// Neither UntilNs nor keepalive reaches this: a healthy connection whose
+// filer has stopped producing.
+func TestRun_StalledSubscriptionTimesOutThePass(t *testing.T) {
+	e := engine.New()
+	e.Compile([]engine.CompileInput{
+		{Bucket: "b1", Rules: []*s3lifecycle.Rule{
+			{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 30},
+		}},
+	}, engine.CompileOptions{})
+
+	cfg := Config{
+		Shards:                     []int{0, 1},
+		BucketsPath:                "/buckets",
+		Engine:                     e,
+		FilerClient:                &blockingSubscribeClient{},
+		Client:                     stubLifecycleClient{},
+		Persister:                  newMemPersister(),
+		Lister:                     stubSiblingLister{},
+		SubscriptionReceiveTimeout: 50 * time.Millisecond,
+	}
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.ErrorIs(t, err, reader.ErrReceiveTimeout)
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run did not return on a stalled subscription")
+	}
+}
+
+func TestValidate_RejectsNegativeSubscriptionReceiveTimeout(t *testing.T) {
+	cfg := validatableConfig()
+	cfg.SubscriptionReceiveTimeout = -time.Second
+	require.ErrorContains(t, validate(cfg), "SubscriptionReceiveTimeout")
+	cfg.SubscriptionReceiveTimeout = 0
+	require.NoError(t, validate(cfg))
+}
+
+// slowSavePersister stretches teardown past the caller's deadline.
+type slowSavePersister struct {
+	*memPersister
+	delay time.Duration
+}
+
+func (p *slowSavePersister) Save(ctx context.Context, shardID int, c Cursor) error {
+	time.Sleep(p.delay)
+	return p.memPersister.Save(ctx, shardID, c)
+}
+
+// A reader that fails while the caller's deadline is still live, on a
+// pass whose teardown then outlives that deadline. Sampling ctx.Err()
+// after the drains and cursor saves would read the late deadline as
+// intent and report the truncated pass as a success.
+func TestRun_LateDeadlineDoesNotMaskReaderFailure(t *testing.T) {
+	e := engine.New()
+	e.Compile([]engine.CompileInput{
+		{Bucket: "b1", Rules: []*s3lifecycle.Rule{
+			{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 30},
+		}},
+	}, engine.CompileOptions{})
+
+	cfg := Config{
+		Shards:      []int{0, 1},
+		BucketsPath: "/buckets",
+		Engine:      e,
+		FilerClient: &failingSubscribeClient{err: status.Error(codes.Unavailable, "filer down")},
+		Client:      stubLifecycleClient{},
+		Persister:   &slowSavePersister{memPersister: newMemPersister(), delay: 300 * time.Millisecond},
+		Lister:      stubSiblingLister{},
+	}
+
+	// Deadline lands during the cursor saves, well after the reader failed.
+	ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
+	defer cancel()
+	done := make(chan error, 1)
+	go func() { done <- Run(ctx, cfg) }()
+	select {
+	case err := <-done:
+		require.Error(t, err, "a deadline expiring during teardown must not excuse an earlier reader failure")
+		require.NotNil(t, ctx.Err(), "test is meaningless unless the deadline did expire")
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run did not return")
+	}
+}
+
+// The case status-code classification could not express: the filer
+// cancels while the caller's context is untouched.
+func TestRun_ServerSideCancelFailsThePass(t *testing.T) {
+	e := engine.New()
+	e.Compile([]engine.CompileInput{
+		{Bucket: "b1", Rules: []*s3lifecycle.Rule{
+			{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 30},
+		}},
+	}, engine.CompileOptions{})
+
+	cfg := Config{
+		Shards:      []int{0, 1},
+		BucketsPath: "/buckets",
+		Engine:      e,
+		FilerClient: &failingSubscribeClient{err: status.Error(codes.Canceled, "context canceled")},
+		Client:      stubLifecycleClient{},
+		Persister:   newMemPersister(),
+		Lister:      stubSiblingLister{},
+	}
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.Error(t, err, "a peer-canceled stream truncates the pass and must not report success")
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run did not return after the subscription was canceled")
+	}
+}

weed/s3api/s3lifecycle/dailyrun/run_fanout_test.go

@@ -0,0 +1,148 @@
+package dailyrun
+
+import (
+	"context"
+	"io"
+	"strconv"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"google.golang.org/grpc"
+)
+
+// scriptedSubscribeStream replays a fixed list, then EOFs like a filer
+// honoring UntilNs.
+type scriptedSubscribeStream struct {
+	grpc.ClientStream
+	responses []*filer_pb.SubscribeMetadataResponse
+	next      int
+}
+
+func (s *scriptedSubscribeStream) Recv() (*filer_pb.SubscribeMetadataResponse, error) {
+	if s.next >= len(s.responses) {
+		return nil, io.EOF
+	}
+	resp := s.responses[s.next]
+	s.next++
+	return resp, nil
+}
+
+type scriptedSubscribeClient struct {
+	filer_pb.SeaweedFilerClient
+	responses []*filer_pb.SubscribeMetadataResponse
+}
+
+func (c *scriptedSubscribeClient) SubscribeMetadata(_ context.Context, _ *filer_pb.SubscribeMetadataRequest, _ ...grpc.CallOption) (filer_pb.SeaweedFiler_SubscribeMetadataClient, error) {
+	return &scriptedSubscribeStream{responses: c.responses}, nil
+}
+
+func subscribeResponse(bucket, key string, mtime time.Time, tsNs int64) *filer_pb.SubscribeMetadataResponse {
+	return &filer_pb.SubscribeMetadataResponse{
+		TsNs: tsNs,
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/buckets/" + bucket,
+			NewEntry: &filer_pb.Entry{
+				Name:       key,
+				Attributes: &filer_pb.FuseAttributes{Mtime: mtime.Unix(), FileSize: 1},
+			},
+		},
+	}
+}
+
+// keyForShard finds an object key in bucket that hashes to shardID.
+func keyForShard(t *testing.T, bucket string, shardID int) string {
+	t.Helper()
+	for i := 0; i < 10000; i++ {
+		key := "k" + strconv.Itoa(i)
+		if s3lifecycle.ShardID(bucket, key) == shardID {
+			return key
+		}
+	}
+	t.Fatalf("no key hashes to shard %d", shardID)
+	return ""
+}
+
+// The second wedge: a drain halted by a BLOCKED dispatch stops reading
+// its channel, the fan-out blocks once the 256 buffer fills, and every
+// other shard starves behind it.
+func TestRun_HaltedShardDoesNotStarveOthers(t *testing.T) {
+	const bucket = "bk"
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 1}
+	prior := map[s3lifecycle.ActionKey]engine.PriorState{}
+	ruleHash := s3lifecycle.RuleHash(rule)
+	for _, k := range s3lifecycle.RuleActionKinds(rule) {
+		prior[s3lifecycle.ActionKey{Bucket: bucket, RuleHash: ruleHash, ActionKind: k}] = engine.PriorState{
+			BootstrapComplete: true,
+			Mode:              engine.ModeEventDriven,
+		}
+	}
+	e := engine.New()
+	snap := e.Compile([]engine.CompileInput{{Bucket: bucket, Rules: []*s3lifecycle.Rule{rule}}},
+		engine.CompileOptions{PriorStates: prior})
+
+	haltedShard, starvedShard := 0, 1
+	haltedKey := keyForShard(t, bucket, haltedShard)
+	starvedKey := keyForShard(t, bucket, starvedShard)
+
+	runNow := time.Now().UTC()
+	evTime := runNow.Add(-48 * time.Hour) // past the 1-day expiration
+	// Past the 256 buffer, so the fan-out blocks on the halted shard
+	// before reaching the starved shard's event.
+	var responses []*filer_pb.SubscribeMetadataResponse
+	for i := 0; i < 300; i++ {
+		responses = append(responses, subscribeResponse(bucket, haltedKey, evTime, evTime.UnixNano()+int64(i)))
+	}
+	responses = append(responses, subscribeResponse(bucket, starvedKey, evTime, runNow.Add(-time.Hour).UnixNano()))
+
+	// Pinned per object, not call index: the shards dispatch concurrently.
+	client := &recordingClient{outcomeByObject: map[string]s3_lifecycle_pb.LifecycleDeleteOutcome{
+		haltedKey: s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED,
+	}}
+
+	// Resume from before the events; the cold-start floor would skip them.
+	// Hashes must match or runShard takes the recovery branch instead.
+	persister := newMemPersister()
+	seeded := Cursor{
+		TsNs:         runNow.Add(-72 * time.Hour).UnixNano(),
+		RuleSetHash:  engine.ReplayContentHash(snap),
+		PromotedHash: engine.PromotedHash(snap, engine.MaxEffectiveTTL(snap)),
+	}
+	for _, sh := range []int{haltedShard, starvedShard} {
+		require.NoError(t, persister.Save(context.Background(), sh, seeded))
+	}
+
+	cfg := Config{
+		Shards:      []int{haltedShard, starvedShard},
+		BucketsPath: "/buckets",
+		Engine:      e,
+		FilerClient: &scriptedSubscribeClient{responses: responses},
+		Client:      client,
+		Persister:   persister,
+		Lister:      stubSiblingLister{},
+		Now:         func() time.Time { return runNow },
+	}
+
+	done := make(chan error, 1)
+	go func() { done <- Run(context.Background(), cfg) }()
+	select {
+	case err := <-done:
+		require.NoError(t, err)
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run wedged: the fan-out blocked on a shard that had stopped draining")
+	}
+
+	// Returning isn't enough: dropping the starved shard's events would
+	// return just as cleanly.
+	assert.Contains(t, client.seenObjects(), starvedKey,
+		"the shard behind the halted one must still get its events")
+	starvedCursor, found, err := persister.Load(context.Background(), starvedShard)
+	require.NoError(t, err)
+	require.True(t, found)
+	assert.Greater(t, starvedCursor.TsNs, seeded.TsNs, "starved shard's cursor must advance")
+}

weed/s3api/s3lifecycle/dailyrun/walk_buckets.go

@@ -0,0 +1,76 @@
+package dailyrun
+
+import (
+	"context"
+	"errors"
+	"fmt"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/bootstrap"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+)
+
+// WalkBuckets runs bootstrap.Walk for each bucket using a per-bucket
+// ListFunc wrapped to drop entries whose ShardID doesn't belong to
+// shardID. Returns the first bucket error; remaining buckets log and
+// continue so one bucket's filer error doesn't kill the whole walk.
+//
+// Composable shape: callers (the worker handler) supply a real
+// filer-backed ListFunc and a real Dispatcher (WalkerDispatcher).
+// Tests pass bootstrap.EntryCallback and a stub Dispatcher.
+func WalkBuckets(ctx context.Context, view *engine.Snapshot, shardID int, buckets []string, list bootstrap.ListFunc, dispatch bootstrap.Dispatcher) error {
+	if view == nil {
+		return errors.New("WalkBuckets: nil view")
+	}
+	if list == nil {
+		return errors.New("WalkBuckets: nil list")
+	}
+	if dispatch == nil {
+		return errors.New("WalkBuckets: nil dispatch")
+	}
+	filtered := perShardListFunc(list, shardID)
+	var firstErr error
+	for _, b := range buckets {
+		if err := ctx.Err(); err != nil {
+			return err
+		}
+		if _, err := bootstrap.Walk(ctx, view, b, filtered, dispatch, bootstrap.WalkOptions{}); err != nil {
+			if firstErr == nil {
+				firstErr = fmt.Errorf("walk %s: %w", b, err)
+			} else {
+				glog.V(1).Infof("walker: additional bucket %s: %v", b, err)
+			}
+		}
+	}
+	return firstErr
+}
+
+// perShardListFunc wraps an inner ListFunc so only entries whose
+// logical-key shard matches shardID reach the callback. The walker
+// emits one entry per logical object — versioned siblings share the
+// same logical key so they're either all in-shard or all out — and
+// MPU init records carry the user's destination in DestKey, so the
+// shard predicate must use DestKey there to match what the dispatcher
+// will send.
+func perShardListFunc(inner bootstrap.ListFunc, shardID int) bootstrap.ListFunc {
+	return func(ctx context.Context, bucket, start string, cb func(*bootstrap.Entry) error) error {
+		return inner(ctx, bucket, start, func(e *bootstrap.Entry) error {
+			if e == nil {
+				return nil
+			}
+			if entryShardID(bucket, e) != shardID {
+				return nil
+			}
+			return cb(e)
+		})
+	}
+}
+
+func entryShardID(bucket string, e *bootstrap.Entry) int {
+	key := e.Path
+	if e.IsMPUInit && e.DestKey != "" {
+		key = e.DestKey
+	}
+	return s3lifecycle.ShardID(bucket, key)
+}

weed/s3api/s3lifecycle/dailyrun/walk_buckets_test.go

@@ -0,0 +1,175 @@
+package dailyrun
+
+import (
+	"context"
+	"errors"
+	"sort"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/bootstrap"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// recordingDispatcher captures every (action, entry) pair the walker
+// emits so tests can assert dispatch order and shard filtering.
+type recordingDispatcher struct {
+	calls []recordedDispatch
+	err   error
+}
+
+type recordedDispatch struct {
+	bucket string
+	path   string
+}
+
+func (d *recordingDispatcher) Delete(_ context.Context, action *engine.CompiledAction, entry *bootstrap.Entry) error {
+	d.calls = append(d.calls, recordedDispatch{bucket: action.Bucket, path: entry.Path})
+	return d.err
+}
+
+// findShardForPath returns the shard ID for an entry with given path
+// in given bucket. Helper for tests that want to force entries onto
+// a known shard.
+func findShardForPath(bucket, path string) int {
+	return s3lifecycle.ShardID(bucket, path)
+}
+
+// fixedShardEntries returns a slice of bootstrap.Entries whose paths
+// all live in the same shard. Used so a per-shard filter test has
+// deterministic expectations.
+func fixedShardEntries(t *testing.T, bucket string, shardID int, count int) []*bootstrap.Entry {
+	t.Helper()
+	var out []*bootstrap.Entry
+	for i := 0; tries(i, count); i++ {
+		path := "obj-" + intToStr(i)
+		if findShardForPath(bucket, path) == shardID {
+			out = append(out, &bootstrap.Entry{
+				Path: path,
+				// Old enough to expire under a 7-day ExpirationDays rule.
+				ModTime:  time.Now().Add(-90 * 24 * time.Hour),
+				Size:     1,
+				IsLatest: true,
+			})
+			if len(out) >= count {
+				return out
+			}
+		}
+	}
+	t.Fatalf("could not find %d entries in shard %d for bucket %s after 4096 attempts", count, shardID, bucket)
+	return nil
+}
+
+func intToStr(i int) string {
+	const hex = "0123456789abcdef"
+	if i == 0 {
+		return "0"
+	}
+	var out []byte
+	for n := i; n > 0; n /= 16 {
+		out = append([]byte{hex[n%16]}, out...)
+	}
+	return string(out)
+}
+
+func tries(i, count int) bool { return i < count*256+256 }
+
+func snapshotForBucketRule(t *testing.T, bucket string, days int) *engine.Snapshot {
+	t.Helper()
+	e := engine.New()
+	e.Compile([]engine.CompileInput{
+		{Bucket: bucket, Rules: []*s3lifecycle.Rule{
+			{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: days},
+		}},
+	}, engine.CompileOptions{})
+	snap := e.Snapshot()
+	for _, a := range snap.AllActions() {
+		snap.MarkActive(a.Key)
+	}
+	return snap
+}
+
+func TestWalkBuckets_DispatchesOnlyShardMatchingEntries(t *testing.T) {
+	bucket := "b1"
+	const targetShard = 0
+	// Build entries: half in shard 0, half elsewhere. The walker's
+	// per-shard filter must drop the non-matching ones.
+	inShard := fixedShardEntries(t, bucket, targetShard, 3)
+	otherShard := fixedShardEntries(t, bucket, (targetShard+1)%16, 3)
+	all := append([]*bootstrap.Entry{}, inShard...)
+	all = append(all, otherShard...)
+
+	snap := snapshotForBucketRule(t, bucket, 7) // 30-day-old objects expire
+	d := &recordingDispatcher{}
+	err := WalkBuckets(context.Background(), snap, targetShard, []string{bucket},
+		bootstrap.EntryCallback(all), d)
+	require.NoError(t, err)
+
+	gotPaths := make([]string, 0, len(d.calls))
+	for _, c := range d.calls {
+		gotPaths = append(gotPaths, c.path)
+		assert.Equal(t, bucket, c.bucket)
+	}
+	sort.Strings(gotPaths)
+
+	wantPaths := make([]string, 0, len(inShard))
+	for _, e := range inShard {
+		wantPaths = append(wantPaths, e.Path)
+	}
+	sort.Strings(wantPaths)
+	assert.Equal(t, wantPaths, gotPaths, "walker must dispatch exactly the in-shard entries")
+}
+
+func TestWalkBuckets_NilGuards(t *testing.T) {
+	require.Error(t, WalkBuckets(context.Background(), nil, 0, nil, bootstrap.EntryCallback(nil), &recordingDispatcher{}))
+	require.Error(t, WalkBuckets(context.Background(), snapshotForBucketRule(t, "b", 7), 0, nil, nil, &recordingDispatcher{}))
+	require.Error(t, WalkBuckets(context.Background(), snapshotForBucketRule(t, "b", 7), 0, nil, bootstrap.EntryCallback(nil), nil))
+}
+
+func TestWalkBuckets_OneBucketErrorDoesNotStopOthers(t *testing.T) {
+	// Two buckets, first ListFunc errors. WalkBuckets should still
+	// process the second bucket but return the first bucket's error.
+	listErr := errors.New("filer flake")
+	list := func(_ context.Context, bucket, _ string, _ func(*bootstrap.Entry) error) error {
+		if bucket == "bad" {
+			return listErr
+		}
+		return nil
+	}
+
+	snap := snapshotForBucketRule(t, "good", 7)
+	err := WalkBuckets(context.Background(), snap, 0, []string{"bad", "good"}, list, &recordingDispatcher{})
+	require.Error(t, err)
+	assert.ErrorIs(t, err, listErr)
+}
+
+func TestWalkBuckets_HonorsContextCancellationBetweenBuckets(t *testing.T) {
+	// Pre-cancel ctx before invocation. Even an empty bucket list
+	// must surface ctx.Err early so a long walk in progress can
+	// short-circuit.
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	err := WalkBuckets(ctx, snapshotForBucketRule(t, "b", 7), 0, []string{"b"},
+		bootstrap.EntryCallback(nil), &recordingDispatcher{})
+	assert.ErrorIs(t, err, context.Canceled)
+}
+
+func TestEntryShardID_MPUUsesDestKey(t *testing.T) {
+	bucket := "bkt"
+	// Force a (Path, DestKey) pair that maps to different shards
+	// so we can prove the filter picks the right one.
+	for i := 0; i < 4096; i++ {
+		path := ".uploads/" + intToStr(i)
+		destKey := "user/" + intToStr(i)
+		if s3lifecycle.ShardID(bucket, path) != s3lifecycle.ShardID(bucket, destKey) {
+			e := &bootstrap.Entry{Path: path, DestKey: destKey, IsMPUInit: true}
+			assert.Equal(t, s3lifecycle.ShardID(bucket, destKey), entryShardID(bucket, e),
+				"MPU init must use DestKey for shard, not the .uploads/<id> path")
+			return
+		}
+	}
+	t.Fatal("could not find an MPU path/destkey pair with diverging shards in 4096 attempts")
+}

weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go

@@ -0,0 +1,103 @@
+package dailyrun
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/bootstrap"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/seaweedfs/seaweedfs/weed/stats"
+	"golang.org/x/time/rate"
+)
+
+// WalkerDispatcher adapts LifecycleClient to bootstrap.Dispatcher so
+// the Phase 4b walker can drive the same LifecycleDelete RPC the
+// meta-log replay path uses. No CAS witness is supplied; the server's
+// identityMatches treats nil ExpectedIdentity as "bootstrap call, skip
+// witness" (see weed/s3api/s3api_internal_lifecycle.go), which is the
+// right contract for a full-tree walk that has just observed the entry.
+type WalkerDispatcher struct {
+	Client LifecycleClient
+	// Limiter throttles LifecycleDelete dispatch under the cluster's
+	// shared deletes/sec budget. Should be the same *rate.Limiter the
+	// daily-run's processMatches uses so the walker and replay paths
+	// can't combine to burst past the cap. nil disables throttling.
+	Limiter *rate.Limiter
+}
+
+// Compile-time check.
+var _ bootstrap.Dispatcher = (*WalkerDispatcher)(nil)
+
+// Delete classifies non-DONE server outcomes as errors so the walker
+// halts and the caller persists progress under the bootstrap
+// checkpoint rather than silently skipping objects.
+func (d *WalkerDispatcher) Delete(ctx context.Context, action *engine.CompiledAction, entry *bootstrap.Entry) error {
+	if d == nil || d.Client == nil {
+		return fmt.Errorf("walker dispatch: nil client")
+	}
+	if action == nil || entry == nil {
+		return fmt.Errorf("walker dispatch: nil action or entry")
+	}
+	objectPath := entry.Path
+	if entry.IsMPUInit {
+		// Rule-prefix matching uses DestKey (the user's intended
+		// object key); dispatch uses entry.Path (.uploads/<id>),
+		// which is what the server's ABORT_MPU handler expects in
+		// req.ObjectPath — it strips the .uploads/ prefix to get
+		// the upload id and reads the init record from that
+		// directory. DestKey is the dispatch ANTI-pattern here: it
+		// looks like a regular object path, the server's check for
+		// the .uploads/ prefix fails, and the dispatch comes back
+		// as BLOCKED FATAL_EVENT_ERROR.
+		if entry.DestKey == "" {
+			return fmt.Errorf("walker dispatch: MPU init entry with empty DestKey: %s", entry.Path)
+		}
+	}
+	rh := action.Key.RuleHash
+	req := &s3_lifecycle_pb.LifecycleDeleteRequest{
+		Bucket:     action.Bucket,
+		ObjectPath: objectPath,
+		VersionId:  entry.VersionID,
+		RuleHash:   rh[:],
+		ActionKind: toProtoActionKind(action.Key.ActionKind),
+		// ExpectedIdentity intentionally nil; server bootstraps from
+		// the live entry on this code path.
+	}
+	kindLabel := action.Key.ActionKind.String()
+	if d.Limiter != nil {
+		waitStart := time.Now()
+		if waitErr := d.Limiter.Wait(ctx); waitErr != nil {
+			return fmt.Errorf("walker dispatch %s/%s %s: limiter: %w", action.Bucket, objectPath, action.Key.ActionKind, waitErr)
+		}
+		stats.S3LifecycleDispatchLimiterWaitSeconds.Observe(time.Since(waitStart).Seconds())
+	}
+	resp, err := d.Client.LifecycleDelete(ctx, req)
+	if err != nil {
+		// "RPC_ERROR" matches the streaming dispatcher and processMatches
+		// labels so transport-class failures aggregate under one key.
+		stats.S3LifecycleDispatchCounter.WithLabelValues(action.Bucket, kindLabel, "RPC_ERROR").Inc()
+		return fmt.Errorf("walker dispatch %s/%s %s: %w", action.Bucket, objectPath, action.Key.ActionKind, err)
+	}
+	if resp == nil {
+		// A misbehaving server stub returning (nil, nil) would panic on
+		// the switch below. Bucket under RPC_ERROR rather than a new
+		// label — operationally it's the same class of failure.
+		stats.S3LifecycleDispatchCounter.WithLabelValues(action.Bucket, kindLabel, "RPC_ERROR").Inc()
+		return fmt.Errorf("walker dispatch %s/%s %s: nil response", action.Bucket, objectPath, action.Key.ActionKind)
+	}
+	stats.S3LifecycleDispatchCounter.WithLabelValues(action.Bucket, kindLabel, resp.Outcome.String()).Inc()
+	switch resp.Outcome {
+	case s3_lifecycle_pb.LifecycleDeleteOutcome_DONE,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_SKIPPED_OBJECT_LOCK:
+		return nil
+	default:
+		// RETRY_LATER / BLOCKED / UNSPECIFIED: surface as error so the
+		// walk halts at this entry and resumes from
+		// Checkpoint.LastScannedPath on the next run.
+		return fmt.Errorf("walker dispatch %s/%s %s: outcome=%s reason=%s",
+			action.Bucket, objectPath, action.Key.ActionKind, resp.Outcome, resp.Reason)
+	}
+}

weed/s3api/s3lifecycle/dailyrun/walker_dispatcher_test.go

@@ -0,0 +1,195 @@
+package dailyrun
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/bootstrap"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"golang.org/x/time/rate"
+)
+
+// walkerStubClient captures the last LifecycleDeleteRequest so tests
+// can assert on the request shape produced by WalkerDispatcher.
+type walkerStubClient struct {
+	lastReq *s3_lifecycle_pb.LifecycleDeleteRequest
+	outcome s3_lifecycle_pb.LifecycleDeleteOutcome
+	err     error
+	reason  string
+	nilResp bool // return (nil, nil) — pin the dispatcher's defensive guard
+}
+
+func (c *walkerStubClient) LifecycleDelete(_ context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
+	c.lastReq = req
+	if c.err != nil {
+		return nil, c.err
+	}
+	if c.nilResp {
+		return nil, nil
+	}
+	return &s3_lifecycle_pb.LifecycleDeleteResponse{Outcome: c.outcome, Reason: c.reason}, nil
+}
+
+func sampleAction(t *testing.T, kind s3lifecycle.ActionKind) *engine.CompiledAction {
+	t.Helper()
+	var rh [8]byte
+	for i := range rh {
+		rh[i] = byte(0xa0 + i)
+	}
+	return &engine.CompiledAction{
+		Bucket: "bkt",
+		Key:    s3lifecycle.ActionKey{Bucket: "bkt", ActionKind: kind, RuleHash: rh},
+		Mode:   engine.ModeEventDriven,
+	}
+}
+
+func TestWalkerDispatcher_NonVersionedSendsExpectedRequest(t *testing.T) {
+	c := &walkerStubClient{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE}
+	d := &WalkerDispatcher{Client: c}
+	a := sampleAction(t, s3lifecycle.ActionKindExpirationDays)
+	err := d.Delete(context.Background(), a, &bootstrap.Entry{Path: "obj"})
+	require.NoError(t, err)
+	require.NotNil(t, c.lastReq)
+	assert.Equal(t, "bkt", c.lastReq.Bucket)
+	assert.Equal(t, "obj", c.lastReq.ObjectPath)
+	assert.Equal(t, "", c.lastReq.VersionId)
+	assert.Equal(t, a.Key.RuleHash[:], c.lastReq.RuleHash)
+	assert.Equal(t, s3_lifecycle_pb.ActionKind_EXPIRATION_DAYS, c.lastReq.ActionKind)
+	// Bootstrap-style call: server skips CAS witness when nil.
+	assert.Nil(t, c.lastReq.ExpectedIdentity)
+}
+
+func TestWalkerDispatcher_VersionedPassesVersionID(t *testing.T) {
+	c := &walkerStubClient{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE}
+	d := &WalkerDispatcher{Client: c}
+	a := sampleAction(t, s3lifecycle.ActionKindNoncurrentDays)
+	err := d.Delete(context.Background(), a, &bootstrap.Entry{Path: "obj", VersionID: "v-abc"})
+	require.NoError(t, err)
+	assert.Equal(t, "v-abc", c.lastReq.VersionId)
+}
+
+func TestWalkerDispatcher_MPUInitUsesUploadsPath(t *testing.T) {
+	// Rule-prefix matching uses DestKey to decide IF this MPU init
+	// matches; dispatch uses Path (.uploads/<id>) because the server's
+	// ABORT_MPU handler strips the .uploads/ prefix to get the upload
+	// id. Sending DestKey here would BLOCK with FATAL_EVENT_ERROR.
+	c := &walkerStubClient{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE}
+	d := &WalkerDispatcher{Client: c}
+	a := sampleAction(t, s3lifecycle.ActionKindAbortMPU)
+	err := d.Delete(context.Background(), a, &bootstrap.Entry{
+		Path:      ".uploads/abc123",
+		DestKey:   "user/path/object",
+		IsMPUInit: true,
+	})
+	require.NoError(t, err)
+	assert.Equal(t, ".uploads/abc123", c.lastReq.ObjectPath)
+}
+
+func TestWalkerDispatcher_MPUInitEmptyDestKeyErrors(t *testing.T) {
+	// An MPU init record with no DestKey is mid-write before metadata
+	// landed; skipping silently in the walker is fine, but the
+	// dispatcher must not invent a path.
+	c := &walkerStubClient{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE}
+	d := &WalkerDispatcher{Client: c}
+	a := sampleAction(t, s3lifecycle.ActionKindAbortMPU)
+	err := d.Delete(context.Background(), a, &bootstrap.Entry{
+		Path:      ".uploads/abc123",
+		IsMPUInit: true,
+	})
+	require.Error(t, err)
+	assert.Nil(t, c.lastReq, "no RPC should be sent when DestKey is empty")
+}
+
+func TestWalkerDispatcher_AcceptsAllResolvedOutcomes(t *testing.T) {
+	for _, oc := range []s3_lifecycle_pb.LifecycleDeleteOutcome{
+		s3_lifecycle_pb.LifecycleDeleteOutcome_DONE,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_SKIPPED_OBJECT_LOCK,
+	} {
+		c := &walkerStubClient{outcome: oc}
+		d := &WalkerDispatcher{Client: c}
+		err := d.Delete(context.Background(), sampleAction(t, s3lifecycle.ActionKindExpirationDays), &bootstrap.Entry{Path: "obj"})
+		assert.NoError(t, err, "outcome %s must be treated as resolved", oc)
+	}
+}
+
+func TestWalkerDispatcher_UnresolvedOutcomeReturnsError(t *testing.T) {
+	// RETRY_LATER, BLOCKED, and UNSPECIFIED all halt the walk so it
+	// resumes from Checkpoint.LastScannedPath on the next run.
+	for _, oc := range []s3_lifecycle_pb.LifecycleDeleteOutcome{
+		s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED,
+	} {
+		c := &walkerStubClient{outcome: oc, reason: "server said so"}
+		d := &WalkerDispatcher{Client: c}
+		err := d.Delete(context.Background(), sampleAction(t, s3lifecycle.ActionKindExpirationDays), &bootstrap.Entry{Path: "obj"})
+		require.Error(t, err, "outcome %s must halt the walk", oc)
+		assert.Contains(t, err.Error(), oc.String())
+	}
+}
+
+func TestWalkerDispatcher_TransportErrorReturnsWrappedError(t *testing.T) {
+	c := &walkerStubClient{err: errors.New("transport boom")}
+	d := &WalkerDispatcher{Client: c}
+	err := d.Delete(context.Background(), sampleAction(t, s3lifecycle.ActionKindExpirationDays), &bootstrap.Entry{Path: "obj"})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "transport boom")
+}
+
+func TestWalkerDispatcher_NilResponseReturnsError(t *testing.T) {
+	// A server returning (nil, nil) would otherwise panic on the
+	// outcome switch.
+	c := &walkerStubClient{nilResp: true}
+	d := &WalkerDispatcher{Client: c}
+	err := d.Delete(context.Background(), sampleAction(t, s3lifecycle.ActionKindExpirationDays), &bootstrap.Entry{Path: "obj"})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "nil response")
+}
+
+func TestWalkerDispatcher_LimiterWaitsBeforeDispatch(t *testing.T) {
+	// Build a tiny limiter (1 token, slow refill) and pre-drain it so
+	// the next Wait blocks until the deadline. Dispatcher must respect
+	// the limiter — without the wait the test passes trivially.
+	c := &walkerStubClient{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE}
+	lim := rate.NewLimiter(rate.Every(50*time.Millisecond), 1)
+	_ = lim.AllowN(time.Now(), 1) // burn the burst token
+	d := &WalkerDispatcher{Client: c, Limiter: lim}
+
+	start := time.Now()
+	err := d.Delete(context.Background(), sampleAction(t, s3lifecycle.ActionKindExpirationDays), &bootstrap.Entry{Path: "obj"})
+	elapsed := time.Since(start)
+	require.NoError(t, err)
+	assert.GreaterOrEqual(t, elapsed, 30*time.Millisecond,
+		"limiter must throttle the dispatch; elapsed=%v", elapsed)
+}
+
+func TestWalkerDispatcher_LimiterContextCancelHaltsWalker(t *testing.T) {
+	// Pre-drained limiter + canceled ctx. Limiter.Wait returns the
+	// cancel error; walker must surface it (not silently dispatch).
+	c := &walkerStubClient{outcome: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE}
+	lim := rate.NewLimiter(rate.Every(time.Hour), 1)
+	_ = lim.AllowN(time.Now(), 1)
+	d := &WalkerDispatcher{Client: c, Limiter: lim}
+
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	err := d.Delete(ctx, sampleAction(t, s3lifecycle.ActionKindExpirationDays), &bootstrap.Entry{Path: "obj"})
+	require.Error(t, err)
+	assert.Nil(t, c.lastReq, "no RPC should be sent when ctx was cancelled in Wait")
+}
+
+func TestWalkerDispatcher_NilGuardsReturnError(t *testing.T) {
+	d := &WalkerDispatcher{Client: &walkerStubClient{}}
+	require.Error(t, d.Delete(context.Background(), nil, &bootstrap.Entry{Path: "obj"}))
+	require.Error(t, d.Delete(context.Background(), sampleAction(t, s3lifecycle.ActionKindExpirationDays), nil))
+
+	nilClient := &WalkerDispatcher{}
+	require.Error(t, nilClient.Delete(context.Background(), sampleAction(t, s3lifecycle.ActionKindExpirationDays), &bootstrap.Entry{Path: "obj"}))
+}

weed/s3api/s3lifecycle/dailyrun/walker_interval_test.go

@@ -0,0 +1,276 @@
+package dailyrun
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// readerEventAlias keeps the test free of the reader import collision
+// in the recovery test file when we need a typed nil channel.
+type readerEventAlias = reader.Event
+
+// TestWalkerDue covers the throttle decision matrix in isolation.
+func TestWalkerDue(t *testing.T) {
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+	cases := []struct {
+		name         string
+		lastWalkedNs int64
+		interval     time.Duration
+		want         bool
+	}{
+		// interval=0 keeps the prior "fire every pass" semantics — the
+		// in-repo integration tests and the s3tests fast driver rely on
+		// this so the rule-change-then-walk behavior surfaces within a
+		// single test runtime. Within-pass double-fire suppression
+		// lives in runShard's walkedThisPass local; walkerDue answers
+		// the persisted-state question only.
+		{"interval zero always due", runNow.UnixNano(), 0, true},
+		// LastWalkedNs=0 marks "never walked steady-state" — the post-
+		// upgrade cold-start case for cursor files that predate the
+		// LastWalkedNs field. Fire so the throttle anchor gets seeded.
+		{"never walked is due", 0, time.Hour, true},
+		// Throttle on: interval elapsed → due.
+		{"interval elapsed", runNow.Add(-2 * time.Hour).UnixNano(), time.Hour, true},
+		{"interval exactly elapsed", runNow.Add(-time.Hour).UnixNano(), time.Hour, true},
+		// Throttle on: not yet → skip.
+		{"interval not yet elapsed", runNow.Add(-30 * time.Minute).UnixNano(), time.Hour, false},
+		// Future LastWalkedNs (clock skew or replay of an older runNow
+		// against a newer cursor) must not panic and must not fire —
+		// treat as "throttled" because the cursor claims a future walk.
+		{"future lastWalked", runNow.Add(time.Hour).UnixNano(), time.Hour, false},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			assert.Equal(t, tc.want, walkerDue(tc.lastWalkedNs, runNow, tc.interval))
+		})
+	}
+}
+
+// TestRunShard_WalkerThrottle confirms two back-to-back runShard
+// invocations only fire the steady-state walker once when the second
+// pass starts inside the WalkerInterval window. The interval-zero
+// control case keeps firing on every pass — same shape, different
+// expectation — pinning the prior behavior so a default-zero config
+// (tests, in-repo integration) doesn't regress.
+func TestRunShard_WalkerThrottle(t *testing.T) {
+	cases := []struct {
+		name            string
+		interval        time.Duration
+		secondPassAfter time.Duration
+		wantTotalCalls  int
+		wantSecondAdvNs bool // did LastWalkedNs change between pass 1 and pass 2?
+	}{
+		{"interval=0 fires every pass", 0, 30 * time.Second, 2, true},
+		{"throttled: second pass within interval", time.Hour, 30 * time.Second, 1, false},
+		{"throttled: second pass past interval", time.Hour, 2 * time.Hour, 2, true},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			snap := snapshotWithRule(t, 30) // replay-eligible only; steady-state walker would normally skip
+			// Force a walker-only partition so the steady-state branch
+			// has something to walk: set retentionWindow=0 in cfg, which
+			// makes promoted non-empty and walkView non-empty.
+			p := newMemPersister()
+			calls := 0
+			cfg := Config{
+				Persister: p,
+				Walker: func(_ context.Context, _ *engine.Snapshot, _ int) error {
+					calls++
+					return nil
+				},
+				RetentionWindow: -1, // negative sentinel: falls back to maxTTL — leaves walkView empty
+				WalkerInterval:  tc.interval,
+			}
+			// Pre-seed cursor matching snap's hashes so cold-start /
+			// recovery branches don't fire and we measure ONLY the
+			// steady-state walker.
+			rsh := engine.ReplayContentHash(snap)
+			promoted := engine.PromotedHash(snap, engine.MaxEffectiveTTL(snap))
+			runNow := time.Unix(1_700_000_000, 0).UTC()
+			require.NoError(t, p.Save(context.Background(), 0, Cursor{
+				TsNs:         runNow.UnixNano(),
+				RuleSetHash:  rsh,
+				PromotedHash: promoted,
+			}))
+
+			// With the snapshot's only rule being replay-eligible, the
+			// steady-state walkView is empty and the walker won't fire
+			// regardless of throttle. Use a snapshot that has BOTH a
+			// replay rule AND a scan-only walker rule by forcing
+			// retentionWindow to a very small value via cfg.
+			cfg.RetentionWindow = time.Nanosecond
+			// Recompute promoted with the test retention so persisted
+			// hashes match what runShard sees on each pass.
+			snapPromoted := engine.PromotedHash(snap, time.Nanosecond)
+			require.NoError(t, p.Save(context.Background(), 0, Cursor{
+				TsNs:         runNow.UnixNano(),
+				RuleSetHash:  rsh,
+				PromotedHash: snapPromoted,
+			}))
+
+			// runShard reaches drainShardEvents after the steady-state
+			// walker fires (replay-eligible rules present). A nil events
+			// channel would block forever; a closed one returns
+			// immediately so drain exits cleanly and the cursor save
+			// still records LastWalkedNs.
+			closedEvents := make(chan *readerEventAlias)
+			close(closedEvents)
+
+			// Pass 1.
+			require.NoError(t, runShard(context.Background(), cfg, snap, runNow, 0, closedEvents))
+			afterPass1, _, _ := p.Load(context.Background(), 0)
+			require.Equal(t, 1, calls, "pass 1 must fire the walker (cold-start anchor)")
+			require.Equal(t, runNow.UnixNano(), afterPass1.LastWalkedNs)
+
+			// Pass 2.
+			runNow2 := runNow.Add(tc.secondPassAfter)
+			closedEvents2 := make(chan *readerEventAlias)
+			close(closedEvents2)
+			require.NoError(t, runShard(context.Background(), cfg, snap, runNow2, 0, closedEvents2))
+			afterPass2, _, _ := p.Load(context.Background(), 0)
+			assert.Equal(t, tc.wantTotalCalls, calls, "walker calls after 2 passes")
+			if tc.wantSecondAdvNs {
+				assert.Equal(t, runNow2.UnixNano(), afterPass2.LastWalkedNs, "throttle allowed the second walk")
+			} else {
+				assert.Equal(t, runNow.UnixNano(), afterPass2.LastWalkedNs, "throttle suppressed the second walk; anchor unchanged")
+			}
+		})
+	}
+}
+
+// TestValidate_RejectsNegativeWalkerInterval pins the loud-failure
+// contract for an embedder or test that constructs dailyrun.Config
+// directly with a negative WalkerInterval. The admin-config parser
+// clamps negative input to zero (worker/tasks/s3_lifecycle/config.go),
+// but a caller bypassing that parser would otherwise get the silent
+// fall-through-to-walk-every-pass behavior the throttle is trying to
+// prevent.
+func TestValidate_RejectsNegativeWalkerInterval(t *testing.T) {
+	// validate only checks for nil fields, not behavior — type-assert
+	// each interface to its zero value via a stub. The stubs never run.
+	cfg := validatableConfig()
+	cfg.WalkerInterval = -time.Hour
+	err := validate(cfg)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "WalkerInterval")
+	// And the zero sentinel still passes.
+	cfg.WalkerInterval = 0
+	require.NoError(t, validate(cfg))
+}
+
+// validatableConfig builds a minimal dailyrun.Config that passes the
+// nil checks in validate() so individual fields can be poked. The
+// stubs intentionally don't implement any meaningful behavior — they
+// only need to be non-nil interface values.
+func validatableConfig() Config {
+	return Config{
+		Engine:      engine.New(),
+		FilerClient: stubFilerClient{},
+		Client:      stubLifecycleClient{},
+		Persister:   newMemPersister(),
+		Lister:      stubSiblingLister{},
+		BucketsPath: "/buckets",
+	}
+}
+
+type stubFilerClient struct{ filer_pb.SeaweedFilerClient }
+type stubLifecycleClient struct{}
+
+func (stubLifecycleClient) LifecycleDelete(_ context.Context, _ *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
+	return nil, nil
+}
+
+type stubSiblingLister struct{}
+
+func (stubSiblingLister) Survivors(_ context.Context, _, _ string) (router.Survivors, error) {
+	return router.Survivors{}, nil
+}
+func (stubSiblingLister) LookupVersion(_ context.Context, _, _, _ string) (*filer_pb.Entry, error) {
+	return nil, nil
+}
+func (stubSiblingLister) ListVersions(_ context.Context, _, _ string) ([]*filer_pb.Entry, error) {
+	return nil, nil
+}
+func (stubSiblingLister) LookupNullVersion(_ context.Context, _, _ string) (*filer_pb.Entry, bool, error) {
+	return nil, false, nil
+}
+
+// TestRunShard_ColdStartDoesNotDoubleWalk pins the within-pass guard
+// in runShard's walkedThisPass local. Before the guard, a cold-start
+// pass with WalkerInterval=0 fired the walker twice in one pass:
+// once via the mustWalkColdStart branch (with RecoveryView) and again
+// immediately via the steady-state branch (with the per-shard walk
+// view, which is a subset of RecoveryView). The second walk added no
+// coverage and burned a full bucket scan. Guard against regression.
+func TestRunShard_ColdStartDoesNotDoubleWalk(t *testing.T) {
+	snap := snapshotWithRule(t, 30)
+	p := newMemPersister()
+	// No persisted cursor → mustWalkColdStart=true. WalkerInterval=0
+	// would, before the fix, allow the steady-state branch to fire
+	// again with no time elapsed.
+	calls := 0
+	cfg := Config{
+		Persister: p,
+		Walker: func(_ context.Context, _ *engine.Snapshot, _ int) error {
+			calls++
+			return nil
+		},
+		WalkerInterval: 0,
+	}
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+
+	closedEvents := make(chan *readerEventAlias)
+	close(closedEvents)
+	require.NoError(t, runShard(context.Background(), cfg, snap, runNow, 0, closedEvents))
+
+	assert.Equal(t, 1, calls, "cold-start walker must fire exactly once per pass even with interval=0")
+}
+
+// TestRunShard_RecoveryWalkerSetsLastWalkedAnchor pins that the
+// unconditional recovery-branch walker fire still updates the
+// LastWalkedNs anchor so the NEXT pass's throttle starts counting
+// from this walk rather than re-firing immediately.
+func TestRunShard_RecoveryWalkerSetsLastWalkedAnchor(t *testing.T) {
+	snap := snapshotWithRule(t, 30)
+	p := newMemPersister()
+	var stale [32]byte
+	for i := range stale {
+		stale[i] = 0xAA
+	}
+	require.NoError(t, p.Save(context.Background(), 5, Cursor{
+		TsNs:         1234,
+		RuleSetHash:  stale,
+		LastWalkedNs: 0, // never walked anchor
+	}))
+
+	calls := 0
+	cfg := Config{
+		Persister: p,
+		Walker: func(_ context.Context, _ *engine.Snapshot, _ int) error {
+			calls++
+			return nil
+		},
+		WalkerInterval: time.Hour,
+	}
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+	require.NoError(t, runShard(context.Background(), cfg, snap, runNow, 5, nil))
+
+	require.Equal(t, 1, calls, "recovery branch fires the walker unconditionally")
+	got, ok, _ := p.Load(context.Background(), 5)
+	require.True(t, ok)
+	assert.Equal(t, runNow.UnixNano(), got.LastWalkedNs,
+		"recovery walk must update LastWalkedNs so the steady-state pass after it can throttle")
+}
+
+// Compile-time check that the documented sentinel passes the type system.
+var _ = s3lifecycle.ShardCount

weed/s3api/s3lifecycle/dailyrun/walker_recovery_test.go

@@ -0,0 +1,150 @@
+package dailyrun
+
+import (
+	"context"
+	"errors"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// memPersister is a minimal in-memory CursorPersister. Phase 4b tests
+// drive runShard directly; the recovery-branch path returns before
+// drainShardEvents so the heavier filer/client/lister fakes aren't
+// needed here. Locked because Run fans out a goroutine per shard and
+// they all load and save through the same persister.
+type memPersister struct {
+	mu    sync.Mutex
+	store map[int]Cursor
+}
+
+func newMemPersister() *memPersister { return &memPersister{store: map[int]Cursor{}} }
+
+func (p *memPersister) Load(_ context.Context, shardID int) (Cursor, bool, error) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	c, ok := p.store[shardID]
+	return c, ok, nil
+}
+
+func (p *memPersister) Save(_ context.Context, shardID int, c Cursor) error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	p.store[shardID] = c
+	return nil
+}
+
+func snapshotWithRule(t *testing.T, days int) *engine.Snapshot {
+	t.Helper()
+	e := engine.New()
+	e.Compile([]engine.CompileInput{
+		{Bucket: "b1", Rules: []*s3lifecycle.Rule{
+			{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: days},
+		}},
+	}, engine.CompileOptions{})
+	snap := e.Snapshot()
+	for _, a := range snap.AllActions() {
+		snap.MarkActive(a.Key)
+	}
+	return snap
+}
+
+func TestRunShard_WalkerInvokedOnRecoveryBranch(t *testing.T) {
+	snap := snapshotWithRule(t, 30)
+	p := newMemPersister()
+	// Seed a persisted cursor whose RuleSetHash differs from snap's, so
+	// the rule-change branch fires.
+	var stale [32]byte
+	for i := range stale {
+		stale[i] = 0xAA
+	}
+	require.NoError(t, p.Save(context.Background(), 3, Cursor{TsNs: 1234, RuleSetHash: stale}))
+
+	var gotView *engine.Snapshot
+	var gotShard int
+	calls := 0
+	cfg := Config{
+		Persister: p,
+		Walker: func(_ context.Context, view *engine.Snapshot, shardID int) error {
+			calls++
+			gotView = view
+			gotShard = shardID
+			return nil
+		},
+	}
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+	require.NoError(t, runShard(context.Background(), cfg, snap, runNow, 3, nil))
+
+	assert.Equal(t, 1, calls, "walker must fire exactly once on recovery")
+	require.NotNil(t, gotView, "walker received the RecoveryView")
+	assert.Equal(t, 3, gotShard)
+
+	// Cursor rewound to runNow - maxTTL with the new hashes persisted.
+	got, ok, err := p.Load(context.Background(), 3)
+	require.NoError(t, err)
+	require.True(t, ok)
+	expectedFloor := runNow.Add(-engine.MaxEffectiveTTL(snap)).UnixNano()
+	assert.Equal(t, expectedFloor, got.TsNs, "cursor must rewind to runNow - maxTTL after recovery walk")
+	assert.NotEqual(t, stale, got.RuleSetHash, "stale hash must be replaced")
+}
+
+func TestRunShard_NilWalkerOnRecoveryIsNoop(t *testing.T) {
+	// Phase 4a behavior: a nil Walker must not crash and must still
+	// rewind the cursor.
+	snap := snapshotWithRule(t, 30)
+	p := newMemPersister()
+	var stale [32]byte
+	for i := range stale {
+		stale[i] = 0xBB
+	}
+	require.NoError(t, p.Save(context.Background(), 0, Cursor{TsNs: 9999, RuleSetHash: stale}))
+
+	cfg := Config{Persister: p} // Walker nil
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+	require.NoError(t, runShard(context.Background(), cfg, snap, runNow, 0, nil))
+
+	got, ok, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	require.True(t, ok)
+	assert.Equal(t, runNow.Add(-engine.MaxEffectiveTTL(snap)).UnixNano(), got.TsNs)
+}
+
+func TestRunShard_WalkerErrorPropagates(t *testing.T) {
+	// A walker failure must surface as the runShard error so the daily
+	// run treats the shard as interrupted, and must NOT advance the
+	// cursor (the seeded cursor stays).
+	snap := snapshotWithRule(t, 30)
+	p := newMemPersister()
+	var stale [32]byte
+	for i := range stale {
+		stale[i] = 0xCC
+	}
+	require.NoError(t, p.Save(context.Background(), 7, Cursor{TsNs: 42, RuleSetHash: stale}))
+
+	cfg := Config{
+		Persister: p,
+		Walker: func(_ context.Context, _ *engine.Snapshot, _ int) error {
+			return errors.New("walker boom")
+		},
+	}
+	runNow := time.Unix(1_700_000_000, 0).UTC()
+	err := runShard(context.Background(), cfg, snap, runNow, 7, nil)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "walker boom")
+
+	// Cursor untouched.
+	got, _, _ := p.Load(context.Background(), 7)
+	assert.Equal(t, int64(42), got.TsNs, "walker failure must leave cursor unchanged")
+	assert.Equal(t, stale, got.RuleSetHash)
+}
+
+// The "matching cursor doesn't invoke walker" case is implicit: the
+// walker call is inside the rule-change / partition-flip `if` and
+// can't be reached otherwise. Exercising it end-to-end requires the
+// full filer + lister + client harness; covered by the integration
+// tests once Phase 4b is wired into the handler.

weed/s3api/s3lifecycle/dispatcher/filer_persister.go

@@ -0,0 +1,33 @@
+package dispatcher
+
+import (
+	"context"
+
+	"github.com/seaweedfs/seaweedfs/weed/filer"
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+)
+
+// FilerStore is the small subset of filer-client operations the
+// daily-replay cursor persister needs (dailyrun.FilerCursorPersister).
+// Tests inject an in-memory fake.
+type FilerStore interface {
+	Read(ctx context.Context, dir, name string) ([]byte, error)
+	Save(ctx context.Context, dir, name string, content []byte) error
+}
+
+// NewFilerStoreClient adapts a SeaweedFilerClient into a FilerStore.
+func NewFilerStoreClient(client filer_pb.SeaweedFilerClient) FilerStore {
+	return &filerStoreClient{client: client}
+}
+
+type filerStoreClient struct {
+	client filer_pb.SeaweedFilerClient
+}
+
+func (s *filerStoreClient) Read(ctx context.Context, dir, name string) ([]byte, error) {
+	return filer.ReadInsideFiler(ctx, s.client, dir, name)
+}
+
+func (s *filerStoreClient) Save(ctx context.Context, dir, name string, content []byte) error {
+	return filer.SaveInsideFiler(ctx, s.client, dir, name, content)
+}

weed/s3api/s3lifecycle/dispatcher/sibling_lister.go

@@ -0,0 +1,171 @@
+package dispatcher
+
+import (
+	"context"
+	"errors"
+	"strings"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
+	"github.com/seaweedfs/seaweedfs/weed/util"
+)
+
+// filerSiblingLister inspects the .versions/<key>/ folder and the bare
+// null-version. Listing caps at 2 — callers only distinguish 1 vs >=2.
+type filerSiblingLister struct {
+	client      filer_pb.SeaweedFilerClient
+	bucketsPath string
+}
+
+// NewFilerSiblingLister returns a router.SiblingLister backed by the
+// filer client. Exported so the daily-replay worker (and any future
+// non-dispatcher caller) can reuse the same .versions/ + null-bare
+// lookup logic without re-implementing it. Phase 5 deletes the
+// dispatcher's internal use; this constructor stays.
+func NewFilerSiblingLister(client filer_pb.SeaweedFilerClient, bucketsPath string) router.SiblingLister {
+	return &filerSiblingLister{client: client, bucketsPath: bucketsPath}
+}
+
+func (l *filerSiblingLister) Survivors(ctx context.Context, bucket, objectKey string) (router.Survivors, error) {
+	bucketPath := strings.TrimSuffix(l.bucketsPath, "/") + "/" + bucket
+	versionsDir := bucketPath + "/" + objectKey + s3_constants.VersionsFolder
+
+	var s router.Survivors
+	err := filer_pb.SeaweedList(ctx, l.client, versionsDir, "", func(entry *filer_pb.Entry, _ bool) error {
+		s.Count++
+		if s.Count == 1 && entry != nil {
+			s.LoneEntry = entry
+		} else if s.Count > 1 {
+			s.LoneEntry = nil
+		}
+		return nil
+	}, "", false, 2)
+	if err != nil && !errors.Is(err, filer_pb.ErrNotFound) {
+		return router.Survivors{}, err
+	}
+
+	// NewFullPath strips a trailing slash from objectKey so directory-key
+	// objects (foo/) split the same as regular keys. LookupEntry
+	// normalizes the gRPC string-mapped not-found into ErrNotFound; a
+	// raw client.LookupDirectoryEntry would return that as a generic
+	// error and suppress every otherwise-valid match.
+	parent, name := util.NewFullPath(bucketPath, objectKey).DirAndName()
+	resp, err := filer_pb.LookupEntry(ctx, l.client, &filer_pb.LookupDirectoryEntryRequest{
+		Directory: parent,
+		Name:      name,
+	})
+	if err != nil {
+		if errors.Is(err, filer_pb.ErrNotFound) {
+			return s, nil
+		}
+		return router.Survivors{}, err
+	}
+	// Bare regular file or an explicit S3 directory-marker (an empty
+	// directory entry with Mime set) both count as the null version.
+	if resp.Entry != nil && (!resp.Entry.IsDirectory || resp.Entry.IsDirectoryKeyObject()) {
+		s.HasNullVersion = true
+	}
+	return s, nil
+}
+
+// ListVersions paginates every file under .versions/<key>/. Used by
+// the pointer-transition expansion path when a NewerNoncurrentVersions
+// rule needs accurate per-version ranking. Subdirectories and entries
+// without ExtVersionIdKey are filtered out. NotFound returns
+// (nil, nil) so a hard-deleted .versions/ container collapses cleanly.
+func (l *filerSiblingLister) ListVersions(ctx context.Context, bucket, objectKey string) ([]*filer_pb.Entry, error) {
+	bucketPath := strings.TrimSuffix(l.bucketsPath, "/") + "/" + bucket
+	dir := bucketPath + "/" + objectKey + s3_constants.VersionsFolder
+	const pageSize uint32 = 1024
+	startFrom := ""
+	var versions []*filer_pb.Entry
+	for {
+		var pageCount uint32
+		var lastName string
+		err := filer_pb.SeaweedList(ctx, l.client, dir, "", func(e *filer_pb.Entry, _ bool) error {
+			pageCount++
+			if e == nil {
+				return nil
+			}
+			lastName = e.Name
+			if e.Attributes == nil || e.IsDirectory {
+				return nil
+			}
+			if id, ok := e.Extended[s3_constants.ExtVersionIdKey]; !ok || len(id) == 0 {
+				return nil
+			}
+			versions = append(versions, e)
+			return nil
+		}, startFrom, false, pageSize)
+		if err != nil {
+			if errors.Is(err, filer_pb.ErrNotFound) {
+				return nil, nil
+			}
+			return nil, err
+		}
+		if pageCount < pageSize {
+			return versions, nil
+		}
+		startFrom = lastName
+	}
+}
+
+// LookupNullVersion returns the bare-key entry that represents the
+// null version of objectKey, if any. Both regular files and explicit
+// S3 directory-key markers (an empty directory entry with Mime set)
+// qualify. explicit reports whether the entry's Extended map carries
+// ExtVersionIdKey="null" — the marker the suspended-versioning write
+// path applies. NotFound returns (nil, false, nil).
+func (l *filerSiblingLister) LookupNullVersion(ctx context.Context, bucket, objectKey string) (*filer_pb.Entry, bool, error) {
+	bucketPath := strings.TrimSuffix(l.bucketsPath, "/") + "/" + bucket
+	parent, name := util.NewFullPath(bucketPath, objectKey).DirAndName()
+	resp, err := filer_pb.LookupEntry(ctx, l.client, &filer_pb.LookupDirectoryEntryRequest{
+		Directory: parent,
+		Name:      name,
+	})
+	if err != nil {
+		if errors.Is(err, filer_pb.ErrNotFound) {
+			return nil, false, nil
+		}
+		return nil, false, err
+	}
+	if resp == nil || resp.Entry == nil {
+		return nil, false, nil
+	}
+	e := resp.Entry
+	if e.IsDirectory && !e.IsDirectoryKeyObject() {
+		return nil, false, nil
+	}
+	explicit := false
+	if id, ok := e.Extended[s3_constants.ExtVersionIdKey]; ok && string(id) == "null" {
+		explicit = true
+	}
+	return e, explicit, nil
+}
+
+// LookupVersion fetches <bucketsPath>/<bucket>/<objectKey>.versions/v_<id>
+// for the pointer-transition router branch. NotFound returns (nil, nil)
+// — the displaced version may have been hard-deleted between the
+// pointer update and the lookup.
+func (l *filerSiblingLister) LookupVersion(ctx context.Context, bucket, objectKey, versionID string) (*filer_pb.Entry, error) {
+	if versionID == "" {
+		return nil, nil
+	}
+	bucketPath := strings.TrimSuffix(l.bucketsPath, "/") + "/" + bucket
+	dir := bucketPath + "/" + objectKey + s3_constants.VersionsFolder
+	resp, err := filer_pb.LookupEntry(ctx, l.client, &filer_pb.LookupDirectoryEntryRequest{
+		Directory: dir,
+		Name:      "v_" + versionID,
+	})
+	if err != nil {
+		if errors.Is(err, filer_pb.ErrNotFound) {
+			return nil, nil
+		}
+		return nil, err
+	}
+	if resp == nil {
+		return nil, nil
+	}
+	return resp.Entry, nil
+}

weed/s3api/s3lifecycle/dispatcher/sibling_lister_test.go

@@ -0,0 +1,502 @@
+package dispatcher
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"sort"
+	"sync"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/metadata"
+)
+
+// Tests for filerSiblingLister cover the four routing-critical surfaces
+// the dispatcher pipeline depends on: Survivors (count cap + null
+// detection), ListVersions (pagination + filter), LookupNullVersion
+// (regular vs directory-key marker, explicit-null flag), and
+// LookupVersion (v_ prefix + NotFound collapse).
+
+type fsListStream struct {
+	responses []*filer_pb.ListEntriesResponse
+	index     int
+	// recvErr is returned from the FIRST Recv() call before any
+	// responses are delivered; lets tests inject the gRPC NotFound the
+	// real filer surfaces when the parent directory doesn't exist.
+	recvErr error
+	ctx     context.Context
+}
+
+func (s *fsListStream) Recv() (*filer_pb.ListEntriesResponse, error) {
+	if s.ctx != nil {
+		if err := s.ctx.Err(); err != nil {
+			return nil, err
+		}
+	}
+	if s.recvErr != nil {
+		err := s.recvErr
+		s.recvErr = nil
+		return nil, err
+	}
+	if s.index >= len(s.responses) {
+		return nil, io.EOF
+	}
+	resp := s.responses[s.index]
+	s.index++
+	return resp, nil
+}
+
+func (s *fsListStream) Header() (metadata.MD, error) { return metadata.MD{}, nil }
+func (s *fsListStream) Trailer() metadata.MD         { return metadata.MD{} }
+func (s *fsListStream) CloseSend() error             { return nil }
+func (s *fsListStream) Context() context.Context {
+	if s.ctx != nil {
+		return s.ctx
+	}
+	return context.Background()
+}
+func (s *fsListStream) SendMsg(any) error { return nil }
+func (s *fsListStream) RecvMsg(any) error { return nil }
+
+// fsFakeFiler implements just the SeaweedFilerClient methods that
+// filerSiblingLister calls (ListEntries + LookupDirectoryEntry). Other
+// methods on the embedded interface remain nil; calling them panics,
+// which the tests rely on to catch regressions that try to reach beyond
+// the documented surface.
+type fsFakeFiler struct {
+	filer_pb.SeaweedFilerClient
+
+	mu sync.Mutex
+	// tree maps a directory path to its immediate children. A directory
+	// is "missing" when its key is absent from tree (we return ErrNotFound
+	// from ListEntries / LookupDirectoryEntry to mimic the real filer).
+	tree map[string][]*filer_pb.Entry
+	// listErr / lookupErr force the next call to fail with this error.
+	// Reset to nil after use so tests can mix success + failure runs.
+	listErr   error
+	lookupErr error
+	// missing controls per-directory NotFound for ListEntries: when a path
+	// is in this set, ListEntries returns filer_pb.ErrNotFound regardless
+	// of whether tree has an entry for it. Lets us test the .versions/
+	// not-yet-created vs. empty cases independently.
+	missing map[string]struct{}
+}
+
+func newFakeFiler() *fsFakeFiler {
+	return &fsFakeFiler{
+		tree:    map[string][]*filer_pb.Entry{},
+		missing: map[string]struct{}{},
+	}
+}
+
+func (c *fsFakeFiler) ListEntries(ctx context.Context, in *filer_pb.ListEntriesRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[filer_pb.ListEntriesResponse], error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if c.listErr != nil {
+		err := c.listErr
+		c.listErr = nil
+		return nil, err
+	}
+	// Real-filer behavior on a missing parent: stream creation succeeds,
+	// the first Recv() surfaces ErrNotFound. SeaweedList passes that
+	// through unwrapped, which is what filerSiblingLister's
+	// errors.Is(err, filer_pb.ErrNotFound) check depends on.
+	if _, missing := c.missing[in.Directory]; missing {
+		return &fsListStream{recvErr: filer_pb.ErrNotFound, ctx: ctx}, nil
+	}
+	src := c.tree[in.Directory]
+	filtered := make([]*filer_pb.Entry, 0, len(src))
+	for _, e := range src {
+		if e == nil {
+			continue
+		}
+		if in.StartFromFileName != "" {
+			if in.InclusiveStartFrom {
+				if e.Name < in.StartFromFileName {
+					continue
+				}
+			} else if e.Name <= in.StartFromFileName {
+				continue
+			}
+		}
+		filtered = append(filtered, e)
+	}
+	sort.SliceStable(filtered, func(i, j int) bool { return filtered[i].Name < filtered[j].Name })
+	if in.Limit > 0 && uint32(len(filtered)) > in.Limit {
+		filtered = filtered[:in.Limit]
+	}
+	resps := make([]*filer_pb.ListEntriesResponse, 0, len(filtered))
+	for _, e := range filtered {
+		resps = append(resps, &filer_pb.ListEntriesResponse{Entry: e})
+	}
+	return &fsListStream{responses: resps, ctx: ctx}, nil
+}
+
+func (c *fsFakeFiler) LookupDirectoryEntry(ctx context.Context, in *filer_pb.LookupDirectoryEntryRequest, opts ...grpc.CallOption) (*filer_pb.LookupDirectoryEntryResponse, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if c.lookupErr != nil {
+		err := c.lookupErr
+		c.lookupErr = nil
+		return nil, err
+	}
+	for _, e := range c.tree[in.Directory] {
+		if e != nil && e.Name == in.Name {
+			return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil
+		}
+	}
+	return nil, filer_pb.ErrNotFound
+}
+
+func (c *fsFakeFiler) put(dir string, entries ...*filer_pb.Entry) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.tree[dir] = append(c.tree[dir], entries...)
+}
+
+func (c *fsFakeFiler) markMissing(dir string) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.missing[dir] = struct{}{}
+}
+
+func newLister(c *fsFakeFiler) *filerSiblingLister {
+	return &filerSiblingLister{client: c, bucketsPath: "/buckets"}
+}
+
+func versionEntry(name, versionID string) *filer_pb.Entry {
+	return &filer_pb.Entry{
+		Name:       name,
+		Attributes: &filer_pb.FuseAttributes{},
+		Extended: map[string][]byte{
+			s3_constants.ExtVersionIdKey: []byte(versionID),
+		},
+	}
+}
+
+func TestSurvivors_NoVersionsNoNullReturnsZero(t *testing.T) {
+	f := newFakeFiler()
+	// .versions/ doesn't exist (NotFound) and the bare key doesn't exist either.
+	f.markMissing("/buckets/b/k" + s3_constants.VersionsFolder)
+	l := newLister(f)
+
+	s, err := l.Survivors(context.Background(), "b", "k")
+	require.NoError(t, err)
+	assert.Equal(t, 0, s.Count)
+	assert.False(t, s.HasNullVersion)
+	assert.Nil(t, s.LoneEntry)
+}
+
+func TestSurvivors_OneVersionPopulatesLoneEntry(t *testing.T) {
+	// Count == 1 means LoneEntry must be set; the dispatcher's
+	// sole-survivor path uses it to compare against the marker version.
+	f := newFakeFiler()
+	dir := "/buckets/b/k" + s3_constants.VersionsFolder
+	f.put(dir, versionEntry("v_aaa", "aaa"))
+	f.markMissing("/buckets/b") // bare key absent
+	l := newLister(f)
+
+	s, err := l.Survivors(context.Background(), "b", "k")
+	require.NoError(t, err)
+	assert.Equal(t, 1, s.Count)
+	require.NotNil(t, s.LoneEntry)
+	assert.Equal(t, "v_aaa", s.LoneEntry.Name)
+	assert.False(t, s.HasNullVersion)
+}
+
+func TestSurvivors_MoreThanOneClearsLoneEntry(t *testing.T) {
+	// Listing caps at 2; once Count > 1, LoneEntry must be reset to nil
+	// so the caller doesn't mistakenly treat the second-arrival entry as
+	// the sole survivor.
+	f := newFakeFiler()
+	dir := "/buckets/b/k" + s3_constants.VersionsFolder
+	f.put(dir, versionEntry("v_aaa", "aaa"), versionEntry("v_bbb", "bbb"))
+	f.markMissing("/buckets/b")
+	l := newLister(f)
+
+	s, err := l.Survivors(context.Background(), "b", "k")
+	require.NoError(t, err)
+	assert.Equal(t, 2, s.Count)
+	assert.Nil(t, s.LoneEntry, "Count > 1 must clear LoneEntry")
+}
+
+func TestSurvivors_NullVersionDetectedFromBareRegularFile(t *testing.T) {
+	// The bare <bucket>/<key> path resolves to a regular file when the
+	// object pre-dates versioning; that file is the null version.
+	f := newFakeFiler()
+	f.markMissing("/buckets/b/k" + s3_constants.VersionsFolder)
+	f.put("/buckets/b", &filer_pb.Entry{Name: "k", IsDirectory: false, Attributes: &filer_pb.FuseAttributes{}})
+	l := newLister(f)
+
+	s, err := l.Survivors(context.Background(), "b", "k")
+	require.NoError(t, err)
+	assert.True(t, s.HasNullVersion)
+}
+
+func TestSurvivors_DirectoryKeyMarkerCountsAsNull(t *testing.T) {
+	// An explicit S3 directory-key marker (IsDirectory=true with
+	// Attributes.Mime set) is also the null version. The listing path
+	// downstream treats both the regular-file and the directory-marker
+	// shape as null; Survivors must agree. The lister's NewFullPath
+	// trims the trailing slash from objectKey before splitting, so the
+	// stored entry Name has no trailing slash.
+	f := newFakeFiler()
+	f.markMissing("/buckets/b/dir" + s3_constants.VersionsFolder)
+	f.put("/buckets/b", &filer_pb.Entry{
+		Name:        "dir",
+		IsDirectory: true,
+		Attributes:  &filer_pb.FuseAttributes{Mime: "application/x-directory"},
+	})
+	l := newLister(f)
+
+	s, err := l.Survivors(context.Background(), "b", "dir/")
+	require.NoError(t, err)
+	assert.True(t, s.HasNullVersion, "directory-key marker must count as null version")
+}
+
+func TestSurvivors_PlainDirectoryDoesNotCount(t *testing.T) {
+	// A plain directory (no Mime, just IsDirectory=true) is implicit-only
+	// and must NOT register as a null version — otherwise normal prefix
+	// directories would shadow real null-version detection.
+	f := newFakeFiler()
+	f.markMissing("/buckets/b/dir" + s3_constants.VersionsFolder)
+	f.put("/buckets/b", &filer_pb.Entry{
+		Name:        "dir",
+		IsDirectory: true,
+		Attributes:  &filer_pb.FuseAttributes{}, // no Mime
+	})
+	l := newLister(f)
+
+	s, err := l.Survivors(context.Background(), "b", "dir/")
+	require.NoError(t, err)
+	assert.False(t, s.HasNullVersion, "plain directory must not count as null version")
+}
+
+func TestSurvivors_VersionsListingErrorPropagates(t *testing.T) {
+	// A non-NotFound error from the .versions/ listing must surface; the
+	// dispatcher promotes this to a retry instead of a NOOP.
+	f := newFakeFiler()
+	f.listErr = errors.New("transport boom")
+	l := newLister(f)
+
+	_, err := l.Survivors(context.Background(), "b", "k")
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "transport boom")
+}
+
+func TestSurvivors_NullLookupErrorPropagates(t *testing.T) {
+	// A non-NotFound error from the bare-key lookup must surface; we
+	// can't conclude "no null" from a transport failure.
+	f := newFakeFiler()
+	f.markMissing("/buckets/b/k" + s3_constants.VersionsFolder)
+	f.lookupErr = errors.New("lookup boom")
+	l := newLister(f)
+
+	_, err := l.Survivors(context.Background(), "b", "k")
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "lookup boom")
+}
+
+func TestListVersions_NotFoundReturnsNil(t *testing.T) {
+	// A hard-deleted .versions/ container collapses to (nil, nil) so
+	// callers can treat it as "no versions" without special-casing.
+	f := newFakeFiler()
+	f.markMissing("/buckets/b/k" + s3_constants.VersionsFolder)
+	l := newLister(f)
+
+	versions, err := l.ListVersions(context.Background(), "b", "k")
+	require.NoError(t, err)
+	assert.Nil(t, versions)
+}
+
+func TestListVersions_FiltersOutDirectoriesAndMissingVersionId(t *testing.T) {
+	// Subdirectories and entries without ExtVersionIdKey aren't real
+	// versions; the bootstrap walker writes them sometimes so the
+	// listing must filter rather than treat them as data.
+	f := newFakeFiler()
+	dir := "/buckets/b/k" + s3_constants.VersionsFolder
+	f.put(dir,
+		versionEntry("v_aaa", "aaa"),
+		&filer_pb.Entry{Name: "subdir", IsDirectory: true, Attributes: &filer_pb.FuseAttributes{}},
+		&filer_pb.Entry{Name: "v_no_attrs"},                                       // nil Attributes
+		&filer_pb.Entry{Name: "v_no_ext", Attributes: &filer_pb.FuseAttributes{}}, // missing Extended
+		&filer_pb.Entry{Name: "v_empty_ext", Attributes: &filer_pb.FuseAttributes{}, Extended: map[string][]byte{s3_constants.ExtVersionIdKey: nil}}, // empty value
+		versionEntry("v_zzz", "zzz"),
+	)
+	l := newLister(f)
+
+	versions, err := l.ListVersions(context.Background(), "b", "k")
+	require.NoError(t, err)
+	require.Len(t, versions, 2)
+	assert.Equal(t, "v_aaa", versions[0].Name)
+	assert.Equal(t, "v_zzz", versions[1].Name)
+}
+
+func TestListVersions_PaginatesAcrossPages(t *testing.T) {
+	// pageSize is 1024 inside the lister; populate just over the boundary
+	// to force a second listing call. The fake's StartFromFileName
+	// handling mirrors the real filer.
+	f := newFakeFiler()
+	dir := "/buckets/b/k" + s3_constants.VersionsFolder
+	const total = 1030
+	entries := make([]*filer_pb.Entry, 0, total)
+	for i := 0; i < total; i++ {
+		entries = append(entries, versionEntry(fmt.Sprintf("v_%05d", i), fmt.Sprintf("id_%05d", i)))
+	}
+	f.put(dir, entries...)
+	l := newLister(f)
+
+	versions, err := l.ListVersions(context.Background(), "b", "k")
+	require.NoError(t, err)
+	require.Len(t, versions, total)
+	// Order is preserved across pages.
+	assert.Equal(t, "v_00000", versions[0].Name)
+	assert.Equal(t, fmt.Sprintf("v_%05d", total-1), versions[total-1].Name)
+}
+
+func TestListVersions_ErrorPropagates(t *testing.T) {
+	f := newFakeFiler()
+	f.listErr = errors.New("list boom")
+	l := newLister(f)
+
+	versions, err := l.ListVersions(context.Background(), "b", "k")
+	assert.Nil(t, versions)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "list boom")
+}
+
+func TestLookupNullVersion_RegularFileExplicitFalse(t *testing.T) {
+	// A bare regular file with no ExtVersionIdKey is the null version
+	// but not "explicit" — the suspended-versioning write path flags
+	// explicit nulls with ExtVersionIdKey="null".
+	f := newFakeFiler()
+	f.put("/buckets/b", &filer_pb.Entry{Name: "k", IsDirectory: false, Attributes: &filer_pb.FuseAttributes{}})
+	l := newLister(f)
+
+	entry, explicit, err := l.LookupNullVersion(context.Background(), "b", "k")
+	require.NoError(t, err)
+	require.NotNil(t, entry)
+	assert.Equal(t, "k", entry.Name)
+	assert.False(t, explicit)
+}
+
+func TestLookupNullVersion_ExplicitNullFlag(t *testing.T) {
+	// ExtVersionIdKey="null" is the marker the suspended-versioning
+	// write path sets; the lister surfaces it via the explicit return.
+	f := newFakeFiler()
+	f.put("/buckets/b", &filer_pb.Entry{
+		Name:       "k",
+		Attributes: &filer_pb.FuseAttributes{},
+		Extended:   map[string][]byte{s3_constants.ExtVersionIdKey: []byte("null")},
+	})
+	l := newLister(f)
+
+	entry, explicit, err := l.LookupNullVersion(context.Background(), "b", "k")
+	require.NoError(t, err)
+	require.NotNil(t, entry)
+	assert.True(t, explicit)
+}
+
+func TestLookupNullVersion_DirectoryKeyMarkerQualifies(t *testing.T) {
+	// NewFullPath strips a trailing slash before splitting; the entry
+	// Name as stored has no slash.
+	f := newFakeFiler()
+	f.put("/buckets/b", &filer_pb.Entry{
+		Name:        "dir",
+		IsDirectory: true,
+		Attributes:  &filer_pb.FuseAttributes{Mime: "application/x-directory"},
+	})
+	l := newLister(f)
+
+	entry, _, err := l.LookupNullVersion(context.Background(), "b", "dir/")
+	require.NoError(t, err)
+	require.NotNil(t, entry)
+}
+
+func TestLookupNullVersion_PlainDirectoryRejected(t *testing.T) {
+	// A plain directory must not be returned — otherwise the dispatcher
+	// would treat any prefix directory as a null version.
+	f := newFakeFiler()
+	f.put("/buckets/b", &filer_pb.Entry{
+		Name:        "dir",
+		IsDirectory: true,
+		Attributes:  &filer_pb.FuseAttributes{},
+	})
+	l := newLister(f)
+
+	entry, explicit, err := l.LookupNullVersion(context.Background(), "b", "dir/")
+	require.NoError(t, err)
+	assert.Nil(t, entry)
+	assert.False(t, explicit)
+}
+
+func TestLookupNullVersion_NotFoundCollapses(t *testing.T) {
+	f := newFakeFiler()
+	l := newLister(f) // empty tree -> NotFound on lookup
+
+	entry, explicit, err := l.LookupNullVersion(context.Background(), "b", "k")
+	require.NoError(t, err)
+	assert.Nil(t, entry)
+	assert.False(t, explicit)
+}
+
+func TestLookupNullVersion_TransportErrorPropagates(t *testing.T) {
+	f := newFakeFiler()
+	f.lookupErr = errors.New("lookup boom")
+	l := newLister(f)
+
+	_, _, err := l.LookupNullVersion(context.Background(), "b", "k")
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "lookup boom")
+}
+
+func TestLookupVersion_EmptyVersionIdReturnsNil(t *testing.T) {
+	// An empty versionID is a no-op — neither error nor result.
+	f := newFakeFiler()
+	l := newLister(f)
+
+	entry, err := l.LookupVersion(context.Background(), "b", "k", "")
+	require.NoError(t, err)
+	assert.Nil(t, entry)
+}
+
+func TestLookupVersion_AppliesVPrefix(t *testing.T) {
+	// File name on disk is "v_<id>"; the caller passes only the id.
+	f := newFakeFiler()
+	dir := "/buckets/b/k" + s3_constants.VersionsFolder
+	f.put(dir, versionEntry("v_abc123", "abc123"))
+	l := newLister(f)
+
+	entry, err := l.LookupVersion(context.Background(), "b", "k", "abc123")
+	require.NoError(t, err)
+	require.NotNil(t, entry)
+	assert.Equal(t, "v_abc123", entry.Name)
+}
+
+func TestLookupVersion_NotFoundCollapses(t *testing.T) {
+	// Hard-delete between pointer update and lookup is the documented
+	// race; collapse to (nil, nil) so the dispatcher continues without
+	// retry-storming.
+	f := newFakeFiler()
+	l := newLister(f)
+
+	entry, err := l.LookupVersion(context.Background(), "b", "k", "abc")
+	require.NoError(t, err)
+	assert.Nil(t, entry)
+}
+
+func TestLookupVersion_TransportErrorPropagates(t *testing.T) {
+	f := newFakeFiler()
+	f.lookupErr = errors.New("lookup boom")
+	l := newLister(f)
+
+	_, err := l.LookupVersion(context.Background(), "b", "k", "abc")
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "lookup boom")
+}

weed/s3api/s3lifecycle/due_at.go

@@ -0,0 +1,63 @@
+package s3lifecycle
+
+import (
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/util"
+)
+
+// DaysToDuration converts a "days" lifecycle threshold into a wall-clock
+// duration. Production builds keep one day = 24h; the s3tests build tag
+// shrinks it to LifeCycleInterval (10s) so the upstream s3-tests
+// expiration suite can complete inside its 30s polling window. Exported so
+// tests in sibling packages can pin their math to the same scale.
+func DaysToDuration(days int) time.Duration {
+	return time.Duration(days) * util.LifeCycleInterval
+}
+
+// ComputeDueAt returns the earliest wall-clock time the (rule, kind) action
+// can fire for info. Returns zero time when no action can fire for this
+// entry. Used by reader/bootstrap to decide pending vs. inline-delete.
+func ComputeDueAt(rule *Rule, kind ActionKind, info *ObjectInfo) time.Time {
+	if rule == nil || info == nil || rule.Status != StatusEnabled {
+		return time.Time{}
+	}
+	if !filterMatches(rule, info) {
+		return time.Time{}
+	}
+
+	switch kind {
+	case ActionKindAbortMPU:
+		if info.IsMPUInit && rule.AbortMPUDaysAfterInitiation > 0 {
+			return info.ModTime.Add(DaysToDuration(rule.AbortMPUDaysAfterInitiation))
+		}
+	case ActionKindExpiredDeleteMarker:
+		if info.IsLatest && info.IsDeleteMarker && rule.ExpiredObjectDeleteMarker && info.NumVersions == 1 {
+			return info.ModTime
+		}
+	case ActionKindExpirationDays:
+		if info.IsLatest && !info.IsDeleteMarker && rule.ExpirationDays > 0 {
+			return info.ModTime.Add(DaysToDuration(rule.ExpirationDays))
+		}
+	case ActionKindExpirationDate:
+		if info.IsLatest && !info.IsDeleteMarker && !rule.ExpirationDate.IsZero() {
+			return rule.ExpirationDate
+		}
+	case ActionKindNoncurrentDays:
+		if !info.IsLatest && rule.NoncurrentVersionExpirationDays > 0 {
+			base := info.SuccessorModTime
+			if base.IsZero() {
+				base = info.ModTime
+			}
+			return base.Add(DaysToDuration(rule.NoncurrentVersionExpirationDays))
+		}
+	case ActionKindNewerNoncurrent:
+		if !info.IsLatest && rule.NoncurrentVersionExpirationDays == 0 && rule.NewerNoncurrentVersions > 0 {
+			if !info.SuccessorModTime.IsZero() {
+				return info.SuccessorModTime
+			}
+			return info.ModTime
+		}
+	}
+	return time.Time{}
+}

weed/s3api/s3lifecycle/due_at_test.go

@@ -0,0 +1,102 @@
+package s3lifecycle
+
+import (
+	"testing"
+)
+
+func TestComputeDueAt_ExpirationDays(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, ExpirationDays: 30}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mod}
+	want := mod.Add(DaysToDuration(30))
+	if got := ComputeDueAt(rule, ActionKindExpirationDays, info); !got.Equal(want) {
+		t.Fatalf("want %v, got %v", want, got)
+	}
+}
+
+func TestComputeDueAt_ExpirationDate(t *testing.T) {
+	date := mustTime(t, "2025-06-15T00:00:00Z")
+	rule := &Rule{Status: StatusEnabled, ExpirationDate: date}
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mustTime(t, "2024-01-01T00:00:00Z")}
+	if got := ComputeDueAt(rule, ActionKindExpirationDate, info); !got.Equal(date) {
+		t.Fatalf("want %v, got %v", date, got)
+	}
+}
+
+func TestComputeDueAt_DeleteMarker_NotSoleSurvivor(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, ExpiredObjectDeleteMarker: true}
+	info := &ObjectInfo{Key: "a", IsLatest: true, IsDeleteMarker: true, NumVersions: 3}
+	if got := ComputeDueAt(rule, ActionKindExpiredDeleteMarker, info); !got.IsZero() {
+		t.Fatalf("non-sole marker should be zero, got %v", got)
+	}
+}
+
+func TestComputeDueAt_KindNotDeclared(t *testing.T) {
+	// Asking for a kind the rule doesn't declare returns zero.
+	rule := &Rule{Status: StatusEnabled, ExpirationDays: 1}
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mustTime(t, "2024-01-01T00:00:00Z")}
+	if got := ComputeDueAt(rule, ActionKindAbortMPU, info); !got.IsZero() {
+		t.Fatalf("undeclared kind should be zero, got %v", got)
+	}
+}
+
+func TestComputeDueAt_WrongShapeForKind(t *testing.T) {
+	// Non-current object asked under EXPIRATION_DAYS (which is current-only): zero.
+	rule := &Rule{Status: StatusEnabled, ExpirationDays: 1}
+	info := &ObjectInfo{Key: "a", IsLatest: false, ModTime: mustTime(t, "2024-01-01T00:00:00Z")}
+	if got := ComputeDueAt(rule, ActionKindExpirationDays, info); !got.IsZero() {
+		t.Fatalf("non-current under current rule should be zero, got %v", got)
+	}
+}
+
+func TestComputeDueAt_NoncurrentDeleteMarkerHonorsNoncurrentDays(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, NoncurrentVersionExpirationDays: 7}
+	successor := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{
+		Key:              "a",
+		IsLatest:         false,
+		IsDeleteMarker:   true,
+		NumVersions:      3,
+		SuccessorModTime: successor,
+	}
+	want := successor.Add(DaysToDuration(7))
+	if got := ComputeDueAt(rule, ActionKindNoncurrentDays, info); !got.Equal(want) {
+		t.Fatalf("want %v, got %v", want, got)
+	}
+}
+
+func TestComputeDueAt_NoncurrentSuccessorMtime(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, NoncurrentVersionExpirationDays: 30}
+	successor := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: "a", IsLatest: false, SuccessorModTime: successor}
+	want := successor.Add(DaysToDuration(30))
+	if got := ComputeDueAt(rule, ActionKindNoncurrentDays, info); !got.Equal(want) {
+		t.Fatalf("want %v, got %v", want, got)
+	}
+}
+
+func TestComputeDueAt_FilterRejects(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, Prefix: "logs/", ExpirationDays: 1}
+	info := &ObjectInfo{Key: "data/x", IsLatest: true, ModTime: mustTime(t, "2024-01-01T00:00:00Z")}
+	if got := ComputeDueAt(rule, ActionKindExpirationDays, info); !got.IsZero() {
+		t.Fatalf("filter rejection should be zero, got %v", got)
+	}
+}
+
+func TestComputeDueAt_DisabledRule(t *testing.T) {
+	rule := &Rule{Status: StatusDisabled, ExpirationDays: 1}
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mustTime(t, "2024-01-01T00:00:00Z")}
+	if got := ComputeDueAt(rule, ActionKindExpirationDays, info); !got.IsZero() {
+		t.Fatalf("disabled rule should be zero, got %v", got)
+	}
+}
+
+func TestComputeDueAt_MPUInit(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, AbortMPUDaysAfterInitiation: 7}
+	init := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: ".uploads/u1/", IsMPUInit: true, ModTime: init}
+	want := init.Add(DaysToDuration(7))
+	if got := ComputeDueAt(rule, ActionKindAbortMPU, info); !got.Equal(want) {
+		t.Fatalf("want %v, got %v", want, got)
+	}
+}

weed/s3api/s3lifecycle/engine/compile.go

@@ -0,0 +1,133 @@
+package engine
+
+import (
+	"bytes"
+	"sort"
+	"sync/atomic"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+var snapshotIDSeq atomic.Uint64
+
+// CompileInput: at most one entry per Bucket. Duplicates would overwrite
+// earlier entries' BucketIndex.
+type CompileInput struct {
+	Bucket    string
+	Rules     []*s3lifecycle.Rule
+	Versioned bool
+}
+
+// PriorState carries durable per-action state into Compile. Missing keys
+// are treated as bootstrap_complete=false.
+type PriorState struct {
+	BootstrapComplete bool
+	Mode              RuleMode
+}
+
+// MetaLogRetention=0 means unbounded; the retention gate doesn't trip.
+type CompileOptions struct {
+	MetaLogRetention     time.Duration
+	BootstrapLookbackMin time.Duration
+	PriorStates          map[s3lifecycle.ActionKey]PriorState
+}
+
+const defaultBootstrapLookbackMin = 5 * s3lifecycle.SmallDelay
+
+func (e *Engine) Compile(inputs []CompileInput, opts CompileOptions) *Snapshot {
+	if opts.BootstrapLookbackMin == 0 {
+		opts.BootstrapLookbackMin = defaultBootstrapLookbackMin
+	}
+
+	snap := &Snapshot{
+		id:                  snapshotIDSeq.Add(1),
+		buckets:             make(map[string]*BucketIndex),
+		actions:             make(map[s3lifecycle.ActionKey]*CompiledAction),
+		originalDelayGroups: make(map[time.Duration][]s3lifecycle.ActionKey),
+		dateActions:         make(map[s3lifecycle.ActionKey]time.Time),
+	}
+
+	for _, in := range inputs {
+		bi := &BucketIndex{bucket: in.Bucket, versioned: in.Versioned}
+		snap.buckets[in.Bucket] = bi
+
+		for _, rule := range in.Rules {
+			ruleHash := s3lifecycle.RuleHash(rule)
+			for _, kind := range s3lifecycle.RuleActionKinds(rule) {
+				key := s3lifecycle.ActionKey{Bucket: in.Bucket, RuleHash: ruleHash, ActionKind: kind}
+				// Durable Mode wins; decideMode is only the seed for new
+				// or legacy-zero actions. Otherwise lag-fallback / operator
+				// SCAN_ONLY would re-promote on every rebuild.
+				prior, hasPrior := opts.PriorStates[key]
+				mode := prior.Mode
+				if !hasPrior || mode == ModeUnspecified {
+					mode = decideMode(rule, kind, opts.MetaLogRetention, opts.BootstrapLookbackMin)
+				}
+				// Active gates routing: MatchPath / MatchOriginalWrite skip
+				// !IsActive actions. ScanAtDate's only dispatch path is the
+				// bootstrap walk's MatchPath call, so the action must be
+				// considered active there or its rule is silently a no-op.
+				// Bootstrap-completion state is per-action and event-driven-
+				// shaped; date-based actions don't need a bootstrap rendezvous.
+				active := mode == ModeScanAtDate ||
+					(prior.BootstrapComplete && mode == ModeEventDriven)
+
+				ca := &CompiledAction{
+					Rule:               rule,
+					Bucket:             in.Bucket,
+					Key:                key,
+					Delay:              s3lifecycle.MinTriggerAge(rule, kind),
+					PredicateSensitive: rulePredicateSensitive(rule),
+					Mode:               mode,
+				}
+				if active {
+					ca.markActive()
+				}
+				snap.actions[key] = ca
+				bi.actionKeys = append(bi.actionKeys, key)
+
+				// Index every action regardless of `active`; routing
+				// re-filters on IsActive() so MarkActive flips are visible
+				// without a recompile.
+				if mode == ModeScanAtDate {
+					snap.dateActions[key] = rule.ExpirationDate
+				}
+				if mode == ModeEventDriven {
+					snap.originalDelayGroups[ca.Delay] = append(snap.originalDelayGroups[ca.Delay], key)
+					if ca.PredicateSensitive {
+						snap.predicateActions = append(snap.predicateActions, key)
+					}
+				}
+			}
+		}
+	}
+
+	snap.allActionsSorted = make([]*CompiledAction, 0, len(snap.actions))
+	for _, a := range snap.actions {
+		snap.allActionsSorted = append(snap.allActionsSorted, a)
+	}
+	sort.Slice(snap.allActionsSorted, func(i, j int) bool {
+		a, b := snap.allActionsSorted[i], snap.allActionsSorted[j]
+		if a.Bucket != b.Bucket {
+			return a.Bucket < b.Bucket
+		}
+		if c := bytes.Compare(a.Key.RuleHash[:], b.Key.RuleHash[:]); c != 0 {
+			return c < 0
+		}
+		return a.Key.ActionKind < b.Key.ActionKind
+	})
+
+	e.current.Store(snap)
+	return snap
+}
+
+// rulePredicateSensitive: only tag filters can flip post-PUT. Size is
+// immutable once written; a size change is a fresh write through the
+// original-write stream.
+func rulePredicateSensitive(rule *s3lifecycle.Rule) bool {
+	if rule == nil {
+		return false
+	}
+	return len(rule.FilterTags) > 0
+}

weed/s3api/s3lifecycle/engine/compile_helpers_test.go

@@ -0,0 +1,45 @@
+package engine
+
+import (
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+)
+
+// rulePredicateSensitive's non-nil branches are exercised by the
+// existing Compile tests; this test pins the defensive nil-rule
+// branch directly. Compile shouldn't ever pass a nil rule, but a
+// future caller that does shouldn't panic.
+
+func TestRulePredicateSensitive_NilRuleReturnsFalse(t *testing.T) {
+	assert.False(t, rulePredicateSensitive(nil))
+}
+
+func TestRulePredicateSensitive_NoFilterTagsReturnsFalse(t *testing.T) {
+	// A rule without FilterTags is not predicate-sensitive; the
+	// router's MatchPredicateChange path skips actions whose rule
+	// returns false here.
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	assert.False(t, rulePredicateSensitive(rule))
+}
+
+func TestRulePredicateSensitive_EmptyFilterTagsReturnsFalse(t *testing.T) {
+	// An empty (non-nil) FilterTags map must classify as non-sensitive
+	// — the function checks len, so empty == 0 == false.
+	rule := &s3lifecycle.Rule{
+		ID:         "r",
+		Status:     s3lifecycle.StatusEnabled,
+		FilterTags: map[string]string{},
+	}
+	assert.False(t, rulePredicateSensitive(rule))
+}
+
+func TestRulePredicateSensitive_PopulatedFilterTagsReturnsTrue(t *testing.T) {
+	rule := &s3lifecycle.Rule{
+		ID:         "r",
+		Status:     s3lifecycle.StatusEnabled,
+		FilterTags: map[string]string{"env": "prod"},
+	}
+	assert.True(t, rulePredicateSensitive(rule))
+}

weed/s3api/s3lifecycle/engine/engine.go

@@ -0,0 +1,132 @@
+// Package engine compiles per-bucket lifecycle rules into a CompiledAction
+// snapshot. One XML <Rule> expands into N compiled actions (one per populated
+// action sub-element); each is keyed by ActionKey{bucket, rule_hash,
+// action_kind} so sibling actions schedule and degrade independently.
+package engine
+
+import (
+	"sync/atomic"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+type Engine struct {
+	current atomic.Pointer[Snapshot]
+}
+
+func New() *Engine {
+	e := &Engine{}
+	e.current.Store(&Snapshot{actions: map[s3lifecycle.ActionKey]*CompiledAction{}})
+	return e
+}
+
+func (e *Engine) Snapshot() *Snapshot { return e.current.Load() }
+
+// Snapshot fields are append-only after Compile except CompiledAction.engineState,
+// which transitions inactive -> active atomically via markActive.
+type Snapshot struct {
+	id      uint64
+	buckets map[string]*BucketIndex
+	actions map[s3lifecycle.ActionKey]*CompiledAction
+
+	allActionsSorted []*CompiledAction
+
+	// Routing indexes hold every ActionKey by mode regardless of activation;
+	// dispatch filters on IsActive().
+	originalDelayGroups map[time.Duration][]s3lifecycle.ActionKey
+	predicateActions    []s3lifecycle.ActionKey
+	dateActions         map[s3lifecycle.ActionKey]time.Time
+}
+
+func (s *Snapshot) SnapshotID() uint64 { return s.id }
+
+type BucketIndex struct {
+	bucket     string
+	versioned  bool
+	actionKeys []s3lifecycle.ActionKey
+}
+
+type CompiledAction struct {
+	Rule               *s3lifecycle.Rule
+	Bucket             string
+	Key                s3lifecycle.ActionKey
+	Delay              time.Duration
+	PredicateSensitive bool
+	Mode               RuleMode
+	engineState        atomic.Uint32
+}
+
+const (
+	engineStateInactive uint32 = 0
+	engineStateActive   uint32 = 1
+)
+
+func (a *CompiledAction) IsActive() bool {
+	return a.engineState.Load() == engineStateActive
+}
+
+func (a *CompiledAction) markActive() {
+	a.engineState.Store(engineStateActive)
+}
+
+// MarkActive mirrors the in-memory hint after a durable bootstrap_complete
+// write. No-op if the key isn't in the snapshot.
+func (s *Snapshot) MarkActive(key s3lifecycle.ActionKey) {
+	if a, ok := s.actions[key]; ok {
+		a.markActive()
+	}
+}
+
+func (s *Snapshot) Action(key s3lifecycle.ActionKey) *CompiledAction {
+	return s.actions[key]
+}
+
+// AllActions: caller must not mutate.
+func (s *Snapshot) AllActions() []*CompiledAction {
+	return s.allActionsSorted
+}
+
+// OriginalDelayGroups / PredicateActions / DateActions return defensive copies.
+func (s *Snapshot) OriginalDelayGroups() map[time.Duration][]s3lifecycle.ActionKey {
+	out := make(map[time.Duration][]s3lifecycle.ActionKey, len(s.originalDelayGroups))
+	for d, keys := range s.originalDelayGroups {
+		copied := make([]s3lifecycle.ActionKey, len(keys))
+		copy(copied, keys)
+		out[d] = copied
+	}
+	return out
+}
+
+func (s *Snapshot) PredicateActions() []s3lifecycle.ActionKey {
+	out := make([]s3lifecycle.ActionKey, len(s.predicateActions))
+	copy(out, s.predicateActions)
+	return out
+}
+
+func (s *Snapshot) DateActions() map[s3lifecycle.ActionKey]time.Time {
+	out := make(map[s3lifecycle.ActionKey]time.Time, len(s.dateActions))
+	for k, v := range s.dateActions {
+		out[k] = v
+	}
+	return out
+}
+
+func (s *Snapshot) BucketVersioned(bucket string) bool {
+	if bi, ok := s.buckets[bucket]; ok {
+		return bi.versioned
+	}
+	return false
+}
+
+// BucketActionKeys returns a defensive copy of the action keys for bucket,
+// or nil if the bucket has no compiled actions in this snapshot.
+func (s *Snapshot) BucketActionKeys(bucket string) []s3lifecycle.ActionKey {
+	bi, ok := s.buckets[bucket]
+	if !ok {
+		return nil
+	}
+	out := make([]s3lifecycle.ActionKey, len(bi.actionKeys))
+	copy(out, bi.actionKeys)
+	return out
+}

weed/s3api/s3lifecycle/engine/engine_test.go

@@ -0,0 +1,381 @@
+package engine
+
+import (
+	"strconv"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+func ruleExpDays(id, prefix string, days int) *s3lifecycle.Rule {
+	return &s3lifecycle.Rule{
+		ID:             id,
+		Status:         s3lifecycle.StatusEnabled,
+		Prefix:         prefix,
+		ExpirationDays: days,
+	}
+}
+
+func TestCompile_SingleRuleSingleAction(t *testing.T) {
+	e := New()
+	rule := ruleExpDays("r1", "logs/", 30)
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{
+		PriorStates: map[s3lifecycle.ActionKey]PriorState{
+			{Bucket: "b1", RuleHash: s3lifecycle.RuleHash(rule), ActionKind: s3lifecycle.ActionKindExpirationDays}: {
+				BootstrapComplete: true,
+			},
+		},
+	})
+	if got := len(snap.actions); got != 1 {
+		t.Fatalf("want 1 action, got %d", got)
+	}
+	for _, a := range snap.actions {
+		if !a.IsActive() {
+			t.Fatalf("bootstrap_complete + EVENT_DRIVEN should activate, got mode=%v", a.Mode)
+		}
+		if a.Mode != ModeEventDriven {
+			t.Fatalf("want EVENT_DRIVEN, got %v", a.Mode)
+		}
+		if a.Delay != s3lifecycle.DaysToDuration(30) {
+			t.Fatalf("want 30d delay, got %v", a.Delay)
+		}
+	}
+}
+
+func TestCompile_MultiAction_SiblingsHaveOwnEntries(t *testing.T) {
+	// One XML rule with three actions -> three CompiledActions, three
+	// distinct ActionKeys, three independent delay group memberships.
+	rule := &s3lifecycle.Rule{
+		ID:                              "multi",
+		Status:                          s3lifecycle.StatusEnabled,
+		Prefix:                          "data/",
+		ExpirationDays:                  90,
+		NoncurrentVersionExpirationDays: 30,
+		AbortMPUDaysAfterInitiation:     7,
+	}
+	rh := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for _, k := range s3lifecycle.RuleActionKinds(rule) {
+		prior[s3lifecycle.ActionKey{Bucket: "b1", RuleHash: rh, ActionKind: k}] = PriorState{BootstrapComplete: true}
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{PriorStates: prior})
+
+	if got := len(snap.actions); got != 3 {
+		t.Fatalf("want 3 actions, got %d", got)
+	}
+	wantDelays := map[time.Duration]bool{
+		s3lifecycle.DaysToDuration(90): true, // ExpirationDays
+		s3lifecycle.DaysToDuration(30): true, // NoncurrentDays
+		s3lifecycle.DaysToDuration(7):  true, // AbortMPU
+	}
+	for delay, keys := range snap.originalDelayGroups {
+		if !wantDelays[delay] {
+			t.Fatalf("unexpected delay group %v", delay)
+		}
+		if len(keys) != 1 {
+			t.Fatalf("delay %v should hold exactly its own action, got %d", delay, len(keys))
+		}
+	}
+	if len(snap.originalDelayGroups) != 3 {
+		t.Fatalf("want 3 delay groups, got %d", len(snap.originalDelayGroups))
+	}
+}
+
+func TestCompile_BootstrapPendingIndexedButInactive(t *testing.T) {
+	// Without prior bootstrap_complete=true the action is pending_bootstrap.
+	// It IS indexed in originalDelayGroups (so a later MarkActive flip is
+	// routable without recompile), but IsActive() reads false so the
+	// reader's IsActive-filter skips dispatch.
+	rule := ruleExpDays("r1", "logs/", 30)
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{})
+
+	for _, a := range snap.actions {
+		if a.IsActive() {
+			t.Fatalf("pending_bootstrap action should not be active")
+		}
+	}
+	if len(snap.originalDelayGroups) != 1 {
+		t.Fatalf("EVENT_DRIVEN action should be indexed even when inactive, got %v", snap.originalDelayGroups)
+	}
+}
+
+func TestCompile_RetentionGate(t *testing.T) {
+	// Retention gate compares EventLogHorizon (scales with the day unit)
+	// against MetaLogRetention - BootstrapLookbackMin (where lookback is a
+	// fixed wall-clock minute count). Under the s3tests build tag the day
+	// unit shrinks to 10s, which collapses the margin to zero or negative
+	// and the gate degrades every rule. Skip there; the prod build still
+	// covers the gate semantics this test cares about.
+	if s3lifecycle.DaysToDuration(1) < time.Hour {
+		t.Skip("retention gate semantics require day-unit > lookback margin")
+	}
+	// A 90d ExpirationDays rule with 30d retention should land in scan_only.
+	// A 7d rule (under retention - lookback) stays event_driven.
+	long := ruleExpDays("long", "x/", 90)
+	short := ruleExpDays("short", "y/", 1)
+	rules := []*s3lifecycle.Rule{long, short}
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for _, r := range rules {
+		k := s3lifecycle.ActionKey{Bucket: "b1", RuleHash: s3lifecycle.RuleHash(r), ActionKind: s3lifecycle.ActionKindExpirationDays}
+		prior[k] = PriorState{BootstrapComplete: true}
+	}
+
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: rules}}, CompileOptions{
+		MetaLogRetention:     s3lifecycle.DaysToDuration(30),
+		BootstrapLookbackMin: 5 * time.Minute,
+		PriorStates:          prior,
+	})
+
+	for _, a := range snap.actions {
+		if a.Rule.ID == "long" && a.Mode != ModeScanOnly {
+			t.Fatalf("90d rule under 30d retention should be scan_only, got %v", a.Mode)
+		}
+		if a.Rule.ID == "short" && a.Mode != ModeEventDriven {
+			t.Fatalf("1d rule should be event_driven, got %v", a.Mode)
+		}
+	}
+}
+
+func TestCompile_RetentionUnboundedNeverGates(t *testing.T) {
+	// MetaLogRetention=0 means unbounded (default deployment); even a 100y
+	// rule should stay event_driven.
+	rule := ruleExpDays("decade", "x/", 36500)
+	prior := map[s3lifecycle.ActionKey]PriorState{
+		{RuleHash: s3lifecycle.RuleHash(rule), ActionKind: s3lifecycle.ActionKindExpirationDays}: {BootstrapComplete: true},
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{PriorStates: prior})
+	for _, a := range snap.actions {
+		if a.Mode != ModeEventDriven {
+			t.Fatalf("unbounded retention should keep event_driven, got %v", a.Mode)
+		}
+	}
+}
+
+func TestCompile_SiblingsDegradeIndependently(t *testing.T) {
+	// See TestCompile_RetentionGate for why the s3tests build can't honor
+	// the gate's absolute-time math.
+	if s3lifecycle.DaysToDuration(1) < time.Hour {
+		t.Skip("retention gate semantics require day-unit > lookback margin")
+	}
+	// 90d ExpirationDays + 7d AbortMPU under 30d retention: ExpirationDays
+	// degrades to scan_only, AbortMPU stays event_driven. The whole point
+	// of per-action keying.
+	rule := &s3lifecycle.Rule{
+		ID:                          "mixed",
+		Status:                      s3lifecycle.StatusEnabled,
+		Prefix:                      "x/",
+		ExpirationDays:              90,
+		AbortMPUDaysAfterInitiation: 7,
+	}
+	rh := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]PriorState{
+		{RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}: {BootstrapComplete: true},
+		{RuleHash: rh, ActionKind: s3lifecycle.ActionKindAbortMPU}:       {BootstrapComplete: true},
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{
+		MetaLogRetention:     s3lifecycle.DaysToDuration(30),
+		BootstrapLookbackMin: 5 * time.Minute,
+		PriorStates:          prior,
+	})
+	expDaysKey := s3lifecycle.ActionKey{Bucket: "b1", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	mpuKey := s3lifecycle.ActionKey{Bucket: "b1", RuleHash: rh, ActionKind: s3lifecycle.ActionKindAbortMPU}
+	if snap.actions[expDaysKey].Mode != ModeScanOnly {
+		t.Fatalf("ExpirationDays should be scan_only under 30d retention")
+	}
+	if snap.actions[mpuKey].Mode != ModeEventDriven {
+		t.Fatalf("AbortMPU should stay event_driven (sibling degrades independently)")
+	}
+}
+
+func TestCompile_PriorModePreservedOverDecideMode(t *testing.T) {
+	// A durably-persisted SCAN_ONLY (or DISABLED, or any degraded mode)
+	// must not be re-promoted to EVENT_DRIVEN by decideMode on every
+	// Compile. Otherwise lag-fallback / operator pause / manual scan-only
+	// don't survive an engine rebuild.
+	rule := ruleExpDays("r", "x/", 30)
+	rh := s3lifecycle.RuleHash(rule)
+	key := s3lifecycle.ActionKey{Bucket: "b1", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{
+		PriorStates: map[s3lifecycle.ActionKey]PriorState{
+			key: {BootstrapComplete: true, Mode: ModeScanOnly},
+		},
+	})
+	if snap.actions[key].Mode != ModeScanOnly {
+		t.Fatalf("durable Mode=ScanOnly should win over decideMode, got %v", snap.actions[key].Mode)
+	}
+	if snap.actions[key].IsActive() {
+		t.Fatalf("ScanOnly action must not be active")
+	}
+	// And: a missing PriorState falls through to decideMode as before.
+	rule2 := ruleExpDays("fresh", "x/", 30)
+	key2 := s3lifecycle.ActionKey{Bucket: "b1", RuleHash: s3lifecycle.RuleHash(rule2), ActionKind: s3lifecycle.ActionKindExpirationDays}
+	snap2 := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule2}}}, CompileOptions{})
+	if snap2.actions[key2].Mode != ModeEventDriven {
+		t.Fatalf("missing prior should fall through to decideMode (EventDriven), got %v", snap2.actions[key2].Mode)
+	}
+}
+
+func TestCompile_ExpirationDateScansAtDate(t *testing.T) {
+	date := time.Date(2025, 6, 15, 0, 0, 0, 0, time.UTC)
+	rule := &s3lifecycle.Rule{
+		ID:             "d",
+		Status:         s3lifecycle.StatusEnabled,
+		Prefix:         "x/",
+		ExpirationDate: date,
+	}
+	prior := map[s3lifecycle.ActionKey]PriorState{
+		{RuleHash: s3lifecycle.RuleHash(rule), ActionKind: s3lifecycle.ActionKindExpirationDate}: {BootstrapComplete: true},
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{PriorStates: prior})
+	if len(snap.dateActions) != 1 {
+		t.Fatalf("want 1 date action, got %d", len(snap.dateActions))
+	}
+	for _, d := range snap.dateActions {
+		if !d.Equal(date) {
+			t.Fatalf("date want %v, got %v", date, d)
+		}
+	}
+}
+
+func TestCompile_DisabledRuleNeverActivates(t *testing.T) {
+	rule := ruleExpDays("d", "x/", 30)
+	rule.Status = s3lifecycle.StatusDisabled
+	prior := map[s3lifecycle.ActionKey]PriorState{
+		{RuleHash: s3lifecycle.RuleHash(rule), ActionKind: s3lifecycle.ActionKindExpirationDays}: {BootstrapComplete: true},
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{PriorStates: prior})
+	for _, a := range snap.actions {
+		if a.Mode != ModeDisabled || a.IsActive() {
+			t.Fatalf("disabled rule must be ModeDisabled and inactive")
+		}
+	}
+}
+
+func TestSnapshot_MarkActiveFlipsRoutingFilter(t *testing.T) {
+	rule := ruleExpDays("r", "x/", 30)
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{})
+	key := s3lifecycle.ActionKey{Bucket: "b1", RuleHash: s3lifecycle.RuleHash(rule), ActionKind: s3lifecycle.ActionKindExpirationDays}
+
+	if snap.actions[key].IsActive() {
+		t.Fatalf("should start inactive")
+	}
+	snap.MarkActive(key)
+	if !snap.actions[key].IsActive() {
+		t.Fatalf("MarkActive should flip the bit")
+	}
+	// MarkActive on a missing key is a no-op (stale callback after rebuild).
+	snap.MarkActive(s3lifecycle.ActionKey{})
+}
+
+func TestCompile_CrossBucketIdenticalRulesDoNotCollide(t *testing.T) {
+	// Two buckets carry rules whose XML — and therefore RuleHash — is
+	// identical. ActionKey must be bucket-scoped so the second bucket's
+	// CompiledAction does not overwrite the first's in snap.actions.
+	rule := ruleExpDays("shared", "x/", 30)
+	rh := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]PriorState{
+		{Bucket: "alpha", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}: {BootstrapComplete: true},
+		{Bucket: "beta", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}:  {BootstrapComplete: true},
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{
+		{Bucket: "alpha", Rules: []*s3lifecycle.Rule{rule}},
+		{Bucket: "beta", Rules: []*s3lifecycle.Rule{rule}},
+	}, CompileOptions{PriorStates: prior})
+
+	if got := len(snap.actions); got != 2 {
+		t.Fatalf("want 2 actions (one per bucket), got %d", got)
+	}
+	alphaKey := s3lifecycle.ActionKey{Bucket: "alpha", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	betaKey := s3lifecycle.ActionKey{Bucket: "beta", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	if snap.actions[alphaKey] == nil || snap.actions[alphaKey].Bucket != "alpha" {
+		t.Fatalf("alpha bucket action missing or wrong bucket")
+	}
+	if snap.actions[betaKey] == nil || snap.actions[betaKey].Bucket != "beta" {
+		t.Fatalf("beta bucket action missing or wrong bucket")
+	}
+}
+
+func TestEngine_SnapshotAtomicSwap(t *testing.T) {
+	e := New()
+	r1 := ruleExpDays("r1", "a/", 1)
+	snap1 := e.Compile([]CompileInput{{Bucket: "b", Rules: []*s3lifecycle.Rule{r1}}}, CompileOptions{})
+	if snap1.SnapshotID() == 0 {
+		t.Fatalf("snapshot id should be > 0")
+	}
+	r2 := ruleExpDays("r2", "b/", 2)
+	snap2 := e.Compile([]CompileInput{{Bucket: "b", Rules: []*s3lifecycle.Rule{r2}}}, CompileOptions{})
+	if snap2.SnapshotID() <= snap1.SnapshotID() {
+		t.Fatalf("snapshot id should be monotonic")
+	}
+	if e.Snapshot() != snap2 {
+		t.Fatalf("Engine.Snapshot should return the latest")
+	}
+}
+
+// TestCompile_DelayGroupsDedupeAcrossBuckets pins the scaling property
+// the dispatcher relies on: originalDelayGroups is keyed by Delay, so N
+// rules sharing the same Days threshold across M buckets collapse into
+// one delay group with N*M action keys — not N*M groups. The dispatch
+// path can then index by delay group rather than per-rule.
+func TestCompile_DelayGroupsDedupeAcrossBuckets(t *testing.T) {
+	const buckets = 100
+	const rulesPerBucket = 5
+	// Five rules per bucket, but only three distinct day values, so two
+	// pairs of rules per bucket share a delay group.
+	dayValues := []int{1, 1, 7, 7, 30}
+	distinctDays := map[int]bool{}
+	for _, d := range dayValues {
+		distinctDays[d] = true
+	}
+
+	inputs := make([]CompileInput, buckets)
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for b := 0; b < buckets; b++ {
+		bucket := "b" + strconv.Itoa(b)
+		rules := make([]*s3lifecycle.Rule, rulesPerBucket)
+		for r, days := range dayValues {
+			rule := ruleExpDays("r"+strconv.Itoa(r), "p"+strconv.Itoa(r)+"/", days)
+			rules[r] = rule
+			prior[s3lifecycle.ActionKey{
+				Bucket:     bucket,
+				RuleHash:   s3lifecycle.RuleHash(rule),
+				ActionKind: s3lifecycle.ActionKindExpirationDays,
+			}] = PriorState{BootstrapComplete: true}
+		}
+		inputs[b] = CompileInput{Bucket: bucket, Rules: rules}
+	}
+
+	e := New()
+	snap := e.Compile(inputs, CompileOptions{PriorStates: prior})
+
+	if got := len(snap.actions); got != buckets*rulesPerBucket {
+		t.Fatalf("want %d actions (no collapse), got %d", buckets*rulesPerBucket, got)
+	}
+	if got := len(snap.originalDelayGroups); got != len(distinctDays) {
+		t.Fatalf("want %d delay groups (one per distinct Days), got %d",
+			len(distinctDays), got)
+	}
+	// Per-group key count: each distinct day appears in `count` rules per
+	// bucket, so the group holds count*buckets keys.
+	wantPerGroup := map[time.Duration]int{}
+	for _, d := range dayValues {
+		wantPerGroup[s3lifecycle.DaysToDuration(d)] += buckets
+	}
+	for delay, want := range wantPerGroup {
+		if got := len(snap.originalDelayGroups[delay]); got != want {
+			t.Fatalf("delay %v: want %d action keys, got %d", delay, want, got)
+		}
+	}
+}

weed/s3api/s3lifecycle/engine/hashes.go

@@ -0,0 +1,205 @@
+package engine
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/binary"
+	"hash"
+	"sort"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+// hashItem pairs a ruleset member with its parent action so the
+// sort+hash helpers below can be type-agnostic. Internal type — every
+// hash callsite collects items into a slice, then hands it off.
+type hashItem struct {
+	key    s3lifecycle.ActionKey
+	action *CompiledAction
+}
+
+// sortHashItems orders items by (RuleHash, ActionKind, Bucket). The
+// composition matches the ActionKey identity model: RuleHash is the
+// primary content-derived identifier, ActionKind disambiguates siblings
+// of one rule, Bucket scopes by bucket so the same XML in two buckets
+// hashes distinctly. Shared between ReplayContentHash and PromotedHash
+// so the two helpers agree on the on-wire ordering — if one drifted,
+// the cursor could see "rule changed" on a no-op snapshot rebuild.
+func sortHashItems(items []hashItem) {
+	sort.Slice(items, func(i, j int) bool {
+		if c := bytes.Compare(items[i].key.RuleHash[:], items[j].key.RuleHash[:]); c != 0 {
+			return c < 0
+		}
+		if items[i].key.ActionKind != items[j].key.ActionKind {
+			return items[i].key.ActionKind < items[j].key.ActionKind
+		}
+		return items[i].key.Bucket < items[j].key.Bucket
+	})
+}
+
+// hashWriter is the small varint-tagged writer the hash helpers use.
+// Each field gets a one-byte tag so a future schema change (new field)
+// can extend the on-wire format without colliding with existing values.
+type hashWriter struct {
+	h      hash.Hash
+	lenbuf [binary.MaxVarintLen64]byte
+}
+
+func newHashWriter() *hashWriter {
+	return &hashWriter{h: sha256.New()}
+}
+
+// writeField writes a length-prefixed byte field under tag.
+func (w *hashWriter) writeField(tag byte, b []byte) {
+	_, _ = w.h.Write([]byte{tag})
+	n := binary.PutUvarint(w.lenbuf[:], uint64(len(b)))
+	_, _ = w.h.Write(w.lenbuf[:n])
+	_, _ = w.h.Write(b)
+}
+
+// writeInt writes a varint-encoded signed integer under tag.
+func (w *hashWriter) writeInt(tag byte, v int64) {
+	_, _ = w.h.Write([]byte{tag})
+	n := binary.PutVarint(w.lenbuf[:], v)
+	_, _ = w.h.Write(w.lenbuf[:n])
+}
+
+func (w *hashWriter) sum() [32]byte {
+	var out [32]byte
+	copy(out[:], w.h.Sum(nil))
+	return out
+}
+
+// ReplayContentHash hashes the content (action kind, predicate, TTL value)
+// of every replay-eligible compiled action in the base snapshot, returning
+// the empty hash when no replay-eligible action exists. The hash is:
+//   - Partition-independent. A retention-driven scan_only promotion does
+//     NOT change this hash; only the dispatch path changes, not the rule
+//     content. (PromotedHash exists to catch partition flips separately.)
+//   - Stable across snapshot reorderings. Actions are pre-sorted by
+//     RuleHash + ActionKind + Bucket so two snapshots with the same rules
+//     compiled in any order hash identically.
+//   - Disabled-rule-aware. ModeDisabled actions are excluded so disabling
+//     a rule changes the hash (it changed the rule set the worker is
+//     scanning under).
+//
+// Used as cursor.RuleSetHash. A mismatch between persisted and current
+// triggers the recovery branch on next daily_run.
+func ReplayContentHash(s *Snapshot) [32]byte {
+	var empty [32]byte
+	if s == nil {
+		return empty
+	}
+	var items []hashItem
+	for k, a := range s.actions {
+		if a == nil || a.Mode == ModeDisabled {
+			continue
+		}
+		if !isReplayKind(k.ActionKind) {
+			continue
+		}
+		items = append(items, hashItem{key: k, action: a})
+	}
+	if len(items) == 0 {
+		return empty
+	}
+	sortHashItems(items)
+
+	w := newHashWriter()
+	for _, it := range items {
+		w.writeField(0x01, []byte(it.key.Bucket))
+		w.writeField(0x02, it.key.RuleHash[:])
+		w.writeInt(0x03, int64(it.key.ActionKind))
+		// RuleHash already covers the predicate (Prefix + FilterTags + size
+		// filters) and per-kind TTLs, so we don't need to re-canonicalise
+		// the *Rule. But we also include the action's effective TTL
+		// directly so that an "effective TTL of 0" (a malformed rule where
+		// the kind doesn't match the populated field) is distinguishable
+		// from a valid one.
+		w.writeInt(0x04, int64(effectiveTTL(it.action)))
+	}
+	return w.sum()
+}
+
+// PromotedHash hashes the set of replay-eligible actions that *would* land
+// in walk (rather than replay) for the given retentionWindow, due to TTL >
+// retentionWindow. Empty hash when no rules are promoted. Takes the SAME
+// retentionWindow value as RulesForShard so the two helpers cannot disagree
+// about partition membership.
+//
+// Detects partition flips in either direction:
+//   - replay → walk (retention dropped): rule appears in this hash but
+//     didn't before.
+//   - walk → replay (retention recovered): rule used to appear here but no
+//     longer does.
+//
+// In both cases the persisted hash differs from the freshly computed one,
+// firing the recovery branch.
+//
+// A mismatch with the persisted PromotedHash triggers recovery even when
+// rule content is unchanged.
+func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte {
+	var empty [32]byte
+	if s == nil {
+		return empty
+	}
+	var items []hashItem
+	for k, a := range s.actions {
+		if a == nil || a.Mode == ModeDisabled {
+			continue
+		}
+		if !isReplayKind(k.ActionKind) {
+			continue
+		}
+		ttl := effectiveTTL(a)
+		// Mirror RulesForShard's partition predicate exactly: a replay
+		// kind lands in walk when ttl is 0 (malformed) or ttl >
+		// retentionWindow. PromotedHash hashes that walk-bound subset.
+		if ttl > 0 && ttl <= retentionWindow {
+			continue
+		}
+		items = append(items, hashItem{key: k, action: a})
+	}
+	if len(items) == 0 {
+		return empty
+	}
+	sortHashItems(items)
+
+	w := newHashWriter()
+	for _, it := range items {
+		w.writeField(0x01, []byte(it.key.Bucket))
+		w.writeField(0x02, it.key.RuleHash[:])
+		w.writeInt(0x03, int64(it.key.ActionKind))
+	}
+	return w.sum()
+}
+
+// MaxEffectiveTTL returns the maximum effective TTL across the *active*
+// replay-eligible actions in s. Returns 0 for a nil snapshot or one with no
+// active replay actions; the caller is expected to be in the empty-replay
+// branch already (per the design's sentinel-cursor logic).
+//
+// "Effective TTL" mirrors the partition predicate in views.go: derived from
+// the rule field that matches the action kind. Walker-only action kinds
+// (ExpirationDate / ExpiredDeleteMarker / NewerNoncurrent) contribute
+// nothing — they're either not replay-eligible or, in a walk view, not
+// active in the replay sense.
+func MaxEffectiveTTL(s *Snapshot) time.Duration {
+	if s == nil {
+		return 0
+	}
+	var max time.Duration
+	for k, a := range s.actions {
+		if a == nil || !a.IsActive() {
+			continue
+		}
+		if !isReplayKind(k.ActionKind) {
+			continue
+		}
+		if ttl := effectiveTTL(a); ttl > max {
+			max = ttl
+		}
+	}
+	return max
+}

weed/s3api/s3lifecycle/engine/hashes_test.go

@@ -0,0 +1,273 @@
+package engine
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestReplayContentHash_EmptyWhenNoReplayActions(t *testing.T) {
+	var empty [32]byte
+	// Nil snapshot.
+	assert.Equal(t, empty, ReplayContentHash(nil))
+
+	// Walker-only rule: ExpirationDate compiles to a single ScanAtDate
+	// action which is excluded from ReplayContentHash.
+	when := time.Now().Add(24 * time.Hour)
+	rule := &s3lifecycle.Rule{
+		ID:             "d",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDate: when,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	assert.Equal(t, empty, ReplayContentHash(snap))
+}
+
+func TestReplayContentHash_PartitionIndependent(t *testing.T) {
+	// Critical invariant: a retention shift that promotes a rule from
+	// replay to walk (or back) MUST NOT change ReplayContentHash. The hash
+	// is over the rule content as it appears in the base snapshot, not
+	// over the partition it ends up in.
+	rule := &s3lifecycle.Rule{
+		ID:             "shift",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	hashWithLargeRetention := ReplayContentHash(snap)
+
+	// Whether retention is 1d (promotes to walk) or 365d (stays in
+	// replay), the base snapshot's rule content didn't move — only its
+	// partition did. ReplayContentHash hashes the base, so this is
+	// trivially constant. We re-call here to pin the contract.
+	_, walkSmall := snap.RulesForShard(0, s3lifecycle.DaysToDuration(1))
+	require.NotNil(t, walkSmall, "precondition: 1d retention promotes 30d rule to walk")
+	replayLarge, _ := snap.RulesForShard(0, s3lifecycle.DaysToDuration(365))
+	require.NotNil(t, replayLarge, "precondition: 365d retention keeps 30d rule in replay")
+
+	hashAgain := ReplayContentHash(snap)
+	assert.Equal(t, hashWithLargeRetention, hashAgain,
+		"ReplayContentHash must be partition-independent")
+}
+
+func TestReplayContentHash_ChangesOnRuleContentEdit(t *testing.T) {
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	snap1 := buildSnapshotForViews(t, "b1", rule)
+	h1 := ReplayContentHash(snap1)
+
+	// Edit the TTL — RuleHash changes, so the action's key changes, and
+	// the content hash must move.
+	rule2 := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 60,
+	}
+	snap2 := buildSnapshotForViews(t, "b1", rule2)
+	h2 := ReplayContentHash(snap2)
+
+	assert.NotEqual(t, h1, h2, "TTL edit must change ReplayContentHash")
+}
+
+func TestReplayContentHash_StableAcrossRuleReorder(t *testing.T) {
+	r1 := &s3lifecycle.Rule{ID: "a", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7, Prefix: "a/"}
+	r2 := &s3lifecycle.Rule{ID: "b", Status: s3lifecycle.StatusEnabled, ExpirationDays: 14, Prefix: "b/"}
+	snapAB := buildSnapshotForViews(t, "b1", r1, r2)
+	snapBA := buildSnapshotForViews(t, "b1", r2, r1)
+	assert.Equal(t, ReplayContentHash(snapAB), ReplayContentHash(snapBA),
+		"reordering rules at compile time must not change ReplayContentHash")
+}
+
+func TestReplayContentHash_ExcludesWalkerOnlyAndDisabled(t *testing.T) {
+	// Walker-only kinds and disabled rules don't affect ReplayContentHash;
+	// only replay-eligible action kinds count.
+	replayRule := &s3lifecycle.Rule{
+		ID:             "rep",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 7,
+	}
+	snapOnlyReplay := buildSnapshotForViews(t, "b1", replayRule)
+	baseline := ReplayContentHash(snapOnlyReplay)
+
+	walkerRule := &s3lifecycle.Rule{
+		ID:                        "walker",
+		Status:                    s3lifecycle.StatusEnabled,
+		ExpiredObjectDeleteMarker: true,
+	}
+	disabledRule := &s3lifecycle.Rule{
+		ID:             "off",
+		Status:         s3lifecycle.StatusDisabled,
+		ExpirationDays: 90,
+	}
+	snapMixed := buildSnapshotForViews(t, "b1", replayRule, walkerRule, disabledRule)
+	mixed := ReplayContentHash(snapMixed)
+
+	assert.Equal(t, baseline, mixed,
+		"adding walker-only and disabled rules must not change ReplayContentHash")
+}
+
+func TestPromotedHash_EmptyWhenNoneAreInWalk(t *testing.T) {
+	var empty [32]byte
+	assert.Equal(t, empty, PromotedHash(nil, 0))
+
+	rule := &s3lifecycle.Rule{
+		ID:             "fits",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 7,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	// Large retention → rule stays in replay → no promotion.
+	assert.Equal(t, empty, PromotedHash(snap, s3lifecycle.DaysToDuration(365)))
+}
+
+func TestPromotedHash_ChangesOnReplayToWalkPromotion(t *testing.T) {
+	// Retention drops below the rule's TTL → rule promotes from replay to
+	// walk → PromotedHash changes from empty to non-empty.
+	rule := &s3lifecycle.Rule{
+		ID:             "long",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+
+	before := PromotedHash(snap, s3lifecycle.DaysToDuration(365)) // replay
+	after := PromotedHash(snap, s3lifecycle.DaysToDuration(7))    // promoted to walk
+
+	var empty [32]byte
+	assert.Equal(t, empty, before)
+	assert.NotEqual(t, empty, after)
+	assert.NotEqual(t, before, after, "promotion must change PromotedHash")
+}
+
+func TestPromotedHash_ChangesOnWalkToReplayDemotion(t *testing.T) {
+	// Retention recovers from < TTL to >= TTL → rule demotes from walk to
+	// replay → PromotedHash changes from non-empty to empty.
+	rule := &s3lifecycle.Rule{
+		ID:             "demo",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+
+	before := PromotedHash(snap, s3lifecycle.DaysToDuration(7))  // walk
+	after := PromotedHash(snap, s3lifecycle.DaysToDuration(365)) // demoted back to replay
+
+	var empty [32]byte
+	assert.NotEqual(t, empty, before)
+	assert.Equal(t, empty, after)
+	assert.NotEqual(t, before, after, "demotion must change PromotedHash")
+}
+
+func TestPromotedHash_StableWhenContentUnchangedAndPartitionStays(t *testing.T) {
+	// Same snapshot + same retentionWindow → same PromotedHash.
+	rule := &s3lifecycle.Rule{
+		ID:             "stable",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	rw := s3lifecycle.DaysToDuration(7) // promoted
+	assert.Equal(t, PromotedHash(snap, rw), PromotedHash(snap, rw))
+}
+
+func TestPromotedHash_MatchesRulesForShardWalkMembership(t *testing.T) {
+	// PromotedHash must agree with RulesForShard about which replay-
+	// eligible actions are in walk for the same retentionWindow.
+	rules := []*s3lifecycle.Rule{
+		{ID: "short", Status: s3lifecycle.StatusEnabled, ExpirationDays: 1},
+		{ID: "long", Status: s3lifecycle.StatusEnabled, ExpirationDays: 90},
+		{ID: "noncurrent", Status: s3lifecycle.StatusEnabled, NoncurrentVersionExpirationDays: 200},
+	}
+	snap := buildSnapshotForViews(t, "b1", rules...)
+	rw := s3lifecycle.DaysToDuration(30)
+
+	replay, walk := snap.RulesForShard(0, rw)
+	require.NotNil(t, replay)
+	require.NotNil(t, walk)
+
+	// Expected walk members (from replay-eligible kinds only): long, noncurrent.
+	wantPromoted := map[s3lifecycle.ActionKey]struct{}{}
+	for k := range walk.actions {
+		if isReplayKind(k.ActionKind) {
+			wantPromoted[k] = struct{}{}
+		}
+	}
+	assert.Len(t, wantPromoted, 2, "two of three replay-eligible rules should promote")
+
+	hash := PromotedHash(snap, rw)
+	var empty [32]byte
+	assert.NotEqual(t, empty, hash)
+}
+
+func TestMaxEffectiveTTL_NilAndEmpty(t *testing.T) {
+	assert.Equal(t, time.Duration(0), MaxEffectiveTTL(nil))
+
+	// Snapshot with only walker-only kinds → no active replay action →
+	// MaxEffectiveTTL returns 0.
+	rule := &s3lifecycle.Rule{
+		ID:                        "walk",
+		Status:                    s3lifecycle.StatusEnabled,
+		ExpiredObjectDeleteMarker: true,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	assert.Equal(t, time.Duration(0), MaxEffectiveTTL(snap))
+}
+
+func TestMaxEffectiveTTL_ReturnsMaxAcrossActiveReplay(t *testing.T) {
+	// Three replay-eligible actions with mixed TTLs; MaxEffectiveTTL picks
+	// the largest.
+	rule := &s3lifecycle.Rule{
+		ID:                              "mix",
+		Status:                          s3lifecycle.StatusEnabled,
+		ExpirationDays:                  10,
+		NoncurrentVersionExpirationDays: 60,
+		AbortMPUDaysAfterInitiation:     3,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	got := MaxEffectiveTTL(snap)
+	assert.Equal(t, s3lifecycle.DaysToDuration(60), got)
+}
+
+func TestMaxEffectiveTTL_OperatesOnReplayView(t *testing.T) {
+	// The caller passes the *replay* snapshot here in production. Verify
+	// that on a replay view the answer is the max over the view's
+	// (active) replay actions and excludes anything that was routed away
+	// to walk via scan_only promotion.
+	rule := &s3lifecycle.Rule{
+		ID:                              "mix",
+		Status:                          s3lifecycle.StatusEnabled,
+		ExpirationDays:                  10,
+		NoncurrentVersionExpirationDays: 200, // would promote at small retention
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	// Retention 30d: ExpirationDays(10) stays in replay; NoncurrentDays(200)
+	// promotes to walk. The replay view's MaxEffectiveTTL is therefore 10d.
+	replay, _ := snap.RulesForShard(0, s3lifecycle.DaysToDuration(30))
+	require.NotNil(t, replay)
+	assert.Equal(t, s3lifecycle.DaysToDuration(10), MaxEffectiveTTL(replay))
+}
+
+func TestMaxEffectiveTTL_IgnoresInactiveActions(t *testing.T) {
+	// Pre-BootstrapComplete event-driven action is in the snapshot but
+	// inactive. MaxEffectiveTTL only counts active actions.
+	rule := &s3lifecycle.Rule{
+		ID:             "pending",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{}, // no PriorState → inactive
+	)
+	for _, a := range snap.actions {
+		require.False(t, a.IsActive(), "precondition")
+	}
+	assert.Equal(t, time.Duration(0), MaxEffectiveTTL(snap),
+		"inactive replay actions must not contribute to MaxEffectiveTTL")
+}

weed/s3api/s3lifecycle/engine/match.go

@@ -0,0 +1,134 @@
+package engine
+
+import (
+	"strings"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+type EventShape int
+
+const (
+	EventShapeUnknown EventShape = iota
+	// EventShapeOriginalWrite: any event resetting AWS LastModified
+	// (fresh PUT, overwrite, MPU init, version flip).
+	EventShapeOriginalWrite
+	// EventShapePredicateChange: tags / Extended changed without an mtime
+	// reset.
+	EventShapePredicateChange
+)
+
+// Event is the routing-relevant slice of a meta-log event. Kept minimal so
+// the engine doesn't depend on filer_pb.
+type Event struct {
+	Shape          EventShape
+	Bucket         string
+	Path           string
+	Tags           map[string]string
+	Size           int64
+	IsLatest       bool
+	IsDeleteMarker bool
+	IsMPUInit      bool
+	EventTime      time.Time
+}
+
+// MatchOriginalWrite returns active ActionKeys in the given delay group
+// whose filter matches the event.
+func (s *Snapshot) MatchOriginalWrite(ev *Event, delay time.Duration) []s3lifecycle.ActionKey {
+	if ev == nil || ev.Shape != EventShapeOriginalWrite {
+		return nil
+	}
+	keys := s.originalDelayGroups[delay]
+	if len(keys) == 0 {
+		return nil
+	}
+	return s.filterMatching(keys, ev)
+}
+
+// MatchPredicateChange returns active predicate-sensitive ActionKeys whose
+// filter matches the event.
+func (s *Snapshot) MatchPredicateChange(ev *Event) []s3lifecycle.ActionKey {
+	if ev == nil || ev.Shape != EventShapePredicateChange {
+		return nil
+	}
+	if len(s.predicateActions) == 0 {
+		return nil
+	}
+	return s.filterMatching(s.predicateActions, ev)
+}
+
+// MatchPath returns active ActionKeys for a specific bucket+path. ev=nil
+// applies prefix matching only; pass a non-nil Event to also gate on tags
+// and size.
+func (s *Snapshot) MatchPath(bucket, path string, ev *Event) []s3lifecycle.ActionKey {
+	bi := s.buckets[bucket]
+	if bi == nil {
+		return nil
+	}
+	out := make([]s3lifecycle.ActionKey, 0, len(bi.actionKeys))
+	for _, k := range bi.actionKeys {
+		a := s.actions[k]
+		if a == nil || !a.IsActive() {
+			continue
+		}
+		if !prefixMatches(a.Rule.Prefix, path) {
+			continue
+		}
+		if ev != nil && !filterAllows(a.Rule, ev) {
+			continue
+		}
+		out = append(out, k)
+	}
+	return out
+}
+
+func (s *Snapshot) filterMatching(keys []s3lifecycle.ActionKey, ev *Event) []s3lifecycle.ActionKey {
+	out := make([]s3lifecycle.ActionKey, 0, len(keys))
+	for _, k := range keys {
+		a := s.actions[k]
+		if a == nil || !a.IsActive() || a.Bucket != ev.Bucket {
+			continue
+		}
+		if !prefixMatches(a.Rule.Prefix, ev.Path) {
+			continue
+		}
+		if !filterAllows(a.Rule, ev) {
+			continue
+		}
+		switch k.ActionKind {
+		case s3lifecycle.ActionKindAbortMPU:
+			if !ev.IsMPUInit {
+				continue
+			}
+		case s3lifecycle.ActionKindExpiredDeleteMarker:
+			if !ev.IsDeleteMarker || !ev.IsLatest {
+				continue
+			}
+		}
+		out = append(out, k)
+	}
+	return out
+}
+
+func prefixMatches(rulePrefix, path string) bool {
+	if rulePrefix == "" {
+		return true
+	}
+	return strings.HasPrefix(path, rulePrefix)
+}
+
+func filterAllows(rule *s3lifecycle.Rule, ev *Event) bool {
+	if rule.FilterSizeGreaterThan > 0 && ev.Size <= rule.FilterSizeGreaterThan {
+		return false
+	}
+	if rule.FilterSizeLessThan > 0 && ev.Size >= rule.FilterSizeLessThan {
+		return false
+	}
+	for k, v := range rule.FilterTags {
+		if got, ok := ev.Tags[k]; !ok || got != v {
+			return false
+		}
+	}
+	return true
+}

weed/s3api/s3lifecycle/engine/match_helpers_test.go

@@ -0,0 +1,321 @@
+package engine
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// prefixMatches and filterAllows are pure helpers exercised indirectly
+// through MatchOriginalWrite / MatchPath; pinning them directly catches
+// a regression at the helper level so a Match-test failure isn't the
+// first signal of a broken filter.
+
+func TestPrefixMatches_EmptyPrefixAllowsEverything(t *testing.T) {
+	// An empty prefix on a rule means "match every key in the bucket"
+	// — the no-op fast path that skips the HasPrefix call.
+	assert.True(t, prefixMatches("", "logs/2026/01/01"))
+	assert.True(t, prefixMatches("", ""))
+	assert.True(t, prefixMatches("", "/"))
+}
+
+func TestPrefixMatches_HonorsExactPrefix(t *testing.T) {
+	assert.True(t, prefixMatches("logs/", "logs/2026/01/01"))
+	assert.True(t, prefixMatches("logs/", "logs/"))
+}
+
+func TestPrefixMatches_RejectsNonMatchingPath(t *testing.T) {
+	assert.False(t, prefixMatches("logs/", "metrics/2026/01/01"))
+	assert.False(t, prefixMatches("logs/", "log/2026/01/01")) // no trailing slash
+}
+
+func TestPrefixMatches_PathShorterThanPrefixIsReject(t *testing.T) {
+	// Path shorter than the rule prefix can't satisfy HasPrefix; the
+	// helper must say no rather than treat it as a partial match.
+	assert.False(t, prefixMatches("logs/2026/", "logs/"))
+}
+
+func TestFilterAllows_NoFiltersAcceptsAnything(t *testing.T) {
+	// A rule without any filter knobs allows every event regardless of
+	// size or tag map. Pins the documented "no filter" fast path.
+	rule := &s3lifecycle.Rule{}
+	assert.True(t, filterAllows(rule, &Event{Size: 0}))
+	assert.True(t, filterAllows(rule, &Event{Size: 1024}))
+	assert.True(t, filterAllows(rule, &Event{Tags: map[string]string{"a": "b"}}))
+}
+
+func TestFilterAllows_SizeGreaterThanGate(t *testing.T) {
+	// FilterSizeGreaterThan is "strictly greater"; events at or below
+	// the threshold are rejected.
+	rule := &s3lifecycle.Rule{FilterSizeGreaterThan: 1024}
+	assert.False(t, filterAllows(rule, &Event{Size: 1024}), "boundary value rejected (gate is >, not >=)")
+	assert.False(t, filterAllows(rule, &Event{Size: 100}))
+	assert.True(t, filterAllows(rule, &Event{Size: 1025}))
+}
+
+func TestFilterAllows_SizeLessThanGate(t *testing.T) {
+	// FilterSizeLessThan is "strictly less"; events at or above the
+	// threshold are rejected.
+	rule := &s3lifecycle.Rule{FilterSizeLessThan: 1024}
+	assert.False(t, filterAllows(rule, &Event{Size: 1024}), "boundary value rejected (gate is <, not <=)")
+	assert.False(t, filterAllows(rule, &Event{Size: 4096}))
+	assert.True(t, filterAllows(rule, &Event{Size: 1023}))
+}
+
+func TestFilterAllows_ZeroSizeFilterIsDisabled(t *testing.T) {
+	// FilterSizeGreaterThan/LessThan = 0 means "not set"; the filter
+	// must let any size through. A regression treating 0 as a real
+	// threshold would reject every event at or above 0 (i.e., all of
+	// them).
+	greater := &s3lifecycle.Rule{FilterSizeGreaterThan: 0}
+	less := &s3lifecycle.Rule{FilterSizeLessThan: 0}
+	for _, size := range []int64{0, 1, 1024, 1 << 30} {
+		assert.True(t, filterAllows(greater, &Event{Size: size}), "Gt 0 must allow size %d", size)
+		assert.True(t, filterAllows(less, &Event{Size: size}), "Lt 0 must allow size %d", size)
+	}
+}
+
+func TestFilterAllows_RequiredTagPresent(t *testing.T) {
+	// A FilterTags entry is "must equal"; the event's Tags map must
+	// have the same key with the same value.
+	rule := &s3lifecycle.Rule{FilterTags: map[string]string{"env": "prod"}}
+	assert.True(t, filterAllows(rule, &Event{Tags: map[string]string{"env": "prod"}}))
+	assert.True(t, filterAllows(rule, &Event{Tags: map[string]string{"env": "prod", "team": "a"}}))
+}
+
+func TestFilterAllows_RequiredTagMissingOrWrong(t *testing.T) {
+	rule := &s3lifecycle.Rule{FilterTags: map[string]string{"env": "prod"}}
+	assert.False(t, filterAllows(rule, &Event{}), "nil tags map rejected")
+	assert.False(t, filterAllows(rule, &Event{Tags: map[string]string{}}), "empty tags rejected")
+	assert.False(t, filterAllows(rule, &Event{Tags: map[string]string{"team": "a"}}), "different key rejected")
+	assert.False(t, filterAllows(rule, &Event{Tags: map[string]string{"env": "stage"}}), "wrong value rejected")
+}
+
+func TestFilterAllows_AllRequiredTagsMustMatch(t *testing.T) {
+	// Multiple FilterTags are AND'd; missing any one rejects.
+	rule := &s3lifecycle.Rule{FilterTags: map[string]string{"env": "prod", "team": "data"}}
+	assert.True(t, filterAllows(rule, &Event{Tags: map[string]string{"env": "prod", "team": "data"}}))
+	assert.False(t, filterAllows(rule, &Event{Tags: map[string]string{"env": "prod"}}), "missing one tag rejected")
+	assert.False(t, filterAllows(rule, &Event{Tags: map[string]string{"team": "data"}}), "missing other tag rejected")
+	assert.False(t, filterAllows(rule, &Event{Tags: map[string]string{"env": "prod", "team": "ops"}}), "wrong value on second tag rejected")
+}
+
+func TestFilterAllows_SizeAndTagAreANDed(t *testing.T) {
+	// A rule with both size and tag filters requires BOTH to pass;
+	// either one failing rejects the event.
+	rule := &s3lifecycle.Rule{
+		FilterSizeGreaterThan: 100,
+		FilterTags:            map[string]string{"env": "prod"},
+	}
+	// Both pass.
+	assert.True(t, filterAllows(rule, &Event{Size: 200, Tags: map[string]string{"env": "prod"}}))
+	// Size fails.
+	assert.False(t, filterAllows(rule, &Event{Size: 50, Tags: map[string]string{"env": "prod"}}))
+	// Tag fails.
+	assert.False(t, filterAllows(rule, &Event{Size: 200, Tags: map[string]string{"env": "stage"}}))
+}
+
+// ---------- MatchOriginalWrite / MatchPredicateChange / MatchPath edge cases ----------
+
+// helperSnapshot constructs a Snapshot via the engine's Compile path so
+// the action map and bucket index are populated correctly. Tests below
+// poke at the public Match functions rather than the internal maps.
+func helperSnapshot(t *testing.T) (*Snapshot, *s3lifecycle.Rule, s3lifecycle.ActionKey) {
+	t.Helper()
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		Prefix:         "logs/",
+		ExpirationDays: 7,
+	}
+	hash := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for _, k := range s3lifecycle.RuleActionKinds(rule) {
+		prior[s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: k}] = PriorState{
+			BootstrapComplete: true,
+			Mode:              ModeEventDriven,
+		}
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{PriorStates: prior},
+	)
+	key := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	return snap, rule, key
+}
+
+func TestMatchOriginalWrite_NilEventReturnsNil(t *testing.T) {
+	// A nil event is treated as "no match"; the routing path should
+	// not panic on a defensive caller.
+	snap, _, _ := helperSnapshot(t)
+	assert.Nil(t, snap.MatchOriginalWrite(nil, 7*24*time.Hour))
+}
+
+func TestMatchOriginalWrite_WrongShapeReturnsNil(t *testing.T) {
+	// Only EventShapeOriginalWrite goes through this path; a
+	// PredicateChange event must not produce delay-group matches.
+	snap, _, _ := helperSnapshot(t)
+	got := snap.MatchOriginalWrite(&Event{Shape: EventShapePredicateChange, Bucket: "bk"}, 7*24*time.Hour)
+	assert.Nil(t, got)
+}
+
+func TestMatchOriginalWrite_DelayMismatchReturnsNil(t *testing.T) {
+	// The action is in the 7-day delay group; asking for a different
+	// delay must produce no matches.
+	snap, _, _ := helperSnapshot(t)
+	got := snap.MatchOriginalWrite(&Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "logs/x"}, 30*24*time.Hour)
+	assert.Empty(t, got)
+}
+
+func TestMatchPredicateChange_NilEventReturnsNil(t *testing.T) {
+	snap, _, _ := helperSnapshot(t)
+	assert.Nil(t, snap.MatchPredicateChange(nil))
+}
+
+func TestMatchPredicateChange_WrongShapeReturnsNil(t *testing.T) {
+	// EventShapeOriginalWrite events go through MatchOriginalWrite; a
+	// caller passing one to MatchPredicateChange gets nothing back.
+	snap, _, _ := helperSnapshot(t)
+	got := snap.MatchPredicateChange(&Event{Shape: EventShapeOriginalWrite, Bucket: "bk"})
+	assert.Nil(t, got)
+}
+
+func TestMatchPredicateChange_NoTagSensitiveActionsReturnsNil(t *testing.T) {
+	// The default helperSnapshot rule has no FilterTags, so it's not
+	// tag-sensitive — predicate changes must yield no matches.
+	snap, _, _ := helperSnapshot(t)
+	got := snap.MatchPredicateChange(&Event{Shape: EventShapePredicateChange, Bucket: "bk", Path: "logs/x"})
+	assert.Nil(t, got, "non-tag-sensitive snapshot returns nil from predicate change")
+}
+
+func TestMatchPath_UnknownBucketReturnsNil(t *testing.T) {
+	snap, _, _ := helperSnapshot(t)
+	got := snap.MatchPath("other-bucket", "logs/x", nil)
+	assert.Nil(t, got)
+}
+
+func TestMatchPath_NilEventStillAppliesPrefix(t *testing.T) {
+	// MatchPath with ev=nil applies prefix matching only; tags/size
+	// don't gate. Pinned because the bootstrap walker calls it that
+	// way.
+	snap, _, key := helperSnapshot(t)
+	matches := snap.MatchPath("bk", "logs/2026/01/01", nil)
+	assert.Contains(t, matches, key)
+	assert.Empty(t, snap.MatchPath("bk", "metrics/x", nil), "prefix mismatch must reject even with nil event")
+}
+
+func TestMatchPath_FilterRespectedWhenEventProvided(t *testing.T) {
+	// When the caller supplies an Event, both prefix and filter gates
+	// fire. A size-filter rule rejects an event below the threshold.
+	rule := &s3lifecycle.Rule{
+		ID:                    "r",
+		Status:                s3lifecycle.StatusEnabled,
+		ExpirationDays:        7,
+		FilterSizeGreaterThan: 1000,
+	}
+	hash := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for _, k := range s3lifecycle.RuleActionKinds(rule) {
+		prior[s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: k}] = PriorState{
+			BootstrapComplete: true,
+			Mode:              ModeEventDriven,
+		}
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{PriorStates: prior},
+	)
+	belowThreshold := &Event{Size: 500}
+	aboveThreshold := &Event{Size: 2000}
+	assert.Empty(t, snap.MatchPath("bk", "x", belowThreshold), "size below threshold must reject")
+	assert.NotEmpty(t, snap.MatchPath("bk", "x", aboveThreshold), "size above threshold must allow")
+}
+
+// ---------- filterMatching direct coverage ----------
+
+func TestFilterMatching_BucketScoping(t *testing.T) {
+	// Two buckets sharing identical rules: filterMatching must reject
+	// keys whose action.Bucket doesn't match ev.Bucket so a match in
+	// "alpha" doesn't bleed into "beta".
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	hash := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for _, b := range []string{"alpha", "beta"} {
+		for _, k := range s3lifecycle.RuleActionKinds(rule) {
+			prior[s3lifecycle.ActionKey{Bucket: b, RuleHash: hash, ActionKind: k}] = PriorState{
+				BootstrapComplete: true,
+				Mode:              ModeEventDriven,
+			}
+		}
+	}
+	snap := New().Compile(
+		[]CompileInput{
+			{Bucket: "alpha", Rules: []*s3lifecycle.Rule{rule}},
+			{Bucket: "beta", Rules: []*s3lifecycle.Rule{rule}},
+		},
+		CompileOptions{PriorStates: prior},
+	)
+	got := snap.MatchOriginalWrite(&Event{
+		Shape: EventShapeOriginalWrite, Bucket: "alpha", Path: "x", Size: 1,
+	}, 7*24*time.Hour)
+	require.NotEmpty(t, got)
+	for _, k := range got {
+		assert.Equal(t, "alpha", k.Bucket, "match leaked across bucket")
+	}
+}
+
+func TestFilterMatching_AbortMPURequiresMPUInit(t *testing.T) {
+	// AbortMPU actions only fire on MPU-init events; a normal write to
+	// the same prefix must not trigger them.
+	rule := &s3lifecycle.Rule{
+		ID:                          "r",
+		Status:                      s3lifecycle.StatusEnabled,
+		AbortMPUDaysAfterInitiation: 7,
+	}
+	hash := s3lifecycle.RuleHash(rule)
+	abortKey := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindAbortMPU}
+	prior := map[s3lifecycle.ActionKey]PriorState{
+		abortKey: {BootstrapComplete: true, Mode: ModeEventDriven},
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{PriorStates: prior},
+	)
+	regular := &Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "x", IsMPUInit: false}
+	mpu := &Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "x", IsMPUInit: true}
+	assert.Empty(t, snap.MatchOriginalWrite(regular, 7*24*time.Hour), "regular write must not trigger AbortMPU")
+	assert.Contains(t, snap.MatchOriginalWrite(mpu, 7*24*time.Hour), abortKey, "MPU init must trigger AbortMPU")
+}
+
+func TestFilterMatching_ExpiredDeleteMarkerRequiresMarkerLatest(t *testing.T) {
+	// EXPIRED_DELETE_MARKER actions only fire when the event is both a
+	// delete marker AND the current latest version. A non-delete event
+	// or a noncurrent delete marker must not trigger.
+	rule := &s3lifecycle.Rule{
+		ID:                        "r",
+		Status:                    s3lifecycle.StatusEnabled,
+		ExpiredObjectDeleteMarker: true,
+	}
+	hash := s3lifecycle.RuleHash(rule)
+	markerKey := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindExpiredDeleteMarker}
+	prior := map[s3lifecycle.ActionKey]PriorState{
+		markerKey: {BootstrapComplete: true, Mode: ModeEventDriven},
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{PriorStates: prior},
+	)
+	// ExpiredDeleteMarker has MinTriggerAge=0; the action lives in the
+	// zero-duration delay group, so the dispatcher polls it on every
+	// tick.
+	delay := time.Duration(0)
+	notMarker := &Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "x", IsDeleteMarker: false, IsLatest: true}
+	notLatest := &Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "x", IsDeleteMarker: true, IsLatest: false}
+	bothTrue := &Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "x", IsDeleteMarker: true, IsLatest: true}
+	assert.Empty(t, snap.MatchOriginalWrite(notMarker, delay), "non-marker must not trigger")
+	assert.Empty(t, snap.MatchOriginalWrite(notLatest, delay), "noncurrent marker must not trigger")
+	assert.Contains(t, snap.MatchOriginalWrite(bothTrue, delay), markerKey, "marker that is also latest must trigger")
+}

weed/s3api/s3lifecycle/engine/match_test.go

@@ -0,0 +1,194 @@
+package engine
+
+import (
+	"reflect"
+	"sort"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+// activeAll returns a PriorStates map that activates every action of the
+// given canonical rules under bucket.
+func activeAll(bucket string, rules []*s3lifecycle.Rule) map[s3lifecycle.ActionKey]PriorState {
+	out := map[s3lifecycle.ActionKey]PriorState{}
+	for _, r := range rules {
+		rh := s3lifecycle.RuleHash(r)
+		for _, k := range s3lifecycle.RuleActionKinds(r) {
+			out[s3lifecycle.ActionKey{Bucket: bucket, RuleHash: rh, ActionKind: k}] = PriorState{BootstrapComplete: true}
+		}
+	}
+	return out
+}
+
+func sortedKeys(keys []s3lifecycle.ActionKey) []s3lifecycle.ActionKey {
+	out := append([]s3lifecycle.ActionKey(nil), keys...)
+	sort.Slice(out, func(i, j int) bool {
+		return out[i].ActionKind < out[j].ActionKind
+	})
+	return out
+}
+
+func TestMatchOriginalWrite_DelayGroupRoutes(t *testing.T) {
+	r30 := ruleExpDays("a", "x/", 30)
+	r60 := ruleExpDays("b", "x/", 60)
+	rules := []*s3lifecycle.Rule{r30, r60}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "bk", Rules: rules}}, CompileOptions{PriorStates: activeAll("bk", rules)})
+
+	ev := &Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "x/o"}
+	got30 := snap.MatchOriginalWrite(ev, s3lifecycle.DaysToDuration(30))
+	if len(got30) != 1 || got30[0].ActionKind != s3lifecycle.ActionKindExpirationDays {
+		t.Fatalf("30d sweep should match r30, got %v", got30)
+	}
+	got60 := snap.MatchOriginalWrite(ev, s3lifecycle.DaysToDuration(60))
+	if len(got60) != 1 {
+		t.Fatalf("60d sweep should match r60, got %v", got60)
+	}
+}
+
+func TestMatchOriginalWrite_PrefixFilter(t *testing.T) {
+	r := ruleExpDays("a", "logs/", 30)
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{r}}}, CompileOptions{PriorStates: activeAll("bk", []*s3lifecycle.Rule{r})})
+
+	if got := snap.MatchOriginalWrite(&Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "data/x"}, s3lifecycle.DaysToDuration(30)); len(got) != 0 {
+		t.Fatalf("non-matching prefix should reject, got %v", got)
+	}
+	if got := snap.MatchOriginalWrite(&Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "logs/x"}, s3lifecycle.DaysToDuration(30)); len(got) != 1 {
+		t.Fatalf("matching prefix should fire, got %v", got)
+	}
+}
+
+func TestMatchOriginalWrite_MarkActiveBecomesRoutable(t *testing.T) {
+	// pending_bootstrap actions are indexed but inactive: MatchOriginalWrite
+	// returns nothing. After MarkActive flips engineState, the same key is
+	// routable in subsequent matches without a recompile.
+	r := ruleExpDays("a", "x/", 30)
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{r}}}, CompileOptions{})
+	ev := &Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "x/o"}
+	if got := snap.MatchOriginalWrite(ev, s3lifecycle.DaysToDuration(30)); len(got) != 0 {
+		t.Fatalf("inactive action should not match, got %v", got)
+	}
+	snap.MarkActive(s3lifecycle.ActionKey{Bucket: "bk", RuleHash: s3lifecycle.RuleHash(r), ActionKind: s3lifecycle.ActionKindExpirationDays})
+	if got := snap.MatchOriginalWrite(ev, s3lifecycle.DaysToDuration(30)); len(got) != 1 {
+		t.Fatalf("post-markActive should be routable, got %v", got)
+	}
+}
+
+func TestMatchOriginalWrite_AbortMPUOnlyOnMPUInit(t *testing.T) {
+	r := &s3lifecycle.Rule{
+		ID:                          "mpu",
+		Status:                      s3lifecycle.StatusEnabled,
+		Prefix:                      ".uploads/",
+		AbortMPUDaysAfterInitiation: 7,
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{r}}}, CompileOptions{PriorStates: activeAll("bk", []*s3lifecycle.Rule{r})})
+
+	// Non-MPU event under .uploads/ prefix: filtered out by shape gating.
+	got := snap.MatchOriginalWrite(&Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: ".uploads/u1/", IsMPUInit: false}, s3lifecycle.DaysToDuration(7))
+	if len(got) != 0 {
+		t.Fatalf("non-MPU event should be filtered, got %v", got)
+	}
+	got = snap.MatchOriginalWrite(&Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: ".uploads/u1/", IsMPUInit: true}, s3lifecycle.DaysToDuration(7))
+	if len(got) != 1 {
+		t.Fatalf("MPU init should fire, got %v", got)
+	}
+}
+
+func TestMatchPredicateChange_OnlyTagSensitiveActions(t *testing.T) {
+	rTag := &s3lifecycle.Rule{
+		ID:             "tag",
+		Status:         s3lifecycle.StatusEnabled,
+		Prefix:         "x/",
+		ExpirationDays: 30,
+		FilterTags:     map[string]string{"env": "prod"},
+	}
+	rPlain := ruleExpDays("plain", "x/", 30)
+	rules := []*s3lifecycle.Rule{rTag, rPlain}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "bk", Rules: rules}}, CompileOptions{PriorStates: activeAll("bk", rules)})
+
+	// Predicate-change event: should match only the predicate-sensitive rule.
+	ev := &Event{Shape: EventShapePredicateChange, Bucket: "bk", Path: "x/o", Tags: map[string]string{"env": "prod"}}
+	got := snap.MatchPredicateChange(ev)
+	if len(got) != 1 {
+		t.Fatalf("only the tag-sensitive rule should match, got %v", got)
+	}
+	rh := s3lifecycle.RuleHash(rTag)
+	if got[0].RuleHash != rh {
+		t.Fatalf("expected match on tag-rule")
+	}
+
+	// Wrong shape: never matches.
+	if got := snap.MatchPredicateChange(&Event{Shape: EventShapeOriginalWrite, Bucket: "bk", Path: "x/o"}); len(got) != 0 {
+		t.Fatalf("wrong shape should be empty, got %v", got)
+	}
+}
+
+func TestMatchPath_BootstrapWalkerSeesAllActiveActions(t *testing.T) {
+	// One rule with three actions (90d Expiration, 30d NoncurrentDays,
+	// 7d AbortMPU). MatchPath returns every active action whose filter
+	// matches the path; the bootstrap walker iterates these and calls
+	// EvaluateAction per kind for the entry.
+	rule := &s3lifecycle.Rule{
+		ID:                              "multi",
+		Status:                          s3lifecycle.StatusEnabled,
+		Prefix:                          "x/",
+		ExpirationDays:                  90,
+		NoncurrentVersionExpirationDays: 30,
+		AbortMPUDaysAfterInitiation:     7,
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{PriorStates: activeAll("bk", []*s3lifecycle.Rule{rule})})
+
+	got := snap.MatchPath("bk", "x/obj", nil)
+	if len(got) != 3 {
+		t.Fatalf("want all 3 active actions, got %v", got)
+	}
+	wantKinds := []s3lifecycle.ActionKind{
+		s3lifecycle.ActionKindExpirationDays,
+		s3lifecycle.ActionKindNoncurrentDays,
+		s3lifecycle.ActionKindAbortMPU,
+	}
+	gotKinds := []s3lifecycle.ActionKind{}
+	for _, k := range sortedKeys(got) {
+		gotKinds = append(gotKinds, k.ActionKind)
+	}
+	sort.Slice(wantKinds, func(i, j int) bool { return wantKinds[i] < wantKinds[j] })
+	if !reflect.DeepEqual(gotKinds, wantKinds) {
+		t.Fatalf("want kinds %v, got %v", wantKinds, gotKinds)
+	}
+}
+
+func TestMatchPath_PrefixMismatchExcludes(t *testing.T) {
+	rule := ruleExpDays("a", "logs/", 30)
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{PriorStates: activeAll("bk", []*s3lifecycle.Rule{rule})})
+	if got := snap.MatchPath("bk", "data/x", nil); len(got) != 0 {
+		t.Fatalf("prefix mismatch should reject, got %v", got)
+	}
+}
+
+func TestMatchPath_FilterRejectsByTag(t *testing.T) {
+	rule := &s3lifecycle.Rule{
+		ID:             "tag",
+		Status:         s3lifecycle.StatusEnabled,
+		Prefix:         "x/",
+		ExpirationDays: 30,
+		FilterTags:     map[string]string{"env": "prod"},
+	}
+	e := New()
+	snap := e.Compile([]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{PriorStates: activeAll("bk", []*s3lifecycle.Rule{rule})})
+	// With ev passed, tag mismatch rejects.
+	ev := &Event{Bucket: "bk", Path: "x/obj", Tags: map[string]string{"env": "dev"}}
+	if got := snap.MatchPath("bk", "x/obj", ev); len(got) != 0 {
+		t.Fatalf("tag mismatch should reject, got %v", got)
+	}
+	// With ev nil (caller will fetch live state): prefix-only, returns the action.
+	if got := snap.MatchPath("bk", "x/obj", nil); len(got) != 1 {
+		t.Fatalf("ev=nil prefix-only path should return action, got %v", got)
+	}
+}

weed/s3api/s3lifecycle/engine/mode.go

@@ -0,0 +1,57 @@
+package engine
+
+import (
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+// RuleMode mirrors the durable s3_lifecycle_pb.LifecycleState.RuleMode enum;
+// the worker maps between them when it reads/writes durable state.
+type RuleMode int
+
+const (
+	ModeUnspecified RuleMode = iota
+	ModeEventDriven
+	ModeScanAtDate
+	ModeScanOnly
+	ModeDisabled
+	ModePendingBootstrap
+)
+
+func (m RuleMode) String() string {
+	switch m {
+	case ModeEventDriven:
+		return "event_driven"
+	case ModeScanAtDate:
+		return "scan_at_date"
+	case ModeScanOnly:
+		return "scan_only"
+	case ModeDisabled:
+		return "disabled"
+	case ModePendingBootstrap:
+		return "pending_bootstrap"
+	default:
+		return "unspecified"
+	}
+}
+
+// decideMode: disabled rule -> DISABLED; EXPIRATION_DATE -> SCAN_AT_DATE;
+// reader-driven kind whose horizon exceeds retention -> SCAN_ONLY; else
+// EVENT_DRIVEN. metaLogRetention=0 means unbounded (default), gate doesn't
+// trip.
+func decideMode(rule *s3lifecycle.Rule, kind s3lifecycle.ActionKind, metaLogRetention, bootstrapLookbackMin time.Duration) RuleMode {
+	if rule == nil || rule.Status != s3lifecycle.StatusEnabled {
+		return ModeDisabled
+	}
+	if kind == s3lifecycle.ActionKindExpirationDate {
+		return ModeScanAtDate
+	}
+	if metaLogRetention > 0 {
+		horizon := s3lifecycle.EventLogHorizon(rule, kind)
+		if horizon > 0 && metaLogRetention < horizon+bootstrapLookbackMin {
+			return ModeScanOnly
+		}
+	}
+	return ModeEventDriven
+}

weed/s3api/s3lifecycle/engine/mode_test.go

@@ -0,0 +1,126 @@
+package engine
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+)
+
+// decideMode is the per-action mode classifier the scheduler uses when
+// no durable Mode exists yet. Compile tests cover most paths
+// indirectly; these direct tests pin every branch so a regression in
+// decideMode itself can't slip through behind a more elaborate Compile
+// test failure.
+
+func TestDecideMode_NilRuleIsDisabled(t *testing.T) {
+	got := decideMode(nil, s3lifecycle.ActionKindExpirationDays, 0, 0)
+	assert.Equal(t, ModeDisabled, got)
+}
+
+func TestDecideMode_DisabledStatusIsDisabled(t *testing.T) {
+	rule := &s3lifecycle.Rule{Status: s3lifecycle.StatusDisabled, ExpirationDays: 30}
+	got := decideMode(rule, s3lifecycle.ActionKindExpirationDays, 0, 0)
+	assert.Equal(t, ModeDisabled, got)
+}
+
+func TestDecideMode_ExpirationDateIsScanAtDate(t *testing.T) {
+	// ExpirationDate is a one-shot wall-clock trigger; no event log
+	// involvement, so retention gates don't apply.
+	rule := &s3lifecycle.Rule{
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDate: time.Now().Add(24 * time.Hour),
+	}
+	got := decideMode(rule, s3lifecycle.ActionKindExpirationDate, 0, 0)
+	assert.Equal(t, ModeScanAtDate, got)
+}
+
+func TestDecideMode_UnboundedRetentionIsEventDriven(t *testing.T) {
+	// metaLogRetention=0 means unbounded; the retention gate doesn't
+	// trip, so any horizon falls back to event-driven.
+	rule := &s3lifecycle.Rule{
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 365,
+	}
+	got := decideMode(rule, s3lifecycle.ActionKindExpirationDays, 0, time.Hour)
+	assert.Equal(t, ModeEventDriven, got)
+}
+
+func TestDecideMode_HorizonWithinRetentionIsEventDriven(t *testing.T) {
+	// retention=30d, horizon=7d, lookback=1m → 7d+1m < 30d → no gate.
+	rule := &s3lifecycle.Rule{
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 7,
+	}
+	got := decideMode(rule, s3lifecycle.ActionKindExpirationDays, 30*24*time.Hour, time.Minute)
+	assert.Equal(t, ModeEventDriven, got)
+}
+
+func TestDecideMode_HorizonExceedsRetentionIsScanOnly(t *testing.T) {
+	// retention=7d, horizon=30d → can't drive from event log alone, so
+	// the action is promoted to scan_only and only fires from the
+	// bootstrap walk.
+	rule := &s3lifecycle.Rule{
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	got := decideMode(rule, s3lifecycle.ActionKindExpirationDays, 7*24*time.Hour, time.Minute)
+	assert.Equal(t, ModeScanOnly, got)
+}
+
+func TestDecideMode_LookbackPushesAcrossThreshold(t *testing.T) {
+	// retention=7d, horizon=7d - the bare horizon fits, but adding the
+	// bootstrapLookbackMin must push it past retention so the gate
+	// trips. Pins that lookback is added to horizon (not retention).
+	rule := &s3lifecycle.Rule{
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 7,
+	}
+	retention := 7 * 24 * time.Hour
+	// horizon already ~= retention; even a tiny lookback should trip.
+	got := decideMode(rule, s3lifecycle.ActionKindExpirationDays, retention, 5*time.Minute)
+	assert.Equal(t, ModeScanOnly, got)
+}
+
+func TestDecideMode_ZeroHorizonNeverGates(t *testing.T) {
+	// EventLogHorizon returns 0 for kinds whose rule fields aren't set
+	// (e.g. an EXPIRATION_DAYS kind on a rule with ExpirationDays=0).
+	// A zero horizon must skip the gate so the action falls through to
+	// event-driven rather than incorrectly promote to scan_only.
+	rule := &s3lifecycle.Rule{
+		Status: s3lifecycle.StatusEnabled,
+		// ExpirationDays unset -> EventLogHorizon returns 0
+	}
+	got := decideMode(rule, s3lifecycle.ActionKindExpirationDays, time.Hour, time.Minute)
+	assert.Equal(t, ModeEventDriven, got)
+}
+
+func TestRuleMode_StringRendersDocumentedNames(t *testing.T) {
+	// Operators read these via metrics labels and admin UI; pin the
+	// exact strings so a rename can't break dashboards.
+	cases := []struct {
+		mode RuleMode
+		want string
+	}{
+		{ModeUnspecified, "unspecified"},
+		{ModeEventDriven, "event_driven"},
+		{ModeScanAtDate, "scan_at_date"},
+		{ModeScanOnly, "scan_only"},
+		{ModeDisabled, "disabled"},
+		{ModePendingBootstrap, "pending_bootstrap"},
+	}
+	for _, c := range cases {
+		t.Run(c.want, func(t *testing.T) {
+			assert.Equal(t, c.want, c.mode.String())
+		})
+	}
+}
+
+func TestRuleMode_StringFallsToUnspecifiedForUnknown(t *testing.T) {
+	// A bogus enum value (e.g., a future RuleMode added without a
+	// String() update) must render as "unspecified" rather than empty
+	// or panic.
+	bogus := RuleMode(999)
+	assert.Equal(t, "unspecified", bogus.String())
+}

weed/s3api/s3lifecycle/engine/snapshot_accessors_test.go

@@ -0,0 +1,201 @@
+package engine
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Direct tests for Snapshot.OriginalDelayGroups / PredicateActions /
+// DateActions accessors and MarkActive/IsActive transitions. The Compile
+// tests exercise the construction path; these tests pin the read-side
+// surface the router and dispatcher reach for at runtime.
+
+func TestSnapshot_OriginalDelayGroupsExposesCompiledGroups(t *testing.T) {
+	// Two action kinds with different delays must land in distinct
+	// delay groups so the dispatcher polls each at the right cadence.
+	rule := &s3lifecycle.Rule{
+		ID:                          "r",
+		Status:                      s3lifecycle.StatusEnabled,
+		ExpirationDays:              7,
+		AbortMPUDaysAfterInitiation: 3,
+	}
+	hash := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for _, k := range s3lifecycle.RuleActionKinds(rule) {
+		prior[s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: k}] = PriorState{
+			BootstrapComplete: true,
+			Mode:              ModeEventDriven,
+		}
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{PriorStates: prior},
+	)
+	groups := snap.OriginalDelayGroups()
+	require.NotNil(t, groups)
+
+	// 7d expiration delay group must contain the specific expiration key.
+	expirationDelay := s3lifecycle.MinTriggerAge(rule, s3lifecycle.ActionKindExpirationDays)
+	expirationKey := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	assert.Contains(t, groups[expirationDelay], expirationKey, "expiration delay group must carry its specific key")
+
+	// 3d abort-mpu delay group is distinct and contains the abort key.
+	abortDelay := s3lifecycle.MinTriggerAge(rule, s3lifecycle.ActionKindAbortMPU)
+	assert.NotEqual(t, expirationDelay, abortDelay)
+	abortKey := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindAbortMPU}
+	assert.Contains(t, groups[abortDelay], abortKey, "abort-mpu delay group must carry its specific key")
+}
+
+func TestSnapshot_OriginalDelayGroupsScanOnlyExcluded(t *testing.T) {
+	// Scan-only actions don't go through originalDelayGroups (they
+	// fire from the bootstrap walk only), so the dispatcher's
+	// MatchOriginalWrite path won't see them.
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	hash := s3lifecycle.RuleHash(rule)
+	scanOnlyKey := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	prior := map[s3lifecycle.ActionKey]PriorState{
+		scanOnlyKey: {BootstrapComplete: true, Mode: ModeScanOnly},
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{PriorStates: prior},
+	)
+	groups := snap.OriginalDelayGroups()
+	for delay, keys := range groups {
+		for _, k := range keys {
+			assert.NotEqual(t, scanOnlyKey, k, "scan-only action leaked into delay group %v", delay)
+		}
+	}
+}
+
+func TestSnapshot_PredicateActionsContainsTagSensitive(t *testing.T) {
+	// A tag-sensitive rule's actions must surface in PredicateActions
+	// so MatchPredicateChange routes them.
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 7,
+		FilterTags:     map[string]string{"env": "prod"},
+	}
+	hash := s3lifecycle.RuleHash(rule)
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for _, k := range s3lifecycle.RuleActionKinds(rule) {
+		prior[s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: k}] = PriorState{
+			BootstrapComplete: true,
+			Mode:              ModeEventDriven,
+		}
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{PriorStates: prior},
+	)
+	predicates := snap.PredicateActions()
+	require.NotEmpty(t, predicates, "tag-sensitive rule must populate predicateActions")
+	// Verify the specific key landed (not just non-empty) so a routing
+	// regression that emits a wrong ActionKey is caught.
+	expectedKey := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	assert.Contains(t, predicates, expectedKey, "predicateActions must carry the rule's specific ActionKey")
+}
+
+func TestSnapshot_PredicateActionsEmptyForNonTagSensitiveRule(t *testing.T) {
+	// A rule without FilterTags is not predicate-sensitive; the
+	// predicate-action list must stay empty so MatchPredicateChange
+	// never routes irrelevant kinds.
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{},
+	)
+	assert.Empty(t, snap.PredicateActions())
+}
+
+func TestSnapshot_DateActionsContainsExpirationDate(t *testing.T) {
+	// EXPIRATION_DATE rules go into dateActions (not delay groups);
+	// the scan-at-date scheduler reads from this map.
+	when := time.Now().Add(24 * time.Hour)
+	rule := &s3lifecycle.Rule{
+		ID:             "r",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDate: when,
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{},
+	)
+	dateActions := snap.DateActions()
+	require.NotEmpty(t, dateActions)
+	for _, dt := range dateActions {
+		assert.Equal(t, when, dt, "DateActions must surface the rule's ExpirationDate verbatim")
+	}
+}
+
+func TestSnapshot_DateActionsEmptyForNonDateRule(t *testing.T) {
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{},
+	)
+	assert.Empty(t, snap.DateActions())
+}
+
+func TestSnapshot_MarkActiveUnknownKeyIsNoOp(t *testing.T) {
+	// MarkActive must silently skip keys that aren't in the snapshot;
+	// the durable bootstrap-complete write can race with a recompile
+	// that drops the rule, and a panic here would crash the worker.
+	snap := New().Compile(nil, CompileOptions{})
+	snap.MarkActive(s3lifecycle.ActionKey{Bucket: "ghost", ActionKind: s3lifecycle.ActionKindExpirationDays})
+}
+
+func TestSnapshot_MarkActiveFlipsCompiledActionToActive(t *testing.T) {
+	// Bootstrap-pending actions land inactive; MarkActive transitions
+	// them to active so the routing filter starts emitting matches.
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	hash := s3lifecycle.RuleHash(rule)
+	key := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	// No prior state -> the action lands inactive (BootstrapComplete=false).
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{},
+	)
+	a := snap.Action(key)
+	require.NotNil(t, a)
+	assert.False(t, a.IsActive(), "fresh action without prior state must be inactive")
+	snap.MarkActive(key)
+	assert.True(t, a.IsActive(), "MarkActive must transition the action to active")
+}
+
+func TestSnapshot_BucketIndexedActionKeysCoverAllKinds(t *testing.T) {
+	// Cross-check that BucketActionKeys lists every kind compiled from
+	// the rule, regardless of mode. The router iterates these for
+	// MatchPath, so a missed kind silently disables prefix-only routing
+	// for that action.
+	rule := &s3lifecycle.Rule{
+		ID:                          "r",
+		Status:                      s3lifecycle.StatusEnabled,
+		ExpirationDays:              7,
+		AbortMPUDaysAfterInitiation: 3,
+		ExpiredObjectDeleteMarker:   true,
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{},
+	)
+	keys := snap.BucketActionKeys("bk")
+	wantKinds := s3lifecycle.RuleActionKinds(rule)
+	require.Len(t, keys, len(wantKinds))
+
+	// Verify the contents, not just the count: every key must carry
+	// the right bucket scope, and the kinds must match RuleActionKinds
+	// element-for-element (in any order). Catches an indexing
+	// regression that produces N keys but with wrong kinds.
+	gotKinds := make([]s3lifecycle.ActionKind, 0, len(keys))
+	for _, k := range keys {
+		assert.Equal(t, "bk", k.Bucket, "key leaked across bucket")
+		gotKinds = append(gotKinds, k.ActionKind)
+	}
+	assert.ElementsMatch(t, wantKinds, gotKinds, "kinds must match RuleActionKinds")
+}

weed/s3api/s3lifecycle/engine/views.go

@@ -0,0 +1,234 @@
+package engine
+
+import (
+	"bytes"
+	"sort"
+	"sync/atomic"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+// currentEngine is the package-level pointer the wiring will eventually
+// populate so daily_run can fetch the live snapshot through CurrentSnapshot.
+// Today the worker/scheduler owns its own *Engine and passes the snapshot
+// explicitly to its consumers; nothing calls SetCurrentEngine yet. Phase 4's
+// daily-replay caller will register the worker's engine here on startup so
+// the package-level helper resolves without threading the *Engine through
+// every layer.
+//
+// TODO(phase4-wiring): hook scheduler/worker startup to call SetCurrentEngine.
+var currentEngine atomic.Pointer[Engine]
+
+// SetCurrentEngine installs e as the package-level engine returned by
+// CurrentSnapshot. Intended for the worker bootstrap path; tests may also
+// call it with a locally-constructed engine. Passing nil clears the pointer.
+func SetCurrentEngine(e *Engine) { currentEngine.Store(e) }
+
+// CurrentSnapshot returns the latest *Snapshot from the package-level engine,
+// or nil if no engine has been registered yet. The caller is responsible for
+// nil-checking the return — a missing engine is treated as "no rules" so the
+// daily-run loop short-circuits cleanly instead of panicking during early
+// startup.
+//
+// TODO(phase4-wiring): SetCurrentEngine must be invoked at worker startup
+// for production callers; until then this returns nil.
+func CurrentSnapshot() *Snapshot {
+	e := currentEngine.Load()
+	if e == nil {
+		return nil
+	}
+	return e.Snapshot()
+}
+
+// cloneAction makes an independent *CompiledAction for a per-view snapshot.
+// The clone shares Rule (immutable) and Key (value) with the base, but its
+// engineState (the active bit) is its own atomic — so flipping active on the
+// clone never reaches the base or any sibling view. Mode is set explicitly by
+// the caller to encode the per-view dispatch contract (replay forces
+// ModeEventDriven so router.Route's gate passes; walk and recovery preserve
+// the base Mode so the walker's "any non-Disabled" gate works as-is).
+func cloneAction(src *CompiledAction, mode RuleMode, active bool) *CompiledAction {
+	if src == nil {
+		return nil
+	}
+	dst := &CompiledAction{
+		Rule:               src.Rule,
+		Bucket:             src.Bucket,
+		Key:                src.Key,
+		Delay:              src.Delay,
+		PredicateSensitive: src.PredicateSensitive,
+		Mode:               mode,
+	}
+	if active {
+		dst.markActive()
+	}
+	return dst
+}
+
+// newView constructs a *Snapshot that holds the supplied per-view action
+// clones but otherwise shares the base's immutable rule / index data by
+// pointer (see DESIGN.md "router.Route integration" — only `active` and
+// `Mode` differ per view; everything else is share-by-pointer).
+//
+// Returns nil if actions is empty: callers consume nil as "this partition
+// has no actions, skip the corresponding dispatch path."
+func newView(base *Snapshot, actions map[s3lifecycle.ActionKey]*CompiledAction) *Snapshot {
+	if len(actions) == 0 {
+		return nil
+	}
+	// allActionsSorted mirrors the base order: same comparator (Bucket,
+	// RuleHash, ActionKind) so downstream iteration order is stable per view.
+	sorted := make([]*CompiledAction, 0, len(actions))
+	for _, a := range actions {
+		sorted = append(sorted, a)
+	}
+	sort.Slice(sorted, func(i, j int) bool {
+		a, b := sorted[i], sorted[j]
+		if a.Bucket != b.Bucket {
+			return a.Bucket < b.Bucket
+		}
+		if c := bytes.Compare(a.Key.RuleHash[:], b.Key.RuleHash[:]); c != 0 {
+			return c < 0
+		}
+		return a.Key.ActionKind < b.Key.ActionKind
+	})
+	return &Snapshot{
+		id:                  base.id,
+		buckets:             base.buckets,
+		actions:             actions,
+		allActionsSorted:    sorted,
+		originalDelayGroups: base.originalDelayGroups,
+		predicateActions:    base.predicateActions,
+		dateActions:         base.dateActions,
+	}
+}
+
+// isReplayKind reports whether ActionKind is a candidate for the replay
+// partition: its trigger derives from a single event's TsNs (or a stamped
+// noncurrent_since), so the meta-log scan can drive dispatch directly.
+// ExpirationDate / ExpiredDeleteMarker / NewerNoncurrent are walker-only
+// because their due time depends on current bucket state, not event age.
+func isReplayKind(k s3lifecycle.ActionKind) bool {
+	switch k {
+	case s3lifecycle.ActionKindExpirationDays,
+		s3lifecycle.ActionKindNoncurrentDays,
+		s3lifecycle.ActionKindAbortMPU:
+		return true
+	}
+	return false
+}
+
+// effectiveTTL is the duration used both for partition membership
+// (TTL ≤ retentionWindow → replay) and MaxEffectiveTTL math (driving the
+// cursor's sliding scan window). Walker-only kinds return 0 so they never
+// contribute to MaxEffectiveTTL and never satisfy the partition gate when
+// retentionWindow is finite.
+func effectiveTTL(a *CompiledAction) time.Duration {
+	if a == nil || a.Rule == nil {
+		return 0
+	}
+	switch a.Key.ActionKind {
+	case s3lifecycle.ActionKindExpirationDays:
+		if a.Rule.ExpirationDays > 0 {
+			return s3lifecycle.DaysToDuration(a.Rule.ExpirationDays)
+		}
+	case s3lifecycle.ActionKindNoncurrentDays:
+		if a.Rule.NoncurrentVersionExpirationDays > 0 {
+			return s3lifecycle.DaysToDuration(a.Rule.NoncurrentVersionExpirationDays)
+		}
+	case s3lifecycle.ActionKindAbortMPU:
+		if a.Rule.AbortMPUDaysAfterInitiation > 0 {
+			return s3lifecycle.DaysToDuration(a.Rule.AbortMPUDaysAfterInitiation)
+		}
+	}
+	return 0
+}
+
+// RulesForShard partitions the base snapshot's compiled actions into a
+// replay view and a walk view.
+//
+// shardID is **reserved for forward-compatibility** and not consumed by
+// this implementation. Every shard sees every rule today because the
+// shard filter runs at the entry-iteration site (meta-log subscription
+// per shard, walker entry-loop ShardID check) rather than at view
+// construction. Keeping the parameter in the signature preserves the
+// API surface for a future move to per-shard rule sets (e.g. when
+// bucket-shard ownership is added) — removing it now would be a
+// breaking change at that future point. See DESIGN.md "open questions
+// → per-shard rule snapshots vs. global."
+//
+// retentionWindow is the only window input the engine sees; the
+// daily_run caller computes it as `now - earliest_available` and
+// passes it in so the partition is stable across one daily_run
+// invocation.
+//
+// Membership:
+//   - replay: clones of ExpirationDays / NoncurrentDays / AbortMPU actions
+//     whose TTL ≤ retentionWindow. active=true, Mode forced to
+//     ModeEventDriven (rehabilitating any prior ModeScanOnly lock-in so
+//     router.Route's gate accepts the clone).
+//   - walk: clones of ExpirationDate / ExpiredDeleteMarker / NewerNoncurrent
+//     actions, plus any replay-eligible action whose TTL exceeds
+//     retentionWindow (the "scan_only promotion" path). active=true, Mode
+//     preserved from the base — the walker only rejects ModeDisabled.
+//
+// Either return value may be nil when its partition has no actions.
+// Disabled actions (Mode == ModeDisabled) are excluded from both views.
+func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot) {
+	_ = shardID
+	if s == nil {
+		return nil, nil
+	}
+	replayActions := map[s3lifecycle.ActionKey]*CompiledAction{}
+	walkActions := map[s3lifecycle.ActionKey]*CompiledAction{}
+	for key, a := range s.actions {
+		if a == nil || a.Mode == ModeDisabled {
+			continue
+		}
+		if isReplayKind(key.ActionKind) {
+			ttl := effectiveTTL(a)
+			// retentionWindow == 0 means "no retention info supplied" — the
+			// caller should treat the partition as walk-only to stay safe,
+			// since promoting every rule into replay without retention
+			// proof would defeat the scan_only protection. ttl == 0 is a
+			// malformed rule (kind says replay but no TTL): also routes to
+			// walk so the walker can decide.
+			if ttl > 0 && ttl <= retentionWindow {
+				replayActions[key] = cloneAction(a, ModeEventDriven, true)
+			} else {
+				walkActions[key] = cloneAction(a, a.Mode, true)
+			}
+			continue
+		}
+		// Walker-only kinds: preserve Mode (ModeScanAtDate for
+		// ExpirationDate, ModeEventDriven for the others) so the walker's
+		// per-rule evaluator runs as today.
+		walkActions[key] = cloneAction(a, a.Mode, true)
+	}
+	return newView(s, replayActions), newView(s, walkActions)
+}
+
+// RecoveryView returns a *Snapshot that contains a clone of every action in
+// the base, with active=true regardless of compile-time IsActive/
+// BootstrapComplete and Mode preserved as-is. The recovery walker uses this
+// view to catch already-due objects across the full rule set on cold start,
+// rule edit, retention loss, or partition flip. Disabled actions stay out —
+// recovery shouldn't resurrect an explicitly disabled rule.
+//
+// Unlike RulesForShard, RecoveryView takes no retentionWindow: every
+// non-disabled action is activated, partition-independent, because recovery
+// must reach rules whose subjects sit outside the replay scan window.
+func RecoveryView(s *Snapshot) *Snapshot {
+	if s == nil {
+		return nil
+	}
+	actions := map[s3lifecycle.ActionKey]*CompiledAction{}
+	for key, a := range s.actions {
+		if a == nil || a.Mode == ModeDisabled {
+			continue
+		}
+		actions[key] = cloneAction(a, a.Mode, true)
+	}
+	return newView(s, actions)
+}

weed/s3api/s3lifecycle/engine/views_test.go

@@ -0,0 +1,325 @@
+package engine
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// buildSnapshotForViews compiles a snapshot with a single bucket containing
+// one rule per supplied factory, marking every action bootstrap_complete so
+// the base actions are active. Tests can then call RulesForShard and assert
+// the per-view clones come out independent.
+func buildSnapshotForViews(t *testing.T, bucket string, rules ...*s3lifecycle.Rule) *Snapshot {
+	t.Helper()
+	prior := map[s3lifecycle.ActionKey]PriorState{}
+	for _, r := range rules {
+		rh := s3lifecycle.RuleHash(r)
+		for _, k := range s3lifecycle.RuleActionKinds(r) {
+			prior[s3lifecycle.ActionKey{Bucket: bucket, RuleHash: rh, ActionKind: k}] = PriorState{
+				BootstrapComplete: true,
+			}
+		}
+	}
+	return New().Compile(
+		[]CompileInput{{Bucket: bucket, Rules: rules}},
+		CompileOptions{PriorStates: prior},
+	)
+}
+
+func TestCurrentSnapshot_NilWhenNoEngineRegistered(t *testing.T) {
+	// Save and restore the package-level pointer so tests stay isolated.
+	prev := currentEngine.Load()
+	t.Cleanup(func() { currentEngine.Store(prev) })
+
+	SetCurrentEngine(nil)
+	assert.Nil(t, CurrentSnapshot())
+}
+
+func TestCurrentSnapshot_ReturnsRegisteredEngineSnapshot(t *testing.T) {
+	prev := currentEngine.Load()
+	t.Cleanup(func() { currentEngine.Store(prev) })
+
+	e := New()
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	snap := e.Compile([]CompileInput{{Bucket: "b", Rules: []*s3lifecycle.Rule{rule}}}, CompileOptions{})
+	SetCurrentEngine(e)
+	assert.Same(t, snap, CurrentSnapshot())
+}
+
+func TestRulesForShard_NilSnapshot(t *testing.T) {
+	var s *Snapshot
+	replay, walk := s.RulesForShard(0, 30*24*time.Hour)
+	assert.Nil(t, replay)
+	assert.Nil(t, walk)
+}
+
+func TestRulesForShard_ReplayPartitionMembership(t *testing.T) {
+	// ExpirationDays / NoncurrentDays / AbortMPU below retentionWindow all
+	// land in replay; the replay map carries clones with active=true and
+	// Mode=ModeEventDriven.
+	rule := &s3lifecycle.Rule{
+		ID:                              "all-replay",
+		Status:                          s3lifecycle.StatusEnabled,
+		ExpirationDays:                  3,
+		NoncurrentVersionExpirationDays: 5,
+		AbortMPUDaysAfterInitiation:     2,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	replay, walk := snap.RulesForShard(0, s3lifecycle.DaysToDuration(30))
+	require.NotNil(t, replay, "all three replay-eligible kinds should land in replay")
+	assert.Nil(t, walk, "no walker kinds, no scan_only promotion, walk should be empty")
+	assert.Len(t, replay.actions, 3)
+	for k, a := range replay.actions {
+		assert.True(t, a.IsActive(), "replay clone must be active for kind %v", k.ActionKind)
+		assert.Equal(t, ModeEventDriven, a.Mode, "replay Mode must be ModeEventDriven for kind %v", k.ActionKind)
+	}
+}
+
+func TestRulesForShard_WalkOnlyKindsRouteToWalk(t *testing.T) {
+	// ExpirationDate, ExpiredObjectDeleteMarker, NewerNoncurrent are walker-
+	// only and must land in walk regardless of retentionWindow.
+	when := time.Now().Add(24 * time.Hour)
+	rule := &s3lifecycle.Rule{
+		ID:                        "walker-mix",
+		Status:                    s3lifecycle.StatusEnabled,
+		ExpirationDate:            when,
+		ExpiredObjectDeleteMarker: true,
+		NewerNoncurrentVersions:   2,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	replay, walk := snap.RulesForShard(0, s3lifecycle.DaysToDuration(30))
+	assert.Nil(t, replay, "walker-only rule has no replay partition")
+	require.NotNil(t, walk)
+	assert.Len(t, walk.actions, 3, "all three walker kinds should land in walk")
+	for _, a := range walk.actions {
+		assert.True(t, a.IsActive())
+	}
+}
+
+func TestRulesForShard_MultiActionXMLRuleSplits(t *testing.T) {
+	// A single XML rule that compiles to ExpirationDays + NewerNoncurrent
+	// must split: ExpirationDays → replay, NewerNoncurrent → walk.
+	rule := &s3lifecycle.Rule{
+		ID:                      "split",
+		Status:                  s3lifecycle.StatusEnabled,
+		ExpirationDays:          3,
+		NewerNoncurrentVersions: 5,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	replay, walk := snap.RulesForShard(0, s3lifecycle.DaysToDuration(30))
+	require.NotNil(t, replay)
+	require.NotNil(t, walk)
+	assert.Len(t, replay.actions, 1)
+	assert.Len(t, walk.actions, 1)
+	for k := range replay.actions {
+		assert.Equal(t, s3lifecycle.ActionKindExpirationDays, k.ActionKind)
+	}
+	for k := range walk.actions {
+		assert.Equal(t, s3lifecycle.ActionKindNewerNoncurrent, k.ActionKind)
+	}
+}
+
+func TestRulesForShard_ScanOnlyPromotionRoutesToWalk(t *testing.T) {
+	// A replay-eligible kind whose TTL exceeds retentionWindow promotes to
+	// walk. The clone's active=true is set; Mode is preserved as-is from
+	// the base (the walker only rejects ModeDisabled).
+	rule := &s3lifecycle.Rule{
+		ID:                          "long",
+		Status:                      s3lifecycle.StatusEnabled,
+		ExpirationDays:              90,
+		AbortMPUDaysAfterInitiation: 1,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	// Retention window of 30 days promotes ExpirationDays(90d) to walk;
+	// AbortMPU(1d) stays in replay.
+	replay, walk := snap.RulesForShard(0, s3lifecycle.DaysToDuration(30))
+	require.NotNil(t, replay)
+	require.NotNil(t, walk)
+	assert.Len(t, replay.actions, 1)
+	assert.Len(t, walk.actions, 1)
+	for k, a := range replay.actions {
+		assert.Equal(t, s3lifecycle.ActionKindAbortMPU, k.ActionKind)
+		assert.Equal(t, ModeEventDriven, a.Mode)
+	}
+	for k := range walk.actions {
+		assert.Equal(t, s3lifecycle.ActionKindExpirationDays, k.ActionKind)
+	}
+}
+
+func TestRulesForShard_ReplayRewritesScanOnlyMode(t *testing.T) {
+	// A base action carrying Mode=ModeScanOnly via PriorState must still
+	// land in replay as ModeEventDriven once retention rehabilitates its
+	// TTL. router.Route gates on ModeEventDriven, so this is the
+	// rehabilitation contract.
+	rule := &s3lifecycle.Rule{
+		ID:             "rehab",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 5,
+	}
+	rh := s3lifecycle.RuleHash(rule)
+	key := s3lifecycle.ActionKey{Bucket: "b1", RuleHash: rh, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{
+			PriorStates: map[s3lifecycle.ActionKey]PriorState{
+				// Persistent ModeScanOnly carried over from a prior compile.
+				key: {BootstrapComplete: true, Mode: ModeScanOnly},
+			},
+		},
+	)
+	// Verify the base really did keep ModeScanOnly (Compile preserves it).
+	require.Equal(t, ModeScanOnly, snap.actions[key].Mode)
+
+	replay, _ := snap.RulesForShard(0, s3lifecycle.DaysToDuration(30))
+	require.NotNil(t, replay)
+	clone := replay.actions[key]
+	require.NotNil(t, clone)
+	assert.Equal(t, ModeEventDriven, clone.Mode, "replay must force ModeEventDriven on the clone")
+	assert.True(t, clone.IsActive())
+}
+
+func TestRulesForShard_DisabledRuleExcludedFromBothViews(t *testing.T) {
+	rule := &s3lifecycle.Rule{ID: "off", Status: s3lifecycle.StatusDisabled, ExpirationDays: 7}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	replay, walk := snap.RulesForShard(0, s3lifecycle.DaysToDuration(30))
+	assert.Nil(t, replay)
+	assert.Nil(t, walk)
+}
+
+func TestRulesForShard_ClonesAreIndependentOfBase(t *testing.T) {
+	// Toggling active on a view clone must never propagate to the base
+	// action — that's the whole point of cloning per view.
+	rule := &s3lifecycle.Rule{ID: "iso", Status: s3lifecycle.StatusEnabled, ExpirationDays: 3}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	key := s3lifecycle.ActionKey{
+		Bucket:     "b1",
+		RuleHash:   s3lifecycle.RuleHash(rule),
+		ActionKind: s3lifecycle.ActionKindExpirationDays,
+	}
+	base := snap.Action(key)
+	require.NotNil(t, base)
+	baseActiveBefore := base.IsActive()
+
+	replay, _ := snap.RulesForShard(0, s3lifecycle.DaysToDuration(30))
+	require.NotNil(t, replay)
+	clone := replay.actions[key]
+	require.NotNil(t, clone)
+	assert.NotSame(t, base, clone, "clone must be a distinct *CompiledAction")
+
+	// Force the clone inactive (write directly to its atomic so we exercise
+	// the active bit boundary, not just markActive's setter).
+	clone.engineState.Store(engineStateInactive)
+	assert.False(t, clone.IsActive())
+	assert.Equal(t, baseActiveBefore, base.IsActive(),
+		"toggling the clone must not perturb the base's active bit")
+}
+
+func TestRulesForShard_ZeroRetentionRoutesAllReplayKindsToWalk(t *testing.T) {
+	// retentionWindow == 0 (no proven retention) must route every replay-
+	// eligible action into walk — the partition is "ttl > 0 && ttl <=
+	// retentionWindow", and 0 fails the upper bound. This is the safe
+	// default before the meta-log horizon is known.
+	rule := &s3lifecycle.Rule{
+		ID:             "any",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 1,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	replay, walk := snap.RulesForShard(0, 0)
+	assert.Nil(t, replay)
+	require.NotNil(t, walk)
+	assert.Len(t, walk.actions, 1)
+}
+
+func TestRecoveryView_ActivatesEveryNonDisabledAction(t *testing.T) {
+	// Base snapshot has no PriorState → event-driven actions land inactive
+	// (pre-BootstrapComplete). Recovery view must clone them with active=
+	// true so the recovery walker emits matches for them.
+	rule := &s3lifecycle.Rule{
+		ID:             "pending",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 7,
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{}, // no PriorStates → action lands inactive
+	)
+	key := s3lifecycle.ActionKey{
+		Bucket:     "b1",
+		RuleHash:   s3lifecycle.RuleHash(rule),
+		ActionKind: s3lifecycle.ActionKindExpirationDays,
+	}
+	require.False(t, snap.actions[key].IsActive(), "precondition: base action must be inactive")
+
+	rv := RecoveryView(snap)
+	require.NotNil(t, rv)
+	clone := rv.actions[key]
+	require.NotNil(t, clone)
+	assert.True(t, clone.IsActive(), "recovery view must force-activate even pre-BootstrapComplete actions")
+	assert.Equal(t, ModeEventDriven, clone.Mode, "recovery preserves base Mode")
+}
+
+func TestRecoveryView_PreservesBaseModeForScanAtDate(t *testing.T) {
+	// ScanAtDate actions are active at compile time (no BootstrapComplete
+	// requirement). Recovery view must keep their Mode as ModeScanAtDate so
+	// the walker's date evaluator runs as expected — a Mode rewrite here
+	// would re-route them through the event-driven gate by mistake.
+	when := time.Now().Add(48 * time.Hour)
+	rule := &s3lifecycle.Rule{
+		ID:             "d",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDate: when,
+	}
+	snap := New().Compile(
+		[]CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}},
+		CompileOptions{},
+	)
+	rv := RecoveryView(snap)
+	require.NotNil(t, rv)
+	require.Len(t, rv.actions, 1)
+	for _, a := range rv.actions {
+		assert.Equal(t, ModeScanAtDate, a.Mode)
+		assert.True(t, a.IsActive())
+	}
+}
+
+func TestRecoveryView_ExcludesDisabledRules(t *testing.T) {
+	rule := &s3lifecycle.Rule{
+		ID:             "off",
+		Status:         s3lifecycle.StatusDisabled,
+		ExpirationDays: 7,
+	}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	rv := RecoveryView(snap)
+	assert.Nil(t, rv, "disabled rules must not surface in recovery view")
+}
+
+func TestRecoveryView_ClonesAreIndependentOfBase(t *testing.T) {
+	rule := &s3lifecycle.Rule{ID: "iso", Status: s3lifecycle.StatusEnabled, ExpirationDays: 3}
+	snap := buildSnapshotForViews(t, "b1", rule)
+	key := s3lifecycle.ActionKey{
+		Bucket:     "b1",
+		RuleHash:   s3lifecycle.RuleHash(rule),
+		ActionKind: s3lifecycle.ActionKindExpirationDays,
+	}
+	base := snap.Action(key)
+	baseActiveBefore := base.IsActive()
+
+	rv := RecoveryView(snap)
+	require.NotNil(t, rv)
+	clone := rv.actions[key]
+	require.NotNil(t, clone)
+	assert.NotSame(t, base, clone)
+
+	clone.engineState.Store(engineStateInactive)
+	assert.Equal(t, baseActiveBefore, base.IsActive(),
+		"flipping the recovery clone must not perturb the base")
+}
+
+func TestRecoveryView_NilSnapshot(t *testing.T) {
+	assert.Nil(t, RecoveryView(nil))
+}

weed/s3api/s3lifecycle/evaluate.go

@@ -0,0 +1,117 @@
+package s3lifecycle
+
+import (
+	"strings"
+	"time"
+)
+
+// EvaluateAction returns whether the (rule, kind) action fires for info at
+// now. A non-current delete marker is just another non-current version under
+// NONCURRENT_DAYS / NEWER_NONCURRENT; only a current sole-survivor marker
+// routes to EXPIRED_DELETE_MARKER.
+func EvaluateAction(rule *Rule, kind ActionKind, info *ObjectInfo, now time.Time) EvalResult {
+	none := EvalResult{Action: ActionNone}
+	if rule == nil || info == nil || rule.Status != StatusEnabled {
+		return none
+	}
+	if !filterMatches(rule, info) {
+		return none
+	}
+	// MPU init records carry IsLatest=false (they are not yet versions);
+	// without this guard NoncurrentDays / NewerNoncurrent fire on them
+	// and the dispatcher BLOCKs because version_id is empty, freezing
+	// the cursor. Only ABORT_MPU is meaningful for an in-flight upload.
+	if info.IsMPUInit && kind != ActionKindAbortMPU {
+		return none
+	}
+
+	switch kind {
+	case ActionKindAbortMPU:
+		if !info.IsMPUInit || rule.AbortMPUDaysAfterInitiation <= 0 {
+			return none
+		}
+		due := info.ModTime.Add(DaysToDuration(rule.AbortMPUDaysAfterInitiation))
+		if now.Before(due) {
+			return none
+		}
+		return EvalResult{Action: ActionAbortMultipartUpload, RuleID: rule.ID}
+
+	case ActionKindExpiredDeleteMarker:
+		if !info.IsLatest || !info.IsDeleteMarker || !rule.ExpiredObjectDeleteMarker {
+			return none
+		}
+		if info.NumVersions != 1 {
+			return none
+		}
+		return EvalResult{Action: ActionExpireDeleteMarker, RuleID: rule.ID}
+
+	case ActionKindExpirationDays:
+		if !info.IsLatest || info.IsDeleteMarker || rule.ExpirationDays <= 0 {
+			return none
+		}
+		due := info.ModTime.Add(DaysToDuration(rule.ExpirationDays))
+		if now.Before(due) {
+			return none
+		}
+		return EvalResult{Action: ActionDeleteObject, RuleID: rule.ID}
+
+	case ActionKindExpirationDate:
+		if !info.IsLatest || info.IsDeleteMarker || rule.ExpirationDate.IsZero() {
+			return none
+		}
+		if now.Before(rule.ExpirationDate) {
+			return none
+		}
+		return EvalResult{Action: ActionDeleteObject, RuleID: rule.ID}
+
+	case ActionKindNoncurrentDays:
+		if info.IsLatest || rule.NoncurrentVersionExpirationDays <= 0 {
+			return none
+		}
+		base := info.SuccessorModTime
+		if base.IsZero() {
+			base = info.ModTime
+		}
+		due := base.Add(DaysToDuration(rule.NoncurrentVersionExpirationDays))
+		if now.Before(due) {
+			return none
+		}
+		// nil index = can't evaluate retention; safety-scan revisits.
+		if rule.NewerNoncurrentVersions > 0 {
+			if info.NoncurrentIndex == nil || *info.NoncurrentIndex < rule.NewerNoncurrentVersions {
+				return none
+			}
+		}
+		return EvalResult{Action: ActionDeleteVersion, RuleID: rule.ID}
+
+	case ActionKindNewerNoncurrent:
+		// Count-only; when paired with NoncurrentDays the rule expands to
+		// NONCURRENT_DAYS instead (RuleActionKinds).
+		if info.IsLatest || rule.NoncurrentVersionExpirationDays > 0 || rule.NewerNoncurrentVersions <= 0 {
+			return none
+		}
+		if info.NoncurrentIndex == nil || *info.NoncurrentIndex < rule.NewerNoncurrentVersions {
+			return none
+		}
+		return EvalResult{Action: ActionDeleteVersion, RuleID: rule.ID}
+	}
+	return none
+}
+
+func filterMatches(rule *Rule, info *ObjectInfo) bool {
+	if rule.Prefix != "" && !strings.HasPrefix(info.Key, rule.Prefix) {
+		return false
+	}
+	if rule.FilterSizeGreaterThan > 0 && info.Size <= rule.FilterSizeGreaterThan {
+		return false
+	}
+	if rule.FilterSizeLessThan > 0 && info.Size >= rule.FilterSizeLessThan {
+		return false
+	}
+	for k, v := range rule.FilterTags {
+		if got, ok := info.Tags[k]; !ok || got != v {
+			return false
+		}
+	}
+	return true
+}

weed/s3api/s3lifecycle/evaluate_test.go

@@ -0,0 +1,327 @@
+package s3lifecycle
+
+import (
+	"testing"
+	"time"
+)
+
+// idx is a small helper for tests that need to express NoncurrentIndex as
+// *int (production callers either compute it or leave it nil for current
+// versions; tests want a one-liner).
+func idx(i int) *int { return &i }
+
+func mustTime(t *testing.T, s string) time.Time {
+	t.Helper()
+	tm, err := time.Parse(time.RFC3339, s)
+	if err != nil {
+		t.Fatalf("parse %s: %v", s, err)
+	}
+	return tm
+}
+
+func TestEvaluateAction_DisabledRuleNeverFires(t *testing.T) {
+	rule := &Rule{Status: StatusDisabled, ExpirationDays: 1}
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mustTime(t, "2024-01-01T00:00:00Z")}
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, mustTime(t, "2030-01-01T00:00:00Z")); got.Action != ActionNone {
+		t.Fatalf("disabled rule should be no-op, got %v", got)
+	}
+}
+
+func TestEvaluateAction_NilInputs(t *testing.T) {
+	if EvaluateAction(nil, ActionKindExpirationDays, &ObjectInfo{}, time.Now()).Action != ActionNone {
+		t.Fatalf("nil rule should be no-op")
+	}
+	if EvaluateAction(&Rule{Status: StatusEnabled}, ActionKindExpirationDays, nil, time.Now()).Action != ActionNone {
+		t.Fatalf("nil info should be no-op")
+	}
+}
+
+func TestEvaluateAction_ExpirationDaysBoundary(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, ID: "r", ExpirationDays: 30}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mod}
+
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, mod.Add(DaysToDuration(30)).Add(-time.Nanosecond)); got.Action != ActionNone {
+		t.Fatalf("not yet due, got %v", got)
+	}
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, mod.Add(DaysToDuration(30))); got.Action != ActionDeleteObject || got.RuleID != "r" {
+		t.Fatalf("at boundary, got %v", got)
+	}
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, mod.Add(DaysToDuration(31))); got.Action != ActionDeleteObject {
+		t.Fatalf("past due, got %v", got)
+	}
+}
+
+func TestEvaluateAction_KindFiltersByDeclaredAction(t *testing.T) {
+	// Asking the wrong kind of a rule that doesn't declare that action returns
+	// ActionNone — even when the object would otherwise match.
+	rule := &Rule{Status: StatusEnabled, ExpirationDays: 30}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mod}
+	now := mod.Add(DaysToDuration(31))
+	if got := EvaluateAction(rule, ActionKindAbortMPU, info, now); got.Action != ActionNone {
+		t.Fatalf("AbortMPU not declared, got %v", got)
+	}
+	if got := EvaluateAction(rule, ActionKindNoncurrentDays, info, now); got.Action != ActionNone {
+		t.Fatalf("NoncurrentDays not declared, got %v", got)
+	}
+}
+
+func TestEvaluateAction_MultiActionRule_SiblingsIndependent(t *testing.T) {
+	// The bug this fixes: a rule with both ExpirationDays=90 and AbortMPU=7
+	// is evaluated per-action; the 7d MPU window not being "due" for an
+	// object event has no effect on the 90d expiration's eligibility.
+	rule := &Rule{Status: StatusEnabled, ExpirationDays: 90, AbortMPUDaysAfterInitiation: 7}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mod}
+
+	// Past the 90d threshold: ExpirationDays fires.
+	now := mod.Add(DaysToDuration(91))
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, now); got.Action != ActionDeleteObject {
+		t.Fatalf("90d action should fire, got %v", got)
+	}
+	// AbortMPU asked against a non-MPU entry: no-op.
+	if got := EvaluateAction(rule, ActionKindAbortMPU, info, now); got.Action != ActionNone {
+		t.Fatalf("AbortMPU on non-MPU entry should be no-op, got %v", got)
+	}
+}
+
+func TestEvaluateAction_ExpirationDate(t *testing.T) {
+	date := mustTime(t, "2025-06-15T00:00:00Z")
+	rule := &Rule{Status: StatusEnabled, ExpirationDate: date}
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mustTime(t, "2024-01-01T00:00:00Z")}
+
+	if got := EvaluateAction(rule, ActionKindExpirationDate, info, date.Add(-time.Second)); got.Action != ActionNone {
+		t.Fatalf("before date, got %v", got)
+	}
+	if got := EvaluateAction(rule, ActionKindExpirationDate, info, date); got.Action != ActionDeleteObject {
+		t.Fatalf("at date, got %v", got)
+	}
+}
+
+func TestEvaluateAction_ExpiredObjectDeleteMarker(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, ExpiredObjectDeleteMarker: true}
+	now := mustTime(t, "2025-01-01T00:00:00Z")
+	mod := mustTime(t, "2024-12-01T00:00:00Z")
+
+	// Current sole-survivor delete marker: fires.
+	info := &ObjectInfo{Key: "a", IsLatest: true, IsDeleteMarker: true, NumVersions: 1, ModTime: mod}
+	if got := EvaluateAction(rule, ActionKindExpiredDeleteMarker, info, now); got.Action != ActionExpireDeleteMarker {
+		t.Fatalf("sole-survivor marker, got %v", got)
+	}
+	// Non-sole-survivor: not eligible for EXPIRED_DELETE_MARKER.
+	info.NumVersions = 2
+	if got := EvaluateAction(rule, ActionKindExpiredDeleteMarker, info, now); got.Action != ActionNone {
+		t.Fatalf("non-sole marker, got %v", got)
+	}
+	// Latest non-marker: not in scope of this kind.
+	info = &ObjectInfo{Key: "a", IsLatest: true, IsDeleteMarker: false, NumVersions: 1, ModTime: mod}
+	if got := EvaluateAction(rule, ActionKindExpiredDeleteMarker, info, now); got.Action != ActionNone {
+		t.Fatalf("non-marker, got %v", got)
+	}
+}
+
+func TestEvaluateAction_NoncurrentDeleteMarkerExpiresViaNoncurrentDays(t *testing.T) {
+	// A non-current delete marker is just a version. NoncurrentVersionExpirationDays
+	// must apply to it.
+	rule := &Rule{Status: StatusEnabled, NoncurrentVersionExpirationDays: 7}
+	successor := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{
+		Key:              "a",
+		IsLatest:         false,
+		IsDeleteMarker:   true,
+		NumVersions:      3,
+		SuccessorModTime: successor,
+		ModTime:          successor.Add(DaysToDuration(-1)),
+	}
+	if got := EvaluateAction(rule, ActionKindNoncurrentDays, info, successor.Add(DaysToDuration(7))); got.Action != ActionDeleteVersion {
+		t.Fatalf("noncurrent delete marker should be eligible under NoncurrentDays, got %v", got)
+	}
+}
+
+func TestEvaluateAction_NoncurrentVersionDays(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, NoncurrentVersionExpirationDays: 30}
+	successor := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{
+		Key:              "a",
+		IsLatest:         false,
+		ModTime:          mustTime(t, "2023-01-01T00:00:00Z"),
+		SuccessorModTime: successor,
+		NoncurrentIndex:  idx(0),
+	}
+	if got := EvaluateAction(rule, ActionKindNoncurrentDays, info, successor.Add(DaysToDuration(29))); got.Action != ActionNone {
+		t.Fatalf("not due, got %v", got)
+	}
+	if got := EvaluateAction(rule, ActionKindNoncurrentDays, info, successor.Add(DaysToDuration(30))); got.Action != ActionDeleteVersion {
+		t.Fatalf("due, got %v", got)
+	}
+}
+
+func TestEvaluateAction_NoncurrentDaysFallsBackToModTime(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, NoncurrentVersionExpirationDays: 30}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: "a", IsLatest: false, ModTime: mod, NoncurrentIndex: idx(0)}
+	if got := EvaluateAction(rule, ActionKindNoncurrentDays, info, mod.Add(DaysToDuration(30))); got.Action != ActionDeleteVersion {
+		t.Fatalf("expected fallback to ModTime, got %v", got)
+	}
+}
+
+func TestEvaluateAction_NoncurrentDays_WithKeepN_NilIndexIsNoOp(t *testing.T) {
+	// Pointer-migration safety: a nil NoncurrentIndex paired with a
+	// keep-N retention threshold must short-circuit to ActionNone rather
+	// than guess at the version's position in the keep window.
+	rule := &Rule{Status: StatusEnabled, NoncurrentVersionExpirationDays: 30, NewerNoncurrentVersions: 2}
+	successor := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{
+		Key:              "a",
+		IsLatest:         false,
+		ModTime:          successor,
+		SuccessorModTime: successor,
+		NoncurrentIndex:  nil,
+	}
+	now := successor.Add(DaysToDuration(30))
+	if got := EvaluateAction(rule, ActionKindNoncurrentDays, info, now); got.Action != ActionNone {
+		t.Fatalf("nil index with keep-N must be no-op, got %v", got)
+	}
+}
+
+func TestEvaluateAction_NewerNoncurrent_NilIndexIsNoOp(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, NewerNoncurrentVersions: 3}
+	info := &ObjectInfo{
+		Key:             "a",
+		IsLatest:        false,
+		ModTime:         mustTime(t, "2024-01-01T00:00:00Z"),
+		NoncurrentIndex: nil,
+	}
+	if got := EvaluateAction(rule, ActionKindNewerNoncurrent, info, mustTime(t, "2025-01-01T00:00:00Z")); got.Action != ActionNone {
+		t.Fatalf("nil index pure-count must be no-op, got %v", got)
+	}
+}
+
+func TestEvaluateAction_NewerNoncurrentCountOnly(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, NewerNoncurrentVersions: 3}
+	now := mustTime(t, "2025-01-01T00:00:00Z")
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+
+	for i := 0; i < 3; i++ {
+		info := &ObjectInfo{Key: "a", IsLatest: false, ModTime: mod, NoncurrentIndex: idx(i)}
+		if got := EvaluateAction(rule, ActionKindNewerNoncurrent, info, now); got.Action != ActionNone {
+			t.Fatalf("idx=%d should be retained, got %v", i, got)
+		}
+	}
+	info := &ObjectInfo{Key: "a", IsLatest: false, ModTime: mod, NoncurrentIndex: idx(3)}
+	if got := EvaluateAction(rule, ActionKindNewerNoncurrent, info, now); got.Action != ActionDeleteVersion {
+		t.Fatalf("idx=3 should fire, got %v", got)
+	}
+}
+
+func TestEvaluateAction_NoncurrentDaysAndCount(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, NoncurrentVersionExpirationDays: 30, NewerNoncurrentVersions: 2}
+	successor := mustTime(t, "2024-01-01T00:00:00Z")
+	now := successor.Add(DaysToDuration(30))
+
+	for i := 0; i < 2; i++ {
+		info := &ObjectInfo{Key: "a", IsLatest: false, ModTime: successor, SuccessorModTime: successor, NoncurrentIndex: idx(i)}
+		if got := EvaluateAction(rule, ActionKindNoncurrentDays, info, now); got.Action != ActionNone {
+			t.Fatalf("idx=%d kept by NewerNoncurrent retention, got %v", i, got)
+		}
+	}
+	info := &ObjectInfo{Key: "a", IsLatest: false, ModTime: successor, SuccessorModTime: successor, NoncurrentIndex: idx(2)}
+	if got := EvaluateAction(rule, ActionKindNoncurrentDays, info, now); got.Action != ActionDeleteVersion {
+		t.Fatalf("idx=2 satisfies both, got %v", got)
+	}
+}
+
+func TestEvaluateAction_AbortMultipartUpload(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, AbortMPUDaysAfterInitiation: 7}
+	init := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: ".uploads/u1/", IsMPUInit: true, ModTime: init}
+	if got := EvaluateAction(rule, ActionKindAbortMPU, info, init.Add(DaysToDuration(6))); got.Action != ActionNone {
+		t.Fatalf("not due, got %v", got)
+	}
+	if got := EvaluateAction(rule, ActionKindAbortMPU, info, init.Add(DaysToDuration(7))); got.Action != ActionAbortMultipartUpload {
+		t.Fatalf("due, got %v", got)
+	}
+}
+
+func TestEvaluateAction_PrefixFilter(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, Prefix: "logs/", ExpirationDays: 1}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.Add(DaysToDuration(10))
+	if got := EvaluateAction(rule, ActionKindExpirationDays, &ObjectInfo{Key: "data/x", IsLatest: true, ModTime: mod}, now); got.Action != ActionNone {
+		t.Fatalf("prefix mismatch should reject, got %v", got)
+	}
+	if got := EvaluateAction(rule, ActionKindExpirationDays, &ObjectInfo{Key: "logs/x", IsLatest: true, ModTime: mod}, now); got.Action != ActionDeleteObject {
+		t.Fatalf("prefix match should fire, got %v", got)
+	}
+}
+
+func TestEvaluateAction_TagFilter(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, ExpirationDays: 1, FilterTags: map[string]string{"env": "prod", "tier": "cold"}}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.Add(DaysToDuration(10))
+
+	info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mod}
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, now); got.Action != ActionNone {
+		t.Fatalf("missing tags should reject, got %v", got)
+	}
+	info.Tags = map[string]string{"env": "prod"}
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, now); got.Action != ActionNone {
+		t.Fatalf("partial match should reject, got %v", got)
+	}
+	info.Tags = map[string]string{"env": "prod", "tier": "cold", "extra": "ignored"}
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, now); got.Action != ActionDeleteObject {
+		t.Fatalf("all tags match should fire, got %v", got)
+	}
+}
+
+func TestEvaluateAction_SizeFilter(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, ExpirationDays: 1, FilterSizeGreaterThan: 100, FilterSizeLessThan: 1000}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	now := mod.Add(DaysToDuration(10))
+	cases := []struct {
+		size int64
+		want Action
+	}{
+		{50, ActionNone},
+		{100, ActionNone},
+		{500, ActionDeleteObject},
+		{1000, ActionNone},
+		{2000, ActionNone},
+	}
+	for _, c := range cases {
+		info := &ObjectInfo{Key: "a", IsLatest: true, ModTime: mod, Size: c.size}
+		if got := EvaluateAction(rule, ActionKindExpirationDays, info, now); got.Action != c.want {
+			t.Fatalf("size=%d want=%v got=%v", c.size, c.want, got)
+		}
+	}
+}
+
+func TestEvaluateAction_EmptyPrefixMatchesAll(t *testing.T) {
+	rule := &Rule{Status: StatusEnabled, ExpirationDays: 1, Prefix: ""}
+	mod := mustTime(t, "2024-01-01T00:00:00Z")
+	info := &ObjectInfo{Key: "deeply/nested/path/x", IsLatest: true, ModTime: mod}
+	if got := EvaluateAction(rule, ActionKindExpirationDays, info, mod.Add(DaysToDuration(10))); got.Action != ActionDeleteObject {
+		t.Fatalf("empty prefix should match, got %v", got)
+	}
+}
+
+func TestEvaluateAction_MPUInitDoesNotFireNoncurrent(t *testing.T) {
+	// IsLatest=false on an MPU init must not let NoncurrentDays /
+	// NewerNoncurrent fire. The dispatcher BLOCKs noncurrent kinds with
+	// empty version_id, which would freeze the cursor.
+	rule := &Rule{
+		ID:                              "r",
+		Status:                          StatusEnabled,
+		NoncurrentVersionExpirationDays: 7,
+		NewerNoncurrentVersions:         3,
+	}
+	init := time.Now().Add(-DaysToDuration(30))
+	info := &ObjectInfo{Key: "logs/foo.txt", IsMPUInit: true, ModTime: init}
+
+	for _, kind := range []ActionKind{ActionKindNoncurrentDays, ActionKindNewerNoncurrent} {
+		if got := EvaluateAction(rule, kind, info, time.Now()); got.Action != ActionNone {
+			t.Fatalf("kind=%v on IsMPUInit must return None, got %+v", kind, got)
+		}
+	}
+}

weed/s3api/s3lifecycle/event_log_horizon.go

@@ -0,0 +1,37 @@
+package s3lifecycle
+
+import "time"
+
+// EventLogHorizon returns the max event age the reader needs to drive the
+// (rule, kind) action. Used by the retention mode gate: when
+// metaLogRetention < EventLogHorizon + bootstrapLookbackMin, the action is
+// promoted to scan_only. EXPIRATION_DATE returns 0 (date kind bypasses);
+// count / immediate kinds return SmallDelay.
+func EventLogHorizon(rule *Rule, kind ActionKind) time.Duration {
+	if rule == nil {
+		return 0
+	}
+	switch kind {
+	case ActionKindExpirationDays:
+		if rule.ExpirationDays > 0 {
+			return DaysToDuration(rule.ExpirationDays)
+		}
+	case ActionKindNoncurrentDays:
+		if rule.NoncurrentVersionExpirationDays > 0 {
+			return DaysToDuration(rule.NoncurrentVersionExpirationDays)
+		}
+	case ActionKindAbortMPU:
+		if rule.AbortMPUDaysAfterInitiation > 0 {
+			return DaysToDuration(rule.AbortMPUDaysAfterInitiation)
+		}
+	case ActionKindNewerNoncurrent:
+		if rule.NewerNoncurrentVersions > 0 && rule.NoncurrentVersionExpirationDays == 0 {
+			return SmallDelay
+		}
+	case ActionKindExpiredDeleteMarker:
+		if rule.ExpiredObjectDeleteMarker {
+			return SmallDelay
+		}
+	}
+	return 0
+}

weed/s3api/s3lifecycle/event_log_horizon_test.go

@@ -0,0 +1,79 @@
+package s3lifecycle
+
+import (
+	"testing"
+	"time"
+)
+
+func TestEventLogHorizon_PerActionIsIndependent(t *testing.T) {
+	// A multi-action rule's horizon is computed per kind: the gate runs on
+	// each compiled action separately, so EXPIRATION_DAYS at 90d does NOT
+	// share a horizon with its sibling ABORT_MPU at 7d.
+	rule := &Rule{ExpirationDays: 90, AbortMPUDaysAfterInitiation: 7, NoncurrentVersionExpirationDays: 30}
+	cases := []struct {
+		kind ActionKind
+		want time.Duration
+	}{
+		{ActionKindExpirationDays, DaysToDuration(90)},
+		{ActionKindAbortMPU, DaysToDuration(7)},
+		{ActionKindNoncurrentDays, DaysToDuration(30)},
+		{ActionKindExpirationDate, 0},
+	}
+	for _, c := range cases {
+		t.Run(c.kind.String(), func(t *testing.T) {
+			if got := EventLogHorizon(rule, c.kind); got != c.want {
+				t.Fatalf("want %v, got %v", c.want, got)
+			}
+		})
+	}
+}
+
+func TestEventLogHorizon_NewerNoncurrentCountOnlyIsSmallDelay(t *testing.T) {
+	rule := &Rule{NewerNoncurrentVersions: 5}
+	if got := EventLogHorizon(rule, ActionKindNewerNoncurrent); got != SmallDelay {
+		t.Fatalf("want SmallDelay, got %v", got)
+	}
+}
+
+func TestEventLogHorizon_NewerNoncurrentWithDaysIsZero(t *testing.T) {
+	// Pure NEWER_NONCURRENT only when NoncurrentDays is unset; otherwise the
+	// rule produces a NONCURRENT_DAYS action and asking for NEWER_NONCURRENT
+	// returns 0 (not declared by this rule).
+	rule := &Rule{NoncurrentVersionExpirationDays: 30, NewerNoncurrentVersions: 5}
+	if got := EventLogHorizon(rule, ActionKindNewerNoncurrent); got != 0 {
+		t.Fatalf("NEWER_NONCURRENT not declared when paired with days, got %v", got)
+	}
+	// The actual action this rule declares is NONCURRENT_DAYS — verify horizon there.
+	if got := EventLogHorizon(rule, ActionKindNoncurrentDays); got != DaysToDuration(30) {
+		t.Fatalf("NoncurrentDays horizon, want 30d, got %v", got)
+	}
+}
+
+func TestEventLogHorizon_ExpiredDeleteMarkerIsSmallDelay(t *testing.T) {
+	rule := &Rule{ExpiredObjectDeleteMarker: true}
+	if got := EventLogHorizon(rule, ActionKindExpiredDeleteMarker); got != SmallDelay {
+		t.Fatalf("want SmallDelay, got %v", got)
+	}
+}
+
+func TestEventLogHorizon_DateKindReturnsZero(t *testing.T) {
+	rule := &Rule{ExpirationDate: mustTime(t, "2025-06-15T00:00:00Z")}
+	if got := EventLogHorizon(rule, ActionKindExpirationDate); got != 0 {
+		t.Fatalf("date kind bypasses gate, got %v", got)
+	}
+}
+
+func TestEventLogHorizon_NilRule(t *testing.T) {
+	if got := EventLogHorizon(nil, ActionKindExpirationDays); got != 0 {
+		t.Fatalf("nil rule should return 0, got %v", got)
+	}
+}
+
+func TestEventLogHorizon_KindNotDeclaredReturnsZero(t *testing.T) {
+	// Asking for a kind the rule doesn't declare returns 0; the gate then
+	// trivially passes for that compiled action (which doesn't exist).
+	rule := &Rule{ExpirationDays: 30}
+	if got := EventLogHorizon(rule, ActionKindAbortMPU); got != 0 {
+		t.Fatalf("want 0 for undeclared kind, got %v", got)
+	}
+}

weed/s3api/s3lifecycle/final_cleanup_test.go

@@ -0,0 +1,48 @@
+package s3lifecycle
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+// Final unit-test cleanup ahead of the integration suite. Pins the
+// remaining 0%-and-default-branch helpers in the s3lifecycle package
+// and the lifecycletest builder so a regression doesn't slip in
+// during the integration work that follows.
+
+func TestActionKind_StringUnspecifiedDefault(t *testing.T) {
+	// String's default branch (covers ActionKindUnspecified and any
+	// future enum value not listed) must render "unspecified" rather
+	// than empty or panic. Operators read this via the metrics labels
+	// (see S3LifecycleDispatchCounter "kind").
+	assert.Equal(t, "unspecified", ActionKindUnspecified.String())
+	assert.Equal(t, "unspecified", ActionKind(99).String())
+}
+
+func TestHashExtended_DirectFromLifecyclePackage(t *testing.T) {
+	// HashExtended is exercised from the s3api package's identity
+	// tests; pin it here so the s3lifecycle package's own coverage
+	// reflects the call. A nil/empty map produces no bytes, so the
+	// CAS witness collapses to "no Extended" rather than a synthetic
+	// hash that would mismatch on the server.
+	assert.Empty(t, HashExtended(nil))
+	assert.Empty(t, HashExtended(map[string][]byte{}))
+	got := HashExtended(map[string][]byte{"a": []byte("1")})
+	assert.NotEmpty(t, got)
+	// Same content, different literal/insertion order across multiple
+	// keys: hash must be stable. A single-key check can't catch an
+	// iteration-order regression — multiple keys force the helper's
+	// sort path to actually do work.
+	first := HashExtended(map[string][]byte{
+		"a": []byte("1"),
+		"b": []byte("2"),
+		"c": []byte("3"),
+	})
+	second := HashExtended(map[string][]byte{
+		"c": []byte("3"),
+		"a": []byte("1"),
+		"b": []byte("2"),
+	})
+	assert.Equal(t, first, second, "hash must be insensitive to map iteration order")
+}

weed/s3api/s3lifecycle/identity.go

@@ -0,0 +1,36 @@
+package s3lifecycle
+
+import (
+	"crypto/sha256"
+	"sort"
+	"strconv"
+)
+
+// HashExtended is the canonical fingerprint of an entry's Extended map for
+// use in lifecycle identity-CAS. Length-prefixed so a forged tag value
+// can't collide with a legitimate multi-tag map. Returns nil for empty.
+//
+// Both sides of the LifecycleDelete CAS — the worker that captures the
+// schedule-time identity and the server that re-fetches the live entry —
+// must call this same function so the bytes match exactly.
+func HashExtended(ext map[string][]byte) []byte {
+	if len(ext) == 0 {
+		return nil
+	}
+	keys := make([]string, 0, len(ext))
+	for k := range ext {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+	h := sha256.New()
+	for _, k := range keys {
+		h.Write([]byte(strconv.Itoa(len(k))))
+		h.Write([]byte{':'})
+		h.Write([]byte(k))
+		v := ext[k]
+		h.Write([]byte(strconv.Itoa(len(v))))
+		h.Write([]byte{':'})
+		h.Write(v)
+	}
+	return h.Sum(nil)
+}

weed/s3api/s3lifecycle/lifecycletest/eventbuilder.go

@@ -0,0 +1,286 @@
+package lifecycletest
+
+import (
+	"sync"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
+)
+
+// EventOption mutates a reader.Event during construction. Options compose
+// in order — later options override earlier ones if they touch the same
+// field, which lets a default+override pattern work without surprises.
+type EventOption func(*reader.Event)
+
+// WithSize sets the Size on whichever entry the event populates (NewEntry
+// for Create / Update, OldEntry for Delete). Constructors that build
+// both entries apply the size to NewEntry.
+func WithSize(bytes int64) EventOption {
+	return func(e *reader.Event) {
+		if e.NewEntry != nil && e.NewEntry.Attributes != nil {
+			e.NewEntry.Attributes.FileSize = uint64(bytes)
+		} else if e.OldEntry != nil && e.OldEntry.Attributes != nil {
+			e.OldEntry.Attributes.FileSize = uint64(bytes)
+		}
+	}
+}
+
+// WithModTime sets Mtime/MtimeNs on the populated entry.
+func WithModTime(t time.Time) EventOption {
+	return func(e *reader.Event) {
+		secs, nanos := t.Unix(), int32(t.Nanosecond())
+		if e.NewEntry != nil && e.NewEntry.Attributes != nil {
+			e.NewEntry.Attributes.Mtime = secs
+			e.NewEntry.Attributes.MtimeNs = nanos
+		} else if e.OldEntry != nil && e.OldEntry.Attributes != nil {
+			e.OldEntry.Attributes.Mtime = secs
+			e.OldEntry.Attributes.MtimeNs = nanos
+		}
+	}
+}
+
+// WithTtlSec sets the TTL on the populated entry. Used by tests that
+// exercise the lifecycle metadata-only delete path (TtlSec > 0 means
+// the volume reclaims chunks naturally).
+func WithTtlSec(ttl int32) EventOption {
+	return func(e *reader.Event) {
+		if e.NewEntry != nil && e.NewEntry.Attributes != nil {
+			e.NewEntry.Attributes.TtlSec = ttl
+		} else if e.OldEntry != nil && e.OldEntry.Attributes != nil {
+			e.OldEntry.Attributes.TtlSec = ttl
+		}
+	}
+}
+
+// WithVersionID stamps Seaweed-X-Amz-Version-Id on the populated entry's
+// Extended map. Used by versioning-aware router and dispatcher tests.
+func WithVersionID(versionID string) EventOption {
+	return func(e *reader.Event) {
+		var entry *filer_pb.Entry
+		if e.NewEntry != nil {
+			entry = e.NewEntry
+		} else if e.OldEntry != nil {
+			entry = e.OldEntry
+		}
+		if entry == nil {
+			return
+		}
+		if entry.Extended == nil {
+			entry.Extended = map[string][]byte{}
+		}
+		entry.Extended[s3_constants.ExtVersionIdKey] = []byte(versionID)
+	}
+}
+
+// WithExtended sets an arbitrary Extended key/value on the populated entry.
+func WithExtended(key string, value []byte) EventOption {
+	return func(e *reader.Event) {
+		var entry *filer_pb.Entry
+		if e.NewEntry != nil {
+			entry = e.NewEntry
+		} else if e.OldEntry != nil {
+			entry = e.OldEntry
+		}
+		if entry == nil {
+			return
+		}
+		if entry.Extended == nil {
+			entry.Extended = map[string][]byte{}
+		}
+		entry.Extended[key] = value
+	}
+}
+
+// WithChunks attaches FileChunks to the populated entry. Identity-CAS in
+// LifecycleDelete uses the head chunk's FID, so tests that exercise CAS
+// drift need at least one chunk. For Update events the chunks land on
+// NewEntry; use WithOldChunks to target the pre-update state.
+func WithChunks(chunks ...*filer_pb.FileChunk) EventOption {
+	return func(e *reader.Event) {
+		if e.NewEntry != nil {
+			e.NewEntry.Chunks = append(e.NewEntry.Chunks, chunks...)
+		} else if e.OldEntry != nil {
+			e.OldEntry.Chunks = append(e.OldEntry.Chunks, chunks...)
+		}
+	}
+}
+
+// WithOldSize sets FileSize on OldEntry specifically. Use on Update
+// events to configure the pre-update size (the WithSize default lands
+// on NewEntry when both are populated).
+func WithOldSize(bytes int64) EventOption {
+	return func(e *reader.Event) {
+		if e.OldEntry != nil && e.OldEntry.Attributes != nil {
+			e.OldEntry.Attributes.FileSize = uint64(bytes)
+		}
+	}
+}
+
+// WithOldChunks attaches FileChunks to OldEntry specifically. Use on
+// Update events to configure the pre-update chunk list (WithChunks
+// targets NewEntry when both are populated).
+func WithOldChunks(chunks ...*filer_pb.FileChunk) EventOption {
+	return func(e *reader.Event) {
+		if e.OldEntry != nil {
+			e.OldEntry.Chunks = append(e.OldEntry.Chunks, chunks...)
+		}
+	}
+}
+
+// WithOldModTime sets Mtime/MtimeNs on OldEntry specifically. Update
+// events whose pre-update mtime should differ from the event timestamp
+// use this to override the default.
+func WithOldModTime(t time.Time) EventOption {
+	return func(e *reader.Event) {
+		if e.OldEntry != nil && e.OldEntry.Attributes != nil {
+			e.OldEntry.Attributes.Mtime = t.Unix()
+			e.OldEntry.Attributes.MtimeNs = int32(t.Nanosecond())
+		}
+	}
+}
+
+// WithBootstrapVersion attaches a BootstrapVersion to the event.
+// Bootstrap-walk events use this to carry per-version state the live
+// meta-log doesn't see.
+func WithBootstrapVersion(bv *reader.BootstrapVersion) EventOption {
+	return func(e *reader.Event) { e.BootstrapVersion = bv }
+}
+
+// WithShardID overrides the computed ShardID. Most tests should leave
+// it as the s3lifecycle.ShardID default to mirror production routing.
+func WithShardID(shard int) EventOption {
+	return func(e *reader.Event) { e.ShardID = shard }
+}
+
+// NewCreate builds a Create event (NewEntry populated, OldEntry nil) at
+// the given timestamp. Bucket and key are required; everything else is
+// derived (Mtime defaults to ts, FileSize defaults to 0). Apply options
+// to override.
+func NewCreate(bucket, key string, ts time.Time, opts ...EventOption) *reader.Event {
+	e := &reader.Event{
+		TsNs:    ts.UnixNano(),
+		Bucket:  bucket,
+		Key:     key,
+		ShardID: s3lifecycle.ShardID(bucket, key),
+		NewEntry: &filer_pb.Entry{
+			Name: leafOf(key),
+			Attributes: &filer_pb.FuseAttributes{
+				Mtime:   ts.Unix(),
+				MtimeNs: int32(ts.Nanosecond()),
+			},
+		},
+	}
+	for _, o := range opts {
+		o(e)
+	}
+	return e
+}
+
+// NewDelete builds a Delete event (OldEntry populated, NewEntry nil).
+// Tests usually pass the entry the prior Create produced as a snapshot
+// of pre-delete state; this helper builds a minimal stand-in.
+func NewDelete(bucket, key string, ts time.Time, opts ...EventOption) *reader.Event {
+	e := &reader.Event{
+		TsNs:    ts.UnixNano(),
+		Bucket:  bucket,
+		Key:     key,
+		ShardID: s3lifecycle.ShardID(bucket, key),
+		OldEntry: &filer_pb.Entry{
+			Name: leafOf(key),
+			Attributes: &filer_pb.FuseAttributes{
+				Mtime:   ts.Unix(),
+				MtimeNs: int32(ts.Nanosecond()),
+			},
+		},
+	}
+	for _, o := range opts {
+		o(e)
+	}
+	return e
+}
+
+// NewUpdate builds an Update event (both OldEntry and NewEntry populated).
+// Same defaults as NewCreate; options apply to NewEntry per the rules in
+// each option's doc.
+func NewUpdate(bucket, key string, ts time.Time, opts ...EventOption) *reader.Event {
+	e := &reader.Event{
+		TsNs:    ts.UnixNano(),
+		Bucket:  bucket,
+		Key:     key,
+		ShardID: s3lifecycle.ShardID(bucket, key),
+		OldEntry: &filer_pb.Entry{
+			Name: leafOf(key),
+			Attributes: &filer_pb.FuseAttributes{
+				Mtime:   ts.Unix(),
+				MtimeNs: int32(ts.Nanosecond()),
+			},
+		},
+		NewEntry: &filer_pb.Entry{
+			Name: leafOf(key),
+			Attributes: &filer_pb.FuseAttributes{
+				Mtime:   ts.Unix(),
+				MtimeNs: int32(ts.Nanosecond()),
+			},
+		},
+	}
+	for _, o := range opts {
+		o(e)
+	}
+	return e
+}
+
+// MetaLogClock produces monotonically increasing timestamps for fixture
+// generation. Each call to Next advances by Step (default 1s) so tests
+// don't have to thread a counter through every helper invocation. Safe
+// for concurrent use.
+type MetaLogClock struct {
+	mu   sync.Mutex
+	now  time.Time
+	step time.Duration
+}
+
+// NewMetaLogClock returns a clock that ticks forward Step on every Next
+// call. Step defaults to 1s when zero.
+func NewMetaLogClock(start time.Time, step time.Duration) *MetaLogClock {
+	if step <= 0 {
+		step = time.Second
+	}
+	return &MetaLogClock{now: start, step: step}
+}
+
+// Next returns the current timestamp and advances by Step.
+func (c *MetaLogClock) Next() time.Time {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	t := c.now
+	c.now = c.now.Add(c.step)
+	return t
+}
+
+// Peek returns what Next() would return without advancing.
+func (c *MetaLogClock) Peek() time.Time {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return c.now
+}
+
+// leafOf returns the basename of a slash-separated key. Filer entries
+// store only the leaf name; tests that mirror production layout need
+// the same shape. Trailing slashes are stripped first so directory-key
+// fixtures (e.g. "folder/") get the slashless leaf "folder" — the
+// production directory-marker write path stores the name without the
+// trailing slash.
+func leafOf(key string) string {
+	for len(key) > 0 && key[len(key)-1] == '/' {
+		key = key[:len(key)-1]
+	}
+	for i := len(key) - 1; i >= 0; i-- {
+		if key[i] == '/' {
+			return key[i+1:]
+		}
+	}
+	return key
+}

weed/s3api/s3lifecycle/lifecycletest/eventbuilder_old_entry_test.go

@@ -0,0 +1,102 @@
+package lifecycletest
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Coverage gap-fillers for the With* options' OldEntry-only branches
+// (NewEntry==nil, OldEntry populated — i.e. Delete events). The
+// existing eventbuilder_test.go covers Create events; these tests
+// exercise the alternate fall-through path on each option.
+
+func TestEventOption_WithModTime_AppliesToOldEntryOnDelete(t *testing.T) {
+	t0 := time.Unix(1700000000, 0)
+	override := time.Unix(1699000000, 250)
+	e := NewDelete("bk", "k", t0, WithModTime(override))
+	require.Nil(t, e.NewEntry)
+	require.NotNil(t, e.OldEntry)
+	assert.Equal(t, override.Unix(), e.OldEntry.Attributes.Mtime)
+	assert.Equal(t, int32(250), e.OldEntry.Attributes.MtimeNs)
+}
+
+func TestEventOption_WithTtlSec_AppliesToOldEntryOnDelete(t *testing.T) {
+	e := NewDelete("bk", "k", time.Unix(0, 0), WithTtlSec(7200))
+	require.NotNil(t, e.OldEntry)
+	assert.Equal(t, int32(7200), e.OldEntry.Attributes.TtlSec)
+}
+
+func TestEventOption_WithVersionID_AppliesToOldEntryOnDelete(t *testing.T) {
+	// VersionID-on-delete: the bootstrapper synthesizes deletes for
+	// noncurrent-version sweeps and needs the version-id stamp.
+	e := NewDelete("bk", "k", time.Unix(0, 0), WithVersionID("v_old"))
+	require.NotNil(t, e.OldEntry)
+	require.NotNil(t, e.OldEntry.Extended)
+	assert.Equal(t, []byte("v_old"), e.OldEntry.Extended[s3_constants.ExtVersionIdKey])
+}
+
+func TestEventOption_WithExtended_AppliesToOldEntryOnDelete(t *testing.T) {
+	e := NewDelete("bk", "k", time.Unix(0, 0), WithExtended("Custom-Tag", []byte("v")))
+	require.NotNil(t, e.OldEntry)
+	require.NotNil(t, e.OldEntry.Extended)
+	assert.Equal(t, []byte("v"), e.OldEntry.Extended["Custom-Tag"])
+}
+
+func TestEventOption_WithChunks_AppliesToOldEntryOnDelete(t *testing.T) {
+	c := &filer_pb.FileChunk{FileId: "1,old"}
+	e := NewDelete("bk", "k", time.Unix(0, 0), WithChunks(c))
+	require.NotNil(t, e.OldEntry)
+	require.Len(t, e.OldEntry.Chunks, 1)
+	assert.Equal(t, "1,old", e.OldEntry.Chunks[0].FileId)
+}
+
+func TestEventOption_WithBootstrapVersion(t *testing.T) {
+	// WithBootstrapVersion attaches a BootstrapVersion to the event;
+	// the bootstrap walker uses this for per-version state the live
+	// meta-log doesn't carry. Pin both create and delete shapes.
+	bv := &reader.BootstrapVersion{
+		LogicalKey:      "obj.txt",
+		VersionID:       "v_aaa",
+		IsLatest:        true,
+		IsDeleteMarker:  false,
+		NumVersions:     3,
+		NoncurrentIndex: 2,
+	}
+
+	create := NewCreate("bk", "obj.txt", time.Unix(0, 0), WithBootstrapVersion(bv))
+	require.Same(t, bv, create.BootstrapVersion)
+
+	del := NewDelete("bk", "obj.txt", time.Unix(0, 0), WithBootstrapVersion(bv))
+	require.Same(t, bv, del.BootstrapVersion)
+
+	update := NewUpdate("bk", "obj.txt", time.Unix(0, 0), WithBootstrapVersion(bv))
+	require.Same(t, bv, update.BootstrapVersion)
+}
+
+func TestEventOption_NoPanicOnEmptyEvent(t *testing.T) {
+	// Defense: an event with neither NewEntry nor OldEntry must not
+	// panic on any With* option, even though no constructor produces
+	// this shape today. Entry-targeting options fall through silently
+	// (NewEntry/OldEntry stay nil); WithBootstrapVersion targets the
+	// event itself, not an entry, so it does set BootstrapVersion —
+	// included here for panic safety.
+	e := &reader.Event{}
+	WithModTime(time.Unix(1, 0))(e)
+	WithTtlSec(60)(e)
+	WithVersionID("v")(e)
+	WithExtended("k", []byte("v"))(e)
+	WithChunks(&filer_pb.FileChunk{FileId: "x"})(e)
+	WithSize(1024)(e)
+	bv := &reader.BootstrapVersion{VersionID: "v"}
+	WithBootstrapVersion(bv)(e)
+
+	assert.Nil(t, e.NewEntry, "entry-targeting options must not allocate NewEntry")
+	assert.Nil(t, e.OldEntry, "entry-targeting options must not allocate OldEntry")
+	assert.Same(t, bv, e.BootstrapVersion, "WithBootstrapVersion sets the field regardless of entry state")
+}

weed/s3api/s3lifecycle/lifecycletest/eventbuilder_test.go

@@ -0,0 +1,240 @@
+package lifecycletest
+
+import (
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestNewCreate_PopulatesNewEntryOnly(t *testing.T) {
+	t0 := time.Unix(1700000000, 123)
+	e := NewCreate("bk", "obj.txt", t0)
+	require.NotNil(t, e.NewEntry)
+	assert.Nil(t, e.OldEntry)
+	assert.True(t, e.IsCreate())
+	assert.Equal(t, t0.UnixNano(), e.TsNs)
+	assert.Equal(t, "bk", e.Bucket)
+	assert.Equal(t, "obj.txt", e.Key)
+	assert.Equal(t, "obj.txt", e.NewEntry.Name)
+	// Mtime defaults to the event timestamp; tests that don't care can
+	// rely on this rather than threading time through every call.
+	assert.Equal(t, t0.Unix(), e.NewEntry.Attributes.Mtime)
+	assert.Equal(t, int32(t0.Nanosecond()), e.NewEntry.Attributes.MtimeNs)
+}
+
+func TestNewDelete_PopulatesOldEntryOnly(t *testing.T) {
+	t0 := time.Unix(1700000000, 0)
+	e := NewDelete("bk", "obj.txt", t0)
+	require.NotNil(t, e.OldEntry)
+	assert.Nil(t, e.NewEntry)
+	assert.True(t, e.IsDelete())
+	assert.Equal(t, "obj.txt", e.OldEntry.Name)
+}
+
+func TestNewUpdate_PopulatesBothEntries(t *testing.T) {
+	t0 := time.Unix(1700000000, 0)
+	e := NewUpdate("bk", "obj.txt", t0)
+	require.NotNil(t, e.OldEntry)
+	require.NotNil(t, e.NewEntry)
+	assert.False(t, e.IsCreate())
+	assert.False(t, e.IsDelete())
+	// OldEntry's Mtime defaults to the event ts so its mirror of pre-
+	// update state reflects the same wall-clock origin as NewEntry —
+	// a downstream router that compares mtimes won't see a synthetic
+	// 1970 epoch.
+	assert.Equal(t, t0.Unix(), e.OldEntry.Attributes.Mtime)
+	assert.Equal(t, t0.Unix(), e.NewEntry.Attributes.Mtime)
+}
+
+func TestNewCreate_NestedKeyUsesLeafName(t *testing.T) {
+	// Filer entries store only the leaf name; mirroring that in the
+	// fixture keeps router/dispatcher tests realistic.
+	e := NewCreate("bk", "a/b/c.txt", time.Unix(0, 0))
+	assert.Equal(t, "c.txt", e.NewEntry.Name)
+}
+
+func TestNewCreate_DirectoryKeyUsesSlashlessLeaf(t *testing.T) {
+	// Directory-marker objects (S3 keys ending in "/") store the leaf
+	// name without the trailing slash on the filer side. A pre-fix
+	// regression returned "" here.
+	e := NewCreate("bk", "folder/", time.Unix(0, 0))
+	assert.Equal(t, "folder", e.NewEntry.Name)
+
+	// Nested directory key strips both the trailing slash AND the
+	// parent prefix.
+	e2 := NewCreate("bk", "a/b/folder/", time.Unix(0, 0))
+	assert.Equal(t, "folder", e2.NewEntry.Name)
+
+	// Multiple trailing slashes collapse.
+	e3 := NewCreate("bk", "folder///", time.Unix(0, 0))
+	assert.Equal(t, "folder", e3.NewEntry.Name)
+}
+
+func TestNewCreate_ShardIDDerivedFromKey(t *testing.T) {
+	// ShardID matches s3lifecycle.ShardID(bucket, key) so the event
+	// routes through the same shard the production reader would
+	// classify it under.
+	e := NewCreate("bk", "obj.txt", time.Unix(0, 0))
+	assert.Equal(t, s3lifecycle.ShardID("bk", "obj.txt"), e.ShardID)
+}
+
+func TestEventOption_WithSize(t *testing.T) {
+	e := NewCreate("bk", "k", time.Unix(0, 0), WithSize(4096))
+	assert.Equal(t, uint64(4096), e.NewEntry.Attributes.FileSize)
+}
+
+func TestEventOption_WithSizeAppliesToOldEntryOnDelete(t *testing.T) {
+	// Delete events have OldEntry only; WithSize must apply there.
+	e := NewDelete("bk", "k", time.Unix(0, 0), WithSize(8192))
+	assert.Equal(t, uint64(8192), e.OldEntry.Attributes.FileSize)
+}
+
+func TestEventOption_WithModTimeOverridesDefault(t *testing.T) {
+	// Default Mtime is event ts; explicit WithModTime overrides.
+	t0 := time.Unix(1700000000, 0)
+	override := time.Unix(1700000123, 456)
+	e := NewCreate("bk", "k", t0, WithModTime(override))
+	assert.Equal(t, override.Unix(), e.NewEntry.Attributes.Mtime)
+	assert.Equal(t, int32(override.Nanosecond()), e.NewEntry.Attributes.MtimeNs)
+}
+
+func TestEventOption_WithTtlSec(t *testing.T) {
+	// WithTtlSec drives the lifecycle metadata-only delete path; the
+	// gate fires when the live entry's TtlSec > 0.
+	e := NewCreate("bk", "k", time.Unix(0, 0), WithTtlSec(300))
+	assert.Equal(t, int32(300), e.NewEntry.Attributes.TtlSec)
+}
+
+func TestEventOption_WithVersionID(t *testing.T) {
+	e := NewCreate("bk", "k", time.Unix(0, 0), WithVersionID("v_abc"))
+	assert.Equal(t, []byte("v_abc"), e.NewEntry.Extended[s3_constants.ExtVersionIdKey])
+}
+
+func TestEventOption_WithExtendedKeyValue(t *testing.T) {
+	e := NewCreate("bk", "k", time.Unix(0, 0), WithExtended("Custom-Tag", []byte("v1")))
+	assert.Equal(t, []byte("v1"), e.NewEntry.Extended["Custom-Tag"])
+}
+
+func TestEventOption_WithChunks(t *testing.T) {
+	c1 := &filer_pb.FileChunk{FileId: "1,abc"}
+	c2 := &filer_pb.FileChunk{FileId: "1,def"}
+	e := NewCreate("bk", "k", time.Unix(0, 0), WithChunks(c1, c2))
+	require.Len(t, e.NewEntry.Chunks, 2)
+	assert.Equal(t, "1,abc", e.NewEntry.Chunks[0].FileId)
+	assert.Equal(t, "1,def", e.NewEntry.Chunks[1].FileId)
+}
+
+func TestEventOption_WithShardIDOverrides(t *testing.T) {
+	e := NewCreate("bk", "obj.txt", time.Unix(0, 0), WithShardID(7))
+	assert.Equal(t, 7, e.ShardID)
+}
+
+func TestEventOption_WithOldSizeTargetsOldEntryOnUpdate(t *testing.T) {
+	// On Update events, WithSize lands on NewEntry; WithOldSize lets
+	// the caller configure pre-update state independently. Both options
+	// in a single call should produce distinct sizes on the two entries.
+	t0 := time.Unix(1700000000, 0)
+	e := NewUpdate("bk", "k", t0, WithSize(200), WithOldSize(100))
+	assert.Equal(t, uint64(200), e.NewEntry.Attributes.FileSize, "WithSize lands on NewEntry")
+	assert.Equal(t, uint64(100), e.OldEntry.Attributes.FileSize, "WithOldSize lands on OldEntry")
+}
+
+func TestEventOption_WithOldChunksTargetsOldEntryOnUpdate(t *testing.T) {
+	newChunk := &filer_pb.FileChunk{FileId: "1,new"}
+	oldChunk := &filer_pb.FileChunk{FileId: "1,old"}
+	t0 := time.Unix(1700000000, 0)
+	e := NewUpdate("bk", "k", t0, WithChunks(newChunk), WithOldChunks(oldChunk))
+	require.Len(t, e.NewEntry.Chunks, 1)
+	require.Len(t, e.OldEntry.Chunks, 1)
+	assert.Equal(t, "1,new", e.NewEntry.Chunks[0].FileId)
+	assert.Equal(t, "1,old", e.OldEntry.Chunks[0].FileId)
+}
+
+func TestEventOption_WithOldModTimeTargetsOldEntryOnUpdate(t *testing.T) {
+	// Pre-update mtime can lag the event timestamp; WithOldModTime lets
+	// a router test pin the noncurrent-clock origin.
+	t0 := time.Unix(1700000000, 0)
+	older := time.Unix(1699000000, 500)
+	e := NewUpdate("bk", "k", t0, WithOldModTime(older))
+	assert.Equal(t, older.Unix(), e.OldEntry.Attributes.Mtime)
+	assert.Equal(t, int32(500), e.OldEntry.Attributes.MtimeNs)
+	// NewEntry's mtime stays at the default ts.
+	assert.Equal(t, t0.Unix(), e.NewEntry.Attributes.Mtime)
+}
+
+func TestEventOption_WithOldOptionsAreNoOpsOnCreate(t *testing.T) {
+	// Create events have no OldEntry; the WithOld* options must not
+	// panic and must not leak fields into NewEntry.
+	t0 := time.Unix(1700000000, 0)
+	e := NewCreate("bk", "k", t0,
+		WithOldSize(999),
+		WithOldChunks(&filer_pb.FileChunk{FileId: "phantom"}),
+		WithOldModTime(time.Unix(1, 0)),
+	)
+	assert.Nil(t, e.OldEntry)
+	assert.Equal(t, uint64(0), e.NewEntry.Attributes.FileSize, "WithOldSize must not bleed to NewEntry")
+	assert.Empty(t, e.NewEntry.Chunks, "WithOldChunks must not bleed to NewEntry")
+	assert.Equal(t, t0.Unix(), e.NewEntry.Attributes.Mtime, "WithOldModTime must not bleed to NewEntry")
+}
+
+func TestEventOption_LaterOverridesEarlier(t *testing.T) {
+	// Apply order matters: later options win on the same field. Pins
+	// the documented ordering so tests can compose default+override
+	// patterns without surprises.
+	e := NewCreate("bk", "k", time.Unix(0, 0),
+		WithSize(100),
+		WithSize(200),
+	)
+	assert.Equal(t, uint64(200), e.NewEntry.Attributes.FileSize)
+}
+
+func TestMetaLogClock_DefaultStepIsOneSecond(t *testing.T) {
+	c := NewMetaLogClock(time.Unix(1000, 0), 0)
+	t1 := c.Next()
+	t2 := c.Next()
+	assert.Equal(t, time.Second, t2.Sub(t1))
+}
+
+func TestMetaLogClock_CustomStep(t *testing.T) {
+	c := NewMetaLogClock(time.Unix(1000, 0), 250*time.Millisecond)
+	t1 := c.Next()
+	t2 := c.Next()
+	assert.Equal(t, 250*time.Millisecond, t2.Sub(t1))
+}
+
+func TestMetaLogClock_PeekDoesNotAdvance(t *testing.T) {
+	c := NewMetaLogClock(time.Unix(1000, 0), time.Second)
+	first := c.Peek()
+	again := c.Peek()
+	assert.True(t, first.Equal(again), "Peek must not advance the clock")
+	advanced := c.Next()
+	assert.True(t, first.Equal(advanced), "Next returns what Peek returned")
+}
+
+func TestMetaLogClock_ConcurrentNextNoRace(t *testing.T) {
+	// Tests that produce events from many goroutines (e.g. fan-out
+	// fixtures) need the clock to serialize without deadlock or
+	// duplicate timestamps. -race catches a regression that drops
+	// the lock.
+	c := NewMetaLogClock(time.Unix(0, 0), time.Microsecond)
+	const N = 64
+	seen := sync.Map{}
+	var wg sync.WaitGroup
+	wg.Add(N)
+	for i := 0; i < N; i++ {
+		go func() {
+			defer wg.Done()
+			seen.Store(c.Next(), struct{}{})
+		}()
+	}
+	wg.Wait()
+	count := 0
+	seen.Range(func(_, _ any) bool { count++; return true })
+	assert.Equal(t, N, count, "every Next must produce a unique timestamp")
+}

weed/s3api/s3lifecycle/lifecycletest/fakeserver.go

@@ -0,0 +1,159 @@
+// Package lifecycletest provides reusable test doubles for the lifecycle
+// worker pipeline. The pieces here let component-level tests stand up the
+// gRPC boundary the worker dials at runtime without pulling in a real
+// S3ApiServer or filer.
+package lifecycletest
+
+import (
+	"context"
+	"sync"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"google.golang.org/protobuf/proto"
+)
+
+// Outcome is what the fake returns for a single LifecycleDelete call.
+// Code is required; Reason is echoed verbatim into LifecycleDeleteResponse.
+type Outcome struct {
+	Code   s3_lifecycle_pb.LifecycleDeleteOutcome
+	Reason string
+}
+
+// Done, NoopResolved, RetryLater, Blocked, SkippedObjectLock are
+// constructors for the common outcomes; tests can also build Outcome
+// values directly when they need a specific Reason.
+func Done() Outcome {
+	return Outcome{Code: s3_lifecycle_pb.LifecycleDeleteOutcome_DONE}
+}
+func NoopResolved(reason string) Outcome {
+	return Outcome{Code: s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED, Reason: reason}
+}
+func RetryLater(reason string) Outcome {
+	return Outcome{Code: s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER, Reason: reason}
+}
+func Blocked(reason string) Outcome {
+	return Outcome{Code: s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED, Reason: reason}
+}
+func SkippedObjectLock(reason string) Outcome {
+	return Outcome{Code: s3_lifecycle_pb.LifecycleDeleteOutcome_SKIPPED_OBJECT_LOCK, Reason: reason}
+}
+
+// FakeLifecycleServer implements s3_lifecycle_pb.SeaweedS3LifecycleInternalServer.
+// It returns per-key queued outcomes (FIFO) and falls back to Default when a
+// key has no queued entry. Every received request is recorded; tests assert
+// against Recorded() in arrival order.
+//
+// A non-nil Err short-circuits everything — LifecycleDelete returns (nil, Err)
+// immediately, before the per-key lookup or the request recording. Use it to
+// simulate transport failures.
+//
+// All methods are safe for concurrent use. Outcomes/Default may be set at
+// construction or via Queue/SetDefault between calls; mid-call mutation is
+// supported but ordering across that boundary is undefined.
+type FakeLifecycleServer struct {
+	s3_lifecycle_pb.UnimplementedSeaweedS3LifecycleInternalServer
+
+	mu       sync.Mutex
+	queues   map[requestKey][]Outcome
+	def      Outcome
+	err      error
+	received []*s3_lifecycle_pb.LifecycleDeleteRequest
+}
+
+// requestKey is the map key for queues. A struct rather than a delimited
+// string so bucket/object/versionId values containing "/" or "@" can't
+// collide.
+type requestKey struct {
+	bucket, objectPath, versionId string
+}
+
+// NewFakeLifecycleServer returns a server whose Default outcome is DONE.
+// Most tests want a different default; call SetDefault to change it.
+func NewFakeLifecycleServer() *FakeLifecycleServer {
+	return &FakeLifecycleServer{
+		queues: map[requestKey][]Outcome{},
+		def:    Done(),
+	}
+}
+
+// Queue appends an outcome to the FIFO for (bucket, objectPath, versionId).
+// Subsequent calls matching the same key return outcomes in the order they
+// were queued; once the queue is drained, Default applies.
+func (f *FakeLifecycleServer) Queue(bucket, objectPath, versionId string, outcome Outcome) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	k := key(bucket, objectPath, versionId)
+	f.queues[k] = append(f.queues[k], outcome)
+}
+
+// SetDefault sets the outcome returned when no per-key queue entry remains.
+func (f *FakeLifecycleServer) SetDefault(o Outcome) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.def = o
+}
+
+// SetError makes LifecycleDelete return (nil, err) on every call until
+// SetError(nil) clears it. The request is not recorded while Err is set —
+// transport-error tests should rely on the worker's own bookkeeping.
+func (f *FakeLifecycleServer) SetError(err error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.err = err
+}
+
+// Recorded returns a deep-copied snapshot of every request the server has
+// received (excluding calls that returned a transport error). Both the
+// slice and each element are independent of the fake's internal state, so
+// callers can mutate freely without affecting later Recorded() snapshots.
+func (f *FakeLifecycleServer) Recorded() []*s3_lifecycle_pb.LifecycleDeleteRequest {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	out := make([]*s3_lifecycle_pb.LifecycleDeleteRequest, len(f.received))
+	for i, r := range f.received {
+		out[i] = cloneRequest(r)
+	}
+	return out
+}
+
+// LifecycleDelete is the gRPC handler. It honors Err first, then dequeues
+// the per-key outcome, falling back to Default.
+func (f *FakeLifecycleServer) LifecycleDelete(ctx context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
+	f.mu.Lock()
+	if f.err != nil {
+		err := f.err
+		f.mu.Unlock()
+		return nil, err
+	}
+	// Record a deep copy so a caller mutating a Recorded() entry can't
+	// reach back into the fake's bookkeeping (and so subsequent calls'
+	// view of "what arrived" stays stable across assertions).
+	f.received = append(f.received, cloneRequest(req))
+	if req == nil {
+		out := f.def
+		f.mu.Unlock()
+		return &s3_lifecycle_pb.LifecycleDeleteResponse{Outcome: out.Code, Reason: out.Reason}, nil
+	}
+	k := key(req.Bucket, req.ObjectPath, req.VersionId)
+	q := f.queues[k]
+	var out Outcome
+	if len(q) > 0 {
+		out = q[0]
+		f.queues[k] = q[1:]
+	} else {
+		out = f.def
+	}
+	f.mu.Unlock()
+	return &s3_lifecycle_pb.LifecycleDeleteResponse{Outcome: out.Code, Reason: out.Reason}, nil
+}
+
+func key(bucket, objectPath, versionId string) requestKey {
+	return requestKey{bucket: bucket, objectPath: objectPath, versionId: versionId}
+}
+
+func cloneRequest(req *s3_lifecycle_pb.LifecycleDeleteRequest) *s3_lifecycle_pb.LifecycleDeleteRequest {
+	if req == nil {
+		return nil
+	}
+	return proto.Clone(req).(*s3_lifecycle_pb.LifecycleDeleteRequest)
+}

weed/s3api/s3lifecycle/lifecycletest/fakeserver_test.go

@@ -0,0 +1,217 @@
+package lifecycletest
+
+import (
+	"context"
+	"errors"
+	"sync"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestFake_DefaultIsDoneOutOfTheBox(t *testing.T) {
+	// A test that doesn't queue anything should still get a non-error
+	// response so it can exercise the worker's happy path.
+	f := NewFakeLifecycleServer()
+	resp, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{
+		Bucket:     "b",
+		ObjectPath: "k",
+	})
+	require.NoError(t, err)
+	require.NotNil(t, resp)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_DONE, resp.Outcome)
+	assert.Equal(t, "", resp.Reason)
+}
+
+func TestFake_QueuedOutcomesPopFIFO(t *testing.T) {
+	// Per-key queue is FIFO and one-shot per entry; after the queue
+	// drains, Default kicks in.
+	f := NewFakeLifecycleServer()
+	f.SetDefault(NoopResolved("nothing more queued"))
+	f.Queue("b", "k", "", RetryLater("first"))
+	f.Queue("b", "k", "", Blocked("second"))
+
+	got := []s3_lifecycle_pb.LifecycleDeleteOutcome{}
+	reasons := []string{}
+	for i := 0; i < 3; i++ {
+		resp, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{
+			Bucket: "b", ObjectPath: "k",
+		})
+		require.NoError(t, err)
+		got = append(got, resp.Outcome)
+		reasons = append(reasons, resp.Reason)
+	}
+	assert.Equal(t, []s3_lifecycle_pb.LifecycleDeleteOutcome{
+		s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED,
+		s3_lifecycle_pb.LifecycleDeleteOutcome_NOOP_RESOLVED,
+	}, got)
+	assert.Equal(t, []string{"first", "second", "nothing more queued"}, reasons)
+}
+
+func TestFake_QueuesIsolatedByKey(t *testing.T) {
+	// Queues are partitioned by (bucket, objectPath, versionId); a queued
+	// outcome for one key must not bleed into another's lookup.
+	f := NewFakeLifecycleServer()
+	f.Queue("b", "a", "", Blocked("a-only"))
+	f.Queue("b", "b", "", RetryLater("b-only"))
+
+	respA, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "a"})
+	require.NoError(t, err)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED, respA.Outcome)
+
+	respB, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "b"})
+	require.NoError(t, err)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_RETRY_LATER, respB.Outcome)
+}
+
+func TestFake_VersionIDPartOfKey(t *testing.T) {
+	// Two requests for the same bucket/objectPath but different
+	// versionIds must address different queues.
+	f := NewFakeLifecycleServer()
+	f.Queue("b", "k", "v1", SkippedObjectLock("v1-locked"))
+	f.Queue("b", "k", "v2", Done())
+
+	respV1, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "k", VersionId: "v1"})
+	require.NoError(t, err)
+	respV2, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "k", VersionId: "v2"})
+	require.NoError(t, err)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_SKIPPED_OBJECT_LOCK, respV1.Outcome)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_DONE, respV2.Outcome)
+}
+
+func TestFake_KeyComponentsWithDelimitersDoNotCollide(t *testing.T) {
+	// String-concatenation keys would have made these two requests
+	// indistinguishable. The struct key keeps them separate.
+	f := NewFakeLifecycleServer()
+	f.Queue("b/k", "", "", Blocked("variant-a"))
+	f.Queue("b", "k", "", Done())
+
+	respA, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b/k", ObjectPath: ""})
+	require.NoError(t, err)
+	respB, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "k"})
+	require.NoError(t, err)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED, respA.Outcome)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_DONE, respB.Outcome)
+}
+
+func TestFake_ErrShortCircuitsBeforeRecording(t *testing.T) {
+	// Err makes LifecycleDelete return (nil, err) without recording the
+	// request — transport-error tests rely on the worker's own
+	// bookkeeping, not the fake's.
+	f := NewFakeLifecycleServer()
+	transportErr := errors.New("connection refused")
+	f.SetError(transportErr)
+
+	resp, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "k"})
+	assert.Nil(t, resp)
+	assert.ErrorIs(t, err, transportErr)
+	assert.Empty(t, f.Recorded(), "transport-error calls must not be recorded")
+
+	// Clearing the error returns the server to normal behavior.
+	f.SetError(nil)
+	resp2, err2 := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "k"})
+	require.NoError(t, err2)
+	require.NotNil(t, resp2)
+	assert.Len(t, f.Recorded(), 1)
+}
+
+func TestFake_RecordsRequestsInOrder(t *testing.T) {
+	// Recorded() preserves arrival order so tests can assert that
+	// dispatch happened in the expected sequence.
+	f := NewFakeLifecycleServer()
+	for _, key := range []string{"k1", "k2", "k3"} {
+		_, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{
+			Bucket: "b", ObjectPath: key,
+		})
+		require.NoError(t, err)
+	}
+	rec := f.Recorded()
+	require.Len(t, rec, 3)
+	assert.Equal(t, "k1", rec[0].ObjectPath)
+	assert.Equal(t, "k2", rec[1].ObjectPath)
+	assert.Equal(t, "k3", rec[2].ObjectPath)
+}
+
+func TestFake_RecordedIsSnapshot(t *testing.T) {
+	// Mutating the slice the caller got back must not bleed into the
+	// fake's internal state — otherwise a flaky test could corrupt
+	// bookkeeping for later assertions.
+	f := NewFakeLifecycleServer()
+	_, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "k"})
+	require.NoError(t, err)
+
+	snap := f.Recorded()
+	require.Len(t, snap, 1)
+	snap[0] = nil
+	again := f.Recorded()
+	require.Len(t, again, 1)
+	assert.NotNil(t, again[0], "internal record must survive caller-side mutation of the snapshot")
+}
+
+func TestFake_RecordedRequestsAreDeepCopies(t *testing.T) {
+	// A caller mutating fields on a Recorded() entry must not bleed
+	// into a later Recorded() snapshot. We also confirm the original
+	// request the caller passed in stays decoupled from internal state
+	// — proto.Clone runs at record time, so the caller's pointer is
+	// no longer the one the fake holds.
+	f := NewFakeLifecycleServer()
+	orig := &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "k", VersionId: "v"}
+	_, err := f.LifecycleDelete(context.Background(), orig)
+	require.NoError(t, err)
+
+	snap := f.Recorded()
+	require.Len(t, snap, 1)
+	snap[0].Bucket = "mutated-by-caller"
+	snap[0].ObjectPath = "also-mutated"
+
+	again := f.Recorded()
+	require.Len(t, again, 1)
+	assert.Equal(t, "b", again[0].Bucket, "field mutations on a snapshot must not bleed back")
+	assert.Equal(t, "k", again[0].ObjectPath)
+	assert.Equal(t, "v", again[0].VersionId)
+
+	// The caller's own pointer is independent of the fake's record.
+	orig.Bucket = "caller-mutates-original"
+	yetAgain := f.Recorded()
+	assert.Equal(t, "b", yetAgain[0].Bucket, "caller mutating the input pointer must not bleed in either")
+}
+
+func TestFake_NilRequestUsesDefault(t *testing.T) {
+	// gRPC won't deliver a nil request in practice, but defensive code
+	// in the fake should still produce a deterministic response so a
+	// regressing client doesn't panic the test process.
+	f := NewFakeLifecycleServer()
+	f.SetDefault(Blocked("no request"))
+	resp, err := f.LifecycleDelete(context.Background(), nil)
+	require.NoError(t, err)
+	require.NotNil(t, resp)
+	assert.Equal(t, s3_lifecycle_pb.LifecycleDeleteOutcome_BLOCKED, resp.Outcome)
+}
+
+func TestFake_ConcurrentCallsSerializeWithoutDeadlock(t *testing.T) {
+	// The dispatcher fans dispatch across many goroutines; the fake
+	// must not livelock or drop records under concurrent load. Capture
+	// each goroutine's err so a regression in concurrent paths surfaces
+	// instead of being masked by length-only assertions.
+	f := NewFakeLifecycleServer()
+	const N = 64
+	var wg sync.WaitGroup
+	errCh := make(chan error, N)
+	wg.Add(N)
+	for i := 0; i < N; i++ {
+		go func() {
+			defer wg.Done()
+			_, err := f.LifecycleDelete(context.Background(), &s3_lifecycle_pb.LifecycleDeleteRequest{Bucket: "b", ObjectPath: "k"})
+			errCh <- err
+		}()
+	}
+	wg.Wait()
+	close(errCh)
+	for err := range errCh {
+		require.NoError(t, err)
+	}
+	assert.Len(t, f.Recorded(), N)
+}

weed/s3api/s3lifecycle/min_trigger_age.go

@@ -0,0 +1,27 @@
+package s3lifecycle
+
+import "time"
+
+// MinTriggerAge returns the day threshold defined by kind on rule, or 0 if
+// the kind has no day-style threshold (date / count / immediate). Callers
+// use it as max(MinTriggerAge, kindFloor) when computing safety-scan cadence.
+func MinTriggerAge(rule *Rule, kind ActionKind) time.Duration {
+	if rule == nil {
+		return 0
+	}
+	switch kind {
+	case ActionKindExpirationDays:
+		if rule.ExpirationDays > 0 {
+			return DaysToDuration(rule.ExpirationDays)
+		}
+	case ActionKindNoncurrentDays:
+		if rule.NoncurrentVersionExpirationDays > 0 {
+			return DaysToDuration(rule.NoncurrentVersionExpirationDays)
+		}
+	case ActionKindAbortMPU:
+		if rule.AbortMPUDaysAfterInitiation > 0 {
+			return DaysToDuration(rule.AbortMPUDaysAfterInitiation)
+		}
+	}
+	return 0
+}

weed/s3api/s3lifecycle/min_trigger_age_test.go

@@ -0,0 +1,54 @@
+package s3lifecycle
+
+import (
+	"testing"
+	"time"
+)
+
+func TestMinTriggerAge_PerKind(t *testing.T) {
+	rule := &Rule{
+		ExpirationDays:                  30,
+		NoncurrentVersionExpirationDays: 7,
+		AbortMPUDaysAfterInitiation:     14,
+	}
+	cases := []struct {
+		kind ActionKind
+		want time.Duration
+	}{
+		{ActionKindExpirationDays, DaysToDuration(30)},
+		{ActionKindNoncurrentDays, DaysToDuration(7)},
+		{ActionKindAbortMPU, DaysToDuration(14)},
+		{ActionKindExpirationDate, 0},
+		{ActionKindNewerNoncurrent, 0},
+		{ActionKindExpiredDeleteMarker, 0},
+	}
+	for _, c := range cases {
+		t.Run(c.kind.String(), func(t *testing.T) {
+			if got := MinTriggerAge(rule, c.kind); got != c.want {
+				t.Fatalf("want %v, got %v", c.want, got)
+			}
+		})
+	}
+}
+
+func TestMinTriggerAge_KindNotSetReturnsZero(t *testing.T) {
+	// Asking for a kind that the rule doesn't actually declare returns 0,
+	// so the safety-scan cadence falls through to the kind floor.
+	rule := &Rule{ExpirationDays: 30}
+	if got := MinTriggerAge(rule, ActionKindAbortMPU); got != 0 {
+		t.Fatalf("want 0, got %v", got)
+	}
+}
+
+func TestMinTriggerAge_DateOnlyReturnsZero(t *testing.T) {
+	rule := &Rule{ExpirationDate: mustTime(t, "2025-06-15T00:00:00Z")}
+	if got := MinTriggerAge(rule, ActionKindExpirationDate); got != 0 {
+		t.Fatalf("date kind has no day threshold, got %v", got)
+	}
+}
+
+func TestMinTriggerAge_NilRule(t *testing.T) {
+	if got := MinTriggerAge(nil, ActionKindExpirationDays); got != 0 {
+		t.Fatalf("nil rule should return 0, got %v", got)
+	}
+}

weed/s3api/s3lifecycle/noncurrent_since.go

@@ -0,0 +1,37 @@
+package s3lifecycle
+
+import (
+	"strconv"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+)
+
+// SuccessorFromEntryStamp returns the explicit noncurrent-since timestamp
+// written by the S3 PUT handler at demotion time
+// (s3_constants.ExtNoncurrentSinceNsKey). Returns zero time if the stamp
+// is missing or unparseable — the caller falls back to derived values
+// (sibling mtime or entry mtime) for legacy entries written before the
+// stamp existed.
+//
+// The lifecycle engine prefers this stamp over the legacy
+// "next-newer sibling mtime" derivation because it records the exact
+// moment a version was demoted, immune to later mtime edits on the
+// sibling. The router and bootstrap walker both read it through this
+// helper so the parsing rules — non-positive, unparseable, missing —
+// stay in one place.
+func SuccessorFromEntryStamp(entry *filer_pb.Entry) time.Time {
+	if entry == nil {
+		return time.Time{}
+	}
+	raw, ok := entry.Extended[s3_constants.ExtNoncurrentSinceNsKey]
+	if !ok || len(raw) == 0 {
+		return time.Time{}
+	}
+	ns, err := strconv.ParseInt(string(raw), 10, 64)
+	if err != nil || ns <= 0 {
+		return time.Time{}
+	}
+	return time.Unix(0, ns)
+}

weed/s3api/s3lifecycle/noncurrent_since_test.go

@@ -0,0 +1,75 @@
+package s3lifecycle
+
+import (
+	"strconv"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Direct coverage for the parser. The router and the bootstrap walker
+// both go through this single helper to read the demotion stamp, so
+// pinning every parsing branch here keeps that contract enforced even
+// as the call sites evolve.
+
+func TestSuccessorFromEntryStamp_NilOrMissingReturnsZero(t *testing.T) {
+	assert.True(t, SuccessorFromEntryStamp(nil).IsZero())
+	assert.True(t, SuccessorFromEntryStamp(&filer_pb.Entry{}).IsZero())
+	assert.True(t, SuccessorFromEntryStamp(&filer_pb.Entry{
+		Extended: map[string][]byte{},
+	}).IsZero())
+}
+
+func TestSuccessorFromEntryStamp_EmptyOrInvalidReturnsZero(t *testing.T) {
+	for _, raw := range [][]byte{nil, []byte(""), []byte("nope")} {
+		got := SuccessorFromEntryStamp(&filer_pb.Entry{
+			Extended: map[string][]byte{s3_constants.ExtNoncurrentSinceNsKey: raw},
+		})
+		assert.True(t, got.IsZero(), "value %q must produce zero time", string(raw))
+	}
+}
+
+func TestSuccessorFromEntryStamp_NonPositiveReturnsZero(t *testing.T) {
+	// Stamps are wall-clock UnixNano captured at demotion time; <=0
+	// signals "not set" — caller falls back to derived sibling mtime.
+	for _, raw := range []string{"0", "-1"} {
+		got := SuccessorFromEntryStamp(&filer_pb.Entry{
+			Extended: map[string][]byte{s3_constants.ExtNoncurrentSinceNsKey: []byte(raw)},
+		})
+		assert.True(t, got.IsZero(), "value %q must produce zero time", raw)
+	}
+}
+
+func TestSuccessorFromEntryStamp_PositiveNanosRoundTrip(t *testing.T) {
+	const ns int64 = 1700000000_123456789
+	got := SuccessorFromEntryStamp(&filer_pb.Entry{
+		Extended: map[string][]byte{s3_constants.ExtNoncurrentSinceNsKey: []byte("1700000000123456789")},
+	})
+	assert.Equal(t, time.Unix(0, ns).UTC(), got.UTC())
+}
+
+func TestSuccessorFromEntryStamp_OrderedNanosStayOrdered(t *testing.T) {
+	// Pins the read contract: if two stamps were written in increasing
+	// nanosecond order, the parser must return Times in the same order.
+	// Uses fixed nanosecond values rather than time.Now().UnixNano() —
+	// UnixNano() drops the monotonic component, so back-to-back calls
+	// can decrease if the wall clock steps backward and the property
+	// being verified would then exercise OS clock behavior, not the
+	// parser.
+	require := require.New(t)
+	const earlier int64 = 1700000000_000000001
+	const later int64 = 1700000000_000000002
+
+	earlierEntry := &filer_pb.Entry{
+		Extended: map[string][]byte{s3_constants.ExtNoncurrentSinceNsKey: []byte(strconv.FormatInt(earlier, 10))},
+	}
+	laterEntry := &filer_pb.Entry{
+		Extended: map[string][]byte{s3_constants.ExtNoncurrentSinceNsKey: []byte(strconv.FormatInt(later, 10))},
+	}
+	require.False(SuccessorFromEntryStamp(earlierEntry).After(SuccessorFromEntryStamp(laterEntry)),
+		"parser must preserve write-order between two stamps")
+}

weed/s3api/s3lifecycle/reader/composition_test.go

@@ -0,0 +1,162 @@
+package reader
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Layer 2 contracts the daily-replay drain relies on. These augment
+// reader_test.go (extract + dispatch) and cursor_test.go (Advance +
+// Freeze) by pinning the cursor composition surface that the daily-run
+// drain depends on for resume-point selection, checkpoint persistence,
+// and Run-time input validation.
+
+func TestCursorMinTsNsWithFrozenKeysIncluded(t *testing.T) {
+	// MinTsNs is the resume point Reader.Run feeds to SubscribeMetadata
+	// when StartTsNs is zero. A freeze pins a key at its current TsNs;
+	// the min must still see that pinned position so the subscription
+	// doesn't rewind past it. Mixing frozen + non-frozen keys exercises
+	// the path where the smallest position belongs to a frozen key.
+	c := NewCursor()
+	frozenKey := key("a", s3lifecycle.ActionKindExpirationDays)
+	movingKey := key("b", s3lifecycle.ActionKindNoncurrentDays)
+	otherKey := key("c", s3lifecycle.ActionKindAbortMPU)
+
+	c.Advance(frozenKey, 100)
+	c.Advance(movingKey, 500)
+	c.Advance(otherKey, 300)
+	c.Freeze(frozenKey, 100)
+
+	// Frozen key holds at 100; Advance must not move it.
+	c.Advance(frozenKey, 9_999)
+	require.Equal(t, int64(100), c.Get(frozenKey))
+
+	// MinTsNs reflects the frozen key because it is still the smallest.
+	assert.Equal(t, int64(100), c.MinTsNs())
+
+	// Non-frozen keys keep advancing; min stays anchored to the freeze.
+	c.Advance(movingKey, 700)
+	c.Advance(otherKey, 600)
+	assert.Equal(t, int64(100), c.MinTsNs())
+}
+
+func TestCursorSnapshotIsDeepCopy(t *testing.T) {
+	// Snapshot is persisted via the persister; callers must be able to
+	// hold the returned map without observing live cursor mutations,
+	// and the cursor must not observe caller-side edits to the map.
+	c := NewCursor()
+	a := key("a", s3lifecycle.ActionKindExpirationDays)
+	b := key("b", s3lifecycle.ActionKindNoncurrentDays)
+	c.Advance(a, 100)
+	c.Advance(b, 200)
+
+	snap := c.Snapshot()
+	require.Equal(t, int64(100), snap[a])
+	require.Equal(t, int64(200), snap[b])
+
+	// Mutating the returned map must not bleed into the cursor.
+	snap[a] = 9_999
+	delete(snap, b)
+	snap[key("z", s3lifecycle.ActionKindAbortMPU)] = 42
+	assert.Equal(t, int64(100), c.Get(a), "cursor must be insulated from snapshot writes")
+	assert.Equal(t, int64(200), c.Get(b), "cursor must retain key the caller deleted from snapshot")
+	assert.Equal(t, int64(0), c.Get(key("z", s3lifecycle.ActionKindAbortMPU)), "cursor must not see keys added to snapshot")
+
+	// Mutating the cursor after Snapshot must not bleed into the map
+	// the caller is still holding (e.g., between Snapshot and Save).
+	snap2 := c.Snapshot()
+	c.Advance(a, 1_000)
+	c.Advance(key("d", s3lifecycle.ActionKindAbortMPU), 50)
+	assert.Equal(t, int64(100), snap2[a], "snapshot must not see post-snapshot Advance")
+	_, hasNew := snap2[key("d", s3lifecycle.ActionKindAbortMPU)]
+	assert.False(t, hasNew, "snapshot must not see post-snapshot key insertions")
+}
+
+func TestCursorRestoreReplacesNotMerges(t *testing.T) {
+	// On startup the persister loads the last checkpoint and Restores
+	// the cursor. A subsequent Restore (e.g., after a re-bootstrap)
+	// must fully replace the in-memory map: keys present before but
+	// absent in the new map must vanish, otherwise stale resume
+	// points would survive across restores.
+	c := NewCursor()
+	a := key("a", s3lifecycle.ActionKindExpirationDays)
+	b := key("b", s3lifecycle.ActionKindNoncurrentDays)
+	cc := key("c", s3lifecycle.ActionKindAbortMPU)
+
+	c.Restore(map[s3lifecycle.ActionKey]int64{a: 100, b: 200, cc: 300})
+	require.Equal(t, int64(100), c.Get(a))
+	require.Equal(t, int64(200), c.Get(b))
+	require.Equal(t, int64(300), c.Get(cc))
+
+	// Freeze a key before second Restore; Restore must clear frozen state
+	// alongside the value map so a stale freeze doesn't survive a reload.
+	c.Freeze(a, 100)
+	require.True(t, c.IsFrozen(a))
+
+	// Second Restore drops keys b and c entirely; a is replaced.
+	c.Restore(map[s3lifecycle.ActionKey]int64{a: 50})
+	assert.Equal(t, int64(50), c.Get(a))
+	assert.False(t, c.IsFrozen(a), "Restore must clear frozen state")
+	assert.Equal(t, int64(0), c.Get(b), "key absent from second Restore must be removed")
+	assert.Equal(t, int64(0), c.Get(cc), "key absent from second Restore must be removed")
+
+	// MinTsNs reflects only the surviving key.
+	assert.Equal(t, int64(50), c.MinTsNs())
+
+	// Restoring an empty map clears all state.
+	c.Restore(map[s3lifecycle.ActionKey]int64{})
+	assert.Equal(t, int64(0), c.Get(a))
+	assert.Equal(t, int64(0), c.MinTsNs())
+}
+
+func TestReaderRunValidatesInputsBeforeSubscribing(t *testing.T) {
+	// Run guards: out-of-range ShardID (no predicate), nil Events
+	// channel, empty BucketsPath. Each error path must fire before any
+	// client call, so a nil client suffices to drive the assertions.
+	out := make(chan *Event, 1)
+
+	cases := []struct {
+		name    string
+		reader  *Reader
+		wantSub string
+	}{
+		{
+			name:    "shard id below zero",
+			reader:  &Reader{ShardID: -1, BucketsPath: "/buckets", Events: out},
+			wantSub: "shard_id",
+		},
+		{
+			name:    "shard id at or above ShardCount",
+			reader:  &Reader{ShardID: s3lifecycle.ShardCount, BucketsPath: "/buckets", Events: out},
+			wantSub: "shard_id",
+		},
+		{
+			name:    "nil events channel",
+			reader:  &Reader{ShardID: 0, BucketsPath: "/buckets", Events: nil},
+			wantSub: "nil Events",
+		},
+		{
+			name:    "empty buckets path",
+			reader:  &Reader{ShardID: 0, BucketsPath: "", Events: out},
+			wantSub: "empty BucketsPath",
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			// Bound the call so a regression that lets Run reach the
+			// nil client's blocking subscribe surfaces as a test failure
+			// instead of hanging the suite.
+			ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+			defer cancel()
+			err := tc.reader.Run(ctx, nil, "test", 0)
+			require.Error(t, err)
+			assert.Contains(t, err.Error(), tc.wantSub)
+		})
+	}
+}

weed/s3api/s3lifecycle/reader/cursor.go

@@ -0,0 +1,123 @@
+package reader
+
+import (
+	"sync"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+// Cursor tracks the meta-log position for each ActionKey within one shard.
+// Position = tsNs of the last event the shard's reader+downstream considers
+// "fully resolved" for this ActionKey. Subscription resumes at the minimum
+// position across all keys.
+//
+// A frozen key is pinned at its current position (BLOCKED outcome): Advance
+// is a no-op until Unfreeze. The meta-log retention bounds how long a freeze
+// can last before the entry expires from the log; operator action via the
+// blocker-resolve flow lifts the freeze.
+type Cursor struct {
+	mu     sync.RWMutex
+	state  map[s3lifecycle.ActionKey]int64
+	frozen map[s3lifecycle.ActionKey]struct{}
+}
+
+func NewCursor() *Cursor {
+	return &Cursor{
+		state:  map[s3lifecycle.ActionKey]int64{},
+		frozen: map[s3lifecycle.ActionKey]struct{}{},
+	}
+}
+
+// MinTsNs returns the smallest position across all keys; zero if empty.
+// Used as the resume point for the meta-log subscription.
+func (c *Cursor) MinTsNs() int64 {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	if len(c.state) == 0 {
+		return 0
+	}
+	var min int64 = -1
+	for _, v := range c.state {
+		if min < 0 || v < min {
+			min = v
+		}
+	}
+	if min < 0 {
+		return 0
+	}
+	return min
+}
+
+// Get returns the current position for key, or 0 if unset.
+func (c *Cursor) Get(key s3lifecycle.ActionKey) int64 {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return c.state[key]
+}
+
+// Advance moves key's position forward to tsNs. No-op if key is frozen, if
+// tsNs <= current, or if tsNs <= 0.
+func (c *Cursor) Advance(key s3lifecycle.ActionKey, tsNs int64) {
+	if tsNs <= 0 {
+		return
+	}
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if _, isFrozen := c.frozen[key]; isFrozen {
+		return
+	}
+	if cur := c.state[key]; tsNs > cur {
+		c.state[key] = tsNs
+	}
+}
+
+// Freeze pins key at its current position. If key has no recorded position
+// yet, it is set to tsNs.
+func (c *Cursor) Freeze(key s3lifecycle.ActionKey, tsNs int64) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if _, exists := c.state[key]; !exists {
+		c.state[key] = tsNs
+	}
+	c.frozen[key] = struct{}{}
+}
+
+// Unfreeze releases a freeze. Subsequent Advance calls take effect.
+func (c *Cursor) Unfreeze(key s3lifecycle.ActionKey) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	delete(c.frozen, key)
+}
+
+// IsFrozen reports whether key is currently pinned.
+func (c *Cursor) IsFrozen(key s3lifecycle.ActionKey) bool {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	_, ok := c.frozen[key]
+	return ok
+}
+
+// Snapshot copies the cursor map for persistence. Frozen keys are not
+// distinguished in the snapshot; callers persist freezes via the separate
+// blocker-record store.
+func (c *Cursor) Snapshot() map[s3lifecycle.ActionKey]int64 {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	out := make(map[s3lifecycle.ActionKey]int64, len(c.state))
+	for k, v := range c.state {
+		out[k] = v
+	}
+	return out
+}
+
+// Restore replaces the cursor map. Freezes are not restored here; the
+// caller re-applies them from blocker records on startup.
+func (c *Cursor) Restore(state map[s3lifecycle.ActionKey]int64) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.state = make(map[s3lifecycle.ActionKey]int64, len(state))
+	for k, v := range state {
+		c.state[k] = v
+	}
+	c.frozen = map[s3lifecycle.ActionKey]struct{}{}
+}

weed/s3api/s3lifecycle/reader/cursor_test.go

@@ -0,0 +1,109 @@
+package reader
+
+import (
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+func key(b string, kind s3lifecycle.ActionKind) s3lifecycle.ActionKey {
+	return s3lifecycle.ActionKey{Bucket: b, ActionKind: kind}
+}
+
+func TestCursorMinTsNsEmpty(t *testing.T) {
+	c := NewCursor()
+	if got := c.MinTsNs(); got != 0 {
+		t.Fatalf("empty MinTsNs=%d, want 0", got)
+	}
+}
+
+func TestCursorAdvanceMonotonic(t *testing.T) {
+	c := NewCursor()
+	k := key("b", s3lifecycle.ActionKindExpirationDays)
+	c.Advance(k, 100)
+	c.Advance(k, 50)  // backward, ignored
+	c.Advance(k, 100) // equal, ignored
+	c.Advance(k, 200)
+	if got := c.Get(k); got != 200 {
+		t.Fatalf("Get=%d, want 200", got)
+	}
+}
+
+func TestCursorAdvanceIgnoresZero(t *testing.T) {
+	c := NewCursor()
+	k := key("b", s3lifecycle.ActionKindExpirationDays)
+	c.Advance(k, 0)
+	c.Advance(k, -1)
+	if got := c.Get(k); got != 0 {
+		t.Fatalf("Get=%d, want 0", got)
+	}
+}
+
+func TestCursorMinTsNsAcrossKeys(t *testing.T) {
+	c := NewCursor()
+	c.Advance(key("a", s3lifecycle.ActionKindExpirationDays), 100)
+	c.Advance(key("b", s3lifecycle.ActionKindExpirationDays), 50)
+	c.Advance(key("c", s3lifecycle.ActionKindAbortMPU), 200)
+	if got := c.MinTsNs(); got != 50 {
+		t.Fatalf("MinTsNs=%d, want 50", got)
+	}
+}
+
+func TestCursorFreeze(t *testing.T) {
+	c := NewCursor()
+	k := key("b", s3lifecycle.ActionKindExpirationDays)
+	c.Advance(k, 100)
+	c.Freeze(k, 100)
+	c.Advance(k, 200) // frozen — ignored
+	if got := c.Get(k); got != 100 {
+		t.Fatalf("Get after freeze=%d, want 100", got)
+	}
+	if !c.IsFrozen(k) {
+		t.Fatal("IsFrozen=false, want true")
+	}
+	c.Unfreeze(k)
+	c.Advance(k, 200)
+	if got := c.Get(k); got != 200 {
+		t.Fatalf("Get after unfreeze=%d, want 200", got)
+	}
+}
+
+func TestCursorFreezeOnUnsetKeySeedsPosition(t *testing.T) {
+	// Freezing a key with no recorded position seeds it at the freeze tsNs;
+	// subsequent MinTsNs reflects the freeze, so the subscription doesn't
+	// rewind past it.
+	c := NewCursor()
+	k := key("b", s3lifecycle.ActionKindExpirationDays)
+	c.Freeze(k, 500)
+	if got := c.Get(k); got != 500 {
+		t.Fatalf("Get after freeze-on-unset=%d, want 500", got)
+	}
+	if got := c.MinTsNs(); got != 500 {
+		t.Fatalf("MinTsNs=%d, want 500", got)
+	}
+}
+
+func TestCursorSnapshotRestore(t *testing.T) {
+	c := NewCursor()
+	a := key("a", s3lifecycle.ActionKindExpirationDays)
+	b := key("b", s3lifecycle.ActionKindAbortMPU)
+	c.Advance(a, 100)
+	c.Advance(b, 200)
+
+	snap := c.Snapshot()
+	if len(snap) != 2 || snap[a] != 100 || snap[b] != 200 {
+		t.Fatalf("Snapshot=%v", snap)
+	}
+
+	c2 := NewCursor()
+	c2.Restore(snap)
+	if got := c2.Get(a); got != 100 {
+		t.Fatalf("restored Get(a)=%d, want 100", got)
+	}
+	if got := c2.Get(b); got != 200 {
+		t.Fatalf("restored Get(b)=%d, want 200", got)
+	}
+	if c2.IsFrozen(a) {
+		t.Fatal("Restore should not carry freezes")
+	}
+}

weed/s3api/s3lifecycle/reader/event_predicates_test.go

@@ -0,0 +1,47 @@
+package reader
+
+import (
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/stretchr/testify/assert"
+)
+
+// IsCreate / IsDelete are the small but routing-critical predicates
+// the dispatcher uses to decide which match path applies. Both were
+// previously exercised only through end-to-end Tick/Match tests; pin
+// them directly here.
+
+func TestEventIsCreate_PopulatedNewEntryNoOldEntry(t *testing.T) {
+	e := &Event{NewEntry: &filer_pb.Entry{Name: "k"}}
+	assert.True(t, e.IsCreate())
+	assert.False(t, e.IsDelete())
+}
+
+func TestEventIsDelete_PopulatedOldEntryNoNewEntry(t *testing.T) {
+	e := &Event{OldEntry: &filer_pb.Entry{Name: "k"}}
+	assert.True(t, e.IsDelete())
+	assert.False(t, e.IsCreate())
+}
+
+func TestEventBothEntries_NeitherCreateNorDelete(t *testing.T) {
+	// An update event carries both old and new; the predicates are
+	// strict (one or the other, not both) so the router can route
+	// updates through the dedicated path rather than treating them
+	// as creates or deletes.
+	e := &Event{
+		OldEntry: &filer_pb.Entry{Name: "k"},
+		NewEntry: &filer_pb.Entry{Name: "k"},
+	}
+	assert.False(t, e.IsCreate(), "update event must not classify as create")
+	assert.False(t, e.IsDelete(), "update event must not classify as delete")
+}
+
+func TestEventNoEntries_NeitherCreateNorDelete(t *testing.T) {
+	// A degenerate event with neither side populated must not classify
+	// as either; otherwise a metadata-only filer event with no bucket
+	// payload could trigger spurious dispatches.
+	e := &Event{}
+	assert.False(t, e.IsCreate())
+	assert.False(t, e.IsDelete())
+}

weed/s3api/s3lifecycle/reader/log_startup_test.go

@@ -0,0 +1,44 @@
+package reader
+
+import (
+	"testing"
+)
+
+// LogStartup writes a single glog line summarising the reader's
+// resume position; the only behavioral output is a side effect on the
+// log sink, but exercising both branches still pins compile-time
+// shape (e.g. that ShardPredicate-set readers don't trip on a missing
+// ShardID and vice versa) and lets coverage actually visit the code.
+
+func TestLogStartup_ShardIDOnly(t *testing.T) {
+	// Single-shard configuration: ShardID is set, ShardPredicate is nil,
+	// no Cursor, no StartTsNs. The function must run without panic.
+	r := &Reader{ShardID: 7, EventBudget: 100}
+	r.LogStartup()
+}
+
+func TestLogStartup_ShardPredicate(t *testing.T) {
+	// ShardPredicate-set readers take a different log branch; pinning
+	// the call here catches a regression that returns or panics.
+	r := &Reader{
+		ShardPredicate: func(int) bool { return true },
+		EventBudget:    100,
+	}
+	r.LogStartup()
+}
+
+func TestLogStartup_StartTsNsOverridesCursor(t *testing.T) {
+	// Explicit StartTsNs takes precedence over Cursor.MinTsNs; this
+	// branch is otherwise only hit when a worker is replaying a
+	// specific position. Run it through to make sure the override is
+	// honored without consulting the Cursor.
+	r := &Reader{ShardID: 0, StartTsNs: 1700000000_000_000_000, Cursor: NewCursor()}
+	r.LogStartup()
+}
+
+func TestLogStartup_CursorMinFallback(t *testing.T) {
+	// StartTsNs=0 with a non-nil Cursor falls back to Cursor.MinTsNs.
+	c := NewCursor()
+	r := &Reader{ShardID: 0, Cursor: c}
+	r.LogStartup()
+}

weed/s3api/s3lifecycle/reader/persister.go

@@ -0,0 +1,55 @@
+package reader
+
+import (
+	"context"
+	"sync"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+// Persister loads and stores per-shard cursor state. The contract:
+//   - Load returns an empty map (not an error) when no prior state exists.
+//   - Save replaces the prior state atomically; partial writes must not be
+//     observable on a later Load.
+//   - Concurrent Save calls for the same shardID are serialized by the
+//     caller (one Reader per shard at a time).
+//
+// The filer-backed implementation lives in the worker package alongside the
+// rest of the persistence wiring; this package only ships the in-memory
+// variant used by tests.
+type Persister interface {
+	Load(ctx context.Context, shardID int) (map[s3lifecycle.ActionKey]int64, error)
+	Save(ctx context.Context, shardID int, state map[s3lifecycle.ActionKey]int64) error
+}
+
+// InMemoryPersister is a Persister for tests. Save copies the input so
+// later mutations by the caller don't leak into stored state.
+type InMemoryPersister struct {
+	mu      sync.Mutex
+	storage map[int]map[s3lifecycle.ActionKey]int64
+}
+
+func NewInMemoryPersister() *InMemoryPersister {
+	return &InMemoryPersister{storage: map[int]map[s3lifecycle.ActionKey]int64{}}
+}
+
+func (p *InMemoryPersister) Load(ctx context.Context, shardID int) (map[s3lifecycle.ActionKey]int64, error) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	out := map[s3lifecycle.ActionKey]int64{}
+	for k, v := range p.storage[shardID] {
+		out[k] = v
+	}
+	return out, nil
+}
+
+func (p *InMemoryPersister) Save(ctx context.Context, shardID int, state map[s3lifecycle.ActionKey]int64) error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	copied := make(map[s3lifecycle.ActionKey]int64, len(state))
+	for k, v := range state {
+		copied[k] = v
+	}
+	p.storage[shardID] = copied
+	return nil
+}

weed/s3api/s3lifecycle/reader/persister_test.go

@@ -0,0 +1,161 @@
+package reader
+
+import (
+	"context"
+	"sync"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// InMemoryPersister is the in-memory test double other lifecycle tests
+// rely on for cursor checkpointing. The deep-copy contract documented
+// on the Persister interface is what insulates each test's state from
+// the others — these tests pin every direction of that contract so a
+// regression here can't silently corrupt downstream tests.
+
+func TestInMemoryPersister_LoadOnUnknownShardReturnsEmptyMap(t *testing.T) {
+	// Contract: Load returns an empty map (not an error) when no prior
+	// state exists. The dispatcher startup path relies on this so a
+	// fresh worker doesn't have to special-case the "no checkpoint yet"
+	// branch.
+	p := NewInMemoryPersister()
+	got, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	assert.NotNil(t, got)
+	assert.Empty(t, got)
+}
+
+func TestInMemoryPersister_SaveLoadRoundTrip(t *testing.T) {
+	p := NewInMemoryPersister()
+	want := map[s3lifecycle.ActionKey]int64{
+		key("a", s3lifecycle.ActionKindExpirationDays): 100,
+		key("b", s3lifecycle.ActionKindNoncurrentDays): 200,
+	}
+	require.NoError(t, p.Save(context.Background(), 7, want))
+
+	got, err := p.Load(context.Background(), 7)
+	require.NoError(t, err)
+	assert.Equal(t, want, got)
+}
+
+func TestInMemoryPersister_SaveCopiesInput(t *testing.T) {
+	// Caller mutating the map after Save must not bleed into stored
+	// state — the next Load must return what was saved.
+	p := NewInMemoryPersister()
+	in := map[s3lifecycle.ActionKey]int64{
+		key("a", s3lifecycle.ActionKindExpirationDays): 100,
+	}
+	require.NoError(t, p.Save(context.Background(), 0, in))
+
+	in[key("a", s3lifecycle.ActionKindExpirationDays)] = 9_999
+	in[key("z", s3lifecycle.ActionKindAbortMPU)] = 42
+	delete(in, key("a", s3lifecycle.ActionKindExpirationDays))
+
+	got, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	assert.Equal(t, int64(100), got[key("a", s3lifecycle.ActionKindExpirationDays)],
+		"caller-side mutation must not bleed into stored state")
+	assert.NotContains(t, got, key("z", s3lifecycle.ActionKindAbortMPU),
+		"caller-side insertion must not bleed into stored state")
+}
+
+func TestInMemoryPersister_LoadReturnsACopy(t *testing.T) {
+	// Caller mutating the returned map must not bleed back; otherwise
+	// a test reusing the persister across cases could see drift.
+	p := NewInMemoryPersister()
+	want := map[s3lifecycle.ActionKey]int64{
+		key("a", s3lifecycle.ActionKindExpirationDays): 100,
+	}
+	require.NoError(t, p.Save(context.Background(), 0, want))
+
+	first, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	first[key("a", s3lifecycle.ActionKindExpirationDays)] = 9_999
+	first[key("z", s3lifecycle.ActionKindAbortMPU)] = 42
+
+	second, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	assert.Equal(t, int64(100), second[key("a", s3lifecycle.ActionKindExpirationDays)],
+		"caller-side mutation of the first snapshot must not bleed back")
+	assert.NotContains(t, second, key("z", s3lifecycle.ActionKindAbortMPU),
+		"caller-side insertion into the first snapshot must not bleed back")
+}
+
+func TestInMemoryPersister_SaveReplacesNotMerges(t *testing.T) {
+	// Documented contract: Save replaces prior state atomically. A
+	// merge would silently keep stale resume points across restarts.
+	p := NewInMemoryPersister()
+	require.NoError(t, p.Save(context.Background(), 0, map[s3lifecycle.ActionKey]int64{
+		key("a", s3lifecycle.ActionKindExpirationDays): 100,
+		key("b", s3lifecycle.ActionKindNoncurrentDays): 200,
+	}))
+	require.NoError(t, p.Save(context.Background(), 0, map[s3lifecycle.ActionKey]int64{
+		key("c", s3lifecycle.ActionKindAbortMPU): 300,
+	}))
+
+	got, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	assert.Equal(t, map[s3lifecycle.ActionKey]int64{
+		key("c", s3lifecycle.ActionKindAbortMPU): 300,
+	}, got, "second Save must replace prior state, not merge")
+}
+
+func TestInMemoryPersister_DifferentShardsIsolated(t *testing.T) {
+	// State for shardID=1 must not leak into Load(shardID=2).
+	p := NewInMemoryPersister()
+	require.NoError(t, p.Save(context.Background(), 1, map[s3lifecycle.ActionKey]int64{
+		key("a", s3lifecycle.ActionKindExpirationDays): 100,
+	}))
+	require.NoError(t, p.Save(context.Background(), 2, map[s3lifecycle.ActionKey]int64{
+		key("b", s3lifecycle.ActionKindNoncurrentDays): 200,
+	}))
+
+	got1, err := p.Load(context.Background(), 1)
+	require.NoError(t, err)
+	got2, err := p.Load(context.Background(), 2)
+	require.NoError(t, err)
+	assert.Equal(t, int64(100), got1[key("a", s3lifecycle.ActionKindExpirationDays)])
+	assert.NotContains(t, got1, key("b", s3lifecycle.ActionKindNoncurrentDays))
+	assert.Equal(t, int64(200), got2[key("b", s3lifecycle.ActionKindNoncurrentDays)])
+	assert.NotContains(t, got2, key("a", s3lifecycle.ActionKindExpirationDays))
+}
+
+func TestInMemoryPersister_SaveEmptyMapClearsState(t *testing.T) {
+	// An empty map is valid input; Save must accept it and a subsequent
+	// Load must return an empty map (not the prior contents).
+	p := NewInMemoryPersister()
+	require.NoError(t, p.Save(context.Background(), 0, map[s3lifecycle.ActionKey]int64{
+		key("a", s3lifecycle.ActionKindExpirationDays): 100,
+	}))
+	require.NoError(t, p.Save(context.Background(), 0, map[s3lifecycle.ActionKey]int64{}))
+	got, err := p.Load(context.Background(), 0)
+	require.NoError(t, err)
+	assert.Empty(t, got)
+}
+
+func TestInMemoryPersister_ConcurrentSaveLoadDoesNotRace(t *testing.T) {
+	// The dispatcher fans cursor reads/writes across many goroutines;
+	// the persister must hold under load without livelocking or
+	// data-racing (-race detector enforces that).
+	p := NewInMemoryPersister()
+	var wg sync.WaitGroup
+	const N = 64
+	wg.Add(N * 2)
+	for i := 0; i < N; i++ {
+		i := i
+		go func() {
+			defer wg.Done()
+			_ = p.Save(context.Background(), i%4, map[s3lifecycle.ActionKey]int64{
+				key("k", s3lifecycle.ActionKindExpirationDays): int64(i),
+			})
+		}()
+		go func() {
+			defer wg.Done()
+			_, _ = p.Load(context.Background(), i%4)
+		}()
+	}
+	wg.Wait()
+}

weed/s3api/s3lifecycle/reader/reader.go

@@ -0,0 +1,338 @@
+package reader
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"strings"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+// Event is one in-shard meta-log event delivered to the router.
+//
+// BootstrapVersion is set only by the bucket bootstrapper when it
+// expands a .versions/ directory; the meta-log path leaves it nil.
+// Carries pre-computed sibling state so the router can fire
+// NoncurrentDays / NewerNoncurrent without listing again.
+type Event struct {
+	TsNs             int64
+	Bucket           string
+	Key              string
+	ShardID          int
+	OldEntry         *filer_pb.Entry
+	NewEntry         *filer_pb.Entry
+	NewParent        string
+	BootstrapVersion *BootstrapVersion
+}
+
+// BootstrapVersion is the per-version state computed once per
+// .versions/<key>/ directory at bootstrap time. Key fields shape
+// EvaluateAction: IsLatest gates current vs. noncurrent rules,
+// NoncurrentIndex gates NewerNoncurrentVersions retention,
+// SuccessorModTime sets the noncurrent clock (when this version was
+// replaced).
+type BootstrapVersion struct {
+	LogicalKey       string
+	VersionID        string
+	IsLatest         bool
+	IsDeleteMarker   bool
+	NumVersions      int
+	NoncurrentIndex  int // 0 = newest noncurrent
+	SuccessorModTime time.Time
+}
+
+// IsDelete reports whether this event removes an entry.
+func (e *Event) IsDelete() bool {
+	return e.NewEntry == nil && e.OldEntry != nil
+}
+
+// IsCreate reports whether this event creates an entry.
+func (e *Event) IsCreate() bool {
+	return e.OldEntry == nil && e.NewEntry != nil
+}
+
+// Reader subscribes to the filer meta-log and emits in-range Events to a
+// channel. One subscription handles a contiguous span (or arbitrary set)
+// of shards via ShardPredicate; the downstream router/dispatcher consume
+// events and ack-advance the per-shard cursor for matched ActionKeys
+// when their actions complete.
+type Reader struct {
+	// ShardID and ShardPredicate are alternatives — set at most one.
+	// ShardPredicate wins if both are populated.
+	ShardID        int            // [0, s3lifecycle.ShardCount); used when ShardPredicate is nil
+	ShardPredicate func(int) bool // accepts an event when true; nil falls back to ShardID equality
+
+	BucketsPath string // e.g. "/buckets"
+	// Cursor is the single-shard cursor used for SinceNs when StartTsNs is 0.
+	// Range callers pass StartTsNs directly and leave Cursor nil; SinceNs=0
+	// then means "subscribe from the start of the meta-log".
+	Cursor    *Cursor
+	StartTsNs int64
+	Events    chan<- *Event
+
+	// UntilTsNs ends the stream once the filer has delivered through this
+	// timestamp. Zero follows forever, which on an idle cluster parks a
+	// bounded caller in Recv until something unrelated is written.
+	UntilTsNs int64
+
+	// EventBudget caps how many events Run processes before returning nil.
+	// Zero = unbounded; the run continues until ctx cancellation or stream
+	// error. Used by the worker scheduler to bound a single READ task.
+	EventBudget int
+
+	// ReceiveTimeout bounds the wait for each response, covering a filer
+	// that stops producing while the transport still answers keepalives.
+	// Also opts into idle heartbeats so a caught-up stream stays alive.
+	// Keep above the filer's 15m maxGapStall — a subscriber parked on a
+	// gap sends nothing and is not stuck. Zero disables it.
+	ReceiveTimeout time.Duration
+
+	// bucketsPathSlash is BucketsPath with a guaranteed trailing slash,
+	// computed once on Run and reused per event to avoid recomputing the
+	// normalized prefix in extractBucketKey.
+	bucketsPathSlash string
+}
+
+// ErrReceiveTimeout: stream still open, but no events and no heartbeats.
+var ErrReceiveTimeout = errors.New("reader: metadata receive timeout")
+
+type receiveResult struct {
+	resp *filer_pb.SubscribeMetadataResponse
+	err  error
+}
+
+// awaitResponse waits for the next response under ReceiveTimeout. Started
+// per call, so it times the filer only — a slow Events consumer blocks in
+// dispatchOne, outside this window, and can't trip the watchdog.
+func (r *Reader) awaitResponse(ctx context.Context, received <-chan receiveResult) (*filer_pb.SubscribeMetadataResponse, error, bool) {
+	var timeout <-chan time.Time
+	if r.ReceiveTimeout > 0 {
+		timer := time.NewTimer(r.ReceiveTimeout)
+		defer timer.Stop()
+		timeout = timer.C
+	}
+	select {
+	case result := <-received:
+		return result.resp, result.err, false
+	case <-timeout:
+		return nil, nil, true
+	case <-ctx.Done():
+		return nil, ctx.Err(), false
+	}
+}
+
+// Run subscribes via SubscribeMetadata starting at the configured position,
+// filters to the configured shard set, and emits Events. Returns on
+// ctx.Done(), io.EOF, or stream error. Caller is responsible for closing
+// Events if it owns it.
+func (r *Reader) Run(ctx context.Context, client filer_pb.SeaweedFilerClient, clientName string, clientID int32) error {
+	if r.ShardPredicate == nil {
+		if r.ShardID < 0 || r.ShardID >= s3lifecycle.ShardCount {
+			return fmt.Errorf("reader: shard_id %d out of range and no ShardPredicate", r.ShardID)
+		}
+	}
+	if r.Events == nil {
+		return errors.New("reader: nil Events channel")
+	}
+	if r.BucketsPath == "" {
+		return errors.New("reader: empty BucketsPath")
+	}
+	if r.ReceiveTimeout < 0 {
+		return fmt.Errorf("reader: negative ReceiveTimeout %v", r.ReceiveTimeout)
+	}
+	r.bucketsPathSlash = r.BucketsPath
+	if !strings.HasSuffix(r.bucketsPathSlash, "/") {
+		r.bucketsPathSlash += "/"
+	}
+
+	sinceNs := r.StartTsNs
+	if sinceNs == 0 && r.Cursor != nil {
+		sinceNs = r.Cursor.MinTsNs()
+	}
+	// Own context: aborting the RPC is the only way to unblock Recv.
+	streamCtx, cancelStream := context.WithCancel(ctx)
+	defer cancelStream()
+	stream, err := client.SubscribeMetadata(streamCtx, &filer_pb.SubscribeMetadataRequest{
+		ClientName:             clientName,
+		PathPrefix:             r.BucketsPath,
+		SinceNs:                sinceNs,
+		UntilNs:                r.UntilTsNs,
+		ClientId:               clientID,
+		ClientSupportsBatching: true,
+		// dispatchOne drops these, but arriving at all is the point.
+		ClientSupportsIdleHeartbeat: r.ReceiveTimeout > 0,
+	})
+	if err != nil {
+		return fmt.Errorf("subscribe: %w", err)
+	}
+
+	// Buffered so a response landing as the watchdog fires doesn't strand
+	// this goroutine.
+	received := make(chan receiveResult, 1)
+	go func() {
+		for {
+			resp, recvErr := stream.Recv()
+			select {
+			case received <- receiveResult{resp: resp, err: recvErr}:
+			case <-streamCtx.Done():
+				return
+			}
+			if recvErr != nil {
+				return
+			}
+		}
+	}()
+
+	processed := 0
+	for {
+		resp, recvErr, timedOut := r.awaitResponse(streamCtx, received)
+		if timedOut {
+			cancelStream() // unwind the goroutine blocked in Recv
+			return fmt.Errorf("%w after %s", ErrReceiveTimeout, r.ReceiveTimeout)
+		}
+		if recvErr == io.EOF {
+			return nil
+		}
+		if recvErr != nil {
+			return recvErr
+		}
+
+		// First event in resp is the primary; resp.Events is the batched tail.
+		if err := r.dispatchOne(ctx, resp, &processed); err != nil {
+			return err
+		}
+		for _, ev := range resp.Events {
+			if err := r.dispatchOne(ctx, ev, &processed); err != nil {
+				return err
+			}
+		}
+		if r.EventBudget > 0 && processed >= r.EventBudget {
+			return nil
+		}
+	}
+}
+
+func (r *Reader) dispatchOne(ctx context.Context, resp *filer_pb.SubscribeMetadataResponse, processed *int) error {
+	if resp == nil || resp.EventNotification == nil {
+		return nil
+	}
+	bucket, key, ok := r.extractBucketKey(resp)
+	if !ok {
+		return nil
+	}
+	shardID := s3lifecycle.ShardID(bucket, key)
+	if r.ShardPredicate != nil {
+		if !r.ShardPredicate(shardID) {
+			return nil
+		}
+	} else if shardID != r.ShardID {
+		return nil
+	}
+
+	ev := &Event{
+		TsNs:      resp.TsNs,
+		Bucket:    bucket,
+		Key:       key,
+		ShardID:   shardID,
+		OldEntry:  resp.EventNotification.OldEntry,
+		NewEntry:  resp.EventNotification.NewEntry,
+		NewParent: resp.EventNotification.NewParentPath,
+	}
+	select {
+	case <-ctx.Done():
+		return ctx.Err()
+	case r.Events <- ev:
+		*processed++
+		return nil
+	}
+}
+
+// extractBucketKey turns a meta-log event's path into (bucket, key) when the
+// event lies under BucketsPath. Returns ok=false for events outside that
+// subtree (cluster admin entries, system files, etc.) so the reader can skip
+// them without engaging the routing index.
+//
+// The path is reconstructed as DirectoryPath/Name, where DirectoryPath comes
+// from the entry context and Name from old_entry/new_entry. We prefer
+// new_entry on creates/updates and old_entry on deletes; both carry the same
+// Name on renames where new_parent_path differs.
+func (r *Reader) extractBucketKey(resp *filer_pb.SubscribeMetadataResponse) (string, string, bool) {
+	notif := resp.EventNotification
+	dir := notif.NewParentPath
+	if dir == "" {
+		// On deletes, NewParentPath may be empty; the directory is encoded
+		// in resp.Directory.
+		dir = resp.Directory
+	}
+	var name string
+	switch {
+	case notif.NewEntry != nil:
+		name = notif.NewEntry.Name
+	case notif.OldEntry != nil:
+		name = notif.OldEntry.Name
+	default:
+		return "", "", false
+	}
+
+	// Pre-normalized prefix (BucketsPath with trailing slash) is computed
+	// once in Run; bucket-root events arrive as either "/buckets" or
+	// "/buckets/", so accept both. The fallback path mirrors Run's
+	// normalization for tests that call extractBucketKey directly.
+	prefix := r.bucketsPathSlash
+	if prefix == "" {
+		prefix = r.BucketsPath
+		if !strings.HasSuffix(prefix, "/") {
+			prefix += "/"
+		}
+	}
+	bare := strings.TrimSuffix(prefix, "/")
+	var rest string
+	switch {
+	case dir == bare || dir == prefix:
+		// Bucket create/delete at /buckets root: bucket name is the entry name.
+		if name == "" {
+			return "", "", false
+		}
+		return name, "", true
+	case strings.HasPrefix(dir, prefix):
+		rest = dir[len(prefix):]
+	default:
+		return "", "", false
+	}
+	// rest = "<bucket>" or "<bucket>/<sub>/<sub>..."
+	slash := strings.IndexByte(rest, '/')
+	var bucket, parentInBucket string
+	if slash < 0 {
+		bucket = rest
+	} else {
+		bucket = rest[:slash]
+		parentInBucket = rest[slash+1:]
+	}
+	if bucket == "" {
+		return "", "", false
+	}
+	if parentInBucket != "" {
+		return bucket, parentInBucket + "/" + name, true
+	}
+	return bucket, name, true
+}
+
+// LogStartup is a small helper for callers that want a one-line readable
+// description of where the reader is starting.
+func (r *Reader) LogStartup() {
+	sinceNs := r.StartTsNs
+	if sinceNs == 0 && r.Cursor != nil {
+		sinceNs = r.Cursor.MinTsNs()
+	}
+	if r.ShardPredicate != nil {
+		glog.V(1).Infof("lifecycle reader: shard=range sinceNs=%d budget=%d", sinceNs, r.EventBudget)
+		return
+	}
+	glog.V(1).Infof("lifecycle reader: shard=%d sinceNs=%d budget=%d",
+		r.ShardID, sinceNs, r.EventBudget)
+}

weed/s3api/s3lifecycle/reader/reader_test.go

@@ -0,0 +1,184 @@
+package reader
+
+import (
+	"context"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+func TestExtractBucketKeyCreate(t *testing.T) {
+	r := &Reader{BucketsPath: "/buckets"}
+	resp := &filer_pb.SubscribeMetadataResponse{
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/buckets/mybucket/path/to",
+			NewEntry:      &filer_pb.Entry{Name: "obj.txt"},
+		},
+	}
+	b, k, ok := r.extractBucketKey(resp)
+	if !ok || b != "mybucket" || k != "path/to/obj.txt" {
+		t.Fatalf("got (%q,%q,%v), want (mybucket,path/to/obj.txt,true)", b, k, ok)
+	}
+}
+
+func TestExtractBucketKeyTopLevelObject(t *testing.T) {
+	r := &Reader{BucketsPath: "/buckets"}
+	resp := &filer_pb.SubscribeMetadataResponse{
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/buckets/mybucket",
+			NewEntry:      &filer_pb.Entry{Name: "a.txt"},
+		},
+	}
+	b, k, ok := r.extractBucketKey(resp)
+	if !ok || b != "mybucket" || k != "a.txt" {
+		t.Fatalf("got (%q,%q,%v), want (mybucket,a.txt,true)", b, k, ok)
+	}
+}
+
+func TestExtractBucketKeyDeleteUsesOldEntry(t *testing.T) {
+	r := &Reader{BucketsPath: "/buckets"}
+	resp := &filer_pb.SubscribeMetadataResponse{
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/buckets/b/dir",
+			OldEntry:      &filer_pb.Entry{Name: "gone.txt"},
+		},
+	}
+	b, k, ok := r.extractBucketKey(resp)
+	if !ok || b != "b" || k != "dir/gone.txt" {
+		t.Fatalf("got (%q,%q,%v), want (b,dir/gone.txt,true)", b, k, ok)
+	}
+}
+
+func TestExtractBucketKeyDeleteWithEmptyParent(t *testing.T) {
+	// Some delete events carry the directory in resp.Directory rather than
+	// EventNotification.NewParentPath.
+	r := &Reader{BucketsPath: "/buckets"}
+	resp := &filer_pb.SubscribeMetadataResponse{
+		Directory: "/buckets/b/dir",
+		EventNotification: &filer_pb.EventNotification{
+			OldEntry: &filer_pb.Entry{Name: "gone.txt"},
+		},
+	}
+	b, k, ok := r.extractBucketKey(resp)
+	if !ok || b != "b" || k != "dir/gone.txt" {
+		t.Fatalf("got (%q,%q,%v), want (b,dir/gone.txt,true)", b, k, ok)
+	}
+}
+
+func TestExtractBucketKeyOutsideBucketsSkipped(t *testing.T) {
+	r := &Reader{BucketsPath: "/buckets"}
+	resp := &filer_pb.SubscribeMetadataResponse{
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/etc/something",
+			NewEntry:      &filer_pb.Entry{Name: "config"},
+		},
+	}
+	if _, _, ok := r.extractBucketKey(resp); ok {
+		t.Fatal("expected skip for non-bucket path")
+	}
+}
+
+func TestExtractBucketKeyBucketCreateAtRoot(t *testing.T) {
+	// Creating a new bucket emits an event at /buckets/ with the bucket as
+	// the new entry's Name.
+	r := &Reader{BucketsPath: "/buckets"}
+	resp := &filer_pb.SubscribeMetadataResponse{
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/buckets/",
+			NewEntry:      &filer_pb.Entry{Name: "newbucket"},
+		},
+	}
+	b, k, ok := r.extractBucketKey(resp)
+	if !ok || b != "newbucket" || k != "" {
+		t.Fatalf("got (%q,%q,%v), want (newbucket,,true)", b, k, ok)
+	}
+}
+
+func TestExtractBucketKeyBucketCreateBareRoot(t *testing.T) {
+	// Some delete and create events emit the parent without a trailing
+	// slash. /buckets (no trailing slash) must also resolve to a bucket-
+	// root event with the entry name as the bucket.
+	r := &Reader{BucketsPath: "/buckets"}
+	resp := &filer_pb.SubscribeMetadataResponse{
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/buckets",
+			NewEntry:      &filer_pb.Entry{Name: "newbucket"},
+		},
+	}
+	b, k, ok := r.extractBucketKey(resp)
+	if !ok || b != "newbucket" || k != "" {
+		t.Fatalf("bare /buckets parent: got (%q,%q,%v), want (newbucket,,true)", b, k, ok)
+	}
+}
+
+func TestDispatchOneFiltersByShard(t *testing.T) {
+	// Pick a bucket+key whose shard is known, set Reader to a different
+	// shard, expect skip; same shard, expect emit.
+	bucket, key := "mybucket", "thing.txt"
+	target := s3lifecycle.ShardID(bucket, key)
+	other := (target + 1) % s3lifecycle.ShardCount
+
+	resp := &filer_pb.SubscribeMetadataResponse{
+		TsNs: 12345,
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/buckets/" + bucket,
+			NewEntry:      &filer_pb.Entry{Name: key},
+		},
+	}
+
+	out := make(chan *Event, 1)
+	r := &Reader{
+		ShardID:     other,
+		BucketsPath: "/buckets",
+		Cursor:      NewCursor(),
+		Events:      out,
+	}
+	processed := 0
+	if err := r.dispatchOne(context.Background(), resp, &processed); err != nil {
+		t.Fatalf("dispatchOne err: %v", err)
+	}
+	if processed != 0 || len(out) != 0 {
+		t.Fatalf("wrong-shard event should be skipped: processed=%d emitted=%d", processed, len(out))
+	}
+
+	r.ShardID = target
+	if err := r.dispatchOne(context.Background(), resp, &processed); err != nil {
+		t.Fatalf("dispatchOne err: %v", err)
+	}
+	if processed != 1 || len(out) != 1 {
+		t.Fatalf("matching-shard event should be emitted: processed=%d emitted=%d", processed, len(out))
+	}
+	ev := <-out
+	if ev.Bucket != bucket || ev.Key != key || ev.TsNs != 12345 {
+		t.Fatalf("emitted event mismatch: %+v", ev)
+	}
+}
+
+func TestDispatchOneRespectsCtxCancel(t *testing.T) {
+	// Buffered channel of size 0 (unbuffered) blocks the send; ctx cancel
+	// should unblock and return ctx.Err().
+	bucket, key := "mybucket", "thing.txt"
+	target := s3lifecycle.ShardID(bucket, key)
+	resp := &filer_pb.SubscribeMetadataResponse{
+		TsNs: 1,
+		EventNotification: &filer_pb.EventNotification{
+			NewParentPath: "/buckets/" + bucket,
+			NewEntry:      &filer_pb.Entry{Name: key},
+		},
+	}
+	out := make(chan *Event)
+	r := &Reader{
+		ShardID:     target,
+		BucketsPath: "/buckets",
+		Cursor:      NewCursor(),
+		Events:      out,
+	}
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	processed := 0
+	err := r.dispatchOne(ctx, resp, &processed)
+	if err != context.Canceled {
+		t.Fatalf("expected ctx.Canceled, got %v", err)
+	}
+}

weed/s3api/s3lifecycle/reader/receive_timeout_test.go

@@ -0,0 +1,117 @@
+package reader
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"google.golang.org/grpc"
+)
+
+// stalledStream stays open and never delivers: keepalives answered, no
+// progress.
+type stalledStream struct {
+	grpc.ClientStream
+	ctx context.Context
+}
+
+func (s *stalledStream) Recv() (*filer_pb.SubscribeMetadataResponse, error) {
+	<-s.ctx.Done()
+	return nil, s.ctx.Err()
+}
+
+type stalledClient struct {
+	filer_pb.SeaweedFilerClient
+	req *filer_pb.SubscribeMetadataRequest
+}
+
+func (c *stalledClient) SubscribeMetadata(ctx context.Context, req *filer_pb.SubscribeMetadataRequest, _ ...grpc.CallOption) (filer_pb.SeaweedFiler_SubscribeMetadataClient, error) {
+	c.req = req
+	return &stalledStream{ctx: ctx}, nil
+}
+
+func TestRun_ReceiveTimeoutOnStalledStream(t *testing.T) {
+	client := &stalledClient{}
+	r := &Reader{
+		ShardPredicate: func(int) bool { return true },
+		BucketsPath:    "/buckets",
+		Events:         make(chan *Event, 1),
+		ReceiveTimeout: 50 * time.Millisecond,
+	}
+
+	done := make(chan error, 1)
+	go func() { done <- r.Run(context.Background(), client, "test", 1) }()
+	select {
+	case err := <-done:
+		require.ErrorIs(t, err, ErrReceiveTimeout)
+	case <-time.After(5 * time.Second):
+		t.Fatal("Run did not time out on a stalled stream")
+	}
+	assert.True(t, client.req.ClientSupportsIdleHeartbeat,
+		"a reader with a receive timeout must ask for heartbeats, or a caught-up stream looks stalled")
+}
+
+// heartbeatStream delivers only heartbeats: a ts, no EventNotification.
+type heartbeatStream struct {
+	grpc.ClientStream
+	every time.Duration
+	sent  int
+	max   int
+}
+
+func (s *heartbeatStream) Recv() (*filer_pb.SubscribeMetadataResponse, error) {
+	if s.sent >= s.max {
+		return nil, context.Canceled
+	}
+	time.Sleep(s.every)
+	s.sent++
+	return &filer_pb.SubscribeMetadataResponse{TsNs: int64(s.sent)}, nil
+}
+
+type heartbeatClient struct {
+	filer_pb.SeaweedFilerClient
+	stream *heartbeatStream
+}
+
+func (c *heartbeatClient) SubscribeMetadata(_ context.Context, _ *filer_pb.SubscribeMetadataRequest, _ ...grpc.CallOption) (filer_pb.SeaweedFiler_SubscribeMetadataClient, error) {
+	return c.stream, nil
+}
+
+// Why the timeout is safe to set: a caught-up stream keeps arriving, so
+// the watchdog fires on real silence, not on an idle cluster.
+func TestRun_HeartbeatsHoldTheStreamOpen(t *testing.T) {
+	stream := &heartbeatStream{every: 20 * time.Millisecond, max: 10}
+	r := &Reader{
+		ShardPredicate: func(int) bool { return true },
+		BucketsPath:    "/buckets",
+		Events:         make(chan *Event, 1),
+		ReceiveTimeout: 200 * time.Millisecond,
+	}
+
+	done := make(chan error, 1)
+	go func() { done <- r.Run(context.Background(), &heartbeatClient{stream: stream}, "test", 1) }()
+	select {
+	case err := <-done:
+		assert.False(t, errors.Is(err, ErrReceiveTimeout),
+			"heartbeats spanning more than one timeout window must keep the stream alive")
+	case <-time.After(5 * time.Second):
+		t.Fatal("Run never returned")
+	}
+	assert.Equal(t, 10, stream.sent, "every heartbeat should have been consumed")
+}
+
+func TestRun_RejectsNegativeReceiveTimeout(t *testing.T) {
+	r := &Reader{
+		ShardPredicate: func(int) bool { return true },
+		BucketsPath:    "/buckets",
+		Events:         make(chan *Event, 1),
+		ReceiveTimeout: -time.Second,
+	}
+	err := r.Run(context.Background(), &stalledClient{}, "test", 1)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "ReceiveTimeout")
+}

weed/s3api/s3lifecycle/router/helpers_test.go

@@ -0,0 +1,335 @@
+package router
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Direct coverage for the pure helpers in router.go that the larger
+// Route* tests exercise indirectly. Pinning each one separately catches
+// regressions before they manifest as a broader Route failure that's
+// harder to diagnose.
+
+// ---------- successorModTimeFromContainer ----------
+
+func TestSuccessorModTimeFromContainer_MissingExtReturnsZero(t *testing.T) {
+	assert.True(t, successorModTimeFromContainer(&filer_pb.Entry{}).IsZero())
+	assert.True(t, successorModTimeFromContainer(&filer_pb.Entry{
+		Extended: map[string][]byte{},
+	}).IsZero())
+}
+
+func TestSuccessorModTimeFromContainer_EmptyValueReturnsZero(t *testing.T) {
+	got := successorModTimeFromContainer(&filer_pb.Entry{
+		Extended: map[string][]byte{s3_constants.ExtLatestVersionMtimeKey: nil},
+	})
+	assert.True(t, got.IsZero())
+}
+
+func TestSuccessorModTimeFromContainer_NonNumericReturnsZero(t *testing.T) {
+	// A malformed value is the writer's bug; the helper must not
+	// propagate a parse error or surface a nonsense time.
+	got := successorModTimeFromContainer(&filer_pb.Entry{
+		Extended: map[string][]byte{s3_constants.ExtLatestVersionMtimeKey: []byte("not-a-number")},
+	})
+	assert.True(t, got.IsZero())
+}
+
+func TestSuccessorModTimeFromContainer_NonPositiveReturnsZero(t *testing.T) {
+	// The container ext stores Unix seconds; <=0 means "not set" so
+	// the helper falls back to zero rather than emitting 1970-01-01.
+	for _, raw := range []string{"0", "-1", "-1000"} {
+		t.Run(raw, func(t *testing.T) {
+			got := successorModTimeFromContainer(&filer_pb.Entry{
+				Extended: map[string][]byte{s3_constants.ExtLatestVersionMtimeKey: []byte(raw)},
+			})
+			assert.True(t, got.IsZero(), "value %q must produce zero time", raw)
+		})
+	}
+}
+
+func TestSuccessorModTimeFromContainer_PositiveSecondsRoundTrip(t *testing.T) {
+	got := successorModTimeFromContainer(&filer_pb.Entry{
+		Extended: map[string][]byte{s3_constants.ExtLatestVersionMtimeKey: []byte("1700000000")},
+	})
+	assert.Equal(t, time.Unix(1700000000, 0).UTC(), got.UTC())
+}
+
+// ---------- logicalKeyFromVersionPath ----------
+
+func TestLogicalKeyFromVersionPath_ExtractsLogicalKey(t *testing.T) {
+	logical, ok := logicalKeyFromVersionPath("a/b/c.versions/v_abc")
+	require.True(t, ok)
+	assert.Equal(t, "a/b/c", logical)
+}
+
+func TestLogicalKeyFromVersionPath_RejectsPathWithoutVersionsParent(t *testing.T) {
+	// The parent of the version file must end with .versions; otherwise
+	// it's not a version path.
+	_, ok := logicalKeyFromVersionPath("a/b/c/v_abc")
+	assert.False(t, ok)
+}
+
+func TestLogicalKeyFromVersionPath_RejectsRootLevelPath(t *testing.T) {
+	// LastIndex returning 0 means the only "/" is at index 0, which
+	// would yield an empty parent — not a real version path.
+	_, ok := logicalKeyFromVersionPath("/v_abc")
+	assert.False(t, ok)
+}
+
+func TestLogicalKeyFromVersionPath_RejectsNoSlashes(t *testing.T) {
+	_, ok := logicalKeyFromVersionPath("v_abc")
+	assert.False(t, ok)
+	_, ok = logicalKeyFromVersionPath("")
+	assert.False(t, ok)
+}
+
+func TestLogicalKeyFromVersionPath_RejectsBareVersionsContainer(t *testing.T) {
+	// A path that's only the .versions container has no version-file
+	// child to extract a logical key from.
+	_, ok := logicalKeyFromVersionPath(s3_constants.VersionsFolder + "/v_x")
+	assert.False(t, ok, "logical key cannot be empty after trim")
+}
+
+// ---------- isVersionsContainerKey ----------
+
+func TestIsVersionsContainerKey(t *testing.T) {
+	cases := []struct {
+		key  string
+		want bool
+	}{
+		{"obj" + s3_constants.VersionsFolder, true},
+		{"a/b/obj" + s3_constants.VersionsFolder, true},
+		// Bucket-root .versions is rejected explicitly: it has no
+		// logical object key, so the router can't process it.
+		{s3_constants.VersionsFolder, false},
+		{"obj.versions/v_x", false}, // the version file inside, not the container
+		{"obj", false},
+		{"", false},
+	}
+	for _, c := range cases {
+		t.Run(c.key, func(t *testing.T) {
+			assert.Equal(t, c.want, isVersionsContainerKey(c.key))
+		})
+	}
+}
+
+// ---------- isVersionFolderPath ----------
+
+func TestIsVersionFolderPath(t *testing.T) {
+	// Reports whether a key sits inside a .versions/ folder, looking at
+	// the parent segment specifically. The router uses this to skip
+	// version-file events that need sibling state to classify.
+	cases := []struct {
+		key  string
+		want bool
+	}{
+		{"obj.versions/v_aaa", true},
+		{"a/b/obj.versions/v_aaa", true},
+		{"obj/v_aaa", false},    // parent isn't .versions
+		{"obj.versions", false}, // the container itself, not a child
+		// "obj.versions/" reads as the trailing-slash form of the path
+		// "obj.versions" → leaf is "obj.versions" which ends with the
+		// suffix → true. The router never sees this shape from the
+		// reader (events carry concrete leaf names), but pin it here so
+		// the contract is documented.
+		{"obj.versions/", true},
+		{"v_aaa", false}, // no slash at all
+		{"", false},
+	}
+	for _, c := range cases {
+		t.Run(c.key, func(t *testing.T) {
+			assert.Equal(t, c.want, isVersionFolderPath(c.key))
+		})
+	}
+}
+
+// ---------- isDeleteMarkerEntry ----------
+
+func TestIsDeleteMarkerEntry_NilOrEmptyExtendedReturnsFalse(t *testing.T) {
+	assert.False(t, isDeleteMarkerEntry(nil))
+	assert.False(t, isDeleteMarkerEntry(&filer_pb.Entry{}))
+	assert.False(t, isDeleteMarkerEntry(&filer_pb.Entry{Extended: map[string][]byte{}}))
+}
+
+func TestIsDeleteMarkerEntry_TrueOnlyForLiteralTrue(t *testing.T) {
+	// The marker key must be exactly "true"; any other value is not a
+	// marker. Pinning catches a regression that does case-insensitive
+	// matching or treats "1" as truthy.
+	mk := func(val string) *filer_pb.Entry {
+		return &filer_pb.Entry{Extended: map[string][]byte{s3_constants.ExtDeleteMarkerKey: []byte(val)}}
+	}
+	assert.True(t, isDeleteMarkerEntry(mk("true")))
+	assert.False(t, isDeleteMarkerEntry(mk("True")))
+	assert.False(t, isDeleteMarkerEntry(mk("TRUE")))
+	assert.False(t, isDeleteMarkerEntry(mk("1")))
+	assert.False(t, isDeleteMarkerEntry(mk("")))
+	assert.False(t, isDeleteMarkerEntry(mk("false")))
+}
+
+// ---------- extractTags ----------
+
+func TestExtractTags_NilOrEmptyReturnsNil(t *testing.T) {
+	assert.Nil(t, extractTags(nil))
+	assert.Nil(t, extractTags(map[string][]byte{}))
+}
+
+func TestExtractTags_OnlyKeysWithObjectTaggingPrefix(t *testing.T) {
+	// extractTags returns ExtVersionIdKey, Mime, etc. → no.
+	// It only picks up keys with the AmzObjectTagging prefix and strips
+	// that prefix to produce the tag map.
+	prefix := s3_constants.AmzObjectTagging + "-"
+	ext := map[string][]byte{
+		prefix + "env":       []byte("prod"),
+		prefix + "team":      []byte("data"),
+		"X-Amz-Other":        []byte("ignored"),
+		"Seaweed-X-Internal": []byte("ignored"),
+	}
+	got := extractTags(ext)
+	require.Len(t, got, 2)
+	assert.Equal(t, "prod", got["env"])
+	assert.Equal(t, "data", got["team"])
+}
+
+func TestExtractTags_ReturnsNilWhenNoTaggingPrefixedKeys(t *testing.T) {
+	// Keys with no tagging prefix produce a nil map (not an empty map);
+	// the router treats nil as "no tags" without an extra branch.
+	ext := map[string][]byte{
+		"X-Amz-Other":        []byte("ignored"),
+		"Seaweed-X-Internal": []byte("ignored"),
+	}
+	got := extractTags(ext)
+	assert.Nil(t, got)
+}
+
+// ---------- hasActiveEventDrivenAction ----------
+
+func TestHasActiveEventDrivenAction(t *testing.T) {
+	// Build a snapshot with two action kinds and one of them set to
+	// scan-only (inactive in event-driven view). hasActiveEventDrivenAction
+	// must answer true only for the active event-driven kind.
+	rule := &s3lifecycle.Rule{
+		ID:                          "r",
+		Status:                      s3lifecycle.StatusEnabled,
+		ExpirationDays:              7,
+		AbortMPUDaysAfterInitiation: 3,
+	}
+	hash := s3lifecycle.RuleHash(rule)
+	expirationKey := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindExpirationDays}
+	abortKey := s3lifecycle.ActionKey{Bucket: "bk", RuleHash: hash, ActionKind: s3lifecycle.ActionKindAbortMPU}
+	prior := map[s3lifecycle.ActionKey]engine.PriorState{
+		expirationKey: {BootstrapComplete: true, Mode: engine.ModeEventDriven},
+		abortKey:      {BootstrapComplete: true, Mode: engine.ModeScanOnly},
+	}
+	snap := engine.New().Compile(
+		[]engine.CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		engine.CompileOptions{PriorStates: prior},
+	)
+	keys := []s3lifecycle.ActionKey{expirationKey, abortKey}
+
+	// Active event-driven kind matches.
+	assert.True(t, hasActiveEventDrivenAction(snap, keys, s3lifecycle.ActionKindExpirationDays))
+	// Scan-only kind does NOT match — operator promoted it to scan-only
+	// and the router must respect that by skipping the event-driven path.
+	assert.False(t, hasActiveEventDrivenAction(snap, keys, s3lifecycle.ActionKindAbortMPU))
+	// A kind not in the keys list is false.
+	assert.False(t, hasActiveEventDrivenAction(snap, keys, s3lifecycle.ActionKindNoncurrentDays))
+}
+
+func TestHasActiveEventDrivenAction_NilActionSkipped(t *testing.T) {
+	// A key the snapshot doesn't know about returns nil from Action;
+	// the helper must skip rather than panic.
+	snap := engine.New().Compile(nil, engine.CompileOptions{})
+	keys := []s3lifecycle.ActionKey{
+		{Bucket: "ghost", ActionKind: s3lifecycle.ActionKindExpirationDays},
+	}
+	assert.False(t, hasActiveEventDrivenAction(snap, keys, s3lifecycle.ActionKindExpirationDays))
+}
+
+// ---------- engine.Snapshot accessors ----------
+
+func TestSnapshot_BucketVersionedReportsCompiledFlag(t *testing.T) {
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	snap := engine.New().Compile(
+		[]engine.CompileInput{
+			{Bucket: "vbk", Rules: []*s3lifecycle.Rule{rule}, Versioned: true},
+			{Bucket: "ubk", Rules: []*s3lifecycle.Rule{rule}, Versioned: false},
+		},
+		engine.CompileOptions{},
+	)
+	assert.True(t, snap.BucketVersioned("vbk"))
+	assert.False(t, snap.BucketVersioned("ubk"))
+	// Unknown bucket: not versioned, not a panic.
+	assert.False(t, snap.BucketVersioned("missing"))
+}
+
+func TestSnapshot_BucketActionKeysReturnsCompiledList(t *testing.T) {
+	// Every CompileInput rule emits one action key per RuleActionKind;
+	// BucketActionKeys must surface them all (regardless of Mode) so
+	// MatchPath can iterate.
+	rule := &s3lifecycle.Rule{ID: "r", Status: s3lifecycle.StatusEnabled, ExpirationDays: 7}
+	snap := engine.New().Compile(
+		[]engine.CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		engine.CompileOptions{},
+	)
+	keys := snap.BucketActionKeys("bk")
+	require.NotEmpty(t, keys)
+	for _, k := range keys {
+		assert.Equal(t, "bk", k.Bucket)
+	}
+}
+
+func TestSnapshot_BucketActionKeysUnknownBucketReturnsNil(t *testing.T) {
+	snap := engine.New().Compile(nil, engine.CompileOptions{})
+	assert.Nil(t, snap.BucketActionKeys("missing"))
+}
+
+func TestSnapshot_ActionUnknownKeyReturnsNil(t *testing.T) {
+	snap := engine.New().Compile(nil, engine.CompileOptions{})
+	got := snap.Action(s3lifecycle.ActionKey{Bucket: "ghost", ActionKind: s3lifecycle.ActionKindExpirationDays})
+	assert.Nil(t, got)
+}
+
+func TestSnapshot_AllActionsCoversEveryCompiledKind(t *testing.T) {
+	// AllActions must enumerate every compiled action regardless of
+	// active state, because the dispatcher uses it for full-snapshot
+	// reporting.
+	rule := &s3lifecycle.Rule{
+		ID:                          "r",
+		Status:                      s3lifecycle.StatusEnabled,
+		ExpirationDays:              7,
+		AbortMPUDaysAfterInitiation: 3,
+	}
+	snap := engine.New().Compile(
+		[]engine.CompileInput{{Bucket: "bk", Rules: []*s3lifecycle.Rule{rule}}},
+		engine.CompileOptions{},
+	)
+	all := snap.AllActions()
+	wantKinds := s3lifecycle.RuleActionKinds(rule)
+	require.Len(t, all, len(wantKinds))
+	seen := map[s3lifecycle.ActionKind]bool{}
+	for _, a := range all {
+		seen[a.Key.ActionKind] = true
+	}
+	for _, k := range wantKinds {
+		assert.True(t, seen[k], "missing kind %v in AllActions", k)
+	}
+}
+
+func TestSnapshot_SnapshotIDIsMonotonicAcrossRecompiles(t *testing.T) {
+	// Every Compile call advances snapshotIDSeq; pin that successive
+	// snapshots from the same Engine carry strictly increasing IDs so
+	// the dispatcher's stale-snapshot check works.
+	e := engine.New()
+	first := e.Compile(nil, engine.CompileOptions{})
+	second := e.Compile(nil, engine.CompileOptions{})
+	assert.Greater(t, second.SnapshotID(), first.SnapshotID(),
+		"second compile must produce a strictly greater snapshot id")
+}

weed/s3api/s3lifecycle/router/router.go

@@ -0,0 +1,873 @@
+package router
+
+import (
+	"context"
+	"sort"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
+)
+
+// SiblingLister inspects the surviving versions of a versioned key.
+// nil receiver or an error means "unknown" — callers suppress. Four
+// queries: Survivors paginates the .versions/ container plus the bare
+// null version (used by sole-survivor and bootstrap); LookupVersion
+// fetches a single version file by id (used by pointer-transition
+// routing to read the displaced version's identity and mtime);
+// ListVersions paginates every version file in the .versions/
+// container (used to compute NoncurrentIndex when a NewerNoncurrent
+// rule is active); LookupNullVersion returns the bare-key entry that
+// represents the null version (used by pointer-transition routing
+// when oldID is empty and to include the null in expansion ranks).
+type SiblingLister interface {
+	Survivors(ctx context.Context, bucket, objectKey string) (Survivors, error)
+	LookupVersion(ctx context.Context, bucket, objectKey, versionID string) (*filer_pb.Entry, error)
+	ListVersions(ctx context.Context, bucket, objectKey string) ([]*filer_pb.Entry, error)
+	LookupNullVersion(ctx context.Context, bucket, objectKey string) (entry *filer_pb.Entry, explicit bool, err error)
+}
+
+// Survivors describes the state under .versions/<key>/ plus the bare
+// null-version that exists when versioning was turned on after the
+// object was first PUT (s3api_object_versioning.go treats <bucket>/<key>
+// as a regular file, the null version, in that case).
+type Survivors struct {
+	Count          int             // entries under .versions/<key>/, capped at 2
+	LoneEntry      *filer_pb.Entry // populated when Count == 1
+	HasNullVersion bool            // bare <bucket>/<key> exists as a regular file
+}
+
+// Match is one (event, action) pair where EvaluateAction fired. The
+// dispatcher runs `LifecycleDelete` at DueTime; identity-CAS in the RPC
+// guards against drift between schedule time and dispatch time.
+//
+// For NoncurrentDays / NewerNoncurrent on a versioned bucket, ObjectKey
+// is the seaweedfs storage path (logical-key + ".versions/" + version_id)
+// so the dispatcher can locate the specific version, and VersionID
+// carries the AWS-visible version ID separately.
+type Match struct {
+	Key       s3lifecycle.ActionKey
+	Action    *engine.CompiledAction
+	Result    s3lifecycle.EvalResult
+	EventTs   time.Time
+	DueTime   time.Time
+	Bucket    string
+	ObjectKey string
+	VersionID string
+	Identity  *EntryIdentity
+}
+
+// EntryIdentity is the schedule-time CAS witness; the dispatcher serializes
+// it into the LifecycleDelete request. The fields mirror
+// s3_lifecycle_pb.EntryIdentity but stay in-package so the router doesn't
+// pull a proto dependency.
+type EntryIdentity struct {
+	MtimeNs      int64
+	Size         int64
+	HeadFid      string
+	ExtendedHash []byte
+}
+
+// Route returns the matches that fire for ev against snap. Only active
+// event-driven actions are considered; SCAN_AT_DATE and DISABLED bypass
+// this path.
+func Route(ctx context.Context, snap *engine.Snapshot, ev *reader.Event, now time.Time, lister SiblingLister) []Match {
+	if snap == nil || ev == nil {
+		return nil
+	}
+	keys := snap.BucketActionKeys(ev.Bucket)
+	if len(keys) == 0 {
+		return nil
+	}
+	// Bootstrap-expanded version event: sibling state is pre-computed,
+	// info.Key is the LOGICAL key so rule prefixes match. Skip the
+	// meta-log path's version-folder skip.
+	if ev.BootstrapVersion != nil {
+		return routeBootstrapVersion(snap, ev, keys)
+	}
+
+	versioned := snap.BucketVersioned(ev.Bucket)
+
+	// .versions/ directory metadata update: when ExtLatestVersionIdKey
+	// changes, the OLD pointer value names a version that's now
+	// noncurrent. Route NoncurrentDays / NewerNoncurrent for it without
+	// waiting for the next bootstrap.
+	if versioned && ev.NewEntry != nil && ev.OldEntry != nil && ev.NewEntry.IsDirectory && isVersionsContainerKey(ev.Key) {
+		if !hasActiveEventDrivenAction(snap, keys, s3lifecycle.ActionKindNoncurrentDays) &&
+			!hasActiveEventDrivenAction(snap, keys, s3lifecycle.ActionKindNewerNoncurrent) {
+			return nil
+		}
+		return routePointerTransition(ctx, snap, ev, keys, lister)
+	}
+
+	// EXP_DM can fire on two version-folder events: the marker create
+	// (sole survivor immediately) and a noncurrent hard-delete that
+	// leaves only the marker behind. Both reach routeSoleSurvivorMarker.
+	if versioned && isVersionFolderPath(ev.Key) {
+		if !hasActiveEventDrivenAction(snap, keys, s3lifecycle.ActionKindExpiredDeleteMarker) {
+			return nil
+		}
+		return routeSoleSurvivorMarker(ctx, snap, ev, keys, lister)
+	}
+
+	if ev.NewEntry == nil {
+		return nil
+	}
+	info := buildObjectInfo(ev, versioned)
+	if info == nil {
+		return nil
+	}
+	eventTime := time.Unix(0, ev.TsNs)
+
+	var matches []Match
+	for _, key := range keys {
+		action := snap.Action(key)
+		if action == nil || !action.IsActive() {
+			continue
+		}
+		if action.Mode != engine.ModeEventDriven {
+			continue
+		}
+		// (kind, info) shape gate: ABORT_MPU only fires on MPU init events,
+		// and other kinds never do. Without this an MPU init would be
+		// matched against NONCURRENT_DAYS (IsLatest=false reads as a
+		// non-current version) and the dispatcher would BLOCK on empty
+		// version_id.
+		if info.IsMPUInit && key.ActionKind != s3lifecycle.ActionKindAbortMPU {
+			continue
+		}
+		if !info.IsMPUInit && key.ActionKind == s3lifecycle.ActionKindAbortMPU {
+			continue
+		}
+		// Schedule from the per-kind due moment. ExpirationDate is
+		// rule-relative (the date IS the moment); other kinds are
+		// ModTime-relative. Using ModTime+Delay for ExpirationDate
+		// (Delay=0) puts dueTime at the entry's mtime — a backdated
+		// object's mtime is BEFORE the rule's date, so the eligibility
+		// check below would skip it. ComputeDueAt encapsulates both
+		// shapes; the dispatcher's identity-CAS catches drift if the
+		// object changes meanwhile.
+		dueTime := s3lifecycle.ComputeDueAt(action.Rule, key.ActionKind, info)
+		if dueTime.IsZero() {
+			continue
+		}
+		res := s3lifecycle.EvaluateAction(action.Rule, key.ActionKind, info, dueTime)
+		if res.Action == s3lifecycle.ActionNone {
+			continue
+		}
+		matches = append(matches, Match{
+			Key:       key,
+			Action:    action,
+			Result:    res,
+			EventTs:   eventTime,
+			DueTime:   dueTime,
+			Bucket:    ev.Bucket,
+			ObjectKey: ev.Key,
+			Identity:  buildIdentity(ev),
+		})
+	}
+	return matches
+}
+
+// routePointerTransition handles a .versions/ container update where
+// ExtLatestVersionIdKey changed: the OLD pointer value names a version
+// that just became noncurrent. Two lookup shapes:
+//
+//   - Pure NoncurrentVersionExpirationDays without NewerNoncurrentVersions:
+//     a single LookupVersion of oldID is enough — the displaced version
+//     is the only one that newly entered eligibility for this rule.
+//
+//   - Any active NewerNoncurrentVersions rule: a pointer flip shifts
+//     every prior noncurrent's rank by one, so the version that *just
+//     crossed* the keep-count threshold needs evaluation too. List the
+//     full .versions/ container, rank newest-first, and route every
+//     eligible noncurrent. Identity-CAS handles dedup with earlier
+//     schedules.
+//
+// Without this branch the worker has to wait for the next bootstrap to
+// schedule retention on a freshly-noncurrent version.
+func routePointerTransition(ctx context.Context, snap *engine.Snapshot, ev *reader.Event, keys []s3lifecycle.ActionKey, lister SiblingLister) []Match {
+	if lister == nil {
+		return nil
+	}
+	logical := strings.TrimSuffix(ev.Key, s3_constants.VersionsFolder)
+	if logical == "" {
+		return nil
+	}
+	oldID := string(ev.OldEntry.Extended[s3_constants.ExtLatestVersionIdKey])
+	newID := string(ev.NewEntry.Extended[s3_constants.ExtLatestVersionIdKey])
+	if oldID == newID {
+		// Same id means the update didn't transition the pointer.
+		return nil
+	}
+	// oldID == "" doesn't mean "nothing displaced": a bare null may
+	// have been the implicit/explicit latest before the pointer
+	// flipped to a real id.
+	// newID == "" means a suspended-versioning write cleared the
+	// pointer and made the bare null current. The cached
+	// ExtLatestVersionMtimeKey may still hold the prior latest's
+	// mtime (stale), so we must NOT use successorModTimeFromContainer
+	// in that case — derive the successor clock from the null entry's
+	// mtime instead. latestIDForExpand carries the same substitution
+	// so the expansion path's latestPos lookup matches the null sibling.
+	var successor time.Time
+	latestIDForExpand := newID
+	if newID == "" {
+		nullEntry, _, err := lister.LookupNullVersion(ctx, ev.Bucket, logical)
+		if err != nil {
+			glog.V(2).Infof("lifecycle router: lookup null %s/%s: %v", ev.Bucket, logical, err)
+			return nil
+		}
+		if nullEntry == nil || nullEntry.Attributes == nil {
+			return nil
+		}
+		successor = time.Unix(nullEntry.Attributes.Mtime, int64(nullEntry.Attributes.MtimeNs))
+		latestIDForExpand = "null"
+	} else {
+		successor = successorModTimeFromContainer(ev.NewEntry)
+	}
+	if successor.IsZero() {
+		return nil
+	}
+	if needsFullExpansion(snap, keys) {
+		return routePointerTransitionExpand(ctx, snap, ev, keys, lister, logical, latestIDForExpand, successor)
+	}
+	return routePointerTransitionDisplaced(ctx, snap, ev, keys, lister, logical, oldID, successor)
+}
+
+// needsFullExpansion reports whether any active event-driven rule on
+// this bucket cares about NoncurrentIndex (NewerNoncurrentVersions > 0
+// in either NoncurrentDays or pure-count NewerNoncurrent).
+func needsFullExpansion(snap *engine.Snapshot, keys []s3lifecycle.ActionKey) bool {
+	for _, k := range keys {
+		if k.ActionKind != s3lifecycle.ActionKindNoncurrentDays && k.ActionKind != s3lifecycle.ActionKindNewerNoncurrent {
+			continue
+		}
+		a := snap.Action(k)
+		if a == nil || !a.IsActive() || a.Mode != engine.ModeEventDriven {
+			continue
+		}
+		if a.Rule != nil && a.Rule.NewerNoncurrentVersions > 0 {
+			return true
+		}
+	}
+	return false
+}
+
+// successorModTimeFromContainer reads the cached latest-version mtime
+// from the .versions/ container's Extended map.
+// updateLatestVersionInDirectory writes it via setCachedListMetadata
+// alongside ExtLatestVersionIdKey, but the directory's own
+// Attributes.Mtime is preserved across pointer updates — using it
+// directly would let a stale dir mtime trigger expiration immediately.
+// Returns zero time if the cached mtime is missing or unparseable; the
+// caller suppresses in that case.
+func successorModTimeFromContainer(entry *filer_pb.Entry) time.Time {
+	raw, ok := entry.Extended[s3_constants.ExtLatestVersionMtimeKey]
+	if !ok || len(raw) == 0 {
+		return time.Time{}
+	}
+	secs, err := strconv.ParseInt(string(raw), 10, 64)
+	if err != nil || secs <= 0 {
+		return time.Time{}
+	}
+	return time.Unix(secs, 0)
+}
+
+// routePointerTransitionDisplaced is the single-lookup path: only the
+// displaced version's noncurrent eligibility could have changed, so
+// fetching just its file is enough. oldID == "" routes the bare null
+// version instead — it was the implicit latest before the pointer
+// flipped to a real id.
+func routePointerTransitionDisplaced(ctx context.Context, snap *engine.Snapshot, ev *reader.Event, keys []s3lifecycle.ActionKey, lister SiblingLister, logical, oldID string, successor time.Time) []Match {
+	var displaced *filer_pb.Entry
+	displacedID := oldID
+	if oldID == "" {
+		nullEntry, _, err := lister.LookupNullVersion(ctx, ev.Bucket, logical)
+		if err != nil {
+			glog.V(2).Infof("lifecycle router: lookup null version %s/%s: %v", ev.Bucket, logical, err)
+			return nil
+		}
+		if nullEntry == nil {
+			return nil
+		}
+		displaced = nullEntry
+		displacedID = "null"
+	} else {
+		entry, err := lister.LookupVersion(ctx, ev.Bucket, logical, oldID)
+		if err != nil {
+			glog.V(2).Infof("lifecycle router: lookup displaced version %s/%s/%s: %v", ev.Bucket, logical, oldID, err)
+			return nil
+		}
+		displaced = entry
+	}
+	if displaced == nil || displaced.Attributes == nil {
+		return nil
+	}
+	// Prefer the explicit demotion stamp on the displaced entry over the
+	// container-derived successor. Stamp is written by the S3 PUT handler
+	// at the moment the pointer flipped; container value is derived from
+	// the new latest's mtime and may drift across retries.
+	effectiveSuccessor := successor
+	if stamp := s3lifecycle.SuccessorFromEntryStamp(displaced); !stamp.IsZero() {
+		effectiveSuccessor = stamp
+	}
+	idx := 0
+	info := &s3lifecycle.ObjectInfo{
+		Key:              logical,
+		ModTime:          time.Unix(displaced.Attributes.Mtime, int64(displaced.Attributes.MtimeNs)),
+		Size:             int64(displaced.Attributes.FileSize),
+		IsLatest:         false,
+		IsDeleteMarker:   string(displaced.Extended[s3_constants.ExtDeleteMarkerKey]) == "true",
+		NoncurrentIndex:  &idx,
+		SuccessorModTime: effectiveSuccessor,
+	}
+	if tags := extractTags(displaced.Extended); len(tags) > 0 {
+		info.Tags = tags
+	}
+	return emitNoncurrentMatches(snap, ev, keys, info, displaced, displacedID, effectiveSuccessor)
+}
+
+// routePointerTransitionExpand routes only the versions that newly
+// became eligible by the pointer flip:
+//
+//   - rank 0: the displaced version (newly noncurrent), needed for the
+//     pure-NoncurrentDays clock,
+//   - rank == rule.NewerNoncurrentVersions for each active rule that
+//     gates on count: the version at exactly that rank just crossed
+//     from kept to expired.
+//
+// Emitting every eligible noncurrent on every PUT would push
+// O(versions) heap entries per flip — Schedule.Add doesn't dedup, so
+// identity-CAS at dispatch only stops the wasted RPC, not the heap
+// growth. Bootstrap still owns full backfill.
+func routePointerTransitionExpand(ctx context.Context, snap *engine.Snapshot, ev *reader.Event, keys []s3lifecycle.ActionKey, lister SiblingLister, logical, newID string, successor time.Time) []Match {
+	rawVersions, err := lister.ListVersions(ctx, ev.Bucket, logical)
+	if err != nil {
+		glog.V(2).Infof("lifecycle router: list versions %s/%s: %v", ev.Bucket, logical, err)
+		return nil
+	}
+	// Include the bare null version in the sibling set: count-based
+	// ranks are wrong if a pre-versioning or suspended-null entry
+	// exists outside .versions/. ID is "null" for sort + emit.
+	nullEntry, _, nullErr := lister.LookupNullVersion(ctx, ev.Bucket, logical)
+	if nullErr != nil {
+		glog.V(2).Infof("lifecycle router: lookup null %s/%s: %v", ev.Bucket, logical, nullErr)
+		return nil
+	}
+	type sibling struct {
+		entry *filer_pb.Entry
+		id    string
+	}
+	siblings := make([]sibling, 0, len(rawVersions)+1)
+	for _, v := range rawVersions {
+		if v == nil || v.Attributes == nil {
+			continue
+		}
+		id := string(v.Extended[s3_constants.ExtVersionIdKey])
+		if id == "" {
+			continue
+		}
+		siblings = append(siblings, sibling{entry: v, id: id})
+	}
+	if nullEntry != nil && nullEntry.Attributes != nil {
+		siblings = append(siblings, sibling{entry: nullEntry, id: "null"})
+	}
+	if len(siblings) == 0 {
+		return nil
+	}
+	sort.SliceStable(siblings, func(i, j int) bool {
+		mi := siblings[i].entry.Attributes.Mtime*int64(1e9) + int64(siblings[i].entry.Attributes.MtimeNs)
+		mj := siblings[j].entry.Attributes.Mtime*int64(1e9) + int64(siblings[j].entry.Attributes.MtimeNs)
+		if mi != mj {
+			return mi > mj
+		}
+		return s3lifecycle.CompareVersionIds(siblings[i].id, siblings[j].id) < 0
+	})
+	// Resolve latestPos by finding newID. Default to -1 so a missing
+	// newID (race with the listing, or torn write) suppresses the
+	// expansion: we'd otherwise call the actual newest sibling "latest"
+	// against the pointer's intent and misrank every noncurrent.
+	// Bootstrap repairs state on the next walk.
+	latestPos := -1
+	if newID != "" {
+		for i, s := range siblings {
+			if s.id == newID {
+				latestPos = i
+				break
+			}
+		}
+	}
+	if latestPos < 0 {
+		glog.V(2).Infof("lifecycle router: pointer transition %s/%s: new id %s not found in listing", ev.Bucket, logical, newID)
+		return nil
+	}
+	noncurrentCount := len(siblings) - 1
+
+	// Collect the target noncurrent ranks: 0 (the freshly displaced)
+	// plus N for each active count-gated rule.
+	rankSet := map[int]struct{}{0: {}}
+	for _, k := range keys {
+		if k.ActionKind != s3lifecycle.ActionKindNoncurrentDays && k.ActionKind != s3lifecycle.ActionKindNewerNoncurrent {
+			continue
+		}
+		a := snap.Action(k)
+		if a == nil || !a.IsActive() || a.Mode != engine.ModeEventDriven {
+			continue
+		}
+		if a.Rule != nil && a.Rule.NewerNoncurrentVersions > 0 {
+			rankSet[a.Rule.NewerNoncurrentVersions] = struct{}{}
+		}
+	}
+	ranks := make([]int, 0, len(rankSet))
+	for r := range rankSet {
+		ranks = append(ranks, r)
+	}
+	sort.Ints(ranks)
+
+	var matches []Match
+	for _, rank := range ranks {
+		if rank >= noncurrentCount {
+			continue
+		}
+		// Convert noncurrent rank to position in the sorted slice,
+		// skipping the latest's slot.
+		i := rank
+		if rank >= latestPos {
+			i = rank + 1
+		}
+		s := siblings[i]
+		// Successor mtime: the entry directly newer than this one in
+		// the sorted list. When the next-newer slot is the latest,
+		// use the cached successor (the new latest's mtime); otherwise
+		// the immediate predecessor's mtime.
+		var thisSuccessor time.Time
+		if i > 0 && i-1 != latestPos {
+			thisSuccessor = time.Unix(siblings[i-1].entry.Attributes.Mtime, int64(siblings[i-1].entry.Attributes.MtimeNs))
+		} else {
+			thisSuccessor = successor
+		}
+		// Override with the explicit demotion stamp when present —
+		// PUT-time wall clock beats derived sibling mtime for accuracy
+		// and is immune to mtime edits on the sibling itself.
+		if stamp := s3lifecycle.SuccessorFromEntryStamp(s.entry); !stamp.IsZero() {
+			thisSuccessor = stamp
+		}
+		idx := rank
+		info := &s3lifecycle.ObjectInfo{
+			Key:              logical,
+			ModTime:          time.Unix(s.entry.Attributes.Mtime, int64(s.entry.Attributes.MtimeNs)),
+			Size:             int64(s.entry.Attributes.FileSize),
+			IsLatest:         false,
+			IsDeleteMarker:   string(s.entry.Extended[s3_constants.ExtDeleteMarkerKey]) == "true",
+			NoncurrentIndex:  &idx,
+			SuccessorModTime: thisSuccessor,
+			NumVersions:      len(siblings),
+		}
+		if tags := extractTags(s.entry.Extended); len(tags) > 0 {
+			info.Tags = tags
+		}
+		matches = append(matches, emitNoncurrentMatches(snap, ev, keys, info, s.entry, s.id, thisSuccessor)...)
+	}
+	return matches
+}
+
+// emitNoncurrentMatches walks NoncurrentDays / NewerNoncurrent action
+// keys and emits Matches for each one that fires. Shared between the
+// single-lookup and full-expansion paths.
+func emitNoncurrentMatches(snap *engine.Snapshot, ev *reader.Event, keys []s3lifecycle.ActionKey, info *s3lifecycle.ObjectInfo, entry *filer_pb.Entry, versionID string, successor time.Time) []Match {
+	eventTime := time.Unix(0, ev.TsNs)
+	identity := buildIdentityFromEntry(entry)
+	var matches []Match
+	for _, key := range keys {
+		if key.ActionKind != s3lifecycle.ActionKindNoncurrentDays && key.ActionKind != s3lifecycle.ActionKindNewerNoncurrent {
+			continue
+		}
+		action := snap.Action(key)
+		if action == nil || !action.IsActive() || action.Mode != engine.ModeEventDriven {
+			continue
+		}
+		clock := successor
+		if clock.IsZero() {
+			clock = info.ModTime
+		}
+		dueTime := clock.Add(action.Delay)
+		res := s3lifecycle.EvaluateAction(action.Rule, key.ActionKind, info, dueTime)
+		if res.Action == s3lifecycle.ActionNone {
+			continue
+		}
+		matches = append(matches, Match{
+			Key:       key,
+			Action:    action,
+			Result:    res,
+			EventTs:   eventTime,
+			DueTime:   dueTime,
+			Bucket:    ev.Bucket,
+			ObjectKey: info.Key,
+			VersionID: versionID,
+			Identity:  identity,
+		})
+	}
+	return matches
+}
+
+// routeSoleSurvivorMarker emits an EXP_DM Match against the LOGICAL key
+// (so the dispatcher can call deleteSpecificObjectVersion) with the
+// marker's version_id. Handles two events: a marker create (the new
+// entry IS the marker) and a noncurrent hard-delete that leaves the
+// marker behind (the listing's lone entry IS the marker). The server
+// re-checks before deleting.
+func routeSoleSurvivorMarker(ctx context.Context, snap *engine.Snapshot, ev *reader.Event, keys []s3lifecycle.ActionKey, lister SiblingLister) []Match {
+	if lister == nil {
+		return nil
+	}
+	// Skip the listing RPC for events that can't possibly produce a
+	// sole-survivor marker: a regular non-marker version create/update
+	// always lands at Count >= 2.
+	if ev.NewEntry != nil && !isDeleteMarkerEntry(ev.NewEntry) {
+		return nil
+	}
+	logicalKey, ok := logicalKeyFromVersionPath(ev.Key)
+	if !ok {
+		return nil
+	}
+	s, err := lister.Survivors(ctx, ev.Bucket, logicalKey)
+	if err != nil {
+		glog.V(2).Infof("lifecycle router: survivors %s/%s: %v", ev.Bucket, logicalKey, err)
+		return nil
+	}
+	// Pre-versioning bare-key objects (the "null" version) live outside
+	// .versions/. Treating count==1 as sole-survivor while a null
+	// version exists would let lifecycle delete the marker and re-expose
+	// the old object.
+	if s.Count != 1 || s.HasNullVersion || s.LoneEntry == nil {
+		return nil
+	}
+	if !isDeleteMarkerEntry(s.LoneEntry) {
+		return nil
+	}
+	versionID := string(s.LoneEntry.Extended[s3_constants.ExtVersionIdKey])
+	if versionID == "" {
+		// Empty version_id would BLOCK at dispatch and freeze the cursor.
+		return nil
+	}
+	entry := s.LoneEntry
+	info := &s3lifecycle.ObjectInfo{
+		Key:            logicalKey,
+		ModTime:        time.Unix(entry.Attributes.Mtime, int64(entry.Attributes.MtimeNs)),
+		Size:           int64(entry.Attributes.FileSize),
+		IsLatest:       true,
+		IsDeleteMarker: true,
+		NumVersions:    1,
+	}
+	if tags := extractTags(entry.Extended); len(tags) > 0 {
+		info.Tags = tags
+	}
+	eventTime := time.Unix(0, ev.TsNs)
+	identity := buildIdentityFromEntry(entry)
+	var matches []Match
+	for _, key := range keys {
+		if key.ActionKind != s3lifecycle.ActionKindExpiredDeleteMarker {
+			continue
+		}
+		action := snap.Action(key)
+		if action == nil || !action.IsActive() || action.Mode != engine.ModeEventDriven {
+			continue
+		}
+		dueTime := info.ModTime.Add(action.Delay)
+		res := s3lifecycle.EvaluateAction(action.Rule, key.ActionKind, info, dueTime)
+		if res.Action == s3lifecycle.ActionNone {
+			continue
+		}
+		matches = append(matches, Match{
+			Key:       key,
+			Action:    action,
+			Result:    res,
+			EventTs:   eventTime,
+			DueTime:   dueTime,
+			Bucket:    ev.Bucket,
+			ObjectKey: logicalKey,
+			VersionID: versionID,
+			Identity:  identity,
+		})
+	}
+	return matches
+}
+
+// routeBootstrapVersion handles a synthesized event from BucketBootstrapper.
+// The bootstrap walker has already listed .versions/<key>/, sorted siblings
+// newest-first, and stamped each one's IsLatest / NoncurrentIndex /
+// SuccessorModTime. The router only needs to assemble ObjectInfo and run
+// the match loop with the standard kind gates. ev.NewEntry is the version
+// file itself; ev.Key is the version-folder path; the LOGICAL key from
+// BootstrapVersion drives prefix matching and the dispatcher.
+func routeBootstrapVersion(snap *engine.Snapshot, ev *reader.Event, keys []s3lifecycle.ActionKey) []Match {
+	bv := ev.BootstrapVersion
+	entry := ev.NewEntry
+	if entry == nil || entry.Attributes == nil || bv.LogicalKey == "" {
+		return nil
+	}
+	idx := bv.NoncurrentIndex
+	info := &s3lifecycle.ObjectInfo{
+		Key:              bv.LogicalKey,
+		ModTime:          time.Unix(entry.Attributes.Mtime, int64(entry.Attributes.MtimeNs)),
+		Size:             int64(entry.Attributes.FileSize),
+		IsLatest:         bv.IsLatest,
+		IsDeleteMarker:   bv.IsDeleteMarker,
+		NumVersions:      bv.NumVersions,
+		SuccessorModTime: bv.SuccessorModTime,
+	}
+	if !bv.IsLatest {
+		info.NoncurrentIndex = &idx
+	}
+	if tags := extractTags(entry.Extended); len(tags) > 0 {
+		info.Tags = tags
+	}
+	eventTime := time.Unix(0, ev.TsNs)
+	identity := buildIdentityFromEntry(entry)
+	var matches []Match
+	for _, key := range keys {
+		action := snap.Action(key)
+		if action == nil || !action.IsActive() || action.Mode != engine.ModeEventDriven {
+			continue
+		}
+		// ABORT_MPU never applies to a versioned object.
+		if key.ActionKind == s3lifecycle.ActionKindAbortMPU {
+			continue
+		}
+		// Noncurrent rules clock from when this version was replaced
+		// (SuccessorModTime), not from when it was originally written.
+		// Bootstrap populates SuccessorModTime; fall back to ModTime
+		// for the latest version (no successor exists).
+		clock := info.ModTime
+		if !info.IsLatest && !info.SuccessorModTime.IsZero() {
+			clock = info.SuccessorModTime
+		}
+		dueTime := clock.Add(action.Delay)
+		res := s3lifecycle.EvaluateAction(action.Rule, key.ActionKind, info, dueTime)
+		if res.Action == s3lifecycle.ActionNone {
+			continue
+		}
+		// Pin the version_id only for kinds the dispatcher needs to
+		// target by version: noncurrent retention and the marker
+		// itself. EXPIRATION_DAYS / EXPIRATION_DATE on the latest
+		// must NOT carry it — between schedule and dispatch a fresh
+		// PUT can land, and identity-CAS against the original
+		// version's bytes would still pass even though the latest has
+		// moved on. Empty VersionID makes the dispatcher fetch the
+		// current latest, where identity-CAS resolves to STALE_IDENTITY
+		// and bootstrap re-schedules with the new latest's identity.
+		var matchVersionID string
+		switch key.ActionKind {
+		case s3lifecycle.ActionKindNoncurrentDays,
+			s3lifecycle.ActionKindNewerNoncurrent,
+			s3lifecycle.ActionKindExpiredDeleteMarker:
+			matchVersionID = bv.VersionID
+		}
+		matches = append(matches, Match{
+			Key:       key,
+			Action:    action,
+			Result:    res,
+			EventTs:   eventTime,
+			DueTime:   dueTime,
+			Bucket:    ev.Bucket,
+			ObjectKey: bv.LogicalKey,
+			VersionID: matchVersionID,
+			Identity:  identity,
+		})
+	}
+	return matches
+}
+
+// logicalKeyFromVersionPath extracts <logical> from <logical>.versions/<file>.
+// Returns false for a bucket-root marker (path = ".versions/<file>"), since
+// AWS S3 has no concept of an object at the bucket key itself.
+func logicalKeyFromVersionPath(versionPath string) (string, bool) {
+	lastSlash := strings.LastIndex(versionPath, "/")
+	if lastSlash <= 0 {
+		return "", false
+	}
+	parent := versionPath[:lastSlash]
+	if !strings.HasSuffix(parent, s3_constants.VersionsFolder) {
+		return "", false
+	}
+	logical := strings.TrimSuffix(parent, s3_constants.VersionsFolder)
+	if logical == "" {
+		return "", false
+	}
+	return logical, true
+}
+
+// buildObjectInfo derives an ObjectInfo from a meta-log event. Returns
+// nil for shapes the router can't classify safely: missing attributes,
+// non-MPU directories, version-folder files (those route through
+// routeSoleSurvivorMarker upstream when EXP_DM applies). On a versioned
+// bucket the latest pointer lives in the .versions/ directory's
+// Extended map; without it we leave NumVersions=0 so the bootstrap walk
+// drives noncurrent retention.
+func buildObjectInfo(ev *reader.Event, versioned bool) *s3lifecycle.ObjectInfo {
+	entry := ev.NewEntry
+	if entry == nil || entry.Attributes == nil {
+		return nil
+	}
+	if destKey, ok := mpuInitInfo(ev, entry); ok {
+		return &s3lifecycle.ObjectInfo{
+			Key:       destKey,
+			ModTime:   time.Unix(entry.Attributes.Mtime, int64(entry.Attributes.MtimeNs)),
+			IsMPUInit: true,
+		}
+	}
+	if entry.IsDirectory {
+		return nil
+	}
+	info := &s3lifecycle.ObjectInfo{
+		Key:         ev.Key,
+		ModTime:     time.Unix(entry.Attributes.Mtime, int64(entry.Attributes.MtimeNs)),
+		Size:        int64(entry.Attributes.FileSize),
+		IsLatest:    true,
+		NumVersions: 1,
+	}
+	if versioned {
+		if isVersionFolderPath(ev.Key) {
+			return nil
+		}
+		info.NumVersions = 0
+	}
+	if tags := extractTags(entry.Extended); len(tags) > 0 {
+		info.Tags = tags
+	}
+	if isDeleteMarkerEntry(entry) {
+		info.IsDeleteMarker = true
+	}
+	return info
+}
+
+// isVersionsContainerKey reports whether the bucket-relative key IS a
+// .versions/ container itself (e.g. "logs/foo.versions"), as opposed to
+// a file inside one. Used to recognize directory-level events whose
+// Extended map carries the latest pointer for an object.
+func isVersionsContainerKey(key string) bool {
+	if key == s3_constants.VersionsFolder {
+		// Bucket-root .versions: no logical object key.
+		return false
+	}
+	return strings.HasSuffix(key, s3_constants.VersionsFolder)
+}
+
+// isVersionFolderPath reports whether the bucket-relative key sits inside a
+// .versions/ folder — i.e. the path's parent segment ends with the
+// VersionsFolder suffix. Used by the versioned-bucket gate so the router
+// skips version-file events that need sibling state to be classified
+// safely.
+func isVersionFolderPath(key string) bool {
+	idx := strings.LastIndex(key, "/")
+	if idx <= 0 {
+		return false
+	}
+	parent := key[:idx]
+	parentIdx := strings.LastIndex(parent, "/")
+	leaf := parent[parentIdx+1:]
+	return strings.HasSuffix(leaf, s3_constants.VersionsFolder)
+}
+
+// mpuInitInfo recognizes a multipart-upload init: a directory entry at
+// `.uploads/<upload_id>` carrying the destination key in Extended. Sub-events
+// for part uploads (deeper paths under the upload directory) are deliberately
+// rejected — they ride a different mtime and would over-fire ABORT_MPU.
+func mpuInitInfo(ev *reader.Event, entry *filer_pb.Entry) (destKey string, ok bool) {
+	uploadsPrefix := s3_constants.MultipartUploadsFolder + "/"
+	if !entry.IsDirectory || !strings.HasPrefix(ev.Key, uploadsPrefix) {
+		return "", false
+	}
+	rest := ev.Key[len(uploadsPrefix):]
+	if rest == "" || strings.ContainsRune(rest, '/') {
+		// `.uploads/` itself or `.uploads/<id>/<part>...`; not the init.
+		return "", false
+	}
+	keyBytes, hasKey := entry.Extended[s3_constants.ExtMultipartObjectKey]
+	if !hasKey || len(keyBytes) == 0 {
+		return "", false
+	}
+	return string(keyBytes), true
+}
+
+// buildIdentity captures the entry's schedule-time fingerprint for the CAS
+// witness. Returns nil if the event has no entry to fingerprint (deletes).
+func buildIdentity(ev *reader.Event) *EntryIdentity {
+	return buildIdentityFromEntry(ev.NewEntry)
+}
+
+func buildIdentityFromEntry(entry *filer_pb.Entry) *EntryIdentity {
+	if entry == nil {
+		return nil
+	}
+	id := &EntryIdentity{}
+	if entry.Attributes != nil {
+		// Mirror the server-side computeEntryIdentity encoding: Mtime
+		// (seconds) and MtimeNs (nanosecond component) combine into
+		// EntryIdentity.MtimeNs as nanoseconds-since-epoch.
+		id.MtimeNs = entry.Attributes.Mtime*int64(1e9) + int64(entry.Attributes.MtimeNs)
+		id.Size = int64(entry.Attributes.FileSize)
+	}
+	if len(entry.GetChunks()) > 0 {
+		// Meta-log events arrive with chunk.FileId cleared by
+		// BeforeEntrySerialization; GetFileIdString reconstructs it from
+		// Fid so the worker matches the server-side fingerprint.
+		id.HeadFid = entry.GetChunks()[0].GetFileIdString()
+	}
+	id.ExtendedHash = s3lifecycle.HashExtended(entry.Extended)
+	return id
+}
+
+func extractTags(ext map[string][]byte) map[string]string {
+	if len(ext) == 0 {
+		return nil
+	}
+	prefix := s3_constants.AmzObjectTagging + "-"
+	var out map[string]string
+	for k, v := range ext {
+		if !strings.HasPrefix(k, prefix) {
+			continue
+		}
+		if out == nil {
+			out = map[string]string{}
+		}
+		out[k[len(prefix):]] = string(v)
+	}
+	return out
+}
+
+// hasActiveEventDrivenAction gates I/O (e.g. sibling listing) on whether
+// a match could actually fire. Mirrors the per-key filter in Route so
+// disabled or scan-only actions don't pay the RPC.
+func hasActiveEventDrivenAction(snap *engine.Snapshot, keys []s3lifecycle.ActionKey, kind s3lifecycle.ActionKind) bool {
+	for _, k := range keys {
+		if k.ActionKind != kind {
+			continue
+		}
+		a := snap.Action(k)
+		if a == nil {
+			continue
+		}
+		if a.IsActive() && a.Mode == engine.ModeEventDriven {
+			return true
+		}
+	}
+	return false
+}
+
+// isDeleteMarkerEntry mirrors every read site for ExtDeleteMarkerKey:
+// production writes []byte("true").
+func isDeleteMarkerEntry(entry *filer_pb.Entry) bool {
+	if entry == nil || len(entry.Extended) == 0 {
+		return false
+	}
+	v, ok := entry.Extended[s3_constants.ExtDeleteMarkerKey]
+	return ok && string(v) == "true"
+}

weed/s3api/s3lifecycle/router/schedule.go

@@ -0,0 +1,76 @@
+package router
+
+import (
+	"container/heap"
+	"sync"
+	"time"
+)
+
+// Schedule is the worker's in-memory pending list, ordered by DueTime.
+// The dispatcher polls Drain on each tick to retrieve due Matches.
+//
+// Duplicates are allowed: a re-schedule for the same (ActionKey, ObjectKey)
+// before the prior dispatch ran results in two heap entries. The
+// LifecycleDelete RPC's identity-CAS makes the second dispatch a no-op
+// (NOOP_RESOLVED with STALE_IDENTITY), so dedup at insert time would only
+// be a micro-optimization at the cost of an extra map.
+type Schedule struct {
+	mu sync.Mutex
+	h  scheduleHeap
+}
+
+func NewSchedule() *Schedule {
+	return &Schedule{}
+}
+
+// Add enqueues a Match.
+func (s *Schedule) Add(m Match) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	heap.Push(&s.h, m)
+}
+
+// Len returns the number of pending Matches.
+func (s *Schedule) Len() int {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.h.Len()
+}
+
+// NextDue returns the DueTime of the earliest pending Match. ok=false if
+// the schedule is empty.
+func (s *Schedule) NextDue() (time.Time, bool) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.h.Len() == 0 {
+		return time.Time{}, false
+	}
+	return s.h[0].DueTime, true
+}
+
+// Drain pops and returns all Matches whose DueTime <= now, in DueTime order.
+// Subsequent calls return only newly-due Matches.
+func (s *Schedule) Drain(now time.Time) []Match {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	var out []Match
+	for s.h.Len() > 0 && !s.h[0].DueTime.After(now) {
+		out = append(out, heap.Pop(&s.h).(Match))
+	}
+	return out
+}
+
+// scheduleHeap implements heap.Interface ordered by Match.DueTime ascending.
+type scheduleHeap []Match
+
+func (h scheduleHeap) Len() int            { return len(h) }
+func (h scheduleHeap) Less(i, j int) bool  { return h[i].DueTime.Before(h[j].DueTime) }
+func (h scheduleHeap) Swap(i, j int)       { h[i], h[j] = h[j], h[i] }
+func (h *scheduleHeap) Push(x interface{}) { *h = append(*h, x.(Match)) }
+func (h *scheduleHeap) Pop() interface{} {
+	old := *h
+	n := len(old)
+	x := old[n-1]
+	*h = old[:n-1]
+	return x
+}

weed/s3api/s3lifecycle/router/schedule_test.go

@@ -0,0 +1,203 @@
+package router
+
+import (
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+)
+
+func mkMatch(due time.Time, key string) Match {
+	return Match{
+		Key:       s3lifecycle.ActionKey{Bucket: key},
+		DueTime:   due,
+		ObjectKey: key,
+	}
+}
+
+func TestScheduleEmpty(t *testing.T) {
+	s := NewSchedule()
+	if s.Len() != 0 {
+		t.Fatal("Len != 0")
+	}
+	if _, ok := s.NextDue(); ok {
+		t.Fatal("NextDue ok=true on empty")
+	}
+	if got := s.Drain(time.Now()); got != nil {
+		t.Fatalf("Drain on empty returned %v", got)
+	}
+}
+
+func TestScheduleOrderedByDueTime(t *testing.T) {
+	s := NewSchedule()
+	t0 := time.Now()
+	s.Add(mkMatch(t0.Add(3*time.Second), "c"))
+	s.Add(mkMatch(t0.Add(1*time.Second), "a"))
+	s.Add(mkMatch(t0.Add(2*time.Second), "b"))
+
+	if s.Len() != 3 {
+		t.Fatalf("Len=%d, want 3", s.Len())
+	}
+	due, ok := s.NextDue()
+	if !ok || !due.Equal(t0.Add(1*time.Second)) {
+		t.Fatalf("NextDue=%v ok=%v", due, ok)
+	}
+
+	got := s.Drain(t0.Add(2 * time.Second))
+	if len(got) != 2 || got[0].ObjectKey != "a" || got[1].ObjectKey != "b" {
+		t.Fatalf("Drain order: %+v", got)
+	}
+	if s.Len() != 1 {
+		t.Fatalf("Len after drain=%d, want 1", s.Len())
+	}
+
+	got = s.Drain(t0.Add(5 * time.Second))
+	if len(got) != 1 || got[0].ObjectKey != "c" {
+		t.Fatalf("Drain rest: %+v", got)
+	}
+	if s.Len() != 0 {
+		t.Fatal("Len != 0 after final drain")
+	}
+}
+
+func TestScheduleDrainBoundaryInclusive(t *testing.T) {
+	// DueTime exactly equal to now is drainable (<=).
+	s := NewSchedule()
+	t0 := time.Now()
+	s.Add(mkMatch(t0, "a"))
+	got := s.Drain(t0)
+	if len(got) != 1 {
+		t.Fatalf("expected boundary-inclusive drain, got %d", len(got))
+	}
+}
+
+func TestScheduleAllowsDuplicates(t *testing.T) {
+	s := NewSchedule()
+	t0 := time.Now()
+	s.Add(mkMatch(t0, "a"))
+	s.Add(mkMatch(t0, "a"))
+	if s.Len() != 2 {
+		t.Fatalf("dup count=%d, want 2", s.Len())
+	}
+	got := s.Drain(t0)
+	if len(got) != 2 {
+		t.Fatalf("Drain dup count=%d, want 2", len(got))
+	}
+}
+
+func TestScheduleDrainBeforeAnyDueReturnsNothing(t *testing.T) {
+	// Drain at a time before the earliest DueTime must return an empty
+	// slice and leave the heap intact. Otherwise the dispatcher would
+	// consume future-due matches early.
+	s := NewSchedule()
+	t0 := time.Now()
+	s.Add(mkMatch(t0.Add(5*time.Second), "a"))
+	s.Add(mkMatch(t0.Add(10*time.Second), "b"))
+
+	got := s.Drain(t0)
+	if got != nil {
+		t.Fatalf("Drain before any due should be nil, got %+v", got)
+	}
+	if s.Len() != 2 {
+		t.Fatalf("Len after no-op Drain=%d, want 2", s.Len())
+	}
+}
+
+func TestScheduleNextDueAfterPartialDrain(t *testing.T) {
+	// After draining a prefix, NextDue must point at the earliest
+	// remaining Match — regression catch for a Drain implementation
+	// that forgets to maintain the heap invariant.
+	s := NewSchedule()
+	t0 := time.Now()
+	s.Add(mkMatch(t0.Add(1*time.Second), "a"))
+	s.Add(mkMatch(t0.Add(2*time.Second), "b"))
+	s.Add(mkMatch(t0.Add(3*time.Second), "c"))
+
+	got := s.Drain(t0.Add(1500 * time.Millisecond))
+	if len(got) != 1 || got[0].ObjectKey != "a" {
+		t.Fatalf("Drain prefix=%+v, want [a]", got)
+	}
+	due, ok := s.NextDue()
+	if !ok || !due.Equal(t0.Add(2*time.Second)) {
+		t.Fatalf("NextDue after partial Drain=%v ok=%v, want t+2s", due, ok)
+	}
+}
+
+func TestScheduleAddAfterDrainKeepsOrder(t *testing.T) {
+	// Adding to a non-empty schedule mid-stream — after a real Drain
+	// has popped at least one entry, a fresh Match with an earlier
+	// DueTime than the existing minimum must become the next drainable.
+	// Pins that heap.Pop + Push together preserve the heap invariant
+	// across the Drain → Add boundary.
+	s := NewSchedule()
+	t0 := time.Now()
+	s.Add(mkMatch(t0.Add(10*time.Second), "old"))
+	s.Add(mkMatch(t0.Add(6*time.Second), "drain_me"))
+
+	drained := s.Drain(t0.Add(7 * time.Second))
+	if len(drained) != 1 || drained[0].ObjectKey != "drain_me" {
+		t.Fatalf("pre-drain=%+v, want [drain_me]", drained)
+	}
+
+	s.Add(mkMatch(t0.Add(5*time.Second), "new")) // added after Drain, earlier than old
+
+	due, ok := s.NextDue()
+	if !ok || !due.Equal(t0.Add(5*time.Second)) {
+		t.Fatalf("NextDue=%v ok=%v, want t+5s", due, ok)
+	}
+
+	got := s.Drain(t0.Add(6 * time.Second))
+	if len(got) != 1 || got[0].ObjectKey != "new" {
+		t.Fatalf("Drain=%+v, want [new]", got)
+	}
+}
+
+func TestScheduleDrainOrderIsAscendingDueTime(t *testing.T) {
+	// Drain must return Matches in DueTime order regardless of insert
+	// order — explicit pinning of the contract documented on Drain.
+	s := NewSchedule()
+	t0 := time.Now()
+	for _, off := range []time.Duration{
+		5 * time.Second,
+		1 * time.Second,
+		3 * time.Second,
+		2 * time.Second,
+		4 * time.Second,
+	} {
+		s.Add(mkMatch(t0.Add(off), "k"))
+	}
+
+	got := s.Drain(t0.Add(10 * time.Second))
+	if len(got) != 5 {
+		t.Fatalf("Drain count=%d, want 5", len(got))
+	}
+	for i := 1; i < len(got); i++ {
+		if got[i].DueTime.Before(got[i-1].DueTime) {
+			t.Fatalf("Drain[%d]=%v is before Drain[%d]=%v", i, got[i].DueTime, i-1, got[i-1].DueTime)
+		}
+	}
+}
+
+func TestScheduleConcurrentAddDrainNoRace(t *testing.T) {
+	// The dispatcher's Add and Drain run on separate goroutines; the
+	// schedule's mutex must serialize them without deadlock. -race
+	// catches a regression that drops the lock on either path.
+	s := NewSchedule()
+	t0 := time.Now()
+	const N = 64
+	var wg sync.WaitGroup
+	wg.Add(N * 2)
+	for i := 0; i < N; i++ {
+		i := i
+		go func() {
+			defer wg.Done()
+			s.Add(mkMatch(t0.Add(time.Duration(i)*time.Millisecond), "k"))
+		}()
+		go func() {
+			defer wg.Done()
+			_ = s.Drain(t0.Add(10 * time.Second))
+		}()
+	}
+	wg.Wait()
+}

weed/s3api/s3lifecycle/rule.go

@@ -0,0 +1,74 @@
+package s3lifecycle
+
+import "time"
+
+// Rule is the flat representation built from the XML-parsed s3api.Rule via
+// s3api.LifecycleToCanonical.
+type Rule struct {
+	ID     string
+	Status string // "Enabled" | "Disabled"
+
+	Prefix string
+
+	ExpirationDays            int
+	ExpirationDate            time.Time
+	ExpiredObjectDeleteMarker bool
+
+	NoncurrentVersionExpirationDays int
+	NewerNoncurrentVersions         int
+
+	AbortMPUDaysAfterInitiation int
+
+	FilterTags map[string]string
+
+	// Zero is "not set"; can't represent an explicit
+	// <ObjectSizeGreaterThan>0</...> exclusion of empty objects.
+	FilterSizeGreaterThan int64
+	FilterSizeLessThan    int64
+}
+
+type ObjectInfo struct {
+	Key            string
+	ModTime        time.Time
+	Size           int64
+	IsLatest       bool
+	IsDeleteMarker bool
+	NumVersions    int
+
+	SuccessorModTime time.Time
+
+	// NoncurrentIndex: 0-based among non-current versions, newest first.
+	// nil = current or not yet computed; count-based retention returns
+	// ActionNone rather than guess. Pointer so valid 0 doesn't collide
+	// with zero-value "uninitialised".
+	NoncurrentIndex *int
+
+	Tags map[string]string
+
+	// IsMPUInit: in-flight upload under .uploads/<id>/; ModTime is init time.
+	IsMPUInit bool
+}
+
+const (
+	StatusEnabled  = "Enabled"
+	StatusDisabled = "Disabled"
+)
+
+// SmallDelay is the lookback for predicate-change events and the event-log
+// horizon for count / immediate kinds.
+const SmallDelay = time.Minute
+
+type Action int
+
+const (
+	ActionNone Action = iota
+	ActionDeleteObject
+	ActionDeleteVersion
+	ActionExpireDeleteMarker
+	ActionAbortMultipartUpload
+)
+
+type EvalResult struct {
+	Action Action
+	RuleID string
+}

weed/s3api/s3lifecycle/rule_hash.go

@@ -0,0 +1,104 @@
+package s3lifecycle
+
+import (
+	"crypto/sha256"
+	"encoding/binary"
+	"sort"
+	"time"
+)
+
+// RuleHash returns the first 8 bytes of sha256 over a length-prefixed
+// canonical encoding. Stable across tag-key reorder, ID renames, and Status
+// flips. Prefix is hashed verbatim — "logs" and "logs/" match different
+// objects, so they must hash differently. Length prefixing prevents a
+// forged value (e.g. a tag containing "=") from colliding with a legitimate
+// tuple.
+func RuleHash(rule *Rule) [8]byte {
+	if rule == nil {
+		var zero [8]byte
+		return zero
+	}
+	h := sha256.New()
+	writeBytes(h, fieldPrefix, []byte(rule.Prefix))
+	tagKeys := make([]string, 0, len(rule.FilterTags))
+	for k := range rule.FilterTags {
+		tagKeys = append(tagKeys, k)
+	}
+	sort.Strings(tagKeys)
+	writeUvarint(h, fieldTagCount, uint64(len(tagKeys)))
+	for _, k := range tagKeys {
+		writeBytes(h, fieldTagKey, []byte(k))
+		writeBytes(h, fieldTagValue, []byte(rule.FilterTags[k]))
+	}
+	writeInt64(h, fieldSizeGT, rule.FilterSizeGreaterThan)
+	writeInt64(h, fieldSizeLT, rule.FilterSizeLessThan)
+	writeInt64(h, fieldExpDays, int64(rule.ExpirationDays))
+	writeBytes(h, fieldExpDate, []byte(canonicalTime(rule.ExpirationDate)))
+	writeBool(h, fieldExpDeleteMarker, rule.ExpiredObjectDeleteMarker)
+	writeInt64(h, fieldNoncurDays, int64(rule.NoncurrentVersionExpirationDays))
+	writeInt64(h, fieldNoncurKeep, int64(rule.NewerNoncurrentVersions))
+	writeInt64(h, fieldMPUDays, int64(rule.AbortMPUDaysAfterInitiation))
+
+	sum := h.Sum(nil)
+	var out [8]byte
+	copy(out[:], sum[:8])
+	return out
+}
+
+// Field tags namespace each scalar so a forged string can't substitute for
+// another in a way that hashes the same.
+const (
+	fieldPrefix          byte = 0x01
+	fieldTagCount        byte = 0x02
+	fieldTagKey          byte = 0x03
+	fieldTagValue        byte = 0x04
+	fieldSizeGT          byte = 0x05
+	fieldSizeLT          byte = 0x06
+	fieldExpDays         byte = 0x10
+	fieldExpDate         byte = 0x11
+	fieldExpDeleteMarker byte = 0x12
+	fieldNoncurDays      byte = 0x13
+	fieldNoncurKeep      byte = 0x14
+	fieldMPUDays         byte = 0x15
+)
+
+type byteSink interface {
+	Write(p []byte) (n int, err error)
+}
+
+func writeBytes(w byteSink, tag byte, b []byte) {
+	var lenbuf [binary.MaxVarintLen64]byte
+	n := binary.PutUvarint(lenbuf[:], uint64(len(b)))
+	w.Write([]byte{tag})
+	w.Write(lenbuf[:n])
+	w.Write(b)
+}
+
+func writeUvarint(w byteSink, tag byte, v uint64) {
+	var buf [binary.MaxVarintLen64]byte
+	n := binary.PutUvarint(buf[:], v)
+	w.Write([]byte{tag})
+	w.Write(buf[:n])
+}
+
+func writeInt64(w byteSink, tag byte, v int64) {
+	var buf [binary.MaxVarintLen64]byte
+	n := binary.PutVarint(buf[:], v)
+	w.Write([]byte{tag})
+	w.Write(buf[:n])
+}
+
+func writeBool(w byteSink, tag byte, b bool) {
+	v := byte(0)
+	if b {
+		v = 1
+	}
+	w.Write([]byte{tag, v})
+}
+
+func canonicalTime(t time.Time) string {
+	if t.IsZero() {
+		return ""
+	}
+	return t.UTC().Format(time.RFC3339Nano)
+}

weed/s3api/s3lifecycle/rule_hash_test.go

@@ -0,0 +1,114 @@
+package s3lifecycle
+
+import (
+	"testing"
+)
+
+func TestRuleHash_Stable(t *testing.T) {
+	r := &Rule{Status: StatusEnabled, ExpirationDays: 30, Prefix: "logs/"}
+	a := RuleHash(r)
+	b := RuleHash(r)
+	if a != b {
+		t.Fatalf("hash should be deterministic, got %x vs %x", a, b)
+	}
+}
+
+func TestRuleHash_TagOrderInvariant(t *testing.T) {
+	r1 := &Rule{ExpirationDays: 30, FilterTags: map[string]string{"a": "1", "b": "2"}}
+	r2 := &Rule{ExpirationDays: 30, FilterTags: map[string]string{"b": "2", "a": "1"}}
+	if RuleHash(r1) != RuleHash(r2) {
+		t.Fatalf("tag order should not affect hash")
+	}
+}
+
+func TestRuleHash_PrefixTrailingSlashMattersToHash(t *testing.T) {
+	// "logs" matches "logs", "logsmore/x", "logs/x" (literal HasPrefix);
+	// "logs/" matches only "logs/x". Different match sets -> different rules
+	// -> different hashes. Collapsing would let an edit silently reuse
+	// state for a rule that no longer matches the same objects.
+	r1 := &Rule{ExpirationDays: 30, Prefix: "logs"}
+	r2 := &Rule{ExpirationDays: 30, Prefix: "logs/"}
+	if RuleHash(r1) == RuleHash(r2) {
+		t.Fatalf("trailing slash MUST affect hash; rules match different objects")
+	}
+}
+
+func TestRuleHash_IDIgnored(t *testing.T) {
+	r1 := &Rule{ID: "first", ExpirationDays: 30}
+	r2 := &Rule{ID: "renamed", ExpirationDays: 30}
+	if RuleHash(r1) != RuleHash(r2) {
+		t.Fatalf("ID change should not affect hash")
+	}
+}
+
+func TestRuleHash_StatusIgnored(t *testing.T) {
+	r1 := &Rule{Status: StatusEnabled, ExpirationDays: 30}
+	r2 := &Rule{Status: StatusDisabled, ExpirationDays: 30}
+	if RuleHash(r1) != RuleHash(r2) {
+		t.Fatalf("status flip should not affect hash (state continuity)")
+	}
+}
+
+func TestRuleHash_DifferentDaysHashDifferent(t *testing.T) {
+	r1 := &Rule{ExpirationDays: 30}
+	r2 := &Rule{ExpirationDays: 31}
+	if RuleHash(r1) == RuleHash(r2) {
+		t.Fatalf("different days must hash differently")
+	}
+}
+
+func TestRuleHash_DifferentActionTypesHashDifferent(t *testing.T) {
+	r1 := &Rule{ExpirationDays: 30}
+	r2 := &Rule{NoncurrentVersionExpirationDays: 30}
+	r3 := &Rule{AbortMPUDaysAfterInitiation: 30}
+	if RuleHash(r1) == RuleHash(r2) || RuleHash(r2) == RuleHash(r3) || RuleHash(r1) == RuleHash(r3) {
+		t.Fatalf("different action types must hash differently")
+	}
+}
+
+func TestRuleHash_FilterMatters(t *testing.T) {
+	r1 := &Rule{ExpirationDays: 30, Prefix: "logs/"}
+	r2 := &Rule{ExpirationDays: 30, Prefix: "data/"}
+	r3 := &Rule{ExpirationDays: 30, Prefix: "logs/", FilterTags: map[string]string{"env": "prod"}}
+	r4 := &Rule{ExpirationDays: 30, FilterSizeGreaterThan: 1000}
+	if RuleHash(r1) == RuleHash(r2) || RuleHash(r1) == RuleHash(r3) || RuleHash(r1) == RuleHash(r4) {
+		t.Fatalf("different filters must hash differently")
+	}
+}
+
+func TestRuleHash_NilSafe(t *testing.T) {
+	if h := RuleHash(nil); h != ([8]byte{}) {
+		t.Fatalf("nil rule should yield zero hash, got %x", h)
+	}
+}
+
+func TestRuleHash_TagDelimiterCollisionResistant(t *testing.T) {
+	// With a naive "tag=K=V\n" encoding, ("a=b", "c") and ("a", "b=c")
+	// serialize identically. Length-prefixed encoding must keep them
+	// distinct.
+	r1 := &Rule{ExpirationDays: 30, FilterTags: map[string]string{"a=b": "c"}}
+	r2 := &Rule{ExpirationDays: 30, FilterTags: map[string]string{"a": "b=c"}}
+	if RuleHash(r1) == RuleHash(r2) {
+		t.Fatalf("delimiter collision: tag(a=b,c) and tag(a,b=c) hash equal")
+	}
+}
+
+func TestRuleHash_TagNewlineCollisionResistant(t *testing.T) {
+	// "a\nb" -> single tag with a key that embeds a newline; naively this
+	// could be split across lines and matched by an unrelated 2-tag rule.
+	r1 := &Rule{ExpirationDays: 30, FilterTags: map[string]string{"a\nb": "c"}}
+	r2 := &Rule{ExpirationDays: 30, FilterTags: map[string]string{"a": "b", "c": ""}}
+	if RuleHash(r1) == RuleHash(r2) {
+		t.Fatalf("delimiter collision via embedded newline")
+	}
+}
+
+func TestRuleHash_PrefixSeparatorIsolation(t *testing.T) {
+	// A prefix containing whatever the previous encoder used as a separator
+	// ("=" / "\n") must not be able to "escape" into another field.
+	r1 := &Rule{ExpirationDays: 30, Prefix: "logs\nexp_days=99"}
+	r2 := &Rule{ExpirationDays: 99, Prefix: "logs"}
+	if RuleHash(r1) == RuleHash(r2) {
+		t.Fatalf("prefix-side delimiter forgery hashes equal")
+	}
+}

weed/s3api/s3lifecycle/scheduler/configload.go

@@ -0,0 +1,114 @@
+package scheduler
+
+import (
+	"context"
+	"strings"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/lifecycle_xml"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+)
+
+// The bucket lifecycle configuration lives here rather than in weed/s3api so
+// callers outside the gateway (the admin dashboard, the scheduler itself) and
+// the gateway share one definition of the extended-attribute keys and the
+// size cap. weed/s3api aliases these; nothing redeclares the values.
+const (
+	BucketLifecycleConfigurationXMLKey            = "s3-bucket-lifecycle-configuration-xml"
+	BucketLifecycleTransitionMinimumObjectSizeKey = "s3-bucket-lifecycle-transition-default-minimum-object-size"
+
+	// MaxBucketLifecycleConfigurationSize caps the serialized XML, mirroring
+	// the S3 API's limit.
+	MaxBucketLifecycleConfigurationSize = 1 << 20
+)
+
+// ParseError is returned alongside successfully-loaded inputs so callers can
+// surface malformed bucket configs rather than silently dropping them.
+type ParseError struct {
+	Bucket string
+	Err    error
+}
+
+// LoadCompileInputs walks the buckets directory and returns one
+// engine.CompileInput per bucket that carries a non-empty lifecycle
+// configuration. Pagination loops with startFrom so clusters with more
+// than one page of buckets don't drop the tail.
+func LoadCompileInputs(ctx context.Context, client filer_pb.SeaweedFilerClient, bucketsPath string) ([]engine.CompileInput, []ParseError, error) {
+	var (
+		inputs      []engine.CompileInput
+		parseErrors []ParseError
+		startFrom   string
+	)
+	const pageSize uint32 = 1024
+	for {
+		pageCount := 0
+		var lastName string
+		err := filer_pb.SeaweedList(ctx, client, bucketsPath, "", func(entry *filer_pb.Entry, isLast bool) error {
+			pageCount++
+			lastName = entry.Name
+			if !entry.IsDirectory {
+				return nil
+			}
+			xmlBytes, ok := entry.Extended[BucketLifecycleConfigurationXMLKey]
+			if !ok || len(xmlBytes) == 0 {
+				return nil
+			}
+			rules, err := lifecycle_xml.ParseCanonical(xmlBytes)
+			if err != nil {
+				parseErrors = append(parseErrors, ParseError{Bucket: entry.Name, Err: err})
+				return nil
+			}
+			if len(rules) == 0 {
+				return nil
+			}
+			inputs = append(inputs, engine.CompileInput{
+				Bucket:    entry.Name,
+				Rules:     rules,
+				Versioned: IsBucketVersioned(entry),
+			})
+			return nil
+		}, startFrom, false, pageSize)
+		if err != nil {
+			return nil, nil, err
+		}
+		if uint32(pageCount) < pageSize {
+			break
+		}
+		startFrom = lastName
+	}
+	return inputs, parseErrors, nil
+}
+
+// IsBucketVersioned returns true when the bucket's filer entry has the
+// versioning extended attribute set to Enabled or Suspended.
+func IsBucketVersioned(entry *filer_pb.Entry) bool {
+	v, ok := entry.Extended[s3_constants.ExtVersioningKey]
+	if !ok {
+		return false
+	}
+	s := strings.ToLower(strings.TrimSpace(string(v)))
+	return s == "enabled" || s == "suspended"
+}
+
+// AllActivePriorStates seeds every compiled action as bootstrap-complete +
+// event-driven so the scheduler dispatches whatever fires immediately.
+// Production bootstrap walks set this incrementally per bucket; the
+// scheduler runs out-of-band of that flow for now.
+func AllActivePriorStates(inputs []engine.CompileInput) map[s3lifecycle.ActionKey]engine.PriorState {
+	prior := map[s3lifecycle.ActionKey]engine.PriorState{}
+	for _, in := range inputs {
+		for _, rule := range in.Rules {
+			hash := s3lifecycle.RuleHash(rule)
+			for _, kind := range s3lifecycle.RuleActionKinds(rule) {
+				key := s3lifecycle.ActionKey{Bucket: in.Bucket, RuleHash: hash, ActionKind: kind}
+				prior[key] = engine.PriorState{
+					BootstrapComplete: true,
+					Mode:              engine.ModeEventDriven,
+				}
+			}
+		}
+	}
+	return prior
+}

weed/s3api/s3lifecycle/scheduler/configload_test.go

@@ -0,0 +1,236 @@
+package scheduler
+
+import (
+	"context"
+	"fmt"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
+	"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// Tests for LoadCompileInputs / IsBucketVersioned / AllActivePriorStates.
+// LoadCompileInputs is the bridge between the filer's bucket directory
+// and the engine snapshot the scheduler compiles every refresh —
+// pagination, parse-error surfacing, and the empty/missing-XML skip
+// semantics matter because a missed bucket silently disables lifecycle
+// for that prefix until the next refresh.
+
+const testBucketsRoot = "/buckets"
+
+// minimalLifecycleXML is the smallest XML that ParseCanonical accepts as
+// a non-empty rule set; the actual rule details are exercised by the
+// lifecycle_xml package's own tests.
+const minimalLifecycleXML = `<LifecycleConfiguration>
+  <Rule>
+    <ID>r1</ID>
+    <Status>Enabled</Status>
+    <Filter><Prefix>logs/</Prefix></Filter>
+    <Expiration><Days>30</Days></Expiration>
+  </Rule>
+</LifecycleConfiguration>`
+
+func bucketEntry(name string, extended map[string][]byte) *filer_pb.Entry {
+	return dirEntry(name, extended)
+}
+
+func TestIsBucketVersioned_MissingExtendedReturnsFalse(t *testing.T) {
+	assert.False(t, IsBucketVersioned(&filer_pb.Entry{}))
+	assert.False(t, IsBucketVersioned(&filer_pb.Entry{Extended: map[string][]byte{}}))
+}
+
+func TestIsBucketVersioned_AcceptsEnabledAndSuspended(t *testing.T) {
+	// Casing and surrounding whitespace are normalized so the worker
+	// tolerates whatever casing the writer chose.
+	for _, raw := range []string{"Enabled", "ENABLED", "enabled", "  enabled  ", "Suspended", "suspended"} {
+		t.Run(raw, func(t *testing.T) {
+			e := &filer_pb.Entry{Extended: map[string][]byte{s3_constants.ExtVersioningKey: []byte(raw)}}
+			assert.True(t, IsBucketVersioned(e), "value %q must classify as versioned", raw)
+		})
+	}
+}
+
+func TestIsBucketVersioned_RejectsOtherValues(t *testing.T) {
+	for _, raw := range []string{"", "Disabled", "Off", "true", "1", "garbage"} {
+		t.Run(raw, func(t *testing.T) {
+			e := &filer_pb.Entry{Extended: map[string][]byte{s3_constants.ExtVersioningKey: []byte(raw)}}
+			assert.False(t, IsBucketVersioned(e), "value %q must NOT classify as versioned", raw)
+		})
+	}
+}
+
+func TestAllActivePriorStates_EmptyInputs(t *testing.T) {
+	assert.Empty(t, AllActivePriorStates(nil))
+	assert.Empty(t, AllActivePriorStates([]engine.CompileInput{}))
+}
+
+func TestAllActivePriorStates_SeedsAllActionsAsActive(t *testing.T) {
+	// Every (bucket, ruleHash, actionKind) tuple from the inputs must
+	// receive a PriorState with BootstrapComplete=true and
+	// Mode=ModeEventDriven so the scheduler dispatches matched events
+	// immediately rather than waiting for a per-action bootstrap walk.
+	rule := &s3lifecycle.Rule{
+		ID:             "r1",
+		Status:         s3lifecycle.StatusEnabled,
+		ExpirationDays: 30,
+	}
+	inputs := []engine.CompileInput{{Bucket: "b1", Rules: []*s3lifecycle.Rule{rule}}}
+	prior := AllActivePriorStates(inputs)
+
+	hash := s3lifecycle.RuleHash(rule)
+	kinds := s3lifecycle.RuleActionKinds(rule)
+	require.NotEmpty(t, kinds, "expiration-days rule must yield at least one action kind")
+	for _, kind := range kinds {
+		key := s3lifecycle.ActionKey{Bucket: "b1", RuleHash: hash, ActionKind: kind}
+		state, ok := prior[key]
+		require.True(t, ok, "prior state missing for action kind %v", kind)
+		assert.True(t, state.BootstrapComplete, "bootstrap complete for kind %v", kind)
+		assert.Equal(t, engine.ModeEventDriven, state.Mode, "mode for kind %v", kind)
+	}
+	assert.Len(t, prior, len(kinds), "exactly one prior-state entry per action kind")
+}
+
+func TestAllActivePriorStates_KeysSeparatedByBucket(t *testing.T) {
+	// Two buckets sharing identical rules must still get distinct
+	// PriorState entries, otherwise refresh would conflate their state.
+	rule := &s3lifecycle.Rule{ID: "r1", Status: s3lifecycle.StatusEnabled, ExpirationDays: 30}
+	inputs := []engine.CompileInput{
+		{Bucket: "alpha", Rules: []*s3lifecycle.Rule{rule}},
+		{Bucket: "beta", Rules: []*s3lifecycle.Rule{rule}},
+	}
+	prior := AllActivePriorStates(inputs)
+	hash := s3lifecycle.RuleHash(rule)
+	for _, kind := range s3lifecycle.RuleActionKinds(rule) {
+		assert.Contains(t, prior, s3lifecycle.ActionKey{Bucket: "alpha", RuleHash: hash, ActionKind: kind})
+		assert.Contains(t, prior, s3lifecycle.ActionKey{Bucket: "beta", RuleHash: hash, ActionKind: kind})
+	}
+}
+
+func TestLoadCompileInputs_EmptyBucketDir(t *testing.T) {
+	// No buckets at all -> no inputs, no parse errors, no transport error.
+	client := &fakeFilerClient{tree: map[string][]*filer_pb.Entry{testBucketsRoot: {}}}
+	inputs, perr, err := LoadCompileInputs(context.Background(), client, testBucketsRoot)
+	require.NoError(t, err)
+	assert.Empty(t, inputs)
+	assert.Empty(t, perr)
+}
+
+func TestLoadCompileInputs_FilesAtBucketLevelSkipped(t *testing.T) {
+	// A bare file at the buckets root isn't a bucket; the loader must
+	// skip it without touching the parser.
+	client := &fakeFilerClient{tree: map[string][]*filer_pb.Entry{
+		testBucketsRoot: {fileEntry("stray.txt")},
+	}}
+	inputs, perr, err := LoadCompileInputs(context.Background(), client, testBucketsRoot)
+	require.NoError(t, err)
+	assert.Empty(t, inputs)
+	assert.Empty(t, perr)
+}
+
+func TestLoadCompileInputs_BucketWithoutLifecycleConfigSkipped(t *testing.T) {
+	// Buckets that don't carry the lifecycle XML extended attribute are
+	// silently skipped — operators may have many buckets without rules.
+	client := &fakeFilerClient{tree: map[string][]*filer_pb.Entry{
+		testBucketsRoot: {bucketEntry("b1", nil)},
+	}}
+	inputs, perr, err := LoadCompileInputs(context.Background(), client, testBucketsRoot)
+	require.NoError(t, err)
+	assert.Empty(t, inputs)
+	assert.Empty(t, perr)
+}
+
+func TestLoadCompileInputs_BucketWithEmptyLifecycleXMLSkipped(t *testing.T) {
+	// Empty bytes under the lifecycle key shouldn't reach the parser.
+	client := &fakeFilerClient{tree: map[string][]*filer_pb.Entry{
+		testBucketsRoot: {bucketEntry("b1", map[string][]byte{
+			BucketLifecycleConfigurationXMLKey: nil,
+		})},
+	}}
+	inputs, perr, err := LoadCompileInputs(context.Background(), client, testBucketsRoot)
+	require.NoError(t, err)
+	assert.Empty(t, inputs)
+	assert.Empty(t, perr)
+}
+
+func TestLoadCompileInputs_ValidConfigBecomesInput(t *testing.T) {
+	client := &fakeFilerClient{tree: map[string][]*filer_pb.Entry{
+		testBucketsRoot: {bucketEntry("b1", map[string][]byte{
+			BucketLifecycleConfigurationXMLKey: []byte(minimalLifecycleXML),
+		})},
+	}}
+	inputs, perr, err := LoadCompileInputs(context.Background(), client, testBucketsRoot)
+	require.NoError(t, err)
+	assert.Empty(t, perr)
+	require.Len(t, inputs, 1)
+	assert.Equal(t, "b1", inputs[0].Bucket)
+	assert.False(t, inputs[0].Versioned, "no versioning attr -> not versioned")
+	assert.NotEmpty(t, inputs[0].Rules)
+}
+
+func TestLoadCompileInputs_VersioningAttrPropagates(t *testing.T) {
+	client := &fakeFilerClient{tree: map[string][]*filer_pb.Entry{
+		testBucketsRoot: {bucketEntry("b1", map[string][]byte{
+			BucketLifecycleConfigurationXMLKey: []byte(minimalLifecycleXML),
+			s3_constants.ExtVersioningKey:      []byte("Enabled"),
+		})},
+	}}
+	inputs, perr, err := LoadCompileInputs(context.Background(), client, testBucketsRoot)
+	require.NoError(t, err)
+	assert.Empty(t, perr)
+	require.Len(t, inputs, 1)
+	assert.True(t, inputs[0].Versioned)
+}
+
+func TestLoadCompileInputs_MalformedXMLBecomesParseError(t *testing.T) {
+	// Malformed XML must not abort the whole walk — surfacing it as a
+	// ParseError lets operators see the bucket-level failure while
+	// other buckets keep loading.
+	client := &fakeFilerClient{tree: map[string][]*filer_pb.Entry{
+		testBucketsRoot: {
+			bucketEntry("bad", map[string][]byte{
+				BucketLifecycleConfigurationXMLKey: []byte("<not-valid-xml"),
+			}),
+			bucketEntry("good", map[string][]byte{
+				BucketLifecycleConfigurationXMLKey: []byte(minimalLifecycleXML),
+			}),
+		},
+	}}
+	inputs, perr, err := LoadCompileInputs(context.Background(), client, testBucketsRoot)
+	require.NoError(t, err)
+	require.Len(t, perr, 1)
+	assert.Equal(t, "bad", perr[0].Bucket)
+	assert.Error(t, perr[0].Err)
+
+	require.Len(t, inputs, 1)
+	assert.Equal(t, "good", inputs[0].Bucket)
+}
+
+func TestLoadCompileInputs_PaginatesAcrossPages(t *testing.T) {
+	// pageSize is 1024 inside LoadCompileInputs; populate just over the
+	// boundary to force a second listing call. Every bucket has the same
+	// minimal config so the input count must equal the bucket count.
+	const total = 1030
+	entries := make([]*filer_pb.Entry, 0, total)
+	for i := 0; i < total; i++ {
+		entries = append(entries, bucketEntry(fmt.Sprintf("b%05d", i), map[string][]byte{
+			BucketLifecycleConfigurationXMLKey: []byte(minimalLifecycleXML),
+		}))
+	}
+	client := &fakeFilerClient{tree: map[string][]*filer_pb.Entry{
+		testBucketsRoot: entries,
+	}}
+	inputs, perr, err := LoadCompileInputs(context.Background(), client, testBucketsRoot)
+	require.NoError(t, err)
+	assert.Empty(t, perr)
+	require.Len(t, inputs, total, "pagination must surface every bucket")
+	// Order is preserved across pages, and the page boundary at 1023/1024
+	// (pageSize is 1024) must not skip or duplicate entries.
+	assert.Equal(t, "b00000", inputs[0].Bucket)
+	assert.Equal(t, "b01023", inputs[1023].Bucket, "last entry of first page")
+	assert.Equal(t, "b01024", inputs[1024].Bucket, "first entry of second page")
+	assert.Equal(t, fmt.Sprintf("b%05d", total-1), inputs[total-1].Bucket)
+}

weed/s3api/s3lifecycle/scheduler/testhelpers_test.go

@@ -0,0 +1,119 @@
+package scheduler
+
+import (
+	"context"
+	"io"
+	"sort"
+	"sync"
+	"sync/atomic"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/metadata"
+)
+
+// fakeListStream implements grpc.ServerStreamingClient[filer_pb.ListEntriesResponse]
+// for configload tests.
+type fakeListStream struct {
+	responses []*filer_pb.ListEntriesResponse
+	index     int
+	ctx       context.Context
+}
+
+func (s *fakeListStream) Recv() (*filer_pb.ListEntriesResponse, error) {
+	if s.ctx != nil {
+		if err := s.ctx.Err(); err != nil {
+			return nil, err
+		}
+	}
+	if s.index >= len(s.responses) {
+		return nil, io.EOF
+	}
+	r := s.responses[s.index]
+	s.index++
+	return r, nil
+}
+
+func (s *fakeListStream) Header() (metadata.MD, error) { return metadata.MD{}, nil }
+func (s *fakeListStream) Trailer() metadata.MD         { return metadata.MD{} }
+func (s *fakeListStream) CloseSend() error             { return nil }
+func (s *fakeListStream) Context() context.Context {
+	if s.ctx != nil {
+		return s.ctx
+	}
+	return context.Background()
+}
+func (s *fakeListStream) SendMsg(any) error { return nil }
+func (s *fakeListStream) RecvMsg(any) error { return nil }
+
+// fakeFilerClient is the in-memory filer used by configload tests.
+type fakeFilerClient struct {
+	filer_pb.SeaweedFilerClient
+
+	mu      sync.Mutex
+	tree    map[string][]*filer_pb.Entry
+	listed  []string
+	listedN int32
+}
+
+func (c *fakeFilerClient) LookupDirectoryEntry(_ context.Context, in *filer_pb.LookupDirectoryEntryRequest, _ ...grpc.CallOption) (*filer_pb.LookupDirectoryEntryResponse, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	for _, e := range c.tree[in.Directory] {
+		if e != nil && e.Name == in.Name {
+			return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil
+		}
+	}
+	return nil, filer_pb.ErrNotFound
+}
+
+func (c *fakeFilerClient) ListEntries(ctx context.Context, in *filer_pb.ListEntriesRequest, _ ...grpc.CallOption) (grpc.ServerStreamingClient[filer_pb.ListEntriesResponse], error) {
+	c.mu.Lock()
+	c.listed = append(c.listed, in.Directory)
+	src := c.tree[in.Directory]
+	c.mu.Unlock()
+	atomic.AddInt32(&c.listedN, 1)
+
+	filtered := make([]*filer_pb.Entry, 0, len(src))
+	for _, e := range src {
+		if e == nil {
+			continue
+		}
+		if in.StartFromFileName != "" {
+			if in.InclusiveStartFrom {
+				if e.Name < in.StartFromFileName {
+					continue
+				}
+			} else if e.Name <= in.StartFromFileName {
+				continue
+			}
+		}
+		filtered = append(filtered, e)
+	}
+	sort.SliceStable(filtered, func(i, j int) bool { return filtered[i].Name < filtered[j].Name })
+	if in.Limit > 0 && uint32(len(filtered)) > in.Limit {
+		filtered = filtered[:in.Limit]
+	}
+	resps := make([]*filer_pb.ListEntriesResponse, 0, len(filtered))
+	for _, e := range filtered {
+		resps = append(resps, &filer_pb.ListEntriesResponse{Entry: e})
+	}
+	return &fakeListStream{responses: resps, ctx: ctx}, nil
+}
+
+func dirEntry(name string, extended map[string][]byte) *filer_pb.Entry {
+	return &filer_pb.Entry{
+		Name:        name,
+		IsDirectory: true,
+		Attributes:  &filer_pb.FuseAttributes{},
+		Extended:    extended,
+	}
+}
+
+func fileEntry(name string) *filer_pb.Entry {
+	return &filer_pb.Entry{
+		Name:        name,
+		IsDirectory: false,
+		Attributes:  &filer_pb.FuseAttributes{},
+	}
+}

weed/s3api/s3lifecycle/shard.go

@@ -0,0 +1,33 @@
+package s3lifecycle
+
+import (
+	"crypto/sha256"
+	"hash"
+	"sync"
+)
+
+// ShardCount partitions the (bucket, key) keyspace for the per-shard
+// lifecycle reader. Workers receive one READ task per owned shard;
+// each shard's cursor advances independently.
+const ShardCount = 16
+
+// shardHashPool reuses sha256 hashers — ShardID is in the per-event hot
+// path and a fresh allocation per call shows up under load.
+var shardHashPool = sync.Pool{
+	New: func() interface{} { return sha256.New() },
+}
+
+// ShardID maps (bucket, key) to a shard in [0, ShardCount). Stable across
+// restarts and across processes — identical (bucket, key) always lands on
+// the same shard. Implementation: top 4 bits of sha256(bucket || "/" || key).
+func ShardID(bucket, key string) int {
+	h := shardHashPool.Get().(hash.Hash)
+	defer shardHashPool.Put(h)
+	h.Reset()
+	h.Write([]byte(bucket))
+	h.Write([]byte{'/'})
+	h.Write([]byte(key))
+	var buf [sha256.Size]byte
+	sum := h.Sum(buf[:0])
+	return int(sum[0] >> 4)
+}

weed/s3api/s3lifecycle/shard_test.go

@@ -0,0 +1,46 @@
+package s3lifecycle
+
+import "testing"
+
+func TestShardIDInRange(t *testing.T) {
+	cases := []struct{ bucket, key string }{
+		{"", ""},
+		{"a", "b"},
+		{"my-bucket", "path/to/object.txt"},
+		{"日本語", "キー"},
+		{"bucket-with-dash", ""},
+	}
+	for _, c := range cases {
+		got := ShardID(c.bucket, c.key)
+		if got < 0 || got >= ShardCount {
+			t.Fatalf("ShardID(%q,%q)=%d outside [0,%d)", c.bucket, c.key, got, ShardCount)
+		}
+	}
+}
+
+func TestShardIDDeterministic(t *testing.T) {
+	a := ShardID("bucket", "key")
+	b := ShardID("bucket", "key")
+	if a != b {
+		t.Fatalf("non-deterministic: %d vs %d", a, b)
+	}
+}
+
+func TestShardIDDistinctFromBucket(t *testing.T) {
+	// Same key, different buckets must be free to land on different shards.
+	// (Statistical: with 16 shards, picking 16 distinct buckets should hit
+	// at least 4 different shards.)
+	seen := map[int]struct{}{}
+	for i := 0; i < 16; i++ {
+		buckets := []string{
+			"alpha", "bravo", "charlie", "delta",
+			"echo", "foxtrot", "golf", "hotel",
+			"india", "juliet", "kilo", "lima",
+			"mike", "november", "oscar", "papa",
+		}
+		seen[ShardID(buckets[i], "samekey")] = struct{}{}
+	}
+	if len(seen) < 4 {
+		t.Fatalf("expected >=4 distinct shards across 16 buckets, got %d", len(seen))
+	}
+}

weed/s3api/s3lifecycle/tags.go

@@ -0,0 +1,3 @@
+package s3lifecycle
+
+const tagPrefix = "X-Amz-Tagging-"

weed/s3api/s3lifecycle/version_time.go

@@ -0,0 +1,76 @@
+package s3lifecycle
+
+import (
+	"strconv"
+)
+
+// versionIdFormatThreshold distinguishes old vs new format version IDs.
+// New format (inverted timestamps) produces values above this threshold;
+// old format (raw timestamps) produces values below it.
+const versionIdFormatThreshold = 0x4000000000000000
+
+// CompareVersionIds returns negative if a is newer than b, positive if b
+// is newer, 0 if equal. Mirrors compareVersionIds in s3api_version_id.go
+// (kept duplicated to avoid the s3api -> s3lifecycle import cycle). Used
+// as a tiebreak when version mtimes collide at second resolution.
+func CompareVersionIds(a, b string) int {
+	if a == b {
+		return 0
+	}
+	if a == "null" {
+		return 1 // null sorts last
+	}
+	if b == "null" {
+		return -1
+	}
+	aIsNew := isNewFormatVersionId(a)
+	bIsNew := isNewFormatVersionId(b)
+	if aIsNew == bIsNew {
+		// Same format. New format (inverted timestamps) sorts smaller=newer
+		// lexicographically; old format sorts smaller=older.
+		if aIsNew {
+			if a < b {
+				return -1
+			}
+			return 1
+		}
+		if a < b {
+			return 1
+		}
+		return -1
+	}
+	at := getVersionTimestamp(a)
+	bt := getVersionTimestamp(b)
+	if at > bt {
+		return -1
+	}
+	if at < bt {
+		return 1
+	}
+	return 0
+}
+
+func isNewFormatVersionId(versionId string) bool {
+	if len(versionId) < 16 || versionId == "null" {
+		return false
+	}
+	t, err := strconv.ParseUint(versionId[:16], 16, 64)
+	if err != nil {
+		return false
+	}
+	return t > versionIdFormatThreshold
+}
+
+func getVersionTimestamp(versionId string) int64 {
+	if len(versionId) < 16 || versionId == "null" {
+		return 0
+	}
+	t, err := strconv.ParseUint(versionId[:16], 16, 64)
+	if err != nil {
+		return 0
+	}
+	if t > versionIdFormatThreshold {
+		return int64(^uint64(0)>>1) - int64(t)
+	}
+	return int64(t)
+}

weed/s3api/s3lifecycle/version_time_test.go

@@ -0,0 +1,168 @@
+package s3lifecycle
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+// CompareVersionIds is the version-id tiebreak the router falls back to
+// when two versions land at the same mtime second; getting the format /
+// null / mixed-format axes wrong silently inverts retention rankings,
+// which would resurrect deleted versions or evict live ones. The tests
+// below pin every documented branch.
+
+const (
+	// 16-char hex prefix above 0x4000000000000000 → new-format inverted
+	// timestamp.
+	newFormatPrefix1 = "4123456789abcdef"
+	newFormatPrefix2 = "5123456789abcdef" // strictly larger than prefix1
+	// 16-char hex prefix at or below the threshold → old-format raw
+	// timestamp.
+	oldFormatPrefix1 = "0123456789abcdef"
+	oldFormatPrefix2 = "0223456789abcdef" // strictly larger than prefix1
+)
+
+func TestCompareVersionIds_EqualReturnsZero(t *testing.T) {
+	assert.Equal(t, 0, CompareVersionIds("abc", "abc"))
+	assert.Equal(t, 0, CompareVersionIds("null", "null"))
+	assert.Equal(t, 0, CompareVersionIds("", ""))
+}
+
+func TestCompareVersionIds_NullSortsLast(t *testing.T) {
+	// "null" represents the bare pre-versioning entry; the router treats
+	// it as older than any real version-id so genuine versions take
+	// precedence in tiebreaks.
+	assert.Equal(t, 1, CompareVersionIds("null", newFormatPrefix1))
+	assert.Equal(t, 1, CompareVersionIds("null", oldFormatPrefix1))
+	assert.Equal(t, -1, CompareVersionIds(newFormatPrefix1, "null"))
+	assert.Equal(t, -1, CompareVersionIds(oldFormatPrefix1, "null"))
+}
+
+func TestCompareVersionIds_BothNewFormatSmallerIsNewer(t *testing.T) {
+	// New-format encodes the timestamp inverted, so the lexicographically
+	// smaller string represents a newer wall-clock time.
+	assert.Equal(t, -1, CompareVersionIds(newFormatPrefix1, newFormatPrefix2),
+		"new-format: smaller lex → newer → -1")
+	assert.Equal(t, 1, CompareVersionIds(newFormatPrefix2, newFormatPrefix1),
+		"new-format: larger lex → older → 1")
+}
+
+func TestCompareVersionIds_BothOldFormatLargerIsNewer(t *testing.T) {
+	// Old-format encodes the timestamp raw, so the lexicographically
+	// larger string is the newer wall-clock time.
+	assert.Equal(t, 1, CompareVersionIds(oldFormatPrefix1, oldFormatPrefix2),
+		"old-format: smaller lex → older → 1")
+	assert.Equal(t, -1, CompareVersionIds(oldFormatPrefix2, oldFormatPrefix1),
+		"old-format: larger lex → newer → -1")
+}
+
+func TestCompareVersionIds_MixedFormatComparesByTimestamp(t *testing.T) {
+	// New-format inverted timestamp = MaxInt64 - prefix; old-format raw =
+	// prefix. With prefix1 = 0x4123… and old prefix1 = 0x0123… the
+	// inverted new value is small but still positive, while the raw old
+	// value is also small. Whichever is numerically larger wins.
+	at := getVersionTimestamp(newFormatPrefix1)
+	bt := getVersionTimestamp(oldFormatPrefix1)
+	if at == bt {
+		assert.Equal(t, 0, CompareVersionIds(newFormatPrefix1, oldFormatPrefix1))
+		return
+	}
+	if at > bt {
+		assert.Equal(t, -1, CompareVersionIds(newFormatPrefix1, oldFormatPrefix1))
+		assert.Equal(t, 1, CompareVersionIds(oldFormatPrefix1, newFormatPrefix1))
+	} else {
+		assert.Equal(t, 1, CompareVersionIds(newFormatPrefix1, oldFormatPrefix1))
+		assert.Equal(t, -1, CompareVersionIds(oldFormatPrefix1, newFormatPrefix1))
+	}
+}
+
+func TestCompareVersionIds_MixedFormatEqualTimestampReturnsZero(t *testing.T) {
+	// If a new-format inverted timestamp equals an old-format raw
+	// timestamp, neither is newer — should be 0. We can synthesize this:
+	// pick an old prefix P, the matching new prefix is MaxInt64 - P.
+	old := uint64(0x0000000000010000)
+	new_ := uint64(int64(^uint64(0)>>1) - int64(old))
+	oldHex := uintToHex16(old)
+	newHex := uintToHex16(new_)
+	// Sanity: new_ must be above the threshold, otherwise it'd be parsed
+	// as old-format and the test wouldn't exercise the mixed branch.
+	if !isNewFormatVersionId(newHex) {
+		t.Fatalf("constructed new-format prefix %q didn't classify as new", newHex)
+	}
+	if isNewFormatVersionId(oldHex) {
+		t.Fatalf("constructed old-format prefix %q classified as new", oldHex)
+	}
+	assert.Equal(t, 0, CompareVersionIds(newHex, oldHex), "mixed-format equal timestamps must be 0")
+}
+
+func TestIsNewFormatVersionId_TooShortRejected(t *testing.T) {
+	// Strings shorter than 16 chars can't carry a hex timestamp prefix;
+	// they must classify as old-format so getVersionTimestamp returns 0
+	// rather than panic on slice bounds.
+	assert.False(t, isNewFormatVersionId(""))
+	assert.False(t, isNewFormatVersionId("4123"))
+	assert.False(t, isNewFormatVersionId("4123456789abcde")) // 15 chars
+}
+
+func TestIsNewFormatVersionId_NullRejected(t *testing.T) {
+	assert.False(t, isNewFormatVersionId("null"))
+}
+
+func TestIsNewFormatVersionId_NonHexRejected(t *testing.T) {
+	// Parse failure on the 16-char prefix means the comparator falls
+	// back to old-format treatment. Non-hex characters or surrounding
+	// whitespace must reject without erroring out.
+	assert.False(t, isNewFormatVersionId("zzzzzzzzzzzzzzzz"))
+	assert.False(t, isNewFormatVersionId("0xff000000000000")) // "0x" prefix breaks ParseUint base-16
+	assert.False(t, isNewFormatVersionId("ABC GHIJKLMNOPQR")) // space mid-string
+}
+
+func TestIsNewFormatVersionId_ThresholdBoundary(t *testing.T) {
+	// versionIdFormatThreshold = 0x4000000000000000. The check is
+	// strictly greater than, so the threshold value itself must
+	// classify as old-format.
+	assert.False(t, isNewFormatVersionId("4000000000000000"), "exactly at threshold is old")
+	assert.True(t, isNewFormatVersionId("4000000000000001"), "one above threshold is new")
+	assert.False(t, isNewFormatVersionId("3fffffffffffffff"), "one below threshold is old")
+}
+
+func TestGetVersionTimestamp_ReturnsZeroOnMalformed(t *testing.T) {
+	assert.Equal(t, int64(0), getVersionTimestamp(""))
+	assert.Equal(t, int64(0), getVersionTimestamp("null"))
+	assert.Equal(t, int64(0), getVersionTimestamp("4123456789abcde"))      // too short
+	assert.Equal(t, int64(0), getVersionTimestamp("zzzzzzzzzzzzzzzz"))     // not hex
+	assert.Equal(t, int64(0), getVersionTimestamp("zzzzzzzzzzzzzzzzMORE")) // not hex prefix
+}
+
+func TestGetVersionTimestamp_OldFormatReturnsRawValue(t *testing.T) {
+	// Old-format: prefix below threshold, returned raw.
+	assert.Equal(t, int64(0x0123456789abcdef), getVersionTimestamp(oldFormatPrefix1))
+	assert.Equal(t, int64(0x0223456789abcdef), getVersionTimestamp(oldFormatPrefix2))
+	// Trailing characters past the 16-char prefix are ignored.
+	assert.Equal(t, int64(0x0123456789abcdef), getVersionTimestamp(oldFormatPrefix1+"-suffix"))
+}
+
+func TestGetVersionTimestamp_NewFormatReturnsInvertedValue(t *testing.T) {
+	// New-format inversion: int64 max - parsed value. So a smaller
+	// stored prefix yields a larger inverted timestamp (= newer time).
+	maxI64 := int64(^uint64(0) >> 1)
+	assert.Equal(t, maxI64-int64(0x4123456789abcdef), getVersionTimestamp(newFormatPrefix1))
+	assert.Equal(t, maxI64-int64(0x5123456789abcdef), getVersionTimestamp(newFormatPrefix2))
+	// The smaller-prefix-=-newer-timestamp invariant the comparator
+	// relies on:
+	assert.Greater(t, getVersionTimestamp(newFormatPrefix1), getVersionTimestamp(newFormatPrefix2))
+}
+
+// uintToHex16 turns a uint64 into a 16-char lowercase hex string,
+// padding with leading zeros. Used for synthesizing mixed-format
+// equal-timestamp pairs in TestCompareVersionIds_MixedFormatEqualTimestampReturnsZero.
+func uintToHex16(v uint64) string {
+	const hex = "0123456789abcdef"
+	out := make([]byte, 16)
+	for i := 15; i >= 0; i-- {
+		out[i] = hex[v&0xf]
+		v >>= 4
+	}
+	return string(out)
+}