upstream-seaweedfs/sw-block/design/v3-phase-14-s8-evidence-matrix.md

V3 Phase 14 S8 Evidence Matrix

Date: 2026-04-20 Status: draft (S8 evidence consolidation) Purpose: map every internal P14 claim from v3-phase-14-s8-final-bounded-close.md §5 to concrete tests and commands, classify L0/L1/L2/L3 coverage, and surface residual gaps for 14A / P15 handoff

1. How To Read This Matrix

Each row is one internal P14 claim. Columns:

• L0 Unit — package-local invariant / policy test that proves the claim inside one package.
• L1 Component — in-process multi-package route test (real ObservationHost + TopologyController + Publisher + VolumeBridge + VolumeReplicaAdapter, no shell, no kernel).
• L2 Process — real sparrow binary or in-process Bootstrap() against a real filesystem store.
• L3 Scenario — hardware / YAML-driven scenario. For P14 S8, L3 is classification-only: the rows below say "L3 shape only, not executed" where a V2 YAML scenario describes the same claim shape. L3 entries are NOT executed evidence; they are the scenario shapes that P15 Cluster Validation (Final Gate) will run. A claim with an L3 shape listed is NOT proven at L3 by S8.
• Status — PROVEN (explicit test); PARTIAL (route covered but some sub-claim deferred); DEFERRED (intentionally pushed to later slice / P15); N/A (not applicable at that level).
• Residual — what is NOT proven and where it is carried.

All cited L0/L1/L2 tests live in the committed tree; re-run with:

go test ./core/engine ./core/adapter ./core/authority ./cmd/sparrow -count=1

Baseline on 2026-04-20: engine 0.02s / adapter 2.3s / authority 4.6s / sparrow 2.3s — all green, 193 Go tests total across the S4-S7 scope.

2. Accepted Topology Claim Recap

Pinned by the S8 close doc §4. Reproduced here so every row in the matrix below is read against the correct bounded shape:

1. single active master
2. multiple volumes
3. three bounded replica slots per volume, distinct servers
4. one current authoritative primary, two failover/rebalance candidates
5. publisher-owned Epoch / EndpointVersion
6. durable current authority line, one record per volume
7. passive convergence (publish → observe → confirm / stuck / supersede)
8. real VolumeBridge delivery into adapter/engine

Anything outside this set is P15 or explicit non-goal.

3. Evidence Matrix

Claim 1 — Observation is system-fed, not test-authored

L0	L1	L2	L3	Status	Residual
TestObservation_BuilderStampsAuthorityFromReader / TestObservation_SupportabilityDoesNotReadClusterSnapshotOutput / TestObservation_NeverMintsAuthority_BoundaryGuard	TestObservation_EndToEnd_SystemFedSnapshotReachesController / TestConvergenceRoute_ObservationFedFailover_ConfirmsAndClearsDesired / TestS7_ReloadedAuthority_ReanchorsAdapterViaVolumeBridge	(covered at L1; sparrow run-path uses the same ObservationHost construction)	L3 shape only, not executed — ha-restart-recovery.yaml / ha-failover.yaml carry the L3 shape	PROVEN	none at L0/L1

Claim 2 — Topology supportability is explicit and fail-closed (partial inventory / missing-server / conflict / duplicate / unknown-volume / expired)

L0	L1	L2	L3	Status	Residual
TestObservation_PartialInventory_PendingThenUnsupported / TestObservation_MissingServerObservation_NeverOrdinaryIneligible / TestObservation_ConflictingPrimaryClaim_Unsupported / ..._NoWinnerEverChosen / TestObservation_DuplicateServerTopology_UnsupportedAndEvidenceRecorded / TestObservation_UnknownObservedVolume_ProducesUnsupportedEvidence / TestObservation_ExpiredFact_SemanticallyIneligible / TestObservation_ExpiredDoesNotReachController / TestObservation_FreshnessWindowHonored_NotTickMultiples	TestObservation_SupportabilityCollapsePropagatesToController / TestObservation_SupportabilityCollapseClearsControllerDesired / TestS7_Restart_UnsupportedTopologyRecordsEvidence	N/A (covered by L0/L1)	L3 shape only, not executed — fault-partition.yaml is the L3 shape	PROVEN	none

Claim 3 — Durable authority is single-owner, atomic, and restart-recovered

L0	L1	L2	L3	Status	Residual
TestFileAuthorityStore_RoundTrip / TestFileAuthorityStore_LatestWinsPerVolumeID / TestFileAuthorityStore_EnvelopeShape / TestDurableAuthority_AtomicWrite_NoTornRecord / TestDurableAuthority_CorruptRecord_PerVolumeFailClosed / TestDurableAuthority_ProcessLock_ExclusiveOwner / TestDurableAuthority_ProcessLock_IdempotentRelease / TestDurableAuthority_ProcessLock_SurvivesStaleFileOnDisk / TestDurableAuthority_UnreadableIndex_BootFailsClosed / TestDurableAuthority_StoreNeverMintsAuthority_BoundaryGuard / TestDurableAuthority_StoreWriteCallerAllowlist_BoundaryGuard / TestEncodeVolumeIDForFilename_Injective / TestEncodeVolumeIDForFilename_NoFilesystemUnsafeOutput	TestDurableAuthority_ReloadReproducesCurrentLine / TestDurableAuthority_ControllerSeesReloadedLineAtBoot / TestDurableAuthority_StoreBackedPublisherRunsNormalRoute / TestDurableAuthority_PutFailureRollsBackInMemoryState / TestS7_ReloadedAuthority_ReanchorsAdapterViaVolumeBridge	TestBootstrap_FreshDir_AcquiresLockAndReloads / TestBootstrap_WithExistingRecord_ReloadsPublisherState / TestBootstrap_CorruptRecord_LoggedAtStartup / TestBootstrap_LockContention_RefusesSecondBootstrap / TestS7Process_BootstrapReloadRouteSmoke / subprocess: TestS7Process_RealSubprocessRestartSmoke	L3 shape only, not executed — ha-restart-recovery.yaml / diag-restart-recovery.yaml	PROVEN	Windows os.RemoveAll may race with lock release at teardown — logged as residual per S7 sketch §8.5; does NOT affect correctness

Claim 4 — No old-slot durable revival after restart

L0	L1	L2	L3	Status	Residual
TestDurableAuthority_StalePreFailoverRecordRejectedOnReload	TestS7_Restart_OldSlotNotRevivedFromDurable (explicitly asserts LastAuthorityBasis("vr","r1")==false and ("vr","r2")==Epoch=2 post-restart)	covered by S7 subprocess smoke plus in-process two-run TestS7Process_RestartDoesNotRegressAuthorityLine	L3 shape only, not executed	PROVEN	none — row 4a/4b split in S7 closes the original ambiguity

Claim 5 — Live-route monotonic guard rejects stale old-slot delivery (pre-restart)

L0	L1	L2	L3	Status	Residual
adapter-internal: engine monotonic guards covered in core/adapter/adapter_test.go and core/engine/*_test.go	TestS7_LiveRoute_OldSlotDeliveryRejectedByAdapter (real VolumeBridge, in-memory publisher holds both per-replica keys, lower-epoch catch-up rejected)	N/A (live-route, pre-restart)	N/A	PROVEN	none

Claim 6 — Controller-driven bind / reassign / refresh over accepted topology (system-driven, not test-driven)

L0	L1	L2	L3	Status	Residual
TestTopologyController_InitialPlacementBalancesAcrossVolumes / ..._InitialPlacementUsesEvidenceTieBreakOnEqualLoad / ..._FailoverUsesHighestEvidenceCandidate / ..._RebalanceMovesToLighterServer / ..._RebalanceSkipsWhenLoadAlreadyWithinBound / ..._ObservedAuthorityClearsPending / ..._StalePreConfirmSnapshotDoesNotDuplicateQueuedMove	Placement / failover: TestTopologyControllerToPublisher_E2E_MultiVolumePlacementAndFailover / TestObservationHost_IdenticalSupportedSnapshots_NoDuplicateAsks / TestObservationHost_VolumeRecovers_ClearsStaleDesired. Rebalance: TestTopologyControllerToPublisher_E2E_Rebalance. Refresh: TestConvergenceRoute_RefreshEndpoint_ConfirmsAndClearsDesired (host heartbeat changes slot DataAddr/CtrlAddr while publisher holds old addrs → controller emits IntentRefreshEndpoint → publisher bumps EndpointVersion on same Epoch → bridge delivers refreshed AssignmentInfo → confirmation clears desired).	TestS7Process_RealSubprocessRestartSmoke covers the Bind mint via the real binary (Reassign/Refresh are L1-only in sparrow today; real binary does not ingest heartbeats — see §5 L2 scope note).	L3 shape only, not executed	PROVEN	none

Claim 7 — Convergence: confirmation clears desired

L0	L1	L2	L3	Status	Residual
TestConvergence_Confirmation_ClearsDesired / TestConvergence_Confirmation_RequiresBothSources / TestConvergence_StaleObservation_NoConfirmation / TestConvergence_StuckThenConfirm_ClearsStuckEvidence	TestConvergenceRoute_ObservationFedFailover_ConfirmsAndClearsDesired	N/A	L3 shape only, not executed	PROVEN	none

Claim 8 — Publish-not-observed stuck evidence is bounded and non-churning

L0	L1	L2	L3	Status	Residual
TestConvergence_PublishNotObserved_StuckAfterWindow / TestConvergence_Stuck_DoesNotReMintReassign / TestConvergence_Reassign_CalledAtMostOncePerDesired / TestConvergence_RetryStateIsPerVolume / TestConvergence_SupportedRecovery_ClearsPriorUnsupportedEvidence	TestConvergenceRoute_PublishNotObserved_StuckWithoutChurn	N/A	L3 shape only, not executed	PROVEN	none

Claim 9 — Supersede by newer authority or different decision

L0	L1	L2	L3	Status	Residual
TestConvergence_Supersede_PublisherAdvancedToOtherReplica / TestConvergence_NormalLag_OldObservationDoesNotSupersede / TestConvergence_Supersede_DecisionTableProducesDifferentAsk	TestConvergenceRoute_Supersede_PublisherMovedElsewhere_DropsStaleDesired	N/A	L3 shape only, not executed	PROVEN	Case 2 of the supersede rule was intentionally dropped (see S6 sketch §9) — that decision is tested indirectly by TestConvergence_NormalLag_OldObservationDoesNotSupersede which would otherwise fail

Claim 10 — Restart re-anchors authority via real VolumeBridge into the adapter/engine route

L0	L1	L2	L3	Status	Residual
TestDurableAuthority_PublisherAdvancesFromReloaded	TestS7_ReloadedAuthority_ReanchorsAdapterViaVolumeBridge / TestS7_Restart_RecomputesDesiredNotTransientState (hard precondition prevents vacuous pass)	Not at L2. The sparrow subprocess smoke (TestS7Process_RealSubprocessRestartSmoke) reloads durable Publisher state only; it does NOT construct ObservationHost, TopologyController, VolumeBridge, or VolumeReplicaAdapter, so the bridge/adapter re-anchor cannot be counted at L2. L2 proves a strictly narrower subclaim (durable Publisher reload — Claim 3).	L3 shape only, not executed — ha-restart-recovery.yaml is the L3 shape	PROVEN at L1 (full route); L2 COVERS a subclaim only (durable reload, see Claim 3)	Subprocess heartbeat ingress + real-binary adapter route → P15 T1 (handoff item #2)

Claim 11 — Stale observation cannot move authority backward

L0	L1	L2	L3	Status	Residual
TestConvergence_StaleObservation_NoConfirmation	TestS7_Restart_StaleObservationCannotMoveBackward (4 sub-cases: stale-epoch / stale-replica / stale-endpointVersion / unassigned via snapshottingReader host-reader freeze) / TestConvergenceRoute_PublishNotObserved_StuckWithoutChurn (proves host-reader lag scenario)	covered at L1	L3 shape only, not executed — fault-partition.yaml	PROVEN	stale-basis simulation is via host-reader freeze; heartbeat wire does not carry authority (documented in S7 sketch §9.1)

Claim 12 — Unsupported topology after restart records evidence, not silent idle

L0	L1	L2	L3	Status	Residual
TestConvergence_UnsupportedClearsDesiredWithEvidence / TestConvergence_PendingClearsDesired	TestS7_Restart_UnsupportedTopologyRecordsEvidence	Not at L2. This claim is about the controller's LastUnsupported(vid) surface being populated from conflicting / incomplete observation; that requires ObservationHost + controller composition, which the subprocess does not construct. Bootstrap.ReloadSkips is a DIFFERENT surface (S5 durable-record corruption, see Claim 3 residual path) and is NOT observation-layer unsupported-topology evidence.	L3 shape only, not executed — fault-partition.yaml + diag-restart-recovery.yaml are the L3 shapes	PROVEN at L1	Operator-visible surface over LastUnsupported / LastConvergenceStuck → P15 T5 Diagnostics (handoff item #6)

Claim 13 — Per-volume isolation (one volume's failure does not block others)

L0	L1	L2	L3	Status	Residual
TestObservation_BadVolumeDoesNotBlockHealthyVolume_Isolation / TestObservation_VolumeUnsupportedEvidence_IsolatedPerVolume / TestConvergence_OneStuckVolumeDoesNotBlockOthers / TestTopologyController_UnsupportedVolumeDoesNotBlockOtherVolumes	TestS7_Restart_PerVolumeIsolation	N/A	L3 shape only, not executed	PROVEN	none

Claim 14 — Placement / rebalance / failover within the accepted topology

L0	L1	L2	L3	Status	Residual
TestTopologyController_InitialPlacementBalancesAcrossVolumes / ..._FailoverUsesHighestEvidenceCandidate / ..._RebalanceMovesToLighterServer / ..._RebalanceSkipsWhenLoadAlreadyWithinBound / ..._OutOfTopologyAuthority_NoAskPlusEvidence	TestTopologyControllerToPublisher_E2E_MultiVolumePlacementAndFailover / TestTopologyControllerToPublisher_E2E_Rebalance	N/A	L3 shape only, not executed — ha-failover.yaml carries the L3 shape	PROVEN	none

Claim 15 — Fence path routes through bridge + adapter without spurious Healthy

L0	L1	L2	L3	Status	Residual
adapter fence-slot / monotonic-guard tests in core/adapter/adapter_test.go	TestS7_Restart_FenceRouteBoundedWithoutCallback (Fence command reaches executor via real bridge; withheld callback does not produce Healthy)	N/A	L3 shape only, not executed	PARTIAL	Full timeout-expiry quantitative bound (watchdog fires at exact deadline + structured event) is adapter-owned proof (sketch §10.1 Path B — S7 may not reconfigure the fence path). Carried to the adapter package's own tests; not an L1 S7 proof

Claim 16 — Boundary guards (no unauthorized AssignmentInfo minting, no observation-side asks, no store minting, no hidden adapter ingress)

L0	L1	L2	L3	Status	Residual
TestObservation_NeverMintsAuthority_BoundaryGuard / TestDurableAuthority_StoreNeverMintsAuthority_BoundaryGuard / TestDurableAuthority_StoreWriteCallerAllowlist_BoundaryGuard / TestNonForgeability_* (AST-level guards in authority_test.go)	N/A (structural, not route)	N/A	N/A	PROVEN	none

4. Gap Classification

Per S8 assignment §3.B, every missing proof must be classified. Scanning the matrix above:

4.1 P14 internal blockers

None. Every P14 internal claim has at least one PROVEN row.

4.2 P14 S8 test/harness gap

Claim 15 (fence quantitative bound). S7 deliberately capped at L1 route coverage. The full fence timeout fires at exact deadline proof lives in the adapter package and depends on the uncommitted fence-watchdog branch landing committed. After that commits, a single-file adapter test replaces the current PARTIAL status. Not an S8 gate.

4.3 P15 product-surface gaps (carried forward)

See v3-phase-14-s8-p15-handoff.md for the full mapping (aligned to canonical v3-phase-15-product-plan.md §4 track numbering). Cross-reference summary:

• Frontend data path + subprocess heartbeat ingress → P15 T1 Frontend + Data Path
• CSI lifecycle → P15 T2 CSI / External Lifecycle Surface
• External control API → P15 T3 External Control API
• Security / auth → P15 T4 Security And Auth Posture
• Operator diagnostics for LastUnsupported / LastConvergenceStuck / ReloadSkips → P15 T5 Diagnostics
• Operator workflow (drain / supervised reassign / rebuild) → P15 T6 Operator Workflow
• V2/V3 migration → P15 T7 Migration
• Deployment / hardening → P15 T8 Deployment / Release Hardening
• End-to-end cluster validation → P15 Final Gate

4.4 14A verification follow-ups

See v3-phase-14a-checklist.md for the sidecar list. The S8 matrix above is the input to 14A's final targeted review.

4.5 Explicit non-goals (do NOT carry)

• multi-master / leader election
• performance / soak claims at P14 close
• long-running stability under production load

5. L2 Process Smoke Decision

Per S8 assignment §3.D, S8 must answer: are current sparrow smoke surfaces sufficient?

Answer: scoped YES — L2 proves durable-restart only; the full S4-S7 route remains L1-proven. (Round-2 architect correction.)

5.1 What L2 proves today

Required check	Covered by
real process starts	TestS7Process_RealSubprocessRestartSmoke spawns the real sparrow binary.
durable authority reloads	same test + component TestS7Process_BootstrapReloadRouteSmoke.
restart does not mint backward	subprocess asserts backwardMint: false in the pinned JSON pass-line; component TestS7Process_RestartDoesNotRegressAuthorityLine asserts reloaded Epoch equals pre-restart Epoch.
structured output can be consumed by tester	subprocess pass-line schema pinned in S7 sketch §8.4.

5.2 What L2 deliberately does NOT prove

Not covered at L2	Where it IS proven	Carry-forward
"heartbeat wire → ObservationHost → TopologyController" in a real process. Subprocess drives mints via StaticDirective; there is no heartbeat ingress in sparrow today.	L1: TestObservation_EndToEnd_SystemFedSnapshotReachesController, TestConvergenceRoute_ObservationFedFailover_ConfirmsAndClearsDesired, TestConvergenceRoute_RefreshEndpoint_ConfirmsAndClearsDesired, TestS7_ReloadedAuthority_ReanchorsAdapterViaVolumeBridge (all run real ObservationHost → controller → publisher → bridge → adapter, in-process).	P15 T1 Frontend + Data Path (subprocess heartbeat ingress is part of the frontend/transport contract). Handoff item #2 in v3-phase-14-s8-p15-handoff.md.
"assignment reaches adapter via bridge IN THE REAL BINARY". The subprocess binary does not construct a VolumeReplicaAdapter — only the in-process component test does.	L1: TestS7_ReloadedAuthority_ReanchorsAdapterViaVolumeBridge composes the full stack in-process and waits for adapter ModeHealthy.	P15 T1 Frontend + Data Path — the first real-binary adapter ↔ frontend route is a T1 deliverable.
Multi-process mixed-route scenarios (separate authority + observing processes).	N/A at P14	P15 Final Gate (Cluster Validation).

5.3 Decision

No new S8 smoke is added. The scope split above is honest:

• L2 subprocess smoke correctly proves durable restart truth.
• The full S4-S7 composed route is proven at L1 — including the RefreshEndpoint path added in this pass.
• Subprocess heartbeat ingress and full real-binary adapter route are genuinely P15 T1 surfaces; inventing a hidden --s8-route-smoke heartbeat fixture would be a P15 surface in P14 clothing.

This is a downgrade from the round-1 matrix claim ("L2 covers the S4-S7 internal route"), which was an overclaim.

6. Baseline Command Result (2026-04-20)

$ go test ./core/engine ./core/adapter ./core/authority ./cmd/sparrow -count=1
ok    github.com/seaweedfs/seaweed-block/core/engine      0.016s
ok    github.com/seaweedfs/seaweed-block/core/adapter     2.344s
ok    github.com/seaweedfs/seaweed-block/core/authority   4.569s
ok    github.com/seaweedfs/seaweed-block/cmd/sparrow      2.264s

$ go test ./... -count=1
(14 packages, all PASS; runtime/schema have no tests)

Evidence is reproducible by any reviewer on Windows / Linux from the committed tree on phase-14 at commit a6421f9 or later.

7. Residual Risks

#	Risk	Scope
1	Windows os.RemoveAll on the lock file during t.TempDir() teardown races with lock release. Does NOT affect correctness — only teardown.	S7 sketch §8.5 residual; logged as t.Logf in restart_route_test.go.
2	Claim 15 partial: L1 covers "fence command dispatched via real bridge + withheld callback does not produce Healthy"; full timeout-expiry bound is adapter-package-owned.	Closes when adapter fence-watchdog branch commits. Not an S8 blocker.
3	L2 subprocess smoke drives the directive via StaticDirective, not real heartbeat ingress to ObservationHost. The full heartbeat-path is L1-proven; subprocess widening requires a P15 external surface.	P15 Frontend / Cluster Validation.

No S8-blocking correctness residuals. One adapter-owned quantitative fence proof (Claim 15) remains as a P14-internal follow-up, landing when the fence-watchdog branch commits. No backward-authority-mint risk on restart. No unsupported-topology silent-idle path.

8. Closure Readiness

Against S8 acceptance gate (v3-phase-14-s8-assignment.md §6):

1. evidence matrix exists and is honest — yes (this document)
2. P14 internal blockers fixed or explicitly non-P14 — yes (§4.1 empty)
3. L0 + L1 tests pass — yes (baseline §6)
4. L2 process smoke present or absence classified — yes (§5; present)
5. L3 classified — yes (L3 shape only, not executed this slice; see v3-phase-14-s8-v2-scenario-classification.md)
6. 14A final pass has no open P14 safety blocker — pending (14A review on this matrix)
7. P15 handoff maps every product gap — yes (see v3-phase-14-s8-p15-handoff.md)
8. final closure statement does not claim production readiness — yes (see v3-phase-14-s8-closure.md, draft)

S8 is evidence-complete pending 14A sign-off. Awaiting architect + tester review.