snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

weed/admin/plugin/DESIGN.md

@@ -0,0 +1,205 @@
+# Admin Worker Plugin System (Design)
+
+This document describes the plugin system for admin-managed workers, implemented in parallel with the current maintenance/worker mechanism.
+
+## Scope
+
+- Add a new plugin protocol and runtime model for multi-language workers.
+- Keep all current admin + worker code paths untouched.
+- Use gRPC for all admin-worker communication.
+- Let workers describe job configuration UI declaratively via protobuf.
+- Persist all job type configuration under admin server data directory.
+- Support detector workers and executor workers per job type.
+- Add end-to-end workflow observability (activities, active jobs, progress).
+
+## New Contract
+
+- Proto file: `weed/pb/plugin.proto`
+- gRPC service: `PluginControlService.WorkerStream`
+- Connection model: worker-initiated long-lived bidirectional stream.
+
+Why this model:
+
+- Works for workers in any language with gRPC support.
+- Avoids admin dialing constraints in NAT/private networks.
+- Allows command/response, progress streaming, and heartbeat over one channel.
+
+## Core Runtime Components (Admin Side)
+
+1. `PluginRegistry`
+- Tracks connected workers and their per-job-type capabilities.
+- Maintains liveness via heartbeat timeout.
+
+2. `SchemaCoordinator`
+- For each job type, asks one capable worker for `JobTypeDescriptor`.
+- Caches descriptor version and refresh timestamp.
+
+3. `ConfigStore`
+- Persists descriptor + saved config values in `dataDir`.
+- Stores both:
+  - Admin-owned runtime config (detection interval, dispatch concurrency, retry).
+  - Worker-owned config values (plugin-specific detection/execution knobs).
+
+4. `DetectorScheduler`
+- Per job type, chooses one detector worker (`can_detect=true`).
+- Sends `RunDetectionRequest` with saved configs + cluster context.
+- Accepts `DetectionProposals`, dedupes by `dedupe_key`, inserts jobs.
+
+5. `JobDispatcher`
+- Chooses executor worker (`can_execute=true`) for each pending job.
+- Sends `ExecuteJobRequest`.
+- Consumes `JobProgressUpdate` and `JobCompleted`.
+
+6. `WorkflowMonitor`
+- Builds live counters and timeline from events:
+  - activities per job type,
+  - active jobs,
+  - per-job progress/state,
+  - worker health/load.
+
+## Worker Responsibilities
+
+1. Register capabilities on connect (`WorkerHello`).
+2. Expose job type descriptor (`ConfigSchemaResponse`) including UI schemas:
+- admin config form,
+- worker config form,
+- defaults.
+3. Run detection on demand (`RunDetectionRequest`) and return proposals.
+4. Execute assigned jobs (`ExecuteJobRequest`) and stream progress.
+5. Heartbeat regularly with slot usage and running work.
+6. Handle cancellation requests (`CancelRequest`) for in-flight detection/execution.
+
+## Declarative UI Model
+
+UI is fully derived from protobuf schema:
+
+- `ConfigForm`
+- `ConfigSection`
+- `ConfigField`
+- `ConfigOption`
+- `ValidationRule`
+- `ConfigValue` (typed scalar/list/map/object value container)
+
+Result:
+
+- Admin can render forms without hardcoded task structs.
+- New job types can ship UI schema from worker binary alone.
+- Worker language is irrelevant as long as it can emit protobuf messages.
+
+## Detection and Dispatch Flow
+
+1. Worker connects and registers capabilities.
+2. Admin requests descriptor per job type.
+3. Admin persists descriptor and editable config values.
+4. On detection interval (admin-owned setting):
+- Admin chooses one detector worker for that job type.
+- Sends `RunDetectionRequest` with:
+  - `AdminRuntimeConfig`,
+  - `admin_config_values`,
+  - `worker_config_values`,
+  - `ClusterContext` (master/filer/volume grpc locations, metadata).
+5. Detector emits `DetectionProposals` and `DetectionComplete`.
+6. Admin dedupes and enqueues jobs.
+7. Dispatcher assigns jobs to any eligible executor worker.
+8. Executor emits `JobProgressUpdate` and `JobCompleted`.
+9. Monitor updates workflow UI in near-real-time.
+
+## Persistence Layout (Admin Data Dir)
+
+Current layout under `<admin-data-dir>/plugin/`:
+
+- `job_types/<job_type>/descriptor.pb`
+- `job_types/<job_type>/descriptor.json`
+- `job_types/<job_type>/config.pb`
+- `job_types/<job_type>/config.json`
+- `job_types/<job_type>/runs.json`
+- `jobs/tracked_jobs.json`
+- `activities/activities.json`
+
+`config.pb` should use `PersistedJobTypeConfig` from `plugin.proto`.
+
+## Admin UI
+
+- Route: `/plugin`
+- Includes:
+  - runtime status,
+  - workers/capabilities,
+  - declarative descriptor-driven config forms,
+  - run history (last 10 success + last 10 errors),
+  - tracked jobs and activity stream,
+  - manual actions for schema refresh, detection, and detect+execute workflow.
+
+## Scheduling Policy (Initial)
+
+Detector selection per job type:
+- only workers with `can_detect=true`.
+- prefer healthy worker with highest free detection slots.
+- lease ends when heartbeat timeout or stream drop.
+
+Execution dispatch:
+- only workers with `can_execute=true`.
+- select by available execution slots and least active jobs.
+- retry on failure using admin runtime retry config.
+
+## Safety and Reliability
+
+- Idempotency: dedupe proposals by (`job_type`, `dedupe_key`).
+- Backpressure: enforce max jobs per detection run.
+- Timeouts: detection and execution timeout from admin runtime config.
+- Replay-safe persistence: write job state changes before emitting UI events.
+- Heartbeat-based failover for detector/executor reassignment.
+
+## Backward Compatibility
+
+- Legacy `worker.proto` runtime remains internally available where still referenced.
+- External CLI worker path is moved to plugin runtime behavior.
+- Runtime is enabled by default on admin worker gRPC server.
+
+## Incremental Rollout Plan
+
+Phase 1
+- Introduce protocol and storage models only.
+
+Phase 2
+- Build admin registry/scheduler/dispatcher behind feature flag.
+
+Phase 3
+- Add dedicated plugin UI pages and metrics.
+
+Phase 4
+- Port one existing job type (e.g. vacuum) as external worker plugin.
+
+Phase 4 status (starter)
+- Added `weed worker` command as an external `plugin.proto` worker process.
+- Initial handler implements `vacuum` job type with:
+  - declarative descriptor/config form response (`ConfigSchemaResponse`),
+  - detection via master topology scan (`RunDetectionRequest`),
+  - execution via existing vacuum task logic (`ExecuteJobRequest`),
+  - heartbeat/load reporting for monitor UI.
+- Legacy maintenance-worker-specific CLI path is removed.
+
+Run example:
+- Start admin: `weed admin -master=localhost:9333`
+- Start worker: `weed worker -admin=localhost:23646`
+- Optional explicit job type: `weed worker -admin=localhost:23646 -jobType=vacuum`
+- Optional stable worker ID persistence: `weed worker -admin=localhost:23646 -workingDir=/var/lib/seaweedfs-plugin`
+
+Phase 5
+- Migrate remaining job types and deprecate old mechanism.
+
+## Agreed Defaults
+
+1. Detector multiplicity
+- Exactly one detector worker per job type at a time. Admin selects one worker and runs detection there.
+
+2. Secret handling
+- No encryption at rest required for plugin config in this phase.
+
+3. Schema compatibility
+- No migration policy required yet; this is a new system.
+
+4. Execution ownership
+- Same worker is allowed to do both detection and execution.
+
+5. Retention
+- Keep last 10 successful runs and last 10 error runs per job type.

weed/admin/plugin/cluster_rate_limit.go

@@ -0,0 +1,150 @@
+package plugin
+
+import (
+	"strconv"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+// Job types whose ExecuteJobRequest needs per-worker rate-allocation
+// metadata injected. Keyed by the job-type string so plugin.go's
+// generic dispatch path stays job-agnostic.
+//
+// To add a new job type to the share-allocation pipeline: register an
+// entry here that knows how to read its admin-config field(s) and
+// produce the metadata keys/values the worker reads.
+
+// s3LifecycleClusterDeletesPerSecondKey, s3LifecycleClusterDeletesBurstKey,
+// s3LifecycleMetadataDeletesPerSecond, and s3LifecycleMetadataDeletesBurst
+// are the contract between admin and worker. The values must match the
+// constants exported from weed/worker/tasks/s3_lifecycle/cluster_rate_limit.go
+// — duplicated here as plain strings rather than imported so the admin
+// plugin package doesn't pull a dependency on the worker handler
+// package. A mismatch on either side would silently disable rate
+// limiting; tests pin the constants in both packages against the same
+// values.
+const (
+	s3LifecycleJobType                    = "s3_lifecycle"
+	s3LifecycleClusterDeletesPerSecondKey = "cluster_deletes_per_second"
+	s3LifecycleClusterDeletesBurstKey     = "cluster_deletes_burst"
+	s3LifecycleMetadataDeletesPerSecond   = "s3_lifecycle.deletes_per_second"
+	s3LifecycleMetadataDeletesBurst       = "s3_lifecycle.deletes_burst"
+)
+
+// decorateClusterContextForJob returns a new ClusterContext with any
+// per-job-type metadata the admin needs to inject before the
+// ExecuteJobRequest is sent. Returns the input cc unchanged when no
+// decoration applies.
+//
+// Today only s3_lifecycle decorates; the function exists so a future
+// job type's plumbing slots in alongside without touching
+// executeJobWithExecutor.
+//
+// maxJobsPerDetection is the job-type's AdminRuntimeConfig.MaxJobsPerDetection
+// — the cap on how many parallel instances of this job the scheduler will
+// dispatch per detection cycle. For singleton jobs (s3_lifecycle has
+// MaxJobsPerDetection=1) only one worker is ever active, so the cluster
+// budget must go to that worker undivided. For parallel-dispatch jobs the
+// budget divides across the actually-running set, not across every
+// capable worker. The divisor is min(executors, maxJobsPerDetection),
+// clamped to ≥1.
+func (r *Plugin) decorateClusterContextForJob(cc *plugin_pb.ClusterContext, jobType string, adminConfigValues map[string]*plugin_pb.ConfigValue, maxJobsPerDetection int) *plugin_pb.ClusterContext {
+	if cc == nil {
+		return cc
+	}
+	if jobType != s3LifecycleJobType {
+		return cc
+	}
+	rps := readNonNegativeInt(adminConfigValues, s3LifecycleClusterDeletesPerSecondKey)
+	burst := readNonNegativeInt(adminConfigValues, s3LifecycleClusterDeletesBurstKey)
+	if rps <= 0 {
+		// Operator hasn't configured a cluster cap; nothing to allocate.
+		// The worker treats missing metadata keys as "unlimited," which
+		// is the legacy behavior.
+		return cc
+	}
+	executors := r.registry.CountCapableExecutors(jobType)
+	if executors <= 0 {
+		// No executors means the job won't dispatch at all; metadata
+		// would be discarded. Log so the case is visible in ops.
+		glog.V(2).Infof("decorateClusterContext: %s rps=%d but no execute-capable workers; skipping allocation", jobType, rps)
+		return cc
+	}
+	// Divide by the number of *concurrently-running* workers, not the
+	// number of capable ones. A singleton job (maxJobs=1) gets the full
+	// budget on its single active worker.
+	activeWorkers := executors
+	if maxJobsPerDetection > 0 && maxJobsPerDetection < activeWorkers {
+		activeWorkers = maxJobsPerDetection
+	}
+	perWorkerRps := float64(rps) / float64(activeWorkers)
+	perWorkerBurst := 0
+	if burst > 0 {
+		perWorkerBurst = burst / activeWorkers
+		if perWorkerBurst < 1 {
+			perWorkerBurst = 1
+		}
+	}
+
+	// Clone so we don't mutate the shared base context. The metadata
+	// map is small; a fresh allocation per ExecuteJob is fine.
+	out := cloneClusterContext(cc)
+	if out.Metadata == nil {
+		out.Metadata = map[string]string{}
+	}
+	out.Metadata[s3LifecycleMetadataDeletesPerSecond] = strconv.FormatFloat(perWorkerRps, 'f', -1, 64)
+	if perWorkerBurst > 0 {
+		out.Metadata[s3LifecycleMetadataDeletesBurst] = strconv.Itoa(perWorkerBurst)
+	}
+	glog.V(3).Infof("decorateClusterContext: %s rps=%d burst=%d executors=%d maxJobs=%d active=%d -> per-worker rps=%g burst=%d",
+		jobType, rps, burst, executors, maxJobsPerDetection, activeWorkers, perWorkerRps, perWorkerBurst)
+	return out
+}
+
+// cloneClusterContext returns a shallow-but-safe copy: the top-level
+// fields are reassigned, and the Metadata map is duplicated so the
+// caller can mutate it without racing other consumers of the input.
+// Slices of strings (master/filer/volume/s3 addresses) are copied by
+// reference — those are treated as immutable elsewhere in the codebase.
+func cloneClusterContext(in *plugin_pb.ClusterContext) *plugin_pb.ClusterContext {
+	if in == nil {
+		return nil
+	}
+	out := &plugin_pb.ClusterContext{
+		MasterGrpcAddresses: in.MasterGrpcAddresses,
+		FilerAddresses:      in.FilerAddresses,
+		VolumeGrpcAddresses: in.VolumeGrpcAddresses,
+		S3GrpcAddresses:     in.S3GrpcAddresses,
+	}
+	if in.Metadata != nil {
+		out.Metadata = make(map[string]string, len(in.Metadata))
+		for k, v := range in.Metadata {
+			out.Metadata[k] = v
+		}
+	}
+	return out
+}
+
+// readNonNegativeInt reads an int64 admin config value, treating
+// missing fields and non-int kinds as 0. Negative values are clamped
+// to 0 since the AdminConfigForm declares MinValue=0 on both fields.
+func readNonNegativeInt(values map[string]*plugin_pb.ConfigValue, field string) int {
+	v, ok := values[field]
+	if !ok || v == nil {
+		return 0
+	}
+	switch k := v.Kind.(type) {
+	case *plugin_pb.ConfigValue_Int64Value:
+		if k.Int64Value < 0 {
+			return 0
+		}
+		return int(k.Int64Value)
+	case *plugin_pb.ConfigValue_DoubleValue:
+		if k.DoubleValue < 0 {
+			return 0
+		}
+		return int(k.DoubleValue)
+	}
+	return 0
+}

weed/admin/plugin/cluster_rate_limit_test.go

@@ -0,0 +1,177 @@
+package plugin
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// pluginWithExecutors returns a Plugin whose registry contains n
+// non-stale execute-capable workers for jobType. Helper for the
+// allocator tests. Bypasses UpsertFromHello so tests don't have to
+// build a full Hello message.
+func pluginWithExecutors(t *testing.T, jobType string, n int) *Plugin {
+	t.Helper()
+	reg := NewRegistry()
+	now := time.Now()
+	for i := 0; i < n; i++ {
+		id := "worker-" + string(rune('a'+i))
+		reg.sessions[id] = &WorkerSession{
+			WorkerID:    id,
+			LastSeenAt:  now,
+			ConnectedAt: now,
+			Capabilities: map[string]*plugin_pb.JobTypeCapability{
+				jobType: {CanExecute: true},
+			},
+		}
+	}
+	return &Plugin{registry: reg}
+}
+
+// adminConfig builds an int64 admin config map for the given fields.
+func adminConfig(pairs ...interface{}) map[string]*plugin_pb.ConfigValue {
+	if len(pairs)%2 != 0 {
+		panic("adminConfig expects key/value pairs")
+	}
+	out := map[string]*plugin_pb.ConfigValue{}
+	for i := 0; i < len(pairs); i += 2 {
+		key := pairs[i].(string)
+		switch v := pairs[i+1].(type) {
+		case int:
+			out[key] = &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: int64(v)}}
+		case int64:
+			out[key] = &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: v}}
+		case float64:
+			out[key] = &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_DoubleValue{DoubleValue: v}}
+		default:
+			panic("adminConfig: unsupported value type")
+		}
+	}
+	return out
+}
+
+func TestDecorateClusterContext_NonS3LifecycleIsPassThrough(t *testing.T) {
+	// Any job type other than s3_lifecycle gets the input cc back
+	// unchanged. Future allocators add their own branch; the default
+	// is pass-through.
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	in := &plugin_pb.ClusterContext{Metadata: map[string]string{"unrelated": "v"}}
+	out := r.decorateClusterContextForJob(in, "some_other_job", adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100), 1)
+	assert.Same(t, in, out, "non-allocator job type must return the same pointer")
+}
+
+func TestDecorateClusterContext_RpsZeroSkipsAllocation(t *testing.T) {
+	// rps=0 means "operator hasn't configured a cap"; the worker
+	// treats missing keys as unlimited. We must NOT inject any
+	// metadata (in particular, not "0") because that would force the
+	// worker into a no-throughput state on a misconfigured cluster.
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	in := &plugin_pb.ClusterContext{}
+	out := r.decorateClusterContextForJob(in, s3LifecycleJobType, adminConfig(s3LifecycleClusterDeletesPerSecondKey, 0), 1)
+	if out.Metadata != nil {
+		_, hasRps := out.Metadata[s3LifecycleMetadataDeletesPerSecond]
+		assert.False(t, hasRps, "rps=0 must not write a deletes_per_second key")
+	}
+}
+
+func TestDecorateClusterContext_NoExecutorsSkipsAllocation(t *testing.T) {
+	r := pluginWithExecutors(t, s3LifecycleJobType, 0)
+	in := &plugin_pb.ClusterContext{}
+	out := r.decorateClusterContextForJob(in, s3LifecycleJobType, adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100), 1)
+	if out.Metadata != nil {
+		_, hasRps := out.Metadata[s3LifecycleMetadataDeletesPerSecond]
+		assert.False(t, hasRps, "0 executors must not write share metadata (would divide by zero)")
+	}
+}
+
+func TestDecorateClusterContext_SingletonJobGetsFullBudget(t *testing.T) {
+	// s3_lifecycle has MaxJobsPerDetection=1: only ONE worker runs the
+	// job at a time. The cluster budget must go to that worker undivided
+	// — dividing by N capable executors would starve the active worker
+	// to 1/N of the configured rps. Pin the singleton behavior.
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	in := &plugin_pb.ClusterContext{}
+	out := r.decorateClusterContextForJob(in, s3LifecycleJobType,
+		adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100), 1)
+	require.NotNil(t, out.Metadata)
+	assert.Equal(t, "100", out.Metadata[s3LifecycleMetadataDeletesPerSecond], "singleton job: full budget to the single active worker")
+}
+
+func TestDecorateClusterContext_SharedEvenlyWhenParallelLimited(t *testing.T) {
+	// Hypothetical parallel-dispatch job type (maxJobs=4): budget
+	// divides across the running-set, which equals min(executors,
+	// maxJobs)=4. 100/4=25.
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	in := &plugin_pb.ClusterContext{}
+	out := r.decorateClusterContextForJob(in, s3LifecycleJobType,
+		adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100), 4)
+	require.NotNil(t, out.Metadata)
+	assert.Equal(t, "25", out.Metadata[s3LifecycleMetadataDeletesPerSecond], "maxJobs=4 across 4 executors = 25/worker")
+}
+
+func TestDecorateClusterContext_MaxJobsExceedsExecutors(t *testing.T) {
+	// maxJobs=10 but only 4 executors exist — the divisor is the
+	// smaller value (executors) since you can't run more jobs in
+	// parallel than there are workers to run them.
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	in := &plugin_pb.ClusterContext{}
+	out := r.decorateClusterContextForJob(in, s3LifecycleJobType,
+		adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100), 10)
+	require.NotNil(t, out.Metadata)
+	assert.Equal(t, "25", out.Metadata[s3LifecycleMetadataDeletesPerSecond])
+}
+
+func TestDecorateClusterContext_BurstSharedWhenParallel(t *testing.T) {
+	r := pluginWithExecutors(t, s3LifecycleJobType, 2)
+	in := &plugin_pb.ClusterContext{}
+	out := r.decorateClusterContextForJob(in, s3LifecycleJobType,
+		adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100, s3LifecycleClusterDeletesBurstKey, 20), 2)
+	require.NotNil(t, out.Metadata)
+	assert.Equal(t, "50", out.Metadata[s3LifecycleMetadataDeletesPerSecond])
+	assert.Equal(t, "10", out.Metadata[s3LifecycleMetadataDeletesBurst])
+}
+
+func TestDecorateClusterContext_BurstZeroOmitsKey(t *testing.T) {
+	// burst=0 means "let the worker default it." Don't write the key —
+	// the worker's parsePositiveInt would then take the unset path
+	// and compute 2 × rps automatically.
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	in := &plugin_pb.ClusterContext{}
+	out := r.decorateClusterContextForJob(in, s3LifecycleJobType,
+		adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100, s3LifecycleClusterDeletesBurstKey, 0), 4)
+	_, hasBurst := out.Metadata[s3LifecycleMetadataDeletesBurst]
+	assert.False(t, hasBurst, "burst=0 must NOT write the burst key (worker default kicks in)")
+}
+
+func TestDecorateClusterContext_BurstFloorIsOneWhenDividesBelowOne(t *testing.T) {
+	// burst=1 across 4 active workers would round to 0; clamp to 1 so
+	// the limiter doesn't become "single-token bucket that never refills."
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	in := &plugin_pb.ClusterContext{}
+	out := r.decorateClusterContextForJob(in, s3LifecycleJobType,
+		adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100, s3LifecycleClusterDeletesBurstKey, 1), 4)
+	assert.Equal(t, "1", out.Metadata[s3LifecycleMetadataDeletesBurst])
+}
+
+func TestDecorateClusterContext_DoesNotMutateInput(t *testing.T) {
+	// The same base ClusterContext is shared across many parallel
+	// ExecuteJob calls. The decorator must produce a fresh map so it
+	// can't race / leak per-job metadata into the base.
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	baseMeta := map[string]string{"existing": "value"}
+	in := &plugin_pb.ClusterContext{Metadata: baseMeta}
+	_ = r.decorateClusterContextForJob(in, s3LifecycleJobType,
+		adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100), 1)
+	_, leaked := baseMeta[s3LifecycleMetadataDeletesPerSecond]
+	assert.False(t, leaked, "decorator must not mutate the input metadata map")
+	assert.Equal(t, "value", baseMeta["existing"])
+}
+
+func TestDecorateClusterContext_NilInputPassesThrough(t *testing.T) {
+	r := pluginWithExecutors(t, s3LifecycleJobType, 4)
+	out := r.decorateClusterContextForJob(nil, s3LifecycleJobType, adminConfig(s3LifecycleClusterDeletesPerSecondKey, 100), 1)
+	assert.Nil(t, out)
+}

weed/admin/plugin/config_store.go

@@ -0,0 +1,810 @@
+package plugin
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"os"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+	"google.golang.org/protobuf/encoding/protojson"
+	"google.golang.org/protobuf/proto"
+)
+
+const (
+	pluginDirName           = "plugin"
+	jobTypesDirName         = "job_types"
+	jobsDirName             = "jobs"
+	jobDetailsDirName       = "job_details"
+	activitiesDirName       = "activities"
+	descriptorPBFileName    = "descriptor.pb"
+	descriptorJSONFileName  = "descriptor.json"
+	configPBFileName        = "config.pb"
+	configJSONFileName      = "config.json"
+	runsJSONFileName        = "runs.json"
+	trackedJobsJSONFileName = "tracked_jobs.json"
+	activitiesJSONFileName  = "activities.json"
+	defaultDirPerm          = 0o755
+	defaultFilePerm         = 0o644
+)
+
+// validJobTypePattern is the canonical pattern for safe job type names.
+// Only letters, digits, underscore, dash, and dot are allowed, which prevents
+// path traversal because '/', '\\', and whitespace are rejected.
+var validJobTypePattern = regexp.MustCompile(`^[A-Za-z0-9_.-]+$`)
+
+// ConfigStore persists plugin configuration and bounded run history.
+// If admin data dir is empty, it transparently falls back to in-memory mode.
+type ConfigStore struct {
+	configured bool
+	baseDir    string
+
+	mu sync.RWMutex
+
+	memDescriptors map[string]*plugin_pb.JobTypeDescriptor
+	memConfigs     map[string]*plugin_pb.PersistedJobTypeConfig
+	memRunHistory  map[string]*JobTypeRunHistory
+	memTrackedJobs []TrackedJob
+	memActivities  []JobActivity
+	memJobDetails  map[string]TrackedJob
+}
+
+func NewConfigStore(adminDataDir string) (*ConfigStore, error) {
+	store := &ConfigStore{
+		configured:     adminDataDir != "",
+		memDescriptors: make(map[string]*plugin_pb.JobTypeDescriptor),
+		memConfigs:     make(map[string]*plugin_pb.PersistedJobTypeConfig),
+		memRunHistory:  make(map[string]*JobTypeRunHistory),
+		memJobDetails:  make(map[string]TrackedJob),
+	}
+
+	if adminDataDir == "" {
+		return store, nil
+	}
+
+	store.baseDir = filepath.Join(adminDataDir, pluginDirName)
+	if err := os.MkdirAll(filepath.Join(store.baseDir, jobTypesDirName), defaultDirPerm); err != nil {
+		return nil, fmt.Errorf("create plugin job_types dir: %w", err)
+	}
+	if err := os.MkdirAll(filepath.Join(store.baseDir, jobsDirName), defaultDirPerm); err != nil {
+		return nil, fmt.Errorf("create plugin jobs dir: %w", err)
+	}
+	if err := os.MkdirAll(filepath.Join(store.baseDir, jobsDirName, jobDetailsDirName), defaultDirPerm); err != nil {
+		return nil, fmt.Errorf("create plugin job_details dir: %w", err)
+	}
+	if err := os.MkdirAll(filepath.Join(store.baseDir, activitiesDirName), defaultDirPerm); err != nil {
+		return nil, fmt.Errorf("create plugin activities dir: %w", err)
+	}
+
+	return store, nil
+}
+
+func (s *ConfigStore) IsConfigured() bool {
+	return s.configured
+}
+
+func (s *ConfigStore) BaseDir() string {
+	return s.baseDir
+}
+
+func (s *ConfigStore) SaveDescriptor(jobType string, descriptor *plugin_pb.JobTypeDescriptor) error {
+	if descriptor == nil {
+		return fmt.Errorf("descriptor is nil")
+	}
+	if _, err := sanitizeJobType(jobType); err != nil {
+		return err
+	}
+
+	clone := proto.Clone(descriptor).(*plugin_pb.JobTypeDescriptor)
+	if clone.JobType == "" {
+		clone.JobType = jobType
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if !s.configured {
+		s.memDescriptors[jobType] = clone
+		return nil
+	}
+
+	jobTypeDir, err := s.ensureJobTypeDir(jobType)
+	if err != nil {
+		return err
+	}
+
+	pbPath := filepath.Join(jobTypeDir, descriptorPBFileName)
+	jsonPath := filepath.Join(jobTypeDir, descriptorJSONFileName)
+
+	if err := writeProtoFiles(clone, pbPath, jsonPath); err != nil {
+		return fmt.Errorf("save descriptor for %s: %w", jobType, err)
+	}
+
+	return nil
+}
+
+func (s *ConfigStore) LoadDescriptor(jobType string) (*plugin_pb.JobTypeDescriptor, error) {
+	if _, err := sanitizeJobType(jobType); err != nil {
+		return nil, err
+	}
+
+	s.mu.RLock()
+	if !s.configured {
+		d := s.memDescriptors[jobType]
+		s.mu.RUnlock()
+		if d == nil {
+			return nil, nil
+		}
+		return proto.Clone(d).(*plugin_pb.JobTypeDescriptor), nil
+	}
+	s.mu.RUnlock()
+
+	pbPath := filepath.Join(s.baseDir, jobTypesDirName, jobType, descriptorPBFileName)
+	data, err := os.ReadFile(pbPath)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, nil
+		}
+		return nil, fmt.Errorf("read descriptor for %s: %w", jobType, err)
+	}
+
+	var descriptor plugin_pb.JobTypeDescriptor
+	if err := proto.Unmarshal(data, &descriptor); err != nil {
+		return nil, fmt.Errorf("unmarshal descriptor for %s: %w", jobType, err)
+	}
+	return &descriptor, nil
+}
+
+func (s *ConfigStore) SaveJobTypeConfig(config *plugin_pb.PersistedJobTypeConfig) error {
+	if config == nil {
+		return fmt.Errorf("job type config is nil")
+	}
+	if config.JobType == "" {
+		return fmt.Errorf("job type config has empty job_type")
+	}
+	sanitizedJobType, err := sanitizeJobType(config.JobType)
+	if err != nil {
+		return err
+	}
+	// Use the sanitized job type going forward to ensure it is safe for filesystem paths.
+	config.JobType = sanitizedJobType
+
+	clone := proto.Clone(config).(*plugin_pb.PersistedJobTypeConfig)
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if !s.configured {
+		s.memConfigs[config.JobType] = clone
+		return nil
+	}
+
+	jobTypeDir, err := s.ensureJobTypeDir(config.JobType)
+	if err != nil {
+		return err
+	}
+
+	pbPath := filepath.Join(jobTypeDir, configPBFileName)
+	jsonPath := filepath.Join(jobTypeDir, configJSONFileName)
+
+	if err := writeProtoFiles(clone, pbPath, jsonPath); err != nil {
+		return fmt.Errorf("save job type config for %s: %w", config.JobType, err)
+	}
+
+	return nil
+}
+
+// SaveJobTypeConfigIfNotExists atomically checks whether a config for the
+// given job type already exists and only persists config when none is found.
+// Returns true if the config was saved, false if a config already existed.
+func (s *ConfigStore) SaveJobTypeConfigIfNotExists(config *plugin_pb.PersistedJobTypeConfig) (bool, error) {
+	if config == nil {
+		return false, fmt.Errorf("job type config is nil")
+	}
+	if config.JobType == "" {
+		return false, fmt.Errorf("job type config has empty job_type")
+	}
+	sanitizedJobType, err := sanitizeJobType(config.JobType)
+	if err != nil {
+		return false, err
+	}
+	config.JobType = sanitizedJobType
+
+	clone := proto.Clone(config).(*plugin_pb.PersistedJobTypeConfig)
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if !s.configured {
+		if _, exists := s.memConfigs[config.JobType]; exists {
+			return false, nil
+		}
+		s.memConfigs[config.JobType] = clone
+		return true, nil
+	}
+
+	pbPath := filepath.Join(s.baseDir, jobTypesDirName, config.JobType, configPBFileName)
+	if _, statErr := os.Stat(pbPath); statErr == nil {
+		return false, nil
+	}
+
+	jobTypeDir, err := s.ensureJobTypeDir(config.JobType)
+	if err != nil {
+		return false, err
+	}
+
+	jsonPath := filepath.Join(jobTypeDir, configJSONFileName)
+	if err := writeProtoFiles(clone, filepath.Join(jobTypeDir, configPBFileName), jsonPath); err != nil {
+		return false, fmt.Errorf("save job type config for %s: %w", config.JobType, err)
+	}
+
+	return true, nil
+}
+
+func (s *ConfigStore) LoadJobTypeConfig(jobType string) (*plugin_pb.PersistedJobTypeConfig, error) {
+	if _, err := sanitizeJobType(jobType); err != nil {
+		return nil, err
+	}
+
+	s.mu.RLock()
+	if !s.configured {
+		cfg := s.memConfigs[jobType]
+		s.mu.RUnlock()
+		if cfg == nil {
+			return nil, nil
+		}
+		return proto.Clone(cfg).(*plugin_pb.PersistedJobTypeConfig), nil
+	}
+	s.mu.RUnlock()
+
+	pbPath := filepath.Join(s.baseDir, jobTypesDirName, jobType, configPBFileName)
+	data, err := os.ReadFile(pbPath)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, nil
+		}
+		return nil, fmt.Errorf("read job type config for %s: %w", jobType, err)
+	}
+
+	var config plugin_pb.PersistedJobTypeConfig
+	if err := proto.Unmarshal(data, &config); err != nil {
+		return nil, fmt.Errorf("unmarshal job type config for %s: %w", jobType, err)
+	}
+
+	return &config, nil
+}
+
+func (s *ConfigStore) AppendRunRecord(jobType string, record *JobRunRecord) error {
+	if record == nil {
+		return fmt.Errorf("run record is nil")
+	}
+	if _, err := sanitizeJobType(jobType); err != nil {
+		return err
+	}
+
+	safeRecord := *record
+	if safeRecord.JobType == "" {
+		safeRecord.JobType = jobType
+	}
+	if safeRecord.CompletedAt == nil || safeRecord.CompletedAt.IsZero() {
+		safeRecord.CompletedAt = timeToPtr(time.Now().UTC())
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	history, err := s.loadRunHistoryLocked(jobType)
+	if err != nil {
+		return err
+	}
+
+	if safeRecord.Outcome == RunOutcomeSuccess {
+		history.SuccessfulRuns = append(history.SuccessfulRuns, safeRecord)
+	} else {
+		safeRecord.Outcome = RunOutcomeError
+		history.ErrorRuns = append(history.ErrorRuns, safeRecord)
+	}
+
+	history.SuccessfulRuns = trimRuns(history.SuccessfulRuns, MaxSuccessfulRunHistory)
+	history.ErrorRuns = trimRuns(history.ErrorRuns, MaxErrorRunHistory)
+	history.LastUpdatedTime = timeToPtr(time.Now().UTC())
+
+	return s.saveRunHistoryLocked(jobType, history)
+}
+
+func (s *ConfigStore) LoadRunHistory(jobType string) (*JobTypeRunHistory, error) {
+	if _, err := sanitizeJobType(jobType); err != nil {
+		return nil, err
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	history, err := s.loadRunHistoryLocked(jobType)
+	if err != nil {
+		return nil, err
+	}
+	return cloneRunHistory(history), nil
+}
+
+func (s *ConfigStore) SaveTrackedJobs(jobs []TrackedJob) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	clone := cloneTrackedJobs(jobs)
+
+	if !s.configured {
+		s.memTrackedJobs = clone
+		return nil
+	}
+
+	encoded, err := json.MarshalIndent(clone, "", "  ")
+	if err != nil {
+		return fmt.Errorf("encode tracked jobs: %w", err)
+	}
+
+	path := filepath.Join(s.baseDir, jobsDirName, trackedJobsJSONFileName)
+	if err := atomicWriteFile(path, encoded, defaultFilePerm); err != nil {
+		return fmt.Errorf("write tracked jobs: %w", err)
+	}
+	return nil
+}
+
+func (s *ConfigStore) LoadTrackedJobs() ([]TrackedJob, error) {
+	s.mu.RLock()
+	if !s.configured {
+		out := cloneTrackedJobs(s.memTrackedJobs)
+		s.mu.RUnlock()
+		return out, nil
+	}
+	s.mu.RUnlock()
+
+	path := filepath.Join(s.baseDir, jobsDirName, trackedJobsJSONFileName)
+	data, err := os.ReadFile(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, nil
+		}
+		return nil, fmt.Errorf("read tracked jobs: %w", err)
+	}
+
+	var jobs []TrackedJob
+	if err := json.Unmarshal(data, &jobs); err != nil {
+		return nil, fmt.Errorf("parse tracked jobs: %w", err)
+	}
+	return cloneTrackedJobs(jobs), nil
+}
+
+func (s *ConfigStore) SaveJobDetail(job TrackedJob) error {
+	jobID, err := sanitizeJobID(job.JobID)
+	if err != nil {
+		return err
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	clone := cloneTrackedJob(job)
+	clone.JobID = jobID
+
+	if !s.configured {
+		s.memJobDetails[jobID] = clone
+		return nil
+	}
+
+	encoded, err := json.MarshalIndent(clone, "", "  ")
+	if err != nil {
+		return fmt.Errorf("encode job detail: %w", err)
+	}
+
+	path := filepath.Join(s.baseDir, jobsDirName, jobDetailsDirName, jobDetailFileName(jobID))
+	if err := atomicWriteFile(path, encoded, defaultFilePerm); err != nil {
+		return fmt.Errorf("write job detail: %w", err)
+	}
+
+	return nil
+}
+
+func (s *ConfigStore) LoadJobDetail(jobID string) (*TrackedJob, error) {
+	jobID, err := sanitizeJobID(jobID)
+	if err != nil {
+		return nil, err
+	}
+
+	s.mu.RLock()
+	if !s.configured {
+		job, ok := s.memJobDetails[jobID]
+		s.mu.RUnlock()
+		if !ok {
+			return nil, nil
+		}
+		clone := cloneTrackedJob(job)
+		return &clone, nil
+	}
+	s.mu.RUnlock()
+
+	path := filepath.Join(s.baseDir, jobsDirName, jobDetailsDirName, jobDetailFileName(jobID))
+	data, err := os.ReadFile(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, nil
+		}
+		return nil, fmt.Errorf("read job detail: %w", err)
+	}
+
+	var job TrackedJob
+	if err := json.Unmarshal(data, &job); err != nil {
+		return nil, fmt.Errorf("parse job detail: %w", err)
+	}
+	clone := cloneTrackedJob(job)
+	return &clone, nil
+}
+
+// DeleteJobDetail removes the persisted detail snapshot for a job.
+// It is a no-op when the file does not exist.
+func (s *ConfigStore) DeleteJobDetail(jobID string) error {
+	jobID, err := sanitizeJobID(jobID)
+	if err != nil {
+		return err
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if !s.configured {
+		delete(s.memJobDetails, jobID)
+		return nil
+	}
+
+	path := filepath.Join(s.baseDir, jobsDirName, jobDetailsDirName, jobDetailFileName(jobID))
+	err = os.Remove(path)
+	if err != nil && !os.IsNotExist(err) {
+		return fmt.Errorf("delete job detail: %w", err)
+	}
+	return nil
+}
+
+func (s *ConfigStore) SaveActivities(activities []JobActivity) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	clone := cloneActivities(activities)
+
+	if !s.configured {
+		s.memActivities = clone
+		return nil
+	}
+
+	encoded, err := json.MarshalIndent(clone, "", "  ")
+	if err != nil {
+		return fmt.Errorf("encode activities: %w", err)
+	}
+
+	path := filepath.Join(s.baseDir, activitiesDirName, activitiesJSONFileName)
+	if err := atomicWriteFile(path, encoded, defaultFilePerm); err != nil {
+		return fmt.Errorf("write activities: %w", err)
+	}
+	return nil
+}
+
+func (s *ConfigStore) LoadActivities() ([]JobActivity, error) {
+	s.mu.RLock()
+	if !s.configured {
+		out := cloneActivities(s.memActivities)
+		s.mu.RUnlock()
+		return out, nil
+	}
+	s.mu.RUnlock()
+
+	path := filepath.Join(s.baseDir, activitiesDirName, activitiesJSONFileName)
+	data, err := os.ReadFile(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, nil
+		}
+		return nil, fmt.Errorf("read activities: %w", err)
+	}
+
+	var activities []JobActivity
+	if err := json.Unmarshal(data, &activities); err != nil {
+		return nil, fmt.Errorf("parse activities: %w", err)
+	}
+	return cloneActivities(activities), nil
+}
+
+func (s *ConfigStore) ListJobTypes() ([]string, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	jobTypeSet := make(map[string]struct{})
+
+	if !s.configured {
+		for jobType := range s.memDescriptors {
+			jobTypeSet[jobType] = struct{}{}
+		}
+		for jobType := range s.memConfigs {
+			jobTypeSet[jobType] = struct{}{}
+		}
+		for jobType := range s.memRunHistory {
+			jobTypeSet[jobType] = struct{}{}
+		}
+	} else {
+		jobTypesPath := filepath.Join(s.baseDir, jobTypesDirName)
+		entries, err := os.ReadDir(jobTypesPath)
+		if err != nil {
+			if os.IsNotExist(err) {
+				return []string{}, nil
+			}
+			return nil, fmt.Errorf("list job types: %w", err)
+		}
+		for _, entry := range entries {
+			if !entry.IsDir() {
+				continue
+			}
+			jobType := strings.TrimSpace(entry.Name())
+			if _, err := sanitizeJobType(jobType); err != nil {
+				continue
+			}
+			jobTypeSet[jobType] = struct{}{}
+		}
+	}
+
+	jobTypes := make([]string, 0, len(jobTypeSet))
+	for jobType := range jobTypeSet {
+		jobTypes = append(jobTypes, jobType)
+	}
+	sort.Strings(jobTypes)
+	return jobTypes, nil
+}
+
+func (s *ConfigStore) loadRunHistoryLocked(jobType string) (*JobTypeRunHistory, error) {
+	if !s.configured {
+		history, ok := s.memRunHistory[jobType]
+		if !ok {
+			history = &JobTypeRunHistory{JobType: jobType}
+			s.memRunHistory[jobType] = history
+		}
+		return cloneRunHistory(history), nil
+	}
+
+	runsPath := filepath.Join(s.baseDir, jobTypesDirName, jobType, runsJSONFileName)
+	data, err := os.ReadFile(runsPath)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return &JobTypeRunHistory{JobType: jobType}, nil
+		}
+		return nil, fmt.Errorf("read run history for %s: %w", jobType, err)
+	}
+
+	var history JobTypeRunHistory
+	if err := json.Unmarshal(data, &history); err != nil {
+		return nil, fmt.Errorf("parse run history for %s: %w", jobType, err)
+	}
+	if history.JobType == "" {
+		history.JobType = jobType
+	}
+	return &history, nil
+}
+
+func (s *ConfigStore) saveRunHistoryLocked(jobType string, history *JobTypeRunHistory) error {
+	if !s.configured {
+		s.memRunHistory[jobType] = cloneRunHistory(history)
+		return nil
+	}
+
+	jobTypeDir, err := s.ensureJobTypeDir(jobType)
+	if err != nil {
+		return err
+	}
+
+	encoded, err := json.MarshalIndent(history, "", "  ")
+	if err != nil {
+		return fmt.Errorf("encode run history for %s: %w", jobType, err)
+	}
+
+	runsPath := filepath.Join(jobTypeDir, runsJSONFileName)
+	if err := atomicWriteFile(runsPath, encoded, defaultFilePerm); err != nil {
+		return fmt.Errorf("write run history for %s: %w", jobType, err)
+	}
+	return nil
+}
+
+func (s *ConfigStore) ensureJobTypeDir(jobType string) (string, error) {
+	if !s.configured {
+		return "", nil
+	}
+	jobTypeDir := filepath.Join(s.baseDir, jobTypesDirName, jobType)
+	if err := os.MkdirAll(jobTypeDir, defaultDirPerm); err != nil {
+		return "", fmt.Errorf("create job type dir for %s: %w", jobType, err)
+	}
+	return jobTypeDir, nil
+}
+
+func sanitizeJobType(jobType string) (string, error) {
+	jobType = strings.TrimSpace(jobType)
+	if jobType == "" {
+		return "", fmt.Errorf("job type is empty")
+	}
+	// Enforce a strict, path-safe pattern for job types: only letters, digits, underscore, dash and dot.
+	// This prevents path traversal because '/', '\\' and whitespace are rejected.
+	if !validJobTypePattern.MatchString(jobType) {
+		return "", fmt.Errorf("invalid job type %q: must match %s", jobType, validJobTypePattern.String())
+	}
+	return jobType, nil
+}
+
+// validJobIDPattern allows letters, digits, dash, underscore, and dot.
+// url.PathEscape in jobDetailFileName provides a second layer of defense.
+var validJobIDPattern = regexp.MustCompile(`^[A-Za-z0-9_.-]+$`)
+
+func sanitizeJobID(jobID string) (string, error) {
+	jobID = strings.TrimSpace(jobID)
+	if jobID == "" {
+		return "", fmt.Errorf("job id is empty")
+	}
+	if !validJobIDPattern.MatchString(jobID) {
+		return "", fmt.Errorf("invalid job id %q: must match %s", jobID, validJobIDPattern.String())
+	}
+	return jobID, nil
+}
+
+func jobDetailFileName(jobID string) string {
+	return url.PathEscape(jobID) + ".json"
+}
+
+func trimRuns(runs []JobRunRecord, maxKeep int) []JobRunRecord {
+	if len(runs) == 0 {
+		return runs
+	}
+	sort.Slice(runs, func(i, j int) bool {
+		ti := time.Time{}
+		if runs[i].CompletedAt != nil {
+			ti = *runs[i].CompletedAt
+		}
+		tj := time.Time{}
+		if runs[j].CompletedAt != nil {
+			tj = *runs[j].CompletedAt
+		}
+		return ti.After(tj)
+	})
+	if len(runs) > maxKeep {
+		runs = runs[:maxKeep]
+	}
+	return runs
+}
+
+func cloneRunHistory(in *JobTypeRunHistory) *JobTypeRunHistory {
+	if in == nil {
+		return nil
+	}
+	out := *in
+	if in.SuccessfulRuns != nil {
+		out.SuccessfulRuns = append([]JobRunRecord(nil), in.SuccessfulRuns...)
+	}
+	if in.ErrorRuns != nil {
+		out.ErrorRuns = append([]JobRunRecord(nil), in.ErrorRuns...)
+	}
+	return &out
+}
+
+func cloneTrackedJobs(in []TrackedJob) []TrackedJob {
+	if len(in) == 0 {
+		return nil
+	}
+
+	out := make([]TrackedJob, len(in))
+	for i := range in {
+		out[i] = cloneTrackedJob(in[i])
+	}
+	return out
+}
+
+func cloneTrackedJob(in TrackedJob) TrackedJob {
+	out := in
+	if in.Parameters != nil {
+		out.Parameters = make(map[string]interface{}, len(in.Parameters))
+		for key, value := range in.Parameters {
+			out.Parameters[key] = deepCopyGenericValue(value)
+		}
+	}
+	if in.Labels != nil {
+		out.Labels = make(map[string]string, len(in.Labels))
+		for key, value := range in.Labels {
+			out.Labels[key] = value
+		}
+	}
+	if in.ResultOutputValues != nil {
+		out.ResultOutputValues = make(map[string]interface{}, len(in.ResultOutputValues))
+		for key, value := range in.ResultOutputValues {
+			out.ResultOutputValues[key] = deepCopyGenericValue(value)
+		}
+	}
+	return out
+}
+
+func deepCopyGenericValue(val interface{}) interface{} {
+	switch v := val.(type) {
+	case map[string]interface{}:
+		res := make(map[string]interface{}, len(v))
+		for k, val := range v {
+			res[k] = deepCopyGenericValue(val)
+		}
+		return res
+	case []interface{}:
+		res := make([]interface{}, len(v))
+		for i, val := range v {
+			res[i] = deepCopyGenericValue(val)
+		}
+		return res
+	default:
+		return v
+	}
+}
+
+func cloneActivities(in []JobActivity) []JobActivity {
+	if len(in) == 0 {
+		return nil
+	}
+
+	out := make([]JobActivity, len(in))
+	for i := range in {
+		out[i] = in[i]
+		if in[i].Details != nil {
+			out[i].Details = make(map[string]interface{}, len(in[i].Details))
+			for key, value := range in[i].Details {
+				out[i].Details[key] = deepCopyGenericValue(value)
+			}
+		}
+	}
+	return out
+}
+
+// writeProtoFiles writes message to both a binary protobuf file (pbPath) and a
+// human-readable JSON file (jsonPath) using atomicWriteFile for each.
+// The .pb file is the authoritative source of truth: all reads use proto.Unmarshal
+// on the .pb file. The .json file is for human inspection only, so a partial
+// failure where .pb succeeds but .json fails leaves the store in a consistent state.
+func writeProtoFiles(message proto.Message, pbPath string, jsonPath string) error {
+	pbData, err := proto.Marshal(message)
+	if err != nil {
+		return fmt.Errorf("marshal protobuf: %w", err)
+	}
+	if err := atomicWriteFile(pbPath, pbData, defaultFilePerm); err != nil {
+		return fmt.Errorf("write protobuf file: %w", err)
+	}
+
+	jsonData, err := protojson.MarshalOptions{
+		Multiline:       true,
+		Indent:          "  ",
+		EmitUnpopulated: true,
+	}.Marshal(message)
+	if err != nil {
+		return fmt.Errorf("marshal json: %w", err)
+	}
+	if err := atomicWriteFile(jsonPath, jsonData, defaultFilePerm); err != nil {
+		return fmt.Errorf("write json file: %w", err)
+	}
+
+	return nil
+}
+func atomicWriteFile(filename string, data []byte, perm os.FileMode) error {
+	dir := filepath.Dir(filename)
+	if err := os.MkdirAll(dir, defaultDirPerm); err != nil {
+		return fmt.Errorf("create directory %s: %w", dir, err)
+	}
+	tmpFile := filename + ".tmp"
+	if err := os.WriteFile(tmpFile, data, perm); err != nil {
+		return err
+	}
+	if err := os.Rename(tmpFile, filename); err != nil {
+		_ = os.Remove(tmpFile)
+		return err
+	}
+	return nil
+}

weed/admin/plugin/config_store_test.go

@@ -0,0 +1,379 @@
+package plugin
+
+import (
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+func TestConfigStoreDescriptorRoundTrip(t *testing.T) {
+	t.Parallel()
+
+	tempDir := t.TempDir()
+	store, err := NewConfigStore(tempDir)
+	if err != nil {
+		t.Fatalf("NewConfigStore: %v", err)
+	}
+
+	descriptor := &plugin_pb.JobTypeDescriptor{
+		JobType:           "vacuum",
+		DisplayName:       "Vacuum",
+		Description:       "Vacuum volumes",
+		DescriptorVersion: 1,
+	}
+
+	if err := store.SaveDescriptor("vacuum", descriptor); err != nil {
+		t.Fatalf("SaveDescriptor: %v", err)
+	}
+
+	got, err := store.LoadDescriptor("vacuum")
+	if err != nil {
+		t.Fatalf("LoadDescriptor: %v", err)
+	}
+	if got == nil {
+		t.Fatalf("LoadDescriptor: nil descriptor")
+	}
+	if got.DisplayName != descriptor.DisplayName {
+		t.Fatalf("unexpected display name: got %q want %q", got.DisplayName, descriptor.DisplayName)
+	}
+
+}
+
+func TestConfigStoreRunHistoryRetention(t *testing.T) {
+	t.Parallel()
+
+	store, err := NewConfigStore(t.TempDir())
+	if err != nil {
+		t.Fatalf("NewConfigStore: %v", err)
+	}
+
+	base := time.Now().UTC().Add(-24 * time.Hour)
+	for i := 0; i < 15; i++ {
+		err := store.AppendRunRecord("balance", &JobRunRecord{
+			RunID:       "s" + time.Duration(i).String(),
+			JobID:       "job-success",
+			JobType:     "balance",
+			WorkerID:    "worker-a",
+			Outcome:     RunOutcomeSuccess,
+			CompletedAt: timeToPtr(base.Add(time.Duration(i) * time.Minute)),
+		})
+		if err != nil {
+			t.Fatalf("AppendRunRecord success[%d]: %v", i, err)
+		}
+	}
+
+	for i := 0; i < 12; i++ {
+		err := store.AppendRunRecord("balance", &JobRunRecord{
+			RunID:       "e" + time.Duration(i).String(),
+			JobID:       "job-error",
+			JobType:     "balance",
+			WorkerID:    "worker-b",
+			Outcome:     RunOutcomeError,
+			CompletedAt: timeToPtr(base.Add(time.Duration(i) * time.Minute)),
+		})
+		if err != nil {
+			t.Fatalf("AppendRunRecord error[%d]: %v", i, err)
+		}
+	}
+
+	history, err := store.LoadRunHistory("balance")
+	if err != nil {
+		t.Fatalf("LoadRunHistory: %v", err)
+	}
+	if len(history.SuccessfulRuns) != MaxSuccessfulRunHistory {
+		t.Fatalf("successful retention mismatch: got %d want %d", len(history.SuccessfulRuns), MaxSuccessfulRunHistory)
+	}
+	if len(history.ErrorRuns) != MaxErrorRunHistory {
+		t.Fatalf("error retention mismatch: got %d want %d", len(history.ErrorRuns), MaxErrorRunHistory)
+	}
+
+	for i := 1; i < len(history.SuccessfulRuns); i++ {
+		t1 := time.Time{}
+		if history.SuccessfulRuns[i-1].CompletedAt != nil {
+			t1 = *history.SuccessfulRuns[i-1].CompletedAt
+		}
+		t2 := time.Time{}
+		if history.SuccessfulRuns[i].CompletedAt != nil {
+			t2 = *history.SuccessfulRuns[i].CompletedAt
+		}
+		if t1.Before(t2) {
+			t.Fatalf("successful run order not descending at %d", i)
+		}
+	}
+	for i := 1; i < len(history.ErrorRuns); i++ {
+		t1 := time.Time{}
+		if history.ErrorRuns[i-1].CompletedAt != nil {
+			t1 = *history.ErrorRuns[i-1].CompletedAt
+		}
+		t2 := time.Time{}
+		if history.ErrorRuns[i].CompletedAt != nil {
+			t2 = *history.ErrorRuns[i].CompletedAt
+		}
+		if t1.Before(t2) {
+			t.Fatalf("error run order not descending at %d", i)
+		}
+	}
+}
+
+func TestConfigStoreListJobTypes(t *testing.T) {
+	t.Parallel()
+
+	store, err := NewConfigStore("")
+	if err != nil {
+		t.Fatalf("NewConfigStore: %v", err)
+	}
+
+	if err := store.SaveDescriptor("vacuum", &plugin_pb.JobTypeDescriptor{JobType: "vacuum"}); err != nil {
+		t.Fatalf("SaveDescriptor: %v", err)
+	}
+	if err := store.SaveJobTypeConfig(&plugin_pb.PersistedJobTypeConfig{
+		JobType:      "balance",
+		AdminRuntime: &plugin_pb.AdminRuntimeConfig{Enabled: true},
+	}); err != nil {
+		t.Fatalf("SaveJobTypeConfig: %v", err)
+	}
+	if err := store.AppendRunRecord("ec", &JobRunRecord{Outcome: RunOutcomeSuccess, CompletedAt: timeToPtr(time.Now().UTC())}); err != nil {
+		t.Fatalf("AppendRunRecord: %v", err)
+	}
+
+	got, err := store.ListJobTypes()
+	if err != nil {
+		t.Fatalf("ListJobTypes: %v", err)
+	}
+	want := []string{"balance", "ec", "vacuum"}
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("unexpected job types: got=%v want=%v", got, want)
+	}
+}
+
+func TestConfigStoreMonitorStateRoundTrip(t *testing.T) {
+	t.Parallel()
+
+	store, err := NewConfigStore(t.TempDir())
+	if err != nil {
+		t.Fatalf("NewConfigStore: %v", err)
+	}
+
+	tracked := []TrackedJob{
+		{
+			JobID:     "job-1",
+			JobType:   "vacuum",
+			State:     "running",
+			Progress:  55,
+			WorkerID:  "worker-a",
+			CreatedAt: timeToPtr(time.Now().UTC().Add(-2 * time.Minute)),
+			UpdatedAt: timeToPtr(time.Now().UTC().Add(-1 * time.Minute)),
+		},
+	}
+	activities := []JobActivity{
+		{
+			JobID:      "job-1",
+			JobType:    "vacuum",
+			Source:     "worker_progress",
+			Message:    "processing",
+			Stage:      "running",
+			OccurredAt: timeToPtr(time.Now().UTC()),
+			Details: map[string]interface{}{
+				"step": "scan",
+			},
+		},
+	}
+
+	if err := store.SaveTrackedJobs(tracked); err != nil {
+		t.Fatalf("SaveTrackedJobs: %v", err)
+	}
+	if err := store.SaveActivities(activities); err != nil {
+		t.Fatalf("SaveActivities: %v", err)
+	}
+
+	gotTracked, err := store.LoadTrackedJobs()
+	if err != nil {
+		t.Fatalf("LoadTrackedJobs: %v", err)
+	}
+	if len(gotTracked) != 1 || gotTracked[0].JobID != tracked[0].JobID {
+		t.Fatalf("unexpected tracked jobs: %+v", gotTracked)
+	}
+
+	gotActivities, err := store.LoadActivities()
+	if err != nil {
+		t.Fatalf("LoadActivities: %v", err)
+	}
+	if len(gotActivities) != 1 || gotActivities[0].Message != activities[0].Message {
+		t.Fatalf("unexpected activities: %+v", gotActivities)
+	}
+	if gotActivities[0].Details["step"] != "scan" {
+		t.Fatalf("unexpected activity details: %+v", gotActivities[0].Details)
+	}
+}
+
+func TestConfigStoreSaveJobTypeConfigIfNotExists(t *testing.T) {
+	t.Parallel()
+
+	t.Run("in-memory", func(t *testing.T) {
+		t.Parallel()
+		store, err := NewConfigStore("")
+		if err != nil {
+			t.Fatalf("NewConfigStore: %v", err)
+		}
+		testSaveJobTypeConfigIfNotExists(t, store)
+	})
+
+	t.Run("on-disk", func(t *testing.T) {
+		t.Parallel()
+		store, err := NewConfigStore(t.TempDir())
+		if err != nil {
+			t.Fatalf("NewConfigStore: %v", err)
+		}
+		testSaveJobTypeConfigIfNotExists(t, store)
+	})
+}
+
+func testSaveJobTypeConfigIfNotExists(t *testing.T, store *ConfigStore) {
+	t.Helper()
+
+	cfg := &plugin_pb.PersistedJobTypeConfig{
+		JobType:      "admin_script",
+		AdminRuntime: &plugin_pb.AdminRuntimeConfig{Enabled: true},
+	}
+
+	// First call should save.
+	saved, err := store.SaveJobTypeConfigIfNotExists(cfg)
+	if err != nil {
+		t.Fatalf("first SaveJobTypeConfigIfNotExists: %v", err)
+	}
+	if !saved {
+		t.Fatal("expected first call to save the config")
+	}
+
+	// Second call with same job type should not save.
+	saved, err = store.SaveJobTypeConfigIfNotExists(&plugin_pb.PersistedJobTypeConfig{
+		JobType:      "admin_script",
+		AdminRuntime: &plugin_pb.AdminRuntimeConfig{Enabled: false},
+	})
+	if err != nil {
+		t.Fatalf("second SaveJobTypeConfigIfNotExists: %v", err)
+	}
+	if saved {
+		t.Fatal("expected second call to be a no-op")
+	}
+
+	// Verify the original config was preserved.
+	loaded, err := store.LoadJobTypeConfig("admin_script")
+	if err != nil {
+		t.Fatalf("LoadJobTypeConfig: %v", err)
+	}
+	if loaded == nil {
+		t.Fatal("expected config to exist")
+	}
+	if !loaded.AdminRuntime.Enabled {
+		t.Fatal("expected original config (Enabled=true) to be preserved")
+	}
+
+	// Different job type should still save.
+	saved, err = store.SaveJobTypeConfigIfNotExists(&plugin_pb.PersistedJobTypeConfig{
+		JobType: "vacuum",
+	})
+	if err != nil {
+		t.Fatalf("SaveJobTypeConfigIfNotExists for different type: %v", err)
+	}
+	if !saved {
+		t.Fatal("expected save for a different job type")
+	}
+}
+
+func TestConfigStoreJobDetailRoundTrip(t *testing.T) {
+	t.Parallel()
+
+	store, err := NewConfigStore(t.TempDir())
+	if err != nil {
+		t.Fatalf("NewConfigStore: %v", err)
+	}
+
+	input := TrackedJob{
+		JobID:     "job-detail-1",
+		JobType:   "vacuum",
+		Summary:   "detail summary",
+		Detail:    "detail payload",
+		CreatedAt: timeToPtr(time.Now().UTC().Add(-2 * time.Minute)),
+		UpdatedAt: timeToPtr(time.Now().UTC()),
+		Parameters: map[string]interface{}{
+			"volume_id": map[string]interface{}{"int64_value": "3"},
+		},
+		Labels: map[string]string{
+			"source": "detector",
+		},
+		ResultOutputValues: map[string]interface{}{
+			"moved": map[string]interface{}{"bool_value": true},
+		},
+	}
+
+	if err := store.SaveJobDetail(input); err != nil {
+		t.Fatalf("SaveJobDetail: %v", err)
+	}
+
+	got, err := store.LoadJobDetail(input.JobID)
+	if err != nil {
+		t.Fatalf("LoadJobDetail: %v", err)
+	}
+	if got == nil {
+		t.Fatalf("LoadJobDetail returned nil")
+	}
+	if got.Detail != input.Detail {
+		t.Fatalf("unexpected detail: got=%q want=%q", got.Detail, input.Detail)
+	}
+	if got.Labels["source"] != "detector" {
+		t.Fatalf("unexpected labels: %+v", got.Labels)
+	}
+	if got.ResultOutputValues == nil {
+		t.Fatalf("expected result output values")
+	}
+}
+
+func TestConfigStoreDeleteJobDetail(t *testing.T) {
+	t.Parallel()
+
+	store, err := NewConfigStore(t.TempDir())
+	if err != nil {
+		t.Fatalf("NewConfigStore: %v", err)
+	}
+
+	input := TrackedJob{
+		JobID:   "job-to-delete",
+		JobType: "vacuum",
+		Summary: "will be deleted",
+	}
+
+	if err := store.SaveJobDetail(input); err != nil {
+		t.Fatalf("SaveJobDetail: %v", err)
+	}
+
+	// Verify it was persisted.
+	got, err := store.LoadJobDetail(input.JobID)
+	if err != nil {
+		t.Fatalf("LoadJobDetail before delete: %v", err)
+	}
+	if got == nil {
+		t.Fatalf("expected saved job detail, got nil")
+	}
+
+	// Delete it.
+	if err := store.DeleteJobDetail(input.JobID); err != nil {
+		t.Fatalf("DeleteJobDetail: %v", err)
+	}
+
+	// Verify it is gone.
+	got, err = store.LoadJobDetail(input.JobID)
+	if err != nil {
+		t.Fatalf("LoadJobDetail after delete: %v", err)
+	}
+	if got != nil {
+		t.Fatalf("expected nil after delete, got %+v", got)
+	}
+
+	// Deleting again should be a no-op, not an error.
+	if err := store.DeleteJobDetail(input.JobID); err != nil {
+		t.Fatalf("second DeleteJobDetail: %v", err)
+	}
+}

weed/admin/plugin/job_execution_plan.go

@@ -0,0 +1,236 @@
+package plugin
+
+import (
+	"encoding/base64"
+	"sort"
+	"strconv"
+	"strings"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
+	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
+	"google.golang.org/protobuf/proto"
+)
+
+func enrichTrackedJobParameters(jobType string, parameters map[string]interface{}) map[string]interface{} {
+	if len(parameters) == 0 {
+		return parameters
+	}
+	if _, exists := parameters["execution_plan"]; exists {
+		return parameters
+	}
+
+	taskParams, ok := decodeTaskParamsFromPlainParameters(parameters)
+	if !ok || taskParams == nil {
+		return parameters
+	}
+
+	plan := buildExecutionPlan(strings.TrimSpace(jobType), taskParams)
+	if plan == nil {
+		return parameters
+	}
+
+	enriched := make(map[string]interface{}, len(parameters)+1)
+	for key, value := range parameters {
+		enriched[key] = value
+	}
+	enriched["execution_plan"] = plan
+	return enriched
+}
+
+func decodeTaskParamsFromPlainParameters(parameters map[string]interface{}) (*worker_pb.TaskParams, bool) {
+	rawField, ok := parameters["task_params_pb"]
+	if !ok || rawField == nil {
+		return nil, false
+	}
+
+	fieldMap, ok := rawField.(map[string]interface{})
+	if !ok {
+		return nil, false
+	}
+
+	bytesValue, _ := fieldMap["bytes_value"].(string)
+	bytesValue = strings.TrimSpace(bytesValue)
+	if bytesValue == "" {
+		return nil, false
+	}
+
+	payload, err := base64.StdEncoding.DecodeString(bytesValue)
+	if err != nil {
+		return nil, false
+	}
+
+	params := &worker_pb.TaskParams{}
+	if err := proto.Unmarshal(payload, params); err != nil {
+		return nil, false
+	}
+
+	return params, true
+}
+
+func buildExecutionPlan(jobType string, params *worker_pb.TaskParams) map[string]interface{} {
+	if params == nil {
+		return nil
+	}
+
+	normalizedJobType := strings.TrimSpace(jobType)
+	if normalizedJobType == "" && params.GetErasureCodingParams() != nil {
+		normalizedJobType = "erasure_coding"
+	}
+
+	switch normalizedJobType {
+	case "erasure_coding":
+		return buildErasureCodingExecutionPlan(params)
+	default:
+		return nil
+	}
+}
+
+func buildErasureCodingExecutionPlan(params *worker_pb.TaskParams) map[string]interface{} {
+	if params == nil {
+		return nil
+	}
+
+	ecParams := params.GetErasureCodingParams()
+	if ecParams == nil {
+		return nil
+	}
+
+	dataShards := int(ecParams.DataShards)
+	if dataShards <= 0 {
+		dataShards = int(erasure_coding.DataShardsCount)
+	}
+	parityShards := int(ecParams.ParityShards)
+	if parityShards <= 0 {
+		parityShards = int(erasure_coding.ParityShardsCount)
+	}
+	totalShards := dataShards + parityShards
+
+	sources := make([]map[string]interface{}, 0, len(params.Sources))
+	for _, source := range params.Sources {
+		if source == nil {
+			continue
+		}
+		sources = append(sources, buildExecutionEndpoint(
+			source.Node,
+			source.DataCenter,
+			source.Rack,
+			source.VolumeId,
+			source.DiskId,
+			source.ShardIds,
+			dataShards,
+		))
+	}
+
+	targets := make([]map[string]interface{}, 0, len(params.Targets))
+	shardAssignments := make([]map[string]interface{}, 0, totalShards)
+	for targetIndex, target := range params.Targets {
+		if target == nil {
+			continue
+		}
+
+		targets = append(targets, buildExecutionEndpoint(
+			target.Node,
+			target.DataCenter,
+			target.Rack,
+			target.VolumeId,
+			target.DiskId,
+			target.ShardIds,
+			dataShards,
+		))
+
+		for _, shardID := range normalizeShardIDs(target.ShardIds) {
+			kind, label := classifyShardID(shardID, dataShards)
+			shardAssignments = append(shardAssignments, map[string]interface{}{
+				"shard_id":           shardID,
+				"kind":               kind,
+				"label":              label,
+				"target_index":       targetIndex + 1,
+				"target_node":        strings.TrimSpace(target.Node),
+				"target_data_center": strings.TrimSpace(target.DataCenter),
+				"target_rack":        strings.TrimSpace(target.Rack),
+				"target_volume_id":   int(target.VolumeId),
+				"target_disk_id":     int(target.DiskId),
+			})
+		}
+	}
+	sort.Slice(shardAssignments, func(i, j int) bool {
+		left, _ := shardAssignments[i]["shard_id"].(int)
+		right, _ := shardAssignments[j]["shard_id"].(int)
+		return left < right
+	})
+
+	plan := map[string]interface{}{
+		"job_type":      "erasure_coding",
+		"task_id":       strings.TrimSpace(params.TaskId),
+		"volume_id":     int(params.VolumeId),
+		"collection":    strings.TrimSpace(params.Collection),
+		"data_shards":   dataShards,
+		"parity_shards": parityShards,
+		"total_shards":  totalShards,
+		"sources":       sources,
+		"targets":       targets,
+		"source_count":  len(sources),
+		"target_count":  len(targets),
+	}
+
+	if len(shardAssignments) > 0 {
+		plan["shard_assignments"] = shardAssignments
+	}
+
+	return plan
+}
+
+func buildExecutionEndpoint(
+	node string,
+	dataCenter string,
+	rack string,
+	volumeID uint32,
+	diskID uint32,
+	shardIDs []uint32,
+	dataShardCount int,
+) map[string]interface{} {
+	allShards := normalizeShardIDs(shardIDs)
+	dataShards := make([]int, 0, len(allShards))
+	parityShards := make([]int, 0, len(allShards))
+	for _, shardID := range allShards {
+		if shardID < dataShardCount {
+			dataShards = append(dataShards, shardID)
+		} else {
+			parityShards = append(parityShards, shardID)
+		}
+	}
+
+	return map[string]interface{}{
+		"node":             strings.TrimSpace(node),
+		"data_center":      strings.TrimSpace(dataCenter),
+		"rack":             strings.TrimSpace(rack),
+		"volume_id":        int(volumeID),
+		"disk_id":          int(diskID),
+		"shard_ids":        allShards,
+		"data_shard_ids":   dataShards,
+		"parity_shard_ids": parityShards,
+	}
+}
+
+func normalizeShardIDs(shardIDs []uint32) []int {
+	if len(shardIDs) == 0 {
+		return nil
+	}
+
+	out := make([]int, 0, len(shardIDs))
+	for _, shardID := range shardIDs {
+		out = append(out, int(shardID))
+	}
+	sort.Ints(out)
+	return out
+}
+
+func classifyShardID(shardID int, dataShardCount int) (kind string, label string) {
+	if dataShardCount <= 0 {
+		dataShardCount = int(erasure_coding.DataShardsCount)
+	}
+	if shardID < dataShardCount {
+		return "data", "D" + strconv.Itoa(shardID)
+	}
+	return "parity", "P" + strconv.Itoa(shardID)
+}

weed/admin/plugin/lock_manager.go

@@ -0,0 +1,7 @@
+package plugin
+
+// LockManager provides a shared exclusive lock for admin-managed detection/execution.
+// Acquire returns a release function that must be called when the protected work finishes.
+type LockManager interface {
+	Acquire(reason string) (release func(), err error)
+}

weed/admin/plugin/observations.go

@@ -0,0 +1,179 @@
+package plugin
+
+import (
+	"fmt"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+// maxObservations caps the store. Observations are a convenience for display,
+// so a cluster with more objects than this loses the oldest rather than growing
+// admin's memory without limit.
+const maxObservations = 10000
+
+// observationKey identifies one observed object across workers.
+func observationKey(objectID []string) string {
+	return strings.Join(objectID, "\x1f")
+}
+
+// ObservationStore keeps the last thing a worker said about each object.
+//
+// It is deliberately not authoritative: a worker reports what it saw when it
+// last looked, and admin serves that back with its timestamp so a reader can
+// judge how stale it is. Nothing schedules work from it.
+type ObservationStore struct {
+	mu      sync.RWMutex
+	entries map[string]*Observation
+}
+
+// Observation is one object as a worker last reported it.
+type Observation struct {
+	ObjectID   []string `json:"object_id"`
+	ObjectKind string   `json:"object_kind"`
+	Format     string   `json:"format"`
+	// Attributes are flattened on the way in: they exist to be displayed and
+	// served as JSON, and the typed form is the worker's business.
+	Attributes map[string]interface{} `json:"attributes"`
+	JobType    string                 `json:"job_type"`
+	WorkerID   string                 `json:"worker_id"`
+	ObservedAt time.Time              `json:"observed_at"`
+}
+
+func NewObservationStore() *ObservationStore {
+	return &ObservationStore{entries: make(map[string]*Observation)}
+}
+
+// Record stores what one worker reported, replacing whatever was there for the
+// same object. A later observation from a different worker still wins: the
+// newest look at an object is the useful one.
+func (s *ObservationStore) Record(workerID string, report *plugin_pb.WorkerObservations) {
+	if report == nil {
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	for _, observed := range report.Observations {
+		if observed == nil || len(observed.ObjectId) == 0 {
+			continue
+		}
+		at := time.Now()
+		if observed.ObservedAt != nil {
+			at = observed.ObservedAt.AsTime()
+		}
+		s.entries[observationKey(observed.ObjectId)] = &Observation{
+			ObjectID:   observed.ObjectId,
+			ObjectKind: observed.ObjectKind,
+			Format:     observed.Format,
+			Attributes: flattenAttributes(observed.Attributes),
+			JobType:    report.JobType,
+			WorkerID:   workerID,
+			ObservedAt: at,
+		}
+	}
+	s.evictOldest()
+}
+
+// evictOldest trims the store back under the cap. Called with the lock held.
+func (s *ObservationStore) evictOldest() {
+	if len(s.entries) <= maxObservations {
+		return
+	}
+	keys := make([]string, 0, len(s.entries))
+	for key := range s.entries {
+		keys = append(keys, key)
+	}
+	sort.Slice(keys, func(i, j int) bool {
+		return s.entries[keys[i]].ObservedAt.Before(s.entries[keys[j]].ObservedAt)
+	})
+	for _, key := range keys[:len(s.entries)-maxObservations] {
+		delete(s.entries, key)
+	}
+}
+
+// flattenAttributes unwraps the ConfigValue envelope that configValueMapToPlain
+// preserves. An observation exists to be read, and {"int64_value":"16"} is not
+// a number anyone wants to render; a protojson int64 also arrives as a string,
+// so it is converted back.
+func flattenAttributes(values map[string]*plugin_pb.ConfigValue) map[string]interface{} {
+	plain := configValueMapToPlain(values)
+	if plain == nil {
+		return nil
+	}
+	flat := make(map[string]interface{}, len(plain))
+	for name, value := range plain {
+		wrapper, ok := value.(map[string]interface{})
+		if !ok || len(wrapper) != 1 {
+			flat[name] = value
+			continue
+		}
+		for kind, inner := range wrapper {
+			if kind == "int64_value" {
+				if text, ok := inner.(string); ok {
+					if parsed, err := strconv.ParseInt(text, 10, 64); err == nil {
+						flat[name] = parsed
+						continue
+					}
+				}
+			}
+			flat[name] = inner
+		}
+	}
+	return flat
+}
+
+// Get returns the last observation of one object, if any.
+func (s *ObservationStore) Get(objectID []string) (*Observation, bool) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	observed, ok := s.entries[observationKey(objectID)]
+	return observed, ok
+}
+
+// GetFormat returns the last observation of one object only when it describes
+// the format asked for. A table can be dropped and remade in another format at
+// the same path, and the stale observation would then describe something that
+// no longer exists.
+func (s *ObservationStore) GetFormat(objectID []string, format string) (*Observation, bool) {
+	observed, ok := s.Get(objectID)
+	if !ok || !strings.EqualFold(observed.Format, format) {
+		return nil, false
+	}
+	return observed, true
+}
+
+// List returns every observation, newest first.
+func (s *ObservationStore) List() []*Observation {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	all := make([]*Observation, 0, len(s.entries))
+	for _, observed := range s.entries {
+		all = append(all, observed)
+	}
+	sort.Slice(all, func(i, j int) bool {
+		return all[i].ObservedAt.After(all[j].ObservedAt)
+	})
+	return all
+}
+
+// AttributeString renders one attribute for display, or "" when it is absent.
+func (o *Observation) AttributeString(name string) string {
+	if o == nil {
+		return ""
+	}
+	value, ok := o.Attributes[name]
+	if !ok || value == nil {
+		return ""
+	}
+	return fmt.Sprintf("%v", value)
+}
+
+// Observations exposes the store so admin handlers can serve it.
+func (r *Plugin) Observations() *ObservationStore {
+	return r.observations
+}

weed/admin/plugin/observations_test.go

@@ -0,0 +1,126 @@
+package plugin
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+	"google.golang.org/protobuf/types/known/timestamppb"
+)
+
+func intValue(v int64) *plugin_pb.ConfigValue {
+	return &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: v}}
+}
+
+func stringValue(v string) *plugin_pb.ConfigValue {
+	return &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_StringValue{StringValue: v}}
+}
+
+// An observation exists to be rendered, so the ConfigValue envelope comes off
+// on the way in. A page asking for "fragments" wants 16, not a wrapper holding
+// the string "16".
+func TestObservationAttributesAreFlattened(t *testing.T) {
+	store := NewObservationStore()
+	store.Record("worker-1", &plugin_pb.WorkerObservations{
+		JobType: "lance_compact",
+		Observations: []*plugin_pb.ObjectObservation{{
+			ObjectId:   []string{"bucket", "ns", "table"},
+			ObjectKind: "table",
+			Format:     "LANCE",
+			Attributes: map[string]*plugin_pb.ConfigValue{
+				"fragments": intValue(16),
+				"schema":    stringValue(`[{"name":"id"}]`),
+			},
+			ObservedAt: timestamppb.New(time.Now()),
+		}},
+	})
+
+	observed, ok := store.Get([]string{"bucket", "ns", "table"})
+	if !ok {
+		t.Fatal("observation was not stored")
+	}
+	if got, ok := observed.Attributes["fragments"].(int64); !ok || got != 16 {
+		t.Fatalf("fragments = %#v, want int64 16", observed.Attributes["fragments"])
+	}
+	if got := observed.AttributeString("fragments"); got != "16" {
+		t.Fatalf("AttributeString(fragments) = %q, want \"16\"", got)
+	}
+	if got := observed.AttributeString("schema"); got != `[{"name":"id"}]` {
+		t.Fatalf("AttributeString(schema) = %q", got)
+	}
+	if observed.WorkerID != "worker-1" || observed.JobType != "lance_compact" {
+		t.Fatalf("provenance lost: %+v", observed)
+	}
+}
+
+// The newest look at an object is the useful one, whichever worker took it.
+func TestObservationIsReplacedByALaterOne(t *testing.T) {
+	store := NewObservationStore()
+	record := func(worker string, fragments int64) {
+		store.Record(worker, &plugin_pb.WorkerObservations{
+			JobType: "lance_compact",
+			Observations: []*plugin_pb.ObjectObservation{{
+				ObjectId:   []string{"bucket", "ns", "table"},
+				Format:     "LANCE",
+				Attributes: map[string]*plugin_pb.ConfigValue{"fragments": intValue(fragments)},
+				ObservedAt: timestamppb.New(time.Now()),
+			}},
+		})
+	}
+	record("worker-1", 16)
+	record("worker-2", 1)
+
+	observed, _ := store.Get([]string{"bucket", "ns", "table"})
+	if got := observed.AttributeString("fragments"); got != "1" {
+		t.Fatalf("fragments = %q, want the later observation's 1", got)
+	}
+	if observed.WorkerID != "worker-2" {
+		t.Fatalf("worker = %q, want worker-2", observed.WorkerID)
+	}
+	if len(store.List()) != 1 {
+		t.Fatalf("store holds %d entries, want 1", len(store.List()))
+	}
+}
+
+// Missing attributes read as empty rather than panicking a template.
+func TestObservationAttributeStringHandlesAbsence(t *testing.T) {
+	var absent *Observation
+	if got := absent.AttributeString("fragments"); got != "" {
+		t.Fatalf("nil observation returned %q", got)
+	}
+	store := NewObservationStore()
+	store.Record("worker-1", &plugin_pb.WorkerObservations{
+		Observations: []*plugin_pb.ObjectObservation{{
+			ObjectId: []string{"b", "n", "t"},
+		}},
+	})
+	observed, _ := store.Get([]string{"b", "n", "t"})
+	if got := observed.AttributeString("nothing"); got != "" {
+		t.Fatalf("absent attribute returned %q", got)
+	}
+}
+
+// A table path can be dropped and remade in another format. The observation left
+// behind describes something that is no longer there, so a caller asking about
+// the new format must not be handed it.
+func TestObservationLookupIsScopedToFormat(t *testing.T) {
+	store := NewObservationStore()
+	objectID := []string{"bucket", "ns", "table"}
+	store.Record("worker-1", &plugin_pb.WorkerObservations{
+		Observations: []*plugin_pb.ObjectObservation{{
+			ObjectId:   objectID,
+			Format:     "LANCE",
+			ObservedAt: timestamppb.New(time.Now()),
+		}},
+	})
+
+	if _, ok := store.GetFormat(objectID, "ICEBERG"); ok {
+		t.Fatal("a LANCE observation was returned for an ICEBERG table")
+	}
+	if _, ok := store.GetFormat(objectID, "lance"); !ok {
+		t.Fatal("format matching must not depend on case")
+	}
+	if _, ok := store.GetFormat(objectID, ""); ok {
+		t.Fatal("an unknown format must not match a recorded one")
+	}
+}

weed/admin/plugin/plugin_cancel_test.go

@@ -0,0 +1,276 @@
+package plugin
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+	"google.golang.org/protobuf/types/known/timestamppb"
+)
+
+func TestRunDetectionSendsCancelOnContextDone(t *testing.T) {
+	t.Parallel()
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-detect"
+	const jobType = "vacuum"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: workerID,
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanDetect: true, MaxDetectionConcurrency: 1},
+		},
+	})
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 4), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	errCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.RunDetection(ctx, jobType, &plugin_pb.ClusterContext{}, 10)
+		errCh <- runErr
+	}()
+
+	first := <-session.outgoing
+	if first.GetRunDetectionRequest() == nil {
+		t.Fatalf("expected first message to be run_detection_request")
+	}
+
+	cancel()
+
+	second := <-session.outgoing
+	cancelReq := second.GetCancelRequest()
+	if cancelReq == nil {
+		t.Fatalf("expected second message to be cancel_request")
+	}
+	if cancelReq.TargetId != first.RequestId {
+		t.Fatalf("unexpected cancel target id: got=%s want=%s", cancelReq.TargetId, first.RequestId)
+	}
+	if cancelReq.TargetKind != plugin_pb.WorkKind_WORK_KIND_DETECTION {
+		t.Fatalf("unexpected cancel target kind: %v", cancelReq.TargetKind)
+	}
+
+	runErr := <-errCh
+	if !errors.Is(runErr, context.Canceled) {
+		t.Fatalf("expected context canceled error, got %v", runErr)
+	}
+}
+
+func TestExecuteJobSendsCancelOnContextDone(t *testing.T) {
+	t.Parallel()
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-exec"
+	const jobType = "vacuum"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: workerID,
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanExecute: true, MaxExecutionConcurrency: 1},
+		},
+	})
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 4), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	job := &plugin_pb.JobSpec{JobId: "job-1", JobType: jobType}
+	ctx, cancel := context.WithCancel(context.Background())
+	errCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.ExecuteJob(ctx, job, &plugin_pb.ClusterContext{}, 1)
+		errCh <- runErr
+	}()
+
+	first := <-session.outgoing
+	if first.GetExecuteJobRequest() == nil {
+		t.Fatalf("expected first message to be execute_job_request")
+	}
+
+	cancel()
+
+	second := <-session.outgoing
+	cancelReq := second.GetCancelRequest()
+	if cancelReq == nil {
+		t.Fatalf("expected second message to be cancel_request")
+	}
+	if cancelReq.TargetId != first.RequestId {
+		t.Fatalf("unexpected cancel target id: got=%s want=%s", cancelReq.TargetId, first.RequestId)
+	}
+	if cancelReq.TargetKind != plugin_pb.WorkKind_WORK_KIND_EXECUTION {
+		t.Fatalf("unexpected cancel target kind: %v", cancelReq.TargetKind)
+	}
+
+	runErr := <-errCh
+	if !errors.Is(runErr, context.Canceled) {
+		t.Fatalf("expected context canceled error, got %v", runErr)
+	}
+}
+
+func TestAdminScriptExecutionBlocksOtherDetection(t *testing.T) {
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const adminWorkerID = "worker-admin-script"
+	const otherWorkerID = "worker-vacuum"
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: adminWorkerID,
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "admin_script", CanExecute: true, MaxExecutionConcurrency: 1},
+		},
+	})
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: otherWorkerID,
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true, MaxDetectionConcurrency: 1},
+		},
+	})
+	adminSession := &streamSession{workerID: adminWorkerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	otherSession := &streamSession{workerID: otherWorkerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	pluginSvc.putSession(adminSession)
+	pluginSvc.putSession(otherSession)
+
+	adminErrCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.ExecuteJob(context.Background(), &plugin_pb.JobSpec{
+			JobId:   "job-admin-script-1",
+			JobType: "admin_script",
+		}, &plugin_pb.ClusterContext{}, 1)
+		adminErrCh <- runErr
+	}()
+
+	adminExecMessage := <-adminSession.outgoing
+	if adminExecMessage.GetExecuteJobRequest() == nil {
+		t.Fatalf("expected admin_script execute request")
+	}
+
+	detectErrCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.RunDetection(context.Background(), "vacuum", &plugin_pb.ClusterContext{}, 10)
+		detectErrCh <- runErr
+	}()
+
+	select {
+	case unexpected := <-otherSession.outgoing:
+		t.Fatalf("expected vacuum detection to wait while admin_script runs, got message: %+v", unexpected)
+	case <-time.After(100 * time.Millisecond):
+	}
+
+	pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
+		RequestId:   adminExecMessage.RequestId,
+		JobId:       "job-admin-script-1",
+		JobType:     "admin_script",
+		Success:     true,
+		CompletedAt: timestamppb.Now(),
+	})
+	if runErr := <-adminErrCh; runErr != nil {
+		t.Fatalf("admin_script ExecuteJob error: %v", runErr)
+	}
+
+	detectMessage := <-otherSession.outgoing
+	detectRequest := detectMessage.GetRunDetectionRequest()
+	if detectRequest == nil {
+		t.Fatalf("expected vacuum detection request after admin_script completion")
+	}
+	pluginSvc.handleDetectionComplete(otherWorkerID, &plugin_pb.DetectionComplete{
+		RequestId: detectMessage.RequestId,
+		JobType:   "vacuum",
+		Success:   true,
+	})
+	if runErr := <-detectErrCh; runErr != nil {
+		t.Fatalf("vacuum RunDetection error: %v", runErr)
+	}
+}
+
+func TestAdminScriptExecutionBlocksOtherExecution(t *testing.T) {
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const adminWorkerID = "worker-admin-script"
+	const otherWorkerID = "worker-vacuum"
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: adminWorkerID,
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "admin_script", CanExecute: true, MaxExecutionConcurrency: 1},
+		},
+	})
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: otherWorkerID,
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanExecute: true, MaxExecutionConcurrency: 1},
+		},
+	})
+	adminSession := &streamSession{workerID: adminWorkerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	otherSession := &streamSession{workerID: otherWorkerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	pluginSvc.putSession(adminSession)
+	pluginSvc.putSession(otherSession)
+
+	adminErrCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.ExecuteJob(context.Background(), &plugin_pb.JobSpec{
+			JobId:   "job-admin-script-2",
+			JobType: "admin_script",
+		}, &plugin_pb.ClusterContext{}, 1)
+		adminErrCh <- runErr
+	}()
+
+	adminExecMessage := <-adminSession.outgoing
+	if adminExecMessage.GetExecuteJobRequest() == nil {
+		t.Fatalf("expected admin_script execute request")
+	}
+
+	otherErrCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.ExecuteJob(context.Background(), &plugin_pb.JobSpec{
+			JobId:   "job-vacuum-1",
+			JobType: "vacuum",
+		}, &plugin_pb.ClusterContext{}, 1)
+		otherErrCh <- runErr
+	}()
+
+	select {
+	case unexpected := <-otherSession.outgoing:
+		t.Fatalf("expected vacuum execute to wait while admin_script runs, got message: %+v", unexpected)
+	case <-time.After(100 * time.Millisecond):
+	}
+
+	pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
+		RequestId:   adminExecMessage.RequestId,
+		JobId:       "job-admin-script-2",
+		JobType:     "admin_script",
+		Success:     true,
+		CompletedAt: timestamppb.Now(),
+	})
+	if runErr := <-adminErrCh; runErr != nil {
+		t.Fatalf("admin_script ExecuteJob error: %v", runErr)
+	}
+
+	otherExecMessage := <-otherSession.outgoing
+	if otherExecMessage.GetExecuteJobRequest() == nil {
+		t.Fatalf("expected vacuum execute request after admin_script completion")
+	}
+	pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
+		RequestId:   otherExecMessage.RequestId,
+		JobId:       "job-vacuum-1",
+		JobType:     "vacuum",
+		Success:     true,
+		CompletedAt: timestamppb.Now(),
+	})
+	if runErr := <-otherErrCh; runErr != nil {
+		t.Fatalf("vacuum ExecuteJob error: %v", runErr)
+	}
+}

weed/admin/plugin/plugin_config_bootstrap_test.go

@@ -0,0 +1,125 @@
+package plugin
+
+import (
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+func TestEnsureJobTypeConfigFromDescriptorBootstrapsDefaults(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	descriptor := &plugin_pb.JobTypeDescriptor{
+		JobType:           "vacuum",
+		DescriptorVersion: 3,
+		AdminConfigForm: &plugin_pb.ConfigForm{
+			DefaultValues: map[string]*plugin_pb.ConfigValue{
+				"scan_scope": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "all"}},
+			},
+		},
+		WorkerConfigForm: &plugin_pb.ConfigForm{
+			DefaultValues: map[string]*plugin_pb.ConfigValue{
+				"threshold": {Kind: &plugin_pb.ConfigValue_DoubleValue{DoubleValue: 0.3}},
+			},
+		},
+		AdminRuntimeDefaults: &plugin_pb.AdminRuntimeDefaults{
+			Enabled:                       true,
+			DetectionIntervalMinutes:      60,
+			DetectionTimeoutSeconds:       20,
+			MaxJobsPerDetection:           30,
+			GlobalExecutionConcurrency:    4,
+			PerWorkerExecutionConcurrency: 2,
+			RetryLimit:                    3,
+			RetryBackoffSeconds:           5,
+		},
+	}
+
+	if err := pluginSvc.ensureJobTypeConfigFromDescriptor("vacuum", descriptor); err != nil {
+		t.Fatalf("ensureJobTypeConfigFromDescriptor: %v", err)
+	}
+
+	cfg, err := pluginSvc.LoadJobTypeConfig("vacuum")
+	if err != nil {
+		t.Fatalf("LoadJobTypeConfig: %v", err)
+	}
+	if cfg == nil {
+		t.Fatalf("expected non-nil config")
+	}
+	if cfg.DescriptorVersion != 3 {
+		t.Fatalf("unexpected descriptor version: got=%d", cfg.DescriptorVersion)
+	}
+	if cfg.AdminRuntime == nil || !cfg.AdminRuntime.Enabled {
+		t.Fatalf("expected enabled admin settings")
+	}
+	if cfg.AdminRuntime.GlobalExecutionConcurrency != 4 {
+		t.Fatalf("unexpected global execution concurrency: %d", cfg.AdminRuntime.GlobalExecutionConcurrency)
+	}
+	if _, ok := cfg.AdminConfigValues["scan_scope"]; !ok {
+		t.Fatalf("missing admin default value")
+	}
+	if _, ok := cfg.WorkerConfigValues["threshold"]; !ok {
+		t.Fatalf("missing worker default value")
+	}
+}
+
+func TestEnsureJobTypeConfigFromDescriptorDoesNotOverwriteExisting(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	if err := pluginSvc.SaveJobTypeConfig(&plugin_pb.PersistedJobTypeConfig{
+		JobType: "balance",
+		AdminRuntime: &plugin_pb.AdminRuntimeConfig{
+			Enabled:                    true,
+			GlobalExecutionConcurrency: 9,
+		},
+		AdminConfigValues: map[string]*plugin_pb.ConfigValue{
+			"custom": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "keep"}},
+		},
+	}); err != nil {
+		t.Fatalf("SaveJobTypeConfig: %v", err)
+	}
+
+	descriptor := &plugin_pb.JobTypeDescriptor{
+		JobType:           "balance",
+		DescriptorVersion: 7,
+		AdminConfigForm: &plugin_pb.ConfigForm{
+			DefaultValues: map[string]*plugin_pb.ConfigValue{
+				"custom": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "overwrite"}},
+			},
+		},
+		AdminRuntimeDefaults: &plugin_pb.AdminRuntimeDefaults{
+			Enabled:                    true,
+			GlobalExecutionConcurrency: 1,
+		},
+	}
+
+	if err := pluginSvc.ensureJobTypeConfigFromDescriptor("balance", descriptor); err != nil {
+		t.Fatalf("ensureJobTypeConfigFromDescriptor: %v", err)
+	}
+
+	cfg, err := pluginSvc.LoadJobTypeConfig("balance")
+	if err != nil {
+		t.Fatalf("LoadJobTypeConfig: %v", err)
+	}
+	if cfg == nil {
+		t.Fatalf("expected config")
+	}
+	if cfg.AdminRuntime == nil || cfg.AdminRuntime.GlobalExecutionConcurrency != 9 {
+		t.Fatalf("existing admin settings should be preserved, got=%v", cfg.AdminRuntime)
+	}
+	custom := cfg.AdminConfigValues["custom"]
+	if custom == nil || custom.GetStringValue() != "keep" {
+		t.Fatalf("existing admin config should be preserved")
+	}
+}

weed/admin/plugin/plugin_detection_test.go

@@ -0,0 +1,258 @@
+package plugin
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+func TestRunDetectionIncludesLatestSuccessfulRun(t *testing.T) {
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	jobType := "vacuum"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanDetect: true, MaxDetectionConcurrency: 1},
+		},
+	})
+	session := &streamSession{workerID: "worker-a", outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 1), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	oldSuccess := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	latestSuccess := time.Date(2026, 2, 1, 0, 0, 0, 0, time.UTC)
+	if err := pluginSvc.store.AppendRunRecord(jobType, &JobRunRecord{Outcome: RunOutcomeSuccess, CompletedAt: timeToPtr(oldSuccess)}); err != nil {
+		t.Fatalf("AppendRunRecord old success: %v", err)
+	}
+	if err := pluginSvc.store.AppendRunRecord(jobType, &JobRunRecord{Outcome: RunOutcomeError, CompletedAt: timeToPtr(latestSuccess.Add(2 * time.Hour))}); err != nil {
+		t.Fatalf("AppendRunRecord error run: %v", err)
+	}
+	if err := pluginSvc.store.AppendRunRecord(jobType, &JobRunRecord{Outcome: RunOutcomeSuccess, CompletedAt: timeToPtr(latestSuccess)}); err != nil {
+		t.Fatalf("AppendRunRecord latest success: %v", err)
+	}
+
+	resultCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.RunDetection(context.Background(), jobType, &plugin_pb.ClusterContext{}, 10)
+		resultCh <- runErr
+	}()
+
+	message := <-session.outgoing
+	detectRequest := message.GetRunDetectionRequest()
+	if detectRequest == nil {
+		t.Fatalf("expected run detection request message")
+	}
+	if detectRequest.LastSuccessfulRun == nil {
+		t.Fatalf("expected last_successful_run to be set")
+	}
+	if got := detectRequest.LastSuccessfulRun.AsTime().UTC(); !got.Equal(latestSuccess) {
+		t.Fatalf("unexpected last_successful_run, got=%s want=%s", got, latestSuccess)
+	}
+
+	pluginSvc.handleDetectionComplete("worker-a", &plugin_pb.DetectionComplete{
+		RequestId: message.RequestId,
+		JobType:   jobType,
+		Success:   true,
+	})
+
+	if runErr := <-resultCh; runErr != nil {
+		t.Fatalf("RunDetection error: %v", runErr)
+	}
+}
+
+func TestRunDetectionOmitsLastSuccessfulRunWhenNoSuccessHistory(t *testing.T) {
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	jobType := "vacuum"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanDetect: true, MaxDetectionConcurrency: 1},
+		},
+	})
+	session := &streamSession{workerID: "worker-a", outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 1), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	if err := pluginSvc.store.AppendRunRecord(jobType, &JobRunRecord{
+		Outcome:     RunOutcomeError,
+		CompletedAt: timeToPtr(time.Date(2026, 2, 10, 0, 0, 0, 0, time.UTC)),
+	}); err != nil {
+		t.Fatalf("AppendRunRecord error run: %v", err)
+	}
+
+	resultCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.RunDetection(context.Background(), jobType, &plugin_pb.ClusterContext{}, 10)
+		resultCh <- runErr
+	}()
+
+	message := <-session.outgoing
+	detectRequest := message.GetRunDetectionRequest()
+	if detectRequest == nil {
+		t.Fatalf("expected run detection request message")
+	}
+	if detectRequest.LastSuccessfulRun != nil {
+		t.Fatalf("expected last_successful_run to be nil when no success history")
+	}
+
+	pluginSvc.handleDetectionComplete("worker-a", &plugin_pb.DetectionComplete{
+		RequestId: message.RequestId,
+		JobType:   jobType,
+		Success:   true,
+	})
+
+	if runErr := <-resultCh; runErr != nil {
+		t.Fatalf("RunDetection error: %v", runErr)
+	}
+}
+
+func TestRunDetectionWithReportCapturesDetectionActivities(t *testing.T) {
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	jobType := "vacuum"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanDetect: true, MaxDetectionConcurrency: 1},
+		},
+	})
+	session := &streamSession{workerID: "worker-a", outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 1), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	reportCh := make(chan *DetectionReport, 1)
+	errCh := make(chan error, 1)
+	go func() {
+		report, runErr := pluginSvc.RunDetectionWithReport(context.Background(), jobType, &plugin_pb.ClusterContext{}, 10)
+		reportCh <- report
+		errCh <- runErr
+	}()
+
+	message := <-session.outgoing
+	requestID := message.GetRequestId()
+	if requestID == "" {
+		t.Fatalf("expected request id in detection request")
+	}
+
+	pluginSvc.handleDetectionProposals("worker-a", &plugin_pb.DetectionProposals{
+		RequestId: requestID,
+		JobType:   jobType,
+		Proposals: []*plugin_pb.JobProposal{
+			{
+				ProposalId: "proposal-1",
+				JobType:    jobType,
+				Summary:    "vacuum proposal",
+				Detail:     "based on garbage ratio",
+			},
+		},
+	})
+	pluginSvc.handleDetectionComplete("worker-a", &plugin_pb.DetectionComplete{
+		RequestId:      requestID,
+		JobType:        jobType,
+		Success:        true,
+		TotalProposals: 1,
+	})
+
+	report := <-reportCh
+	if report == nil {
+		t.Fatalf("expected detection report")
+	}
+	if report.RequestID == "" {
+		t.Fatalf("expected detection report request id")
+	}
+	if report.WorkerID != "worker-a" {
+		t.Fatalf("expected worker-a, got %q", report.WorkerID)
+	}
+	if len(report.Proposals) != 1 {
+		t.Fatalf("expected one proposal in report, got %d", len(report.Proposals))
+	}
+	if runErr := <-errCh; runErr != nil {
+		t.Fatalf("RunDetectionWithReport error: %v", runErr)
+	}
+
+	activities := pluginSvc.ListActivities(jobType, 0)
+	stages := map[string]bool{}
+	for _, activity := range activities {
+		if activity.RequestID != report.RequestID {
+			continue
+		}
+		stages[activity.Stage] = true
+	}
+	if !stages["requested"] || !stages["proposal"] || !stages["completed"] {
+		t.Fatalf("expected requested/proposal/completed activities, got stages=%v", stages)
+	}
+}
+
+func TestRunDetectionAdminScriptUsesLastCompletedRun(t *testing.T) {
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	jobType := "admin_script"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-admin-script",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanDetect: true, MaxDetectionConcurrency: 1},
+		},
+	})
+	session := &streamSession{workerID: "worker-admin-script", outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 1), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	successCompleted := time.Date(2026, 2, 1, 10, 0, 0, 0, time.UTC)
+	errorCompleted := successCompleted.Add(45 * time.Minute)
+	if err := pluginSvc.store.AppendRunRecord(jobType, &JobRunRecord{
+		Outcome:     RunOutcomeSuccess,
+		CompletedAt: timeToPtr(successCompleted),
+	}); err != nil {
+		t.Fatalf("AppendRunRecord success run: %v", err)
+	}
+	if err := pluginSvc.store.AppendRunRecord(jobType, &JobRunRecord{
+		Outcome:     RunOutcomeError,
+		CompletedAt: timeToPtr(errorCompleted),
+	}); err != nil {
+		t.Fatalf("AppendRunRecord error run: %v", err)
+	}
+
+	errCh := make(chan error, 1)
+	go func() {
+		_, runErr := pluginSvc.RunDetection(context.Background(), jobType, &plugin_pb.ClusterContext{}, 10)
+		errCh <- runErr
+	}()
+
+	message := <-session.outgoing
+	detectRequest := message.GetRunDetectionRequest()
+	if detectRequest == nil {
+		t.Fatalf("expected run detection request message")
+	}
+	if detectRequest.LastSuccessfulRun == nil {
+		t.Fatalf("expected last_successful_run to be set")
+	}
+	if got := detectRequest.LastSuccessfulRun.AsTime().UTC(); !got.Equal(errorCompleted) {
+		t.Fatalf("unexpected last_successful_run, got=%s want=%s", got, errorCompleted)
+	}
+
+	pluginSvc.handleDetectionComplete("worker-admin-script", &plugin_pb.DetectionComplete{
+		RequestId: message.RequestId,
+		JobType:   jobType,
+		Success:   true,
+	})
+
+	if runErr := <-errCh; runErr != nil {
+		t.Fatalf("RunDetection error: %v", runErr)
+	}
+}

weed/admin/plugin/plugin_monitor_test.go

@@ -0,0 +1,631 @@
+package plugin
+
+import (
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
+	"google.golang.org/protobuf/proto"
+	"google.golang.org/protobuf/types/known/timestamppb"
+)
+
+func TestPluginLoadsPersistedMonitorStateOnStart(t *testing.T) {
+	t.Parallel()
+
+	dataDir := t.TempDir()
+	store, err := NewConfigStore(dataDir)
+	if err != nil {
+		t.Fatalf("NewConfigStore: %v", err)
+	}
+
+	seedJobs := []TrackedJob{
+		{
+			JobID:     "job-seeded",
+			JobType:   "vacuum",
+			State:     "running",
+			CreatedAt: timeToPtr(time.Now().UTC().Add(-2 * time.Minute)),
+			UpdatedAt: timeToPtr(time.Now().UTC().Add(-1 * time.Minute)),
+		},
+	}
+	seedActivities := []JobActivity{
+		{
+			JobID:      "job-seeded",
+			JobType:    "vacuum",
+			Source:     "worker_progress",
+			Message:    "seeded",
+			OccurredAt: timeToPtr(time.Now().UTC().Add(-30 * time.Second)),
+		},
+	}
+
+	if err := store.SaveTrackedJobs(seedJobs); err != nil {
+		t.Fatalf("SaveTrackedJobs: %v", err)
+	}
+	if err := store.SaveActivities(seedActivities); err != nil {
+		t.Fatalf("SaveActivities: %v", err)
+	}
+
+	pluginSvc, err := New(Options{DataDir: dataDir})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	gotJobs := pluginSvc.ListTrackedJobs("", "", 0)
+	if len(gotJobs) != 1 || gotJobs[0].JobID != "job-seeded" {
+		t.Fatalf("unexpected loaded jobs: %+v", gotJobs)
+	}
+
+	gotActivities := pluginSvc.ListActivities("", 0)
+	if len(gotActivities) != 1 || gotActivities[0].Message != "seeded" {
+		t.Fatalf("unexpected loaded activities: %+v", gotActivities)
+	}
+}
+
+func TestPluginPersistsMonitorStateAfterJobUpdates(t *testing.T) {
+	t.Parallel()
+
+	dataDir := t.TempDir()
+	pluginSvc, err := New(Options{DataDir: dataDir})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	job := &plugin_pb.JobSpec{
+		JobId:   "job-persist",
+		JobType: "vacuum",
+		Summary: "persist test",
+	}
+	pluginSvc.trackExecutionStart("req-persist", "worker-a", job, 1)
+
+	pluginSvc.trackExecutionCompletion(&plugin_pb.JobCompleted{
+		RequestId:   "req-persist",
+		JobId:       "job-persist",
+		JobType:     "vacuum",
+		Success:     true,
+		Result:      &plugin_pb.JobResult{Summary: "done"},
+		CompletedAt: timestamppb.New(time.Now().UTC()),
+	})
+	pluginSvc.Shutdown()
+
+	store, err := NewConfigStore(dataDir)
+	if err != nil {
+		t.Fatalf("NewConfigStore: %v", err)
+	}
+
+	trackedJobs, err := store.LoadTrackedJobs()
+	if err != nil {
+		t.Fatalf("LoadTrackedJobs: %v", err)
+	}
+	if len(trackedJobs) == 0 {
+		t.Fatalf("expected persisted tracked jobs")
+	}
+
+	found := false
+	for _, tracked := range trackedJobs {
+		if tracked.JobID == "job-persist" {
+			found = true
+			if tracked.State == "" {
+				t.Fatalf("persisted job state should not be empty")
+			}
+		}
+	}
+	if !found {
+		t.Fatalf("persisted tracked jobs missing job-persist")
+	}
+
+	activities, err := store.LoadActivities()
+	if err != nil {
+		t.Fatalf("LoadActivities: %v", err)
+	}
+	if len(activities) == 0 {
+		t.Fatalf("expected persisted activities")
+	}
+}
+
+func TestTrackExecutionQueuedMarksPendingState(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.trackExecutionQueued(&plugin_pb.JobSpec{
+		JobId:     "job-pending-1",
+		JobType:   "vacuum",
+		DedupeKey: "vacuum:1",
+		Summary:   "pending queue item",
+	})
+
+	jobs := pluginSvc.ListTrackedJobs("vacuum", "", 10)
+	if len(jobs) != 1 {
+		t.Fatalf("expected one tracked pending job, got=%d", len(jobs))
+	}
+	job := jobs[0]
+	if job.JobID != "job-pending-1" {
+		t.Fatalf("unexpected pending job id: %s", job.JobID)
+	}
+	if job.State != "job_state_pending" {
+		t.Fatalf("unexpected pending job state: %s", job.State)
+	}
+	if job.Stage != "queued" {
+		t.Fatalf("unexpected pending job stage: %s", job.Stage)
+	}
+
+	activities := pluginSvc.ListActivities("vacuum", 50)
+	found := false
+	for _, activity := range activities {
+		if activity.JobID == "job-pending-1" && activity.Stage == "queued" && activity.Source == "admin_scheduler" {
+			found = true
+			break
+		}
+	}
+	if !found {
+		t.Fatalf("expected queued activity for pending job")
+	}
+}
+
+func TestHandleJobProgressUpdateCarriesWorkerIDInActivities(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	job := &plugin_pb.JobSpec{
+		JobId:   "job-progress-worker",
+		JobType: "vacuum",
+	}
+	pluginSvc.trackExecutionStart("req-progress-worker", "worker-a", job, 1)
+
+	pluginSvc.handleJobProgressUpdate("worker-a", &plugin_pb.JobProgressUpdate{
+		RequestId:       "req-progress-worker",
+		JobId:           "job-progress-worker",
+		JobType:         "vacuum",
+		State:           plugin_pb.JobState_JOB_STATE_RUNNING,
+		ProgressPercent: 42.0,
+		Stage:           "scan",
+		Message:         "in progress",
+		Activities: []*plugin_pb.ActivityEvent{
+			{
+				Source:  plugin_pb.ActivitySource_ACTIVITY_SOURCE_EXECUTOR,
+				Message: "volume scanned",
+				Stage:   "scan",
+			},
+		},
+	})
+
+	activities := pluginSvc.ListActivities("vacuum", 0)
+	if len(activities) == 0 {
+		t.Fatalf("expected activity entries")
+	}
+
+	foundProgress := false
+	foundEvent := false
+	for _, activity := range activities {
+		if activity.Source == "worker_progress" && activity.Message == "in progress" {
+			foundProgress = true
+			if activity.WorkerID != "worker-a" {
+				t.Fatalf("worker_progress activity worker mismatch: got=%q want=%q", activity.WorkerID, "worker-a")
+			}
+		}
+		if activity.Message == "volume scanned" {
+			foundEvent = true
+			if activity.WorkerID != "worker-a" {
+				t.Fatalf("worker event worker mismatch: got=%q want=%q", activity.WorkerID, "worker-a")
+			}
+		}
+	}
+
+	if !foundProgress {
+		t.Fatalf("expected worker_progress activity")
+	}
+	if !foundEvent {
+		t.Fatalf("expected worker activity event")
+	}
+}
+
+func TestHandleJobProgressUpdateWithoutJobIDTracksDetectionActivities(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.handleJobProgressUpdate("worker-detector", &plugin_pb.JobProgressUpdate{
+		RequestId: "detect-req-1",
+		JobType:   "vacuum",
+		State:     plugin_pb.JobState_JOB_STATE_RUNNING,
+		Stage:     "decision_summary",
+		Message:   "VACUUM: No tasks created for 3 volumes",
+		Activities: []*plugin_pb.ActivityEvent{
+			{
+				Source:  plugin_pb.ActivitySource_ACTIVITY_SOURCE_DETECTOR,
+				Stage:   "decision_summary",
+				Message: "VACUUM: No tasks created for 3 volumes",
+			},
+		},
+	})
+
+	activities := pluginSvc.ListActivities("vacuum", 0)
+	if len(activities) == 0 {
+		t.Fatalf("expected activity entries")
+	}
+
+	foundDetectionProgress := false
+	foundDetectorEvent := false
+	for _, activity := range activities {
+		if activity.RequestID != "detect-req-1" {
+			continue
+		}
+		if activity.Source == "worker_detection" {
+			foundDetectionProgress = true
+			if activity.WorkerID != "worker-detector" {
+				t.Fatalf("worker_detection worker mismatch: got=%q want=%q", activity.WorkerID, "worker-detector")
+			}
+		}
+		if activity.Source == "activity_source_detector" {
+			foundDetectorEvent = true
+			if activity.WorkerID != "worker-detector" {
+				t.Fatalf("detector event worker mismatch: got=%q want=%q", activity.WorkerID, "worker-detector")
+			}
+		}
+	}
+
+	if !foundDetectionProgress {
+		t.Fatalf("expected worker_detection activity")
+	}
+	if !foundDetectorEvent {
+		t.Fatalf("expected detector activity event")
+	}
+}
+
+func TestHandleJobCompletedCarriesWorkerIDInActivitiesAndRunHistory(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	job := &plugin_pb.JobSpec{
+		JobId:   "job-complete-worker",
+		JobType: "vacuum",
+	}
+	pluginSvc.trackExecutionStart("req-complete-worker", "worker-b", job, 1)
+
+	pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
+		RequestId: "req-complete-worker",
+		JobId:     "job-complete-worker",
+		JobType:   "vacuum",
+		Success:   true,
+		Activities: []*plugin_pb.ActivityEvent{
+			{
+				Source:  plugin_pb.ActivitySource_ACTIVITY_SOURCE_EXECUTOR,
+				Message: "finalizer done",
+				Stage:   "finalize",
+			},
+		},
+		CompletedAt: timestamppb.Now(),
+	})
+	pluginSvc.Shutdown()
+
+	activities := pluginSvc.ListActivities("vacuum", 0)
+	foundWorkerEvent := false
+	for _, activity := range activities {
+		if activity.Message == "finalizer done" {
+			foundWorkerEvent = true
+			if activity.WorkerID != "worker-b" {
+				t.Fatalf("worker completion event worker mismatch: got=%q want=%q", activity.WorkerID, "worker-b")
+			}
+		}
+	}
+	if !foundWorkerEvent {
+		t.Fatalf("expected completion worker event activity")
+	}
+
+	history, err := pluginSvc.LoadRunHistory("vacuum")
+	if err != nil {
+		t.Fatalf("LoadRunHistory: %v", err)
+	}
+	if history == nil || len(history.SuccessfulRuns) == 0 {
+		t.Fatalf("expected successful run history entry")
+	}
+	if history.SuccessfulRuns[0].WorkerID != "worker-b" {
+		t.Fatalf("run history worker mismatch: got=%q want=%q", history.SuccessfulRuns[0].WorkerID, "worker-b")
+	}
+}
+
+func TestTrackExecutionStartStoresJobPayloadDetails(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.trackExecutionStart("req-payload", "worker-c", &plugin_pb.JobSpec{
+		JobId:   "job-payload",
+		JobType: "vacuum",
+		Summary: "payload summary",
+		Detail:  "payload detail",
+		Parameters: map[string]*plugin_pb.ConfigValue{
+			"volume_id": {
+				Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 9},
+			},
+		},
+		Labels: map[string]string{
+			"source": "detector",
+		},
+	}, 2)
+	pluginSvc.Shutdown()
+
+	job, found := pluginSvc.GetTrackedJob("job-payload")
+	if !found || job == nil {
+		t.Fatalf("expected tracked job")
+	}
+	if job.Detail != "" {
+		t.Fatalf("expected in-memory tracked job detail to be stripped, got=%q", job.Detail)
+	}
+	if job.Attempt != 2 {
+		t.Fatalf("unexpected attempt: %d", job.Attempt)
+	}
+	if len(job.Labels) != 0 {
+		t.Fatalf("expected in-memory labels to be stripped, got=%+v", job.Labels)
+	}
+	if len(job.Parameters) != 0 {
+		t.Fatalf("expected in-memory parameters to be stripped, got=%+v", job.Parameters)
+	}
+
+	detail, found, err := pluginSvc.BuildJobDetail("job-payload", 100, 0)
+	if err != nil {
+		t.Fatalf("BuildJobDetail: %v", err)
+	}
+	if !found || detail == nil || detail.Job == nil {
+		t.Fatalf("expected disk-backed job detail")
+	}
+	if detail.Job.Detail != "payload detail" {
+		t.Fatalf("unexpected disk-backed detail: %q", detail.Job.Detail)
+	}
+	if got := detail.Job.Labels["source"]; got != "detector" {
+		t.Fatalf("unexpected disk-backed label source: %q", got)
+	}
+	if got, ok := detail.Job.Parameters["volume_id"].(map[string]interface{}); !ok || got["int64_value"] != "9" {
+		t.Fatalf("unexpected disk-backed parameters payload: %#v", detail.Job.Parameters["volume_id"])
+	}
+}
+
+func TestTrackExecutionStartStoresErasureCodingExecutionPlan(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	taskParams := &worker_pb.TaskParams{
+		TaskId:     "task-ec-1",
+		VolumeId:   29,
+		Collection: "photos",
+		Sources: []*worker_pb.TaskSource{
+			{
+				Node:       "source-a:8080",
+				DataCenter: "dc1",
+				Rack:       "rack1",
+				VolumeId:   29,
+				DiskId:     5,
+			},
+		},
+		Targets: []*worker_pb.TaskTarget{
+			{
+				Node:       "target-a:8080",
+				DataCenter: "dc1",
+				Rack:       "rack2",
+				VolumeId:   29,
+				DiskId:     2,
+				ShardIds:   []uint32{0, 10},
+			},
+			{
+				Node:       "target-b:8080",
+				DataCenter: "dc2",
+				Rack:       "rack3",
+				VolumeId:   29,
+				DiskId:     3,
+				ShardIds:   []uint32{1, 11},
+			},
+		},
+		TaskParams: &worker_pb.TaskParams_ErasureCodingParams{
+			ErasureCodingParams: &worker_pb.ErasureCodingTaskParams{
+				DataShards:   10,
+				ParityShards: 4,
+			},
+		},
+	}
+	payload, err := proto.Marshal(taskParams)
+	if err != nil {
+		t.Fatalf("Marshal task params: %v", err)
+	}
+
+	pluginSvc.trackExecutionStart("req-ec-plan", "worker-ec", &plugin_pb.JobSpec{
+		JobId:   "job-ec-plan",
+		JobType: "erasure_coding",
+		Parameters: map[string]*plugin_pb.ConfigValue{
+			"task_params_pb": {
+				Kind: &plugin_pb.ConfigValue_BytesValue{BytesValue: payload},
+			},
+		},
+	}, 1)
+	pluginSvc.Shutdown()
+
+	detail, found, err := pluginSvc.BuildJobDetail("job-ec-plan", 100, 0)
+	if err != nil {
+		t.Fatalf("BuildJobDetail: %v", err)
+	}
+	if !found || detail == nil || detail.Job == nil {
+		t.Fatalf("expected disk-backed detail")
+	}
+
+	rawPlan, ok := detail.Job.Parameters["execution_plan"]
+	if !ok {
+		t.Fatalf("expected execution_plan in parameters, got=%+v", detail.Job.Parameters)
+	}
+	plan, ok := rawPlan.(map[string]interface{})
+	if !ok {
+		t.Fatalf("unexpected execution_plan type: %T", rawPlan)
+	}
+	if plan["job_type"] != "erasure_coding" {
+		t.Fatalf("unexpected execution plan job type: %+v", plan["job_type"])
+	}
+	if plan["volume_id"] != float64(29) {
+		t.Fatalf("unexpected execution plan volume id: %+v", plan["volume_id"])
+	}
+	sourcesRaw, ok := plan["sources"].([]interface{})
+	if !ok || len(sourcesRaw) != 1 {
+		t.Fatalf("unexpected sources in execution plan: %+v", plan["sources"])
+	}
+	firstSource, ok := sourcesRaw[0].(map[string]interface{})
+	if !ok {
+		t.Fatalf("unexpected source payload: %+v", sourcesRaw[0])
+	}
+	if firstSource["disk_id"] != float64(5) {
+		t.Fatalf("unexpected source disk_id: %+v", firstSource["disk_id"])
+	}
+	targets, ok := plan["targets"].([]interface{})
+	if !ok || len(targets) != 2 {
+		t.Fatalf("unexpected targets in execution plan: %+v", plan["targets"])
+	}
+	firstTarget, ok := targets[0].(map[string]interface{})
+	if !ok {
+		t.Fatalf("unexpected target payload: %+v", targets[0])
+	}
+	if firstTarget["disk_id"] != float64(2) {
+		t.Fatalf("unexpected target disk_id: %+v", firstTarget["disk_id"])
+	}
+	assignments, ok := plan["shard_assignments"].([]interface{})
+	if !ok || len(assignments) != 4 {
+		t.Fatalf("unexpected shard assignments in execution plan: %+v", plan["shard_assignments"])
+	}
+	firstAssignment, ok := assignments[0].(map[string]interface{})
+	if !ok {
+		t.Fatalf("unexpected first assignment payload: %+v", assignments[0])
+	}
+	if firstAssignment["shard_id"] != float64(0) || firstAssignment["kind"] != "data" {
+		t.Fatalf("unexpected first assignment: %+v", firstAssignment)
+	}
+	if firstAssignment["target_disk_id"] != float64(2) {
+		t.Fatalf("unexpected first assignment target_disk_id: %+v", firstAssignment["target_disk_id"])
+	}
+	secondAssignment, ok := assignments[1].(map[string]interface{})
+	if !ok {
+		t.Fatalf("unexpected second assignment payload: %+v", assignments[1])
+	}
+	if secondAssignment["shard_id"] != float64(1) || secondAssignment["target_disk_id"] != float64(3) {
+		t.Fatalf("unexpected second assignment: %+v", secondAssignment)
+	}
+}
+
+func TestBuildJobDetailIncludesActivitiesAndRunRecord(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.trackExecutionStart("req-detail", "worker-z", &plugin_pb.JobSpec{
+		JobId:   "job-detail",
+		JobType: "vacuum",
+		Summary: "detail summary",
+	}, 1)
+	pluginSvc.handleJobProgressUpdate("worker-z", &plugin_pb.JobProgressUpdate{
+		RequestId: "req-detail",
+		JobId:     "job-detail",
+		JobType:   "vacuum",
+		State:     plugin_pb.JobState_JOB_STATE_RUNNING,
+		Stage:     "scan",
+		Message:   "scanning volume",
+	})
+	pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
+		RequestId: "req-detail",
+		JobId:     "job-detail",
+		JobType:   "vacuum",
+		Success:   true,
+		Result: &plugin_pb.JobResult{
+			Summary: "done",
+			OutputValues: map[string]*plugin_pb.ConfigValue{
+				"affected": {
+					Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 1},
+				},
+			},
+		},
+		CompletedAt: timestamppb.Now(),
+	})
+	pluginSvc.Shutdown()
+
+	detail, found, err := pluginSvc.BuildJobDetail("job-detail", 100, 5)
+	if err != nil {
+		t.Fatalf("BuildJobDetail error: %v", err)
+	}
+	if !found || detail == nil {
+		t.Fatalf("expected job detail")
+	}
+	if detail.Job == nil || detail.Job.JobID != "job-detail" {
+		t.Fatalf("unexpected job detail payload: %+v", detail.Job)
+	}
+	if detail.RunRecord == nil || detail.RunRecord.JobID != "job-detail" {
+		t.Fatalf("expected run record for job-detail, got=%+v", detail.RunRecord)
+	}
+	if len(detail.Activities) == 0 {
+		t.Fatalf("expected activity timeline entries")
+	}
+	if detail.Job.ResultOutputValues == nil {
+		t.Fatalf("expected result output values")
+	}
+}
+
+func TestBuildJobDetailLoadsFromDiskWhenMemoryCleared(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.trackExecutionStart("req-disk", "worker-d", &plugin_pb.JobSpec{
+		JobId:   "job-disk",
+		JobType: "vacuum",
+		Summary: "disk summary",
+		Detail:  "disk detail payload",
+	}, 1)
+	pluginSvc.Shutdown()
+
+	pluginSvc.jobsMu.Lock()
+	pluginSvc.jobs = map[string]*TrackedJob{}
+	pluginSvc.jobsMu.Unlock()
+	pluginSvc.activitiesMu.Lock()
+	pluginSvc.activities = nil
+	pluginSvc.activitiesMu.Unlock()
+
+	detail, found, err := pluginSvc.BuildJobDetail("job-disk", 100, 0)
+	if err != nil {
+		t.Fatalf("BuildJobDetail: %v", err)
+	}
+	if !found || detail == nil || detail.Job == nil {
+		t.Fatalf("expected detail from disk")
+	}
+	if detail.Job.Detail != "disk detail payload" {
+		t.Fatalf("unexpected disk detail payload: %q", detail.Job.Detail)
+	}
+}

weed/admin/plugin/plugin_scheduler_test.go

@@ -0,0 +1,994 @@
+package plugin
+
+import (
+	"context"
+	"fmt"
+	"math"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+	"google.golang.org/protobuf/types/known/timestamppb"
+)
+
+func TestLoadSchedulerPolicyUsesAdminConfig(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	err = pluginSvc.SaveJobTypeConfig(&plugin_pb.PersistedJobTypeConfig{
+		JobType: "vacuum",
+		AdminRuntime: &plugin_pb.AdminRuntimeConfig{
+			Enabled:                       true,
+			DetectionIntervalMinutes:      30,
+			DetectionTimeoutSeconds:       20,
+			MaxJobsPerDetection:           123,
+			GlobalExecutionConcurrency:    5,
+			PerWorkerExecutionConcurrency: 2,
+			RetryLimit:                    4,
+			RetryBackoffSeconds:           7,
+			JobTypeMaxRuntimeSeconds:      1800,
+		},
+	})
+	if err != nil {
+		t.Fatalf("SaveJobTypeConfig: %v", err)
+	}
+
+	policy, enabled, err := pluginSvc.loadSchedulerPolicy("vacuum")
+	if err != nil {
+		t.Fatalf("loadSchedulerPolicy: %v", err)
+	}
+	if !enabled {
+		t.Fatalf("expected enabled policy")
+	}
+	if policy.MaxResults != 123 {
+		t.Fatalf("unexpected max results: got=%d", policy.MaxResults)
+	}
+	if policy.ExecutionConcurrency != 5 {
+		t.Fatalf("unexpected global concurrency: got=%d", policy.ExecutionConcurrency)
+	}
+	if policy.PerWorkerConcurrency != 2 {
+		t.Fatalf("unexpected per-worker concurrency: got=%d", policy.PerWorkerConcurrency)
+	}
+	if policy.RetryLimit != 4 {
+		t.Fatalf("unexpected retry limit: got=%d", policy.RetryLimit)
+	}
+	if policy.JobTypeMaxRuntime != 30*time.Minute {
+		t.Fatalf("unexpected max runtime: got=%v", policy.JobTypeMaxRuntime)
+	}
+}
+
+func TestLoadSchedulerPolicyUsesDescriptorDefaultsWhenConfigMissing(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	err = pluginSvc.store.SaveDescriptor("ec", &plugin_pb.JobTypeDescriptor{
+		JobType: "ec",
+		AdminRuntimeDefaults: &plugin_pb.AdminRuntimeDefaults{
+			Enabled:                       true,
+			DetectionIntervalMinutes:      60,
+			DetectionTimeoutSeconds:       25,
+			MaxJobsPerDetection:           30,
+			GlobalExecutionConcurrency:    4,
+			PerWorkerExecutionConcurrency: 2,
+			RetryLimit:                    3,
+			RetryBackoffSeconds:           6,
+			JobTypeMaxRuntimeSeconds:      1200,
+		},
+	})
+	if err != nil {
+		t.Fatalf("SaveDescriptor: %v", err)
+	}
+
+	policy, enabled, err := pluginSvc.loadSchedulerPolicy("ec")
+	if err != nil {
+		t.Fatalf("loadSchedulerPolicy: %v", err)
+	}
+	if !enabled {
+		t.Fatalf("expected enabled policy from descriptor defaults")
+	}
+	if policy.MaxResults != 30 {
+		t.Fatalf("unexpected max results: got=%d", policy.MaxResults)
+	}
+	if policy.ExecutionConcurrency != 4 {
+		t.Fatalf("unexpected global concurrency: got=%d", policy.ExecutionConcurrency)
+	}
+	if policy.PerWorkerConcurrency != 2 {
+		t.Fatalf("unexpected per-worker concurrency: got=%d", policy.PerWorkerConcurrency)
+	}
+	if policy.JobTypeMaxRuntime != 20*time.Minute {
+		t.Fatalf("unexpected max runtime: got=%v", policy.JobTypeMaxRuntime)
+	}
+}
+
+func TestReserveScheduledExecutorRespectsPerWorkerLimit(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 4},
+		},
+	})
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-b",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 2},
+		},
+	})
+
+	policy := schedulerPolicy{
+		PerWorkerConcurrency:   1,
+		ExecutorReserveBackoff: time.Millisecond,
+	}
+
+	executor1, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
+	if err != nil {
+		t.Fatalf("reserve executor 1: %v", err)
+	}
+	defer release1()
+
+	executor2, release2, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
+	if err != nil {
+		t.Fatalf("reserve executor 2: %v", err)
+	}
+	defer release2()
+
+	if executor1.WorkerID == executor2.WorkerID {
+		t.Fatalf("expected different executors due per-worker limit, got same worker %s", executor1.WorkerID)
+	}
+}
+
+func TestFilterScheduledProposalsDedupe(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	proposals := []*plugin_pb.JobProposal{
+		{ProposalId: "p1", DedupeKey: "d1"},
+		{ProposalId: "p2", DedupeKey: "d1"}, // same dedupe key
+		{ProposalId: "p3", DedupeKey: "d3"},
+		{ProposalId: "p3"}, // fallback dedupe by proposal id
+		{ProposalId: "p4"},
+		{ProposalId: "p4"}, // same proposal id, no dedupe key
+	}
+
+	filtered := pluginSvc.filterScheduledProposals(proposals)
+	if len(filtered) != 4 {
+		t.Fatalf("unexpected filtered size: got=%d want=4", len(filtered))
+	}
+
+	filtered2 := pluginSvc.filterScheduledProposals(proposals)
+	if len(filtered2) != 4 {
+		t.Fatalf("expected second run dedupe to be per-run only, got=%d", len(filtered2))
+	}
+}
+
+func TestBuildScheduledJobSpecDoesNotReuseProposalID(t *testing.T) {
+	t.Parallel()
+
+	proposal := &plugin_pb.JobProposal{
+		ProposalId: "vacuum-2",
+		DedupeKey:  "vacuum:2",
+		JobType:    "vacuum",
+	}
+
+	jobA := buildScheduledJobSpec("vacuum", proposal, 0)
+	jobB := buildScheduledJobSpec("vacuum", proposal, 1)
+
+	if jobA.JobId == proposal.ProposalId {
+		t.Fatalf("scheduled job id must not reuse proposal id: %s", jobA.JobId)
+	}
+	if jobB.JobId == proposal.ProposalId {
+		t.Fatalf("scheduled job id must not reuse proposal id: %s", jobB.JobId)
+	}
+	if jobA.JobId == jobB.JobId {
+		t.Fatalf("scheduled job ids must be unique across jobs: %s", jobA.JobId)
+	}
+}
+
+func TestFilterProposalsWithActiveJobs(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.trackExecutionStart("req-1", "worker-a", &plugin_pb.JobSpec{
+		JobId:     "job-1",
+		JobType:   "vacuum",
+		DedupeKey: "vacuum:k1",
+	}, 1)
+	pluginSvc.trackExecutionStart("req-2", "worker-b", &plugin_pb.JobSpec{
+		JobId:   "job-2",
+		JobType: "vacuum",
+	}, 1)
+	pluginSvc.trackExecutionQueued(&plugin_pb.JobSpec{
+		JobId:     "job-3",
+		JobType:   "vacuum",
+		DedupeKey: "vacuum:k4",
+	})
+
+	filtered, skipped := pluginSvc.filterProposalsWithActiveJobs("vacuum", []*plugin_pb.JobProposal{
+		{ProposalId: "proposal-1", JobType: "vacuum", DedupeKey: "vacuum:k1"},
+		{ProposalId: "job-2", JobType: "vacuum"},
+		{ProposalId: "proposal-2b", JobType: "vacuum", DedupeKey: "vacuum:k4"},
+		{ProposalId: "proposal-3", JobType: "vacuum", DedupeKey: "vacuum:k3"},
+		{ProposalId: "proposal-4", JobType: "balance", DedupeKey: "balance:k1"},
+	})
+	if skipped != 3 {
+		t.Fatalf("unexpected skipped count: got=%d want=3", skipped)
+	}
+	if len(filtered) != 2 {
+		t.Fatalf("unexpected filtered size: got=%d want=2", len(filtered))
+	}
+	if filtered[0].ProposalId != "proposal-3" || filtered[1].ProposalId != "proposal-4" {
+		t.Fatalf("unexpected filtered proposals: got=%s,%s", filtered[0].ProposalId, filtered[1].ProposalId)
+	}
+}
+
+func TestReserveScheduledExecutorTimesOutWhenNoExecutor(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	policy := schedulerPolicy{
+		ExecutionTimeout:       30 * time.Millisecond,
+		ExecutorReserveBackoff: 5 * time.Millisecond,
+		PerWorkerConcurrency:   1,
+	}
+
+	start := time.Now()
+	pluginSvc.Shutdown()
+	_, _, err = pluginSvc.reserveScheduledExecutor(context.Background(), "missing-job-type", policy)
+	if err == nil {
+		t.Fatalf("expected reservation shutdown error")
+	}
+	if time.Since(start) > 50*time.Millisecond {
+		t.Fatalf("reservation returned too late after shutdown: duration=%v", time.Since(start))
+	}
+}
+
+func TestReserveScheduledExecutorWaitsForWorkerCapacity(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 1},
+		},
+	})
+
+	policy := schedulerPolicy{
+		ExecutionTimeout:       time.Second,
+		PerWorkerConcurrency:   8,
+		ExecutorReserveBackoff: 5 * time.Millisecond,
+	}
+
+	_, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
+	if err != nil {
+		t.Fatalf("reserve executor 1: %v", err)
+	}
+	defer release1()
+
+	type reserveResult struct {
+		err error
+	}
+	secondReserveCh := make(chan reserveResult, 1)
+	go func() {
+		_, release2, reserveErr := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
+		if release2 != nil {
+			release2()
+		}
+		secondReserveCh <- reserveResult{err: reserveErr}
+	}()
+
+	select {
+	case result := <-secondReserveCh:
+		t.Fatalf("expected second reservation to wait for capacity, got=%v", result.err)
+	case <-time.After(25 * time.Millisecond):
+		// Expected: still waiting.
+	}
+
+	release1()
+
+	select {
+	case result := <-secondReserveCh:
+		if result.err != nil {
+			t.Fatalf("second reservation error: %v", result.err)
+		}
+	case <-time.After(200 * time.Millisecond):
+		t.Fatalf("second reservation did not acquire after capacity release")
+	}
+}
+
+func TestShouldSkipDetectionForWaitingJobs(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	policy := schedulerPolicy{
+		ExecutionConcurrency: 2,
+		MaxResults:           100,
+	}
+	threshold := waitingBacklogThreshold(policy)
+	if threshold <= 0 {
+		t.Fatalf("expected positive waiting threshold")
+	}
+
+	for i := 0; i < threshold; i++ {
+		pluginSvc.trackExecutionQueued(&plugin_pb.JobSpec{
+			JobId:     fmt.Sprintf("job-waiting-%d", i),
+			JobType:   "vacuum",
+			DedupeKey: fmt.Sprintf("vacuum:%d", i),
+		})
+	}
+
+	skip, waitingCount, waitingThreshold := pluginSvc.shouldSkipDetectionForWaitingJobs("vacuum", policy)
+	if !skip {
+		t.Fatalf("expected detection to skip when waiting backlog reaches threshold")
+	}
+	if waitingCount != threshold {
+		t.Fatalf("unexpected waiting count: got=%d want=%d", waitingCount, threshold)
+	}
+	if waitingThreshold != threshold {
+		t.Fatalf("unexpected waiting threshold: got=%d want=%d", waitingThreshold, threshold)
+	}
+}
+
+func TestWaitingBacklogThresholdHonorsMaxResultsCap(t *testing.T) {
+	t.Parallel()
+
+	policy := schedulerPolicy{
+		ExecutionConcurrency: 8,
+		MaxResults:           6,
+	}
+	threshold := waitingBacklogThreshold(policy)
+	if threshold != 6 {
+		t.Fatalf("expected threshold to be capped by max results, got=%d", threshold)
+	}
+}
+
+func TestListSchedulerStatesIncludesPolicyAndState(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const jobType = "vacuum"
+	err = pluginSvc.SaveJobTypeConfig(&plugin_pb.PersistedJobTypeConfig{
+		JobType: jobType,
+		AdminRuntime: &plugin_pb.AdminRuntimeConfig{
+			Enabled:                       true,
+			DetectionIntervalMinutes:      45,
+			DetectionTimeoutSeconds:       30,
+			MaxJobsPerDetection:           80,
+			GlobalExecutionConcurrency:    3,
+			PerWorkerExecutionConcurrency: 2,
+			RetryLimit:                    1,
+			RetryBackoffSeconds:           9,
+			JobTypeMaxRuntimeSeconds:      900,
+		},
+	})
+	if err != nil {
+		t.Fatalf("SaveJobTypeConfig: %v", err)
+	}
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanDetect: true, CanExecute: true},
+		},
+	})
+
+	nextDetectionAt := time.Now().UTC().Add(2 * time.Minute).Round(time.Second)
+	pluginSvc.schedulerMu.Lock()
+	pluginSvc.nextDetectionAt[jobType] = nextDetectionAt
+	pluginSvc.detectionInFlight[jobType] = true
+	pluginSvc.schedulerMu.Unlock()
+
+	states, err := pluginSvc.ListSchedulerStates()
+	if err != nil {
+		t.Fatalf("ListSchedulerStates: %v", err)
+	}
+
+	state := findSchedulerState(states, jobType)
+	if state == nil {
+		t.Fatalf("missing scheduler state for %s", jobType)
+	}
+	if !state.Enabled {
+		t.Fatalf("expected enabled scheduler state")
+	}
+	if state.PolicyError != "" {
+		t.Fatalf("unexpected policy error: %s", state.PolicyError)
+	}
+	if !state.DetectionInFlight {
+		t.Fatalf("expected detection in flight")
+	}
+	if state.NextDetectionAt == nil {
+		t.Fatalf("expected next detection time")
+	}
+	if state.NextDetectionAt.Unix() != nextDetectionAt.Unix() {
+		t.Fatalf("unexpected next detection time: got=%v want=%v", state.NextDetectionAt, nextDetectionAt)
+	}
+	if state.DetectionIntervalMinutes != 45 {
+		t.Fatalf("unexpected detection interval: got=%d", state.DetectionIntervalMinutes)
+	}
+	if state.DetectionTimeoutSeconds != 30 {
+		t.Fatalf("unexpected detection timeout: got=%d", state.DetectionTimeoutSeconds)
+	}
+	if state.ExecutionTimeoutSeconds != 90 {
+		t.Fatalf("unexpected execution timeout: got=%d", state.ExecutionTimeoutSeconds)
+	}
+	if state.JobTypeMaxRuntimeSeconds != 900 {
+		t.Fatalf("unexpected job type max runtime: got=%d", state.JobTypeMaxRuntimeSeconds)
+	}
+	if state.MaxJobsPerDetection != 80 {
+		t.Fatalf("unexpected max jobs per detection: got=%d", state.MaxJobsPerDetection)
+	}
+	if state.GlobalExecutionConcurrency != 3 {
+		t.Fatalf("unexpected global execution concurrency: got=%d", state.GlobalExecutionConcurrency)
+	}
+	if state.PerWorkerExecutionConcurrency != 2 {
+		t.Fatalf("unexpected per worker execution concurrency: got=%d", state.PerWorkerExecutionConcurrency)
+	}
+	if state.RetryLimit != 1 {
+		t.Fatalf("unexpected retry limit: got=%d", state.RetryLimit)
+	}
+	if state.RetryBackoffSeconds != 9 {
+		t.Fatalf("unexpected retry backoff: got=%d", state.RetryBackoffSeconds)
+	}
+	if !state.DetectorAvailable || state.DetectorWorkerID != "worker-a" {
+		t.Fatalf("unexpected detector assignment: available=%v worker=%s", state.DetectorAvailable, state.DetectorWorkerID)
+	}
+	if state.ExecutorWorkerCount != 1 {
+		t.Fatalf("unexpected executor worker count: got=%d", state.ExecutorWorkerCount)
+	}
+}
+
+func TestListSchedulerStatesShowsDisabledWhenNoPolicy(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const jobType = "balance"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-b",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanDetect: true, CanExecute: true},
+		},
+	})
+
+	states, err := pluginSvc.ListSchedulerStates()
+	if err != nil {
+		t.Fatalf("ListSchedulerStates: %v", err)
+	}
+
+	state := findSchedulerState(states, jobType)
+	if state == nil {
+		t.Fatalf("missing scheduler state for %s", jobType)
+	}
+	if state.Enabled {
+		t.Fatalf("expected disabled scheduler state")
+	}
+	if state.PolicyError != "" {
+		t.Fatalf("unexpected policy error: %s", state.PolicyError)
+	}
+	if !state.DetectorAvailable || state.DetectorWorkerID != "worker-b" {
+		t.Fatalf("unexpected detector details: available=%v worker=%s", state.DetectorAvailable, state.DetectorWorkerID)
+	}
+	if state.ExecutorWorkerCount != 1 {
+		t.Fatalf("unexpected executor worker count: got=%d", state.ExecutorWorkerCount)
+	}
+}
+
+func findSchedulerState(states []SchedulerJobTypeState, jobType string) *SchedulerJobTypeState {
+	for i := range states {
+		if states[i].JobType == jobType {
+			return &states[i]
+		}
+	}
+	return nil
+}
+
+func TestPickDetectorPrefersLeasedWorker(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true},
+		},
+	})
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-b",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true},
+		},
+	})
+
+	pluginSvc.setDetectorLease("vacuum", "worker-b")
+
+	detector, err := pluginSvc.pickDetector("vacuum")
+	if err != nil {
+		t.Fatalf("pickDetector: %v", err)
+	}
+	if detector.WorkerID != "worker-b" {
+		t.Fatalf("expected leased detector worker-b, got=%s", detector.WorkerID)
+	}
+}
+
+func TestPickDetectorReassignsWhenLeaseIsStale(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true},
+		},
+	})
+	pluginSvc.setDetectorLease("vacuum", "worker-stale")
+
+	detector, err := pluginSvc.pickDetector("vacuum")
+	if err != nil {
+		t.Fatalf("pickDetector: %v", err)
+	}
+	if detector.WorkerID != "worker-a" {
+		t.Fatalf("expected reassigned detector worker-a, got=%s", detector.WorkerID)
+	}
+
+	lease := pluginSvc.getDetectorLease("vacuum")
+	if lease != "worker-a" {
+		t.Fatalf("expected detector lease to be updated to worker-a, got=%s", lease)
+	}
+}
+
+// trackingLockManager records whether Acquire was called and how many times.
+type trackingLockManager struct {
+	mu       sync.Mutex
+	acquired int
+}
+
+func (m *trackingLockManager) Acquire(reason string) (func(), error) {
+	m.mu.Lock()
+	m.acquired++
+	m.mu.Unlock()
+	return func() {}, nil
+}
+
+func (m *trackingLockManager) count() int {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	return m.acquired
+}
+
+func TestRunLaneSchedulerIterationLockBehavior(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name     string
+		lane     SchedulerLane
+		jobType  string
+		wantLock bool
+	}{
+		{"Default", LaneDefault, "vacuum", true},
+		{"Iceberg", LaneIceberg, "iceberg_maintenance", false},
+		{"Lifecycle", LaneLifecycle, "s3_lifecycle", false},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			lm := &trackingLockManager{}
+			// Construct without a cluster-context provider so the background
+			// lane loops do not start: they call runLaneSchedulerIteration on
+			// the same lane and would race this test's own manual call,
+			// consuming the due job before it observes the lock. The provider
+			// is set afterward so the manual iteration can still detect.
+			pluginSvc, err := New(Options{LockManager: lm})
+			if err != nil {
+				t.Fatalf("New: %v", err)
+			}
+			defer pluginSvc.Shutdown()
+			pluginSvc.clusterContextProvider = func(context.Context) (*plugin_pb.ClusterContext, error) {
+				return &plugin_pb.ClusterContext{}, nil
+			}
+
+			// Register a detectable worker for the job type.
+			pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+				WorkerId: "worker-a",
+				Capabilities: []*plugin_pb.JobTypeCapability{
+					{JobType: tt.jobType, CanDetect: true},
+				},
+			})
+
+			// Enable the job type so the scheduler picks it up.
+			err = pluginSvc.SaveJobTypeConfig(&plugin_pb.PersistedJobTypeConfig{
+				JobType: tt.jobType,
+				AdminRuntime: &plugin_pb.AdminRuntimeConfig{
+					Enabled:                  true,
+					DetectionIntervalMinutes: 1,
+				},
+			})
+			if err != nil {
+				t.Fatalf("SaveJobTypeConfig: %v", err)
+			}
+
+			// Make the job type due immediately so the iteration reaches
+			// detection; the lock is only taken around due work now.
+			pluginSvc.schedulerMu.Lock()
+			pluginSvc.nextDetectionAt[tt.jobType] = time.Now().UTC().Add(-time.Second)
+			pluginSvc.schedulerMu.Unlock()
+
+			ls := pluginSvc.lanes[tt.lane]
+			pluginSvc.runLaneSchedulerIteration(ls)
+
+			if got := lm.count(); (got > 0) != tt.wantLock {
+				t.Errorf("lock acquired %d times, wantLock=%v", got, tt.wantLock)
+			}
+		})
+	}
+}
+
+func TestDispatchScheduledProposalsLocksPerJob(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name      string
+		jobType   string
+		wantLocks int
+	}{
+		{"DefaultLaneLocksEachJob", "volume_balance", 3},
+		{"LifecycleLaneNeedsNoLock", "s3_lifecycle", 0},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			lm := &trackingLockManager{}
+			pluginSvc, err := New(Options{LockManager: lm})
+			if err != nil {
+				t.Fatalf("New: %v", err)
+			}
+			defer pluginSvc.Shutdown()
+
+			// The worker has capabilities but no connected session, so each
+			// job reserves capacity, fails to send, and moves on.
+			pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+				WorkerId: "worker-a",
+				Capabilities: []*plugin_pb.JobTypeCapability{
+					{JobType: tt.jobType, CanExecute: true, MaxExecutionConcurrency: 4},
+				},
+			})
+
+			policy := schedulerPolicy{
+				ExecutionConcurrency:   1,
+				PerWorkerConcurrency:   1,
+				ExecutionTimeout:       time.Second,
+				ExecutorReserveBackoff: time.Millisecond,
+			}
+			proposals := []*plugin_pb.JobProposal{
+				{ProposalId: "p1", JobType: tt.jobType, DedupeKey: "k1"},
+				{ProposalId: "p2", JobType: tt.jobType, DedupeKey: "k2"},
+				{ProposalId: "p3", JobType: tt.jobType, DedupeKey: "k3"},
+			}
+
+			success, errors, canceled := pluginSvc.dispatchScheduledProposals(
+				context.Background(), tt.jobType, proposals, &plugin_pb.ClusterContext{}, policy)
+			if success != 0 || canceled != 0 || errors != 3 {
+				t.Fatalf("unexpected dispatch counts: success=%d errors=%d canceled=%d", success, errors, canceled)
+			}
+			if got := lm.count(); got != tt.wantLocks {
+				t.Fatalf("lock acquired %d times, want %d", got, tt.wantLocks)
+			}
+		})
+	}
+}
+
+func TestScheduledAttemptContextDrainGrace(t *testing.T) {
+	t.Parallel()
+
+	assertDeadline := func(t *testing.T, ctx context.Context, want time.Time) {
+		t.Helper()
+		got, ok := ctx.Deadline()
+		if !ok {
+			t.Fatal("expected attempt context to carry a deadline")
+		}
+		if diff := got.Sub(want); diff < -5*time.Second || diff > 5*time.Second {
+			t.Fatalf("unexpected attempt deadline: got=%v want~%v", got, want)
+		}
+	}
+
+	windowEnd := time.Now().Add(time.Minute)
+	window, cancelWindow := context.WithDeadline(context.Background(), windowEnd)
+	defer cancelWindow()
+
+	// Fits inside the window: keeps its own deadline.
+	ctx, cancel := scheduledAttemptContext(window, 30*time.Second, 0)
+	assertDeadline(t, ctx, time.Now().Add(30*time.Second))
+	cancel()
+
+	// Longer: capped at window close plus the grace.
+	ctx, cancel = scheduledAttemptContext(window, 3*time.Hour, 0)
+	assertDeadline(t, ctx, windowEnd.Add(scheduledExecutionDrainGrace))
+	cancel()
+
+	// Estimated runtime: keeps its full deadline.
+	ctx, cancel = scheduledAttemptContext(window, 3*time.Hour, 3*time.Hour)
+	assertDeadline(t, ctx, time.Now().Add(3*time.Hour))
+	cancel()
+
+	// An estimate below the timeout floors the drain cap.
+	ctx, cancel = scheduledAttemptContext(window, 3*time.Hour, 45*time.Minute)
+	assertDeadline(t, ctx, time.Now().Add(45*time.Minute))
+	cancel()
+
+	// No window deadline: the timeout stands alone.
+	ctx, cancel = scheduledAttemptContext(context.Background(), 2*time.Hour, 0)
+	assertDeadline(t, ctx, time.Now().Add(2*time.Hour))
+	cancel()
+}
+
+func TestScheduledEstimatedRuntime(t *testing.T) {
+	t.Parallel()
+
+	estimate := func(v int64) map[string]*plugin_pb.ConfigValue {
+		return map[string]*plugin_pb.ConfigValue{
+			"estimated_runtime_seconds": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: v}},
+		}
+	}
+
+	if got := scheduledEstimatedRuntime(nil); got != 0 {
+		t.Fatalf("nil parameters: got=%v want=0", got)
+	}
+	if got := scheduledEstimatedRuntime(estimate(-5)); got != 0 {
+		t.Fatalf("negative estimate: got=%v want=0", got)
+	}
+	if got := scheduledEstimatedRuntime(estimate(600)); got != 10*time.Minute {
+		t.Fatalf("normal estimate: got=%v want=10m", got)
+	}
+	// Oversized seconds cap before the Duration conversion can overflow.
+	if got := scheduledEstimatedRuntime(estimate(math.MaxInt64)); got != maxEstimatedRuntimeCap {
+		t.Fatalf("oversized estimate: got=%v want=%v", got, maxEstimatedRuntimeCap)
+	}
+}
+
+func TestDispatchScheduledProposalsDrainsStartedJobOnWindowClose(t *testing.T) {
+	t.Parallel()
+
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const workerID = "worker-drain"
+	const jobType = "s3_lifecycle" // lifecycle lane: no admin lock needed
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: workerID,
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanExecute: true, MaxExecutionConcurrency: 1},
+		},
+	})
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 8), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+
+	windowCtx, closeWindow := context.WithCancel(context.Background())
+	defer closeWindow()
+
+	policy := schedulerPolicy{
+		ExecutionConcurrency:   1,
+		PerWorkerConcurrency:   1,
+		ExecutionTimeout:       time.Hour,
+		ExecutorReserveBackoff: time.Millisecond,
+	}
+	proposals := []*plugin_pb.JobProposal{
+		{ProposalId: "p1", JobType: jobType, DedupeKey: "k1"},
+		{ProposalId: "p2", JobType: jobType, DedupeKey: "k2"},
+	}
+
+	resultCh := make(chan [3]int, 1)
+	go func() {
+		success, errCount, canceled := pluginSvc.dispatchScheduledProposals(
+			windowCtx, jobType, proposals, &plugin_pb.ClusterContext{}, policy)
+		resultCh <- [3]int{success, errCount, canceled}
+	}()
+
+	first := <-session.outgoing
+	execReq := first.GetExecuteJobRequest()
+	if execReq == nil {
+		t.Fatalf("expected execute_job_request, got %+v", first)
+	}
+
+	// Close the window mid-job: the job must drain to completion while the
+	// still-queued second proposal is canceled.
+	closeWindow()
+	pluginSvc.handleJobCompleted(&plugin_pb.JobCompleted{
+		RequestId:   first.RequestId,
+		JobId:       execReq.Job.JobId,
+		JobType:     jobType,
+		Success:     true,
+		CompletedAt: timestamppb.Now(),
+	})
+
+	result := <-resultCh
+	if result[0] != 1 || result[1] != 0 || result[2] != 1 {
+		t.Fatalf("unexpected dispatch counts: success=%d errors=%d canceled=%d", result[0], result[1], result[2])
+	}
+
+	select {
+	case unexpected := <-session.outgoing:
+		t.Fatalf("expected no further worker messages (no cancel for the draining job), got %+v", unexpected)
+	default:
+	}
+}
+
+// ---------- lane-scoped prune ----------
+
+func TestPruneSchedulerState_DefaultLaneKeepsForeignLanesAndPrunesOwnStale(t *testing.T) {
+	p, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer p.Shutdown()
+
+	now := time.Now().UTC()
+	p.schedulerMu.Lock()
+	p.nextDetectionAt["s3_lifecycle"] = now.Add(24 * time.Hour) // lifecycle lane
+	p.nextDetectionAt["vacuum"] = now.Add(time.Minute)          // default lane, active
+	p.nextDetectionAt["ec_balance"] = now.Add(time.Minute)      // default lane, stale
+	p.detectionInFlight["ec_balance"] = true
+	p.schedulerMu.Unlock()
+
+	// Default-lane iteration prunes with only its own active job types.
+	p.pruneSchedulerState(LaneDefault, map[string]struct{}{"vacuum": {}})
+
+	p.schedulerMu.Lock()
+	defer p.schedulerMu.Unlock()
+	if _, ok := p.nextDetectionAt["s3_lifecycle"]; !ok {
+		t.Fatal("default-lane prune must not delete lifecycle-lane nextDetectionAt[s3_lifecycle]")
+	}
+	if _, ok := p.nextDetectionAt["vacuum"]; !ok {
+		t.Fatal("active default-lane job (vacuum) must be kept")
+	}
+	if _, ok := p.nextDetectionAt["ec_balance"]; ok {
+		t.Fatal("stale default-lane job (ec_balance) must still be pruned within its own lane")
+	}
+	if _, ok := p.detectionInFlight["ec_balance"]; ok {
+		t.Fatal("pruned job must also drop its detectionInFlight entry")
+	}
+}
+
+func TestPruneSchedulerState_LifecycleLaneLeavesDefaultLane(t *testing.T) {
+	p, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer p.Shutdown()
+
+	now := time.Now().UTC()
+	p.schedulerMu.Lock()
+	p.nextDetectionAt["vacuum"] = now.Add(time.Minute)          // default lane
+	p.nextDetectionAt["s3_lifecycle"] = now.Add(24 * time.Hour) // lifecycle lane, active
+	p.schedulerMu.Unlock()
+
+	p.pruneSchedulerState(LaneLifecycle, map[string]struct{}{"s3_lifecycle": {}})
+
+	p.schedulerMu.Lock()
+	defer p.schedulerMu.Unlock()
+	if _, ok := p.nextDetectionAt["vacuum"]; !ok {
+		t.Fatal("lifecycle-lane prune must not delete default-lane nextDetectionAt[vacuum]")
+	}
+	if _, ok := p.nextDetectionAt["s3_lifecycle"]; !ok {
+		t.Fatal("active lifecycle job (s3_lifecycle) must be kept")
+	}
+}
+
+func TestPruneDetectorLeases_IsLaneScoped(t *testing.T) {
+	p, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer p.Shutdown()
+
+	p.detectorLeaseMu.Lock()
+	p.detectorLeases["s3_lifecycle"] = "worker-a" // lifecycle lane
+	p.detectorLeases["vacuum"] = "worker-b"       // default lane, active
+	p.detectorLeases["ec_balance"] = "worker-c"   // default lane, stale
+	p.detectorLeaseMu.Unlock()
+
+	p.pruneDetectorLeases(LaneDefault, map[string]struct{}{"vacuum": {}})
+
+	p.detectorLeaseMu.Lock()
+	defer p.detectorLeaseMu.Unlock()
+	if _, ok := p.detectorLeases["s3_lifecycle"]; !ok {
+		t.Fatal("default-lane prune must not delete lifecycle-lane detector lease")
+	}
+	if _, ok := p.detectorLeases["vacuum"]; !ok {
+		t.Fatal("active default-lane detector lease (vacuum) must be kept")
+	}
+	if _, ok := p.detectorLeases["ec_balance"]; ok {
+		t.Fatal("stale default-lane detector lease (ec_balance) must still be pruned within its own lane")
+	}
+}
+
+func TestLaneStatus_LifecycleNextDetectionSurvivesDefaultLanePrune(t *testing.T) {
+	p, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer p.Shutdown()
+
+	now := time.Now().UTC()
+	expected := now.Add(24 * time.Hour)
+	p.schedulerMu.Lock()
+	p.nextDetectionAt["s3_lifecycle"] = expected
+	p.nextDetectionAt["vacuum"] = now.Add(time.Minute)
+	p.schedulerMu.Unlock()
+
+	p.pruneSchedulerState(LaneDefault, map[string]struct{}{"vacuum": {}})
+
+	status := p.GetLaneSchedulerStatus(LaneLifecycle)
+	if status.NextDetectionAt == nil {
+		t.Fatal("lifecycle lane status lost next_detection_at after a default-lane prune")
+	}
+	if !status.NextDetectionAt.Equal(expected) {
+		t.Fatalf("next_detection_at = %v, want %v (must be the 24h schedule, not the idle-sleep fallback)",
+			status.NextDetectionAt, expected)
+	}
+}

weed/admin/plugin/plugin_schema_prefetch.go

@@ -0,0 +1,66 @@
+package plugin
+
+import (
+	"context"
+	"sort"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+const descriptorPrefetchTimeout = 20 * time.Second
+
+func (r *Plugin) prefetchDescriptorsFromHello(hello *plugin_pb.WorkerHello) {
+	if hello == nil || len(hello.Capabilities) == 0 {
+		return
+	}
+
+	jobTypeSet := make(map[string]struct{})
+	for _, capability := range hello.Capabilities {
+		if capability == nil || capability.JobType == "" {
+			continue
+		}
+		if !capability.CanDetect && !capability.CanExecute {
+			continue
+		}
+		jobTypeSet[capability.JobType] = struct{}{}
+	}
+
+	if len(jobTypeSet) == 0 {
+		return
+	}
+
+	jobTypes := make([]string, 0, len(jobTypeSet))
+	for jobType := range jobTypeSet {
+		jobTypes = append(jobTypes, jobType)
+	}
+	sort.Strings(jobTypes)
+
+	for _, jobType := range jobTypes {
+		select {
+		case <-r.shutdownCh:
+			return
+		default:
+		}
+
+		descriptor, err := r.store.LoadDescriptor(jobType)
+		if err != nil {
+			glog.Warningf("Plugin descriptor prefetch check failed for %s: %v", jobType, err)
+			continue
+		}
+		if descriptor != nil {
+			continue
+		}
+
+		ctx, cancel := context.WithTimeout(r.ctx, descriptorPrefetchTimeout)
+		_, err = r.RequestConfigSchema(ctx, jobType, false)
+		cancel()
+		if err != nil {
+			glog.V(1).Infof("Plugin descriptor prefetch skipped for %s: %v", jobType, err)
+			continue
+		}
+
+		glog.V(1).Infof("Plugin descriptor prefetched for job_type=%s", jobType)
+	}
+}

weed/admin/plugin/preview.go

@@ -0,0 +1,105 @@
+package plugin
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+	"google.golang.org/protobuf/types/known/timestamppb"
+)
+
+// maxPreviewRows caps what admin will ask for. A preview is a look at the
+// object, not an export, and the rows cross the control stream.
+const maxPreviewRows = 200
+
+// RequestObjectPreview asks the worker that last described an object for sample
+// rows of it, and waits for the answer.
+//
+// The worker is chosen from the observation store rather than by job type: the
+// one that last looked at this object is the one that can read it, and it says
+// so by having reported. That also means a preview is only available once
+// detection has run, which the caller should surface rather than hide.
+func (r *Plugin) RequestObjectPreview(ctx context.Context, objectID []string, format string, rowLimit int) (*plugin_pb.ObjectPreviewResponse, error) {
+	if len(objectID) == 0 {
+		return nil, fmt.Errorf("preview needs an object id")
+	}
+	// The conversion happens only inside the bound check, so what reaches the
+	// wire cannot depend on int being wider than int32 here.
+	requestedRows := int32(maxPreviewRows)
+	if rowLimit >= 1 && rowLimit <= maxPreviewRows {
+		requestedRows = int32(rowLimit)
+	}
+
+	observed, ok := r.observations.GetFormat(objectID, format)
+	if !ok {
+		return nil, fmt.Errorf("no worker has described this object as %s yet", format)
+	}
+	if _, connected := r.registry.Get(observed.WorkerID); !connected {
+		return nil, fmt.Errorf("worker %s is not connected", observed.WorkerID)
+	}
+
+	requestID, err := newRequestID("preview")
+	if err != nil {
+		return nil, err
+	}
+
+	responseCh := make(chan *plugin_pb.ObjectPreviewResponse, 1)
+	r.pendingPreviewMu.Lock()
+	r.pendingPreview[requestID] = responseCh
+	r.pendingPreviewMu.Unlock()
+	defer func() {
+		r.pendingPreviewMu.Lock()
+		delete(r.pendingPreview, requestID)
+		r.pendingPreviewMu.Unlock()
+	}()
+
+	request := &plugin_pb.AdminToWorkerMessage{
+		RequestId: requestID,
+		SentAt:    timestamppb.Now(),
+		Body: &plugin_pb.AdminToWorkerMessage_RequestObjectPreview{
+			RequestObjectPreview: &plugin_pb.RequestObjectPreview{
+				ObjectId: objectID,
+				Format:   format,
+				RowLimit: requestedRows,
+			},
+		},
+	}
+	if err := r.sendToWorker(observed.WorkerID, request); err != nil {
+		return nil, err
+	}
+
+	select {
+	case <-ctx.Done():
+		return nil, ctx.Err()
+	case response, ok := <-responseCh:
+		if !ok {
+			return nil, fmt.Errorf("preview request %s interrupted", requestID)
+		}
+		if response == nil {
+			return nil, fmt.Errorf("preview request %s returned nothing", requestID)
+		}
+		if !response.Success {
+			return nil, fmt.Errorf("worker %s could not preview the object: %s", observed.WorkerID, response.ErrorMessage)
+		}
+		return response, nil
+	}
+}
+
+// handleObjectPreviewResponse routes one reply back to whoever is waiting.
+func (r *Plugin) handleObjectPreviewResponse(response *plugin_pb.ObjectPreviewResponse) {
+	if response == nil {
+		return
+	}
+	// Held across the send: Shutdown closes these channels under the same lock,
+	// and a send that raced it would panic on a closed channel.
+	r.pendingPreviewMu.Lock()
+	defer r.pendingPreviewMu.Unlock()
+	ch := r.pendingPreview[response.RequestId]
+	if ch == nil {
+		return
+	}
+	select {
+	case ch <- response:
+	default:
+	}
+}

weed/admin/plugin/preview_test.go

@@ -0,0 +1,202 @@
+package plugin
+
+import (
+	"context"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+	"google.golang.org/protobuf/types/known/timestamppb"
+)
+
+func previewPlugin(t *testing.T, workerID string) (*Plugin, *streamSession) {
+	t.Helper()
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New plugin error: %v", err)
+	}
+	t.Cleanup(pluginSvc.Shutdown)
+
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{WorkerId: workerID})
+	session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 4), done: make(chan struct{})}
+	pluginSvc.putSession(session)
+	return pluginSvc, session
+}
+
+func observeObject(p *Plugin, workerID string, objectID []string) {
+	p.observations.Record(workerID, &plugin_pb.WorkerObservations{
+		JobType: "lance_compact",
+		Observations: []*plugin_pb.ObjectObservation{{
+			ObjectId:   objectID,
+			Format:     "LANCE",
+			ObservedAt: timestamppb.Now(),
+		}},
+	})
+}
+
+// The round trip the details page makes: ask the worker that described this
+// object, and render what it sends back.
+func TestRequestObjectPreviewReturnsTheWorkersRows(t *testing.T) {
+	t.Parallel()
+	const workerID = "lance-worker-1"
+	objectID := []string{"vectors", "ml", "embeddings"}
+	pluginSvc, session := previewPlugin(t, workerID)
+	observeObject(pluginSvc, workerID, objectID)
+
+	type result struct {
+		response *plugin_pb.ObjectPreviewResponse
+		err      error
+	}
+	results := make(chan result, 1)
+	go func() {
+		response, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 2)
+		results <- result{response, err}
+	}()
+
+	var request *plugin_pb.AdminToWorkerMessage
+	select {
+	case request = <-session.outgoing:
+	case <-time.After(2 * time.Second):
+		t.Fatal("no preview request reached the worker")
+	}
+	asked := request.GetRequestObjectPreview()
+	if asked == nil {
+		t.Fatalf("expected a preview request, got %T", request.Body)
+	}
+	if asked.RowLimit != 2 || asked.Format != "LANCE" {
+		t.Fatalf("request lost its parameters: %+v", asked)
+	}
+
+	pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{
+		WorkerId: workerID,
+		Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{
+			ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{
+				RequestId: request.RequestId,
+				Success:   true,
+				Columns:   []string{"id", "vec"},
+				Rows:      []*plugin_pb.PreviewRow{{Values: []string{"1", "[0.1, 0.2]"}}},
+				TotalRows: 1024,
+			},
+		},
+	})
+
+	select {
+	case got := <-results:
+		if got.err != nil {
+			t.Fatalf("preview failed: %v", got.err)
+		}
+		if len(got.response.Rows) != 1 || got.response.Rows[0].Values[1] != "[0.1, 0.2]" {
+			t.Fatalf("rows did not survive the round trip: %+v", got.response.Rows)
+		}
+		if got.response.TotalRows != 1024 {
+			t.Fatalf("total_rows = %d, want 1024", got.response.TotalRows)
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatal("preview never returned")
+	}
+}
+
+// A worker that cannot read the object says so, and the page shows its reason
+// rather than an empty table.
+func TestRequestObjectPreviewSurfacesTheWorkersError(t *testing.T) {
+	t.Parallel()
+	const workerID = "lance-worker-1"
+	objectID := []string{"vectors", "ml", "embeddings"}
+	pluginSvc, session := previewPlugin(t, workerID)
+	observeObject(pluginSvc, workerID, objectID)
+
+	errs := make(chan error, 1)
+	go func() {
+		_, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10)
+		errs <- err
+	}()
+
+	request := <-session.outgoing
+	pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{
+		WorkerId: workerID,
+		Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{
+			ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{
+				RequestId:    request.RequestId,
+				Success:      false,
+				ErrorMessage: "open lance dataset: access denied",
+			},
+		},
+	})
+
+	select {
+	case err := <-errs:
+		if err == nil || !strings.Contains(err.Error(), "access denied") {
+			t.Fatalf("error = %v, want the worker's reason", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatal("preview never returned")
+	}
+}
+
+// Nothing has looked at the object, so there is nobody to ask. Worth its own
+// message: it means detection has not run, not that the table is unreadable.
+func TestRequestObjectPreviewNeedsAnObservation(t *testing.T) {
+	t.Parallel()
+	pluginSvc, _ := previewPlugin(t, "lance-worker-1")
+
+	_, err := pluginSvc.RequestObjectPreview(context.Background(), []string{"vectors", "ml", "nothing"}, "LANCE", 10)
+	if err == nil || !strings.Contains(err.Error(), "described") {
+		t.Fatalf("error = %v, want one naming the missing description", err)
+	}
+}
+
+// The worker that described the object has since gone away.
+func TestRequestObjectPreviewNeedsAConnectedWorker(t *testing.T) {
+	t.Parallel()
+	objectID := []string{"vectors", "ml", "embeddings"}
+	pluginSvc, _ := previewPlugin(t, "lance-worker-1")
+	observeObject(pluginSvc, "lance-worker-gone", objectID)
+
+	_, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10)
+	if err == nil || !strings.Contains(err.Error(), "not connected") {
+		t.Fatalf("error = %v, want one naming the absent worker", err)
+	}
+}
+
+// Shutdown closes every pending channel, and a reply can arrive at the same
+// moment; delivering it outside the lock panics with "send on closed channel".
+// The window is narrow enough that this test does not reliably reproduce it -
+// widening it with a Gosched between the lookup and the send does, every time -
+// so treat this as exercising the path rather than as a regression alarm.
+func TestObjectPreviewResponseRacesShutdown(t *testing.T) {
+	t.Parallel()
+	const workerID = "lance-worker-1"
+	objectID := []string{"vectors", "ml", "embeddings"}
+
+	for i := 0; i < 50; i++ {
+		pluginSvc, session := previewPlugin(t, workerID)
+		observeObject(pluginSvc, workerID, objectID)
+
+		go func() {
+			_, _ = pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10)
+		}()
+		request := <-session.outgoing
+
+		var wg sync.WaitGroup
+		wg.Add(2)
+		go func() {
+			defer wg.Done()
+			pluginSvc.Shutdown()
+		}()
+		go func() {
+			defer wg.Done()
+			pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{
+				WorkerId: workerID,
+				Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{
+					ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{
+						RequestId: request.RequestId,
+						Success:   true,
+					},
+				},
+			})
+		}()
+		wg.Wait()
+	}
+}

weed/admin/plugin/registry.go

@@ -0,0 +1,512 @@
+package plugin
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+const defaultWorkerStaleTimeout = 2 * time.Minute
+
+// WorkerSession contains tracked worker metadata and plugin status.
+type WorkerSession struct {
+	WorkerID        string
+	WorkerInstance  string
+	Address         string
+	WorkerVersion   string
+	ProtocolVersion string
+	ConnectedAt     time.Time
+	LastSeenAt      time.Time
+	Capabilities    map[string]*plugin_pb.JobTypeCapability
+	Heartbeat       *plugin_pb.WorkerHeartbeat
+}
+
+// Registry tracks connected plugin workers and capability-based selection.
+type Registry struct {
+	mu             sync.RWMutex
+	sessions       map[string]*WorkerSession
+	staleAfter     time.Duration
+	detectorCursor map[string]int
+	executorCursor map[string]int
+}
+
+func NewRegistry() *Registry {
+	return &Registry{
+		sessions:       make(map[string]*WorkerSession),
+		staleAfter:     defaultWorkerStaleTimeout,
+		detectorCursor: make(map[string]int),
+		executorCursor: make(map[string]int),
+	}
+}
+
+func (r *Registry) UpsertFromHello(hello *plugin_pb.WorkerHello) *WorkerSession {
+	now := time.Now()
+	caps := make(map[string]*plugin_pb.JobTypeCapability, len(hello.Capabilities))
+	for _, c := range hello.Capabilities {
+		if c == nil || c.JobType == "" {
+			continue
+		}
+		caps[c.JobType] = cloneJobTypeCapability(c)
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	session, ok := r.sessions[hello.WorkerId]
+	if !ok {
+		session = &WorkerSession{
+			WorkerID:    hello.WorkerId,
+			ConnectedAt: now,
+		}
+		r.sessions[hello.WorkerId] = session
+	}
+
+	session.WorkerInstance = hello.WorkerInstanceId
+	session.Address = hello.Address
+	session.WorkerVersion = hello.WorkerVersion
+	session.ProtocolVersion = hello.ProtocolVersion
+	session.LastSeenAt = now
+	session.Capabilities = caps
+
+	return cloneWorkerSession(session)
+}
+
+func (r *Registry) Remove(workerID string) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	delete(r.sessions, workerID)
+}
+
+func (r *Registry) UpdateHeartbeat(workerID string, heartbeat *plugin_pb.WorkerHeartbeat) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	session, ok := r.sessions[workerID]
+	if !ok {
+		return
+	}
+	session.Heartbeat = cloneWorkerHeartbeat(heartbeat)
+	session.LastSeenAt = time.Now()
+}
+
+func (r *Registry) Get(workerID string) (*WorkerSession, bool) {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+	session, ok := r.sessions[workerID]
+	if !ok || r.isSessionStaleLocked(session, time.Now()) {
+		return nil, false
+	}
+	return cloneWorkerSession(session), true
+}
+
+func (r *Registry) List() []*WorkerSession {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+	out := make([]*WorkerSession, 0, len(r.sessions))
+	now := time.Now()
+	for _, s := range r.sessions {
+		if r.isSessionStaleLocked(s, now) {
+			continue
+		}
+		out = append(out, cloneWorkerSession(s))
+	}
+	sort.Slice(out, func(i, j int) bool {
+		return out[i].WorkerID < out[j].WorkerID
+	})
+	return out
+}
+
+// HasCapableWorker checks if any non-stale worker session has a capability for the given job type.
+// A worker is capable if its capabilities include the job type with CanDetect or CanExecute true.
+func (r *Registry) HasCapableWorker(jobType string) bool {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+
+	now := time.Now()
+	for _, session := range r.sessions {
+		if r.isSessionStaleLocked(session, now) {
+			continue
+		}
+		capability := session.Capabilities[jobType]
+		if capability == nil {
+			continue
+		}
+		if capability.CanDetect || capability.CanExecute {
+			return true
+		}
+	}
+	return false
+}
+
+// CountCapableExecutors returns the number of non-stale workers that
+// can EXECUTE the given job type. Used by per-job-type cluster
+// allocators (e.g. the s3_lifecycle delete-rate divider) to compute a
+// per-worker share at dispatch time. Returns 0 when no executor is
+// available — callers should treat that as "skip allocation" rather
+// than dividing by zero.
+func (r *Registry) CountCapableExecutors(jobType string) int {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+
+	now := time.Now()
+	n := 0
+	for _, session := range r.sessions {
+		if r.isSessionStaleLocked(session, now) {
+			continue
+		}
+		capability := session.Capabilities[jobType]
+		if capability == nil || !capability.CanExecute {
+			continue
+		}
+		n++
+	}
+	return n
+}
+
+// DetectableJobTypes returns sorted job types that currently have at least one detect-capable worker.
+func (r *Registry) DetectableJobTypes() []string {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+
+	jobTypes := make(map[string]struct{})
+	now := time.Now()
+	for _, session := range r.sessions {
+		if r.isSessionStaleLocked(session, now) {
+			continue
+		}
+		for jobType, capability := range session.Capabilities {
+			if capability == nil || !capability.CanDetect {
+				continue
+			}
+			jobTypes[jobType] = struct{}{}
+		}
+	}
+
+	out := make([]string, 0, len(jobTypes))
+	for jobType := range jobTypes {
+		out = append(out, jobType)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// JobTypes returns sorted job types known by connected workers regardless of capability kind.
+func (r *Registry) JobTypes() []string {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+
+	jobTypes := make(map[string]struct{})
+	now := time.Now()
+	for _, session := range r.sessions {
+		if r.isSessionStaleLocked(session, now) {
+			continue
+		}
+		for jobType := range session.Capabilities {
+			if jobType == "" {
+				continue
+			}
+			jobTypes[jobType] = struct{}{}
+		}
+	}
+
+	out := make([]string, 0, len(jobTypes))
+	for jobType := range jobTypes {
+		out = append(out, jobType)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// PickSchemaProvider picks one worker for schema requests.
+// Preference order:
+// 1) workers that can detect this job type
+// 2) workers that can execute this job type
+// tie-break: more free slots, then lexical worker ID.
+func (r *Registry) PickSchemaProvider(jobType string) (*WorkerSession, error) {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+
+	var candidates []*WorkerSession
+	now := time.Now()
+	for _, s := range r.sessions {
+		if r.isSessionStaleLocked(s, now) {
+			continue
+		}
+		capability := s.Capabilities[jobType]
+		if capability == nil {
+			continue
+		}
+		if capability.CanDetect || capability.CanExecute {
+			candidates = append(candidates, s)
+		}
+	}
+
+	if len(candidates) == 0 {
+		return nil, fmt.Errorf("no worker available for schema job_type=%s", jobType)
+	}
+
+	sort.Slice(candidates, func(i, j int) bool {
+		a := candidates[i]
+		b := candidates[j]
+		ac := a.Capabilities[jobType]
+		bc := b.Capabilities[jobType]
+
+		// Prefer detect-capable providers first.
+		if ac.CanDetect != bc.CanDetect {
+			return ac.CanDetect
+		}
+
+		aSlots := availableDetectionSlots(a, ac) + availableExecutionSlots(a, ac)
+		bSlots := availableDetectionSlots(b, bc) + availableExecutionSlots(b, bc)
+		if aSlots != bSlots {
+			return aSlots > bSlots
+		}
+		return a.WorkerID < b.WorkerID
+	})
+
+	return cloneWorkerSession(candidates[0]), nil
+}
+
+// PickDetector picks one detector worker for a job type.
+func (r *Registry) PickDetector(jobType string) (*WorkerSession, error) {
+	return r.pickByKind(jobType, true)
+}
+
+// PickExecutor picks one executor worker for a job type.
+func (r *Registry) PickExecutor(jobType string) (*WorkerSession, error) {
+	return r.pickByKind(jobType, false)
+}
+
+// ListExecutors returns sorted executor candidates for one job type.
+// Ordering is by most available execution slots, then lexical worker ID.
+// The top tie group is rotated round-robin to prevent sticky assignment.
+func (r *Registry) ListExecutors(jobType string) ([]*WorkerSession, error) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	candidates := r.collectByKindLocked(jobType, false, time.Now())
+	if len(candidates) == 0 {
+		return nil, fmt.Errorf("no executor worker available for job_type=%s", jobType)
+	}
+
+	sortByKind(candidates, jobType, false)
+	r.rotateTopCandidatesLocked(candidates, jobType, false)
+
+	out := make([]*WorkerSession, 0, len(candidates))
+	for _, candidate := range candidates {
+		out = append(out, cloneWorkerSession(candidate))
+	}
+	return out, nil
+}
+
+func (r *Registry) pickByKind(jobType string, detect bool) (*WorkerSession, error) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	candidates := r.collectByKindLocked(jobType, detect, time.Now())
+
+	if len(candidates) == 0 {
+		kind := "executor"
+		if detect {
+			kind = "detector"
+		}
+		return nil, fmt.Errorf("no %s worker available for job_type=%s", kind, jobType)
+	}
+
+	sortByKind(candidates, jobType, detect)
+	r.rotateTopCandidatesLocked(candidates, jobType, detect)
+
+	return cloneWorkerSession(candidates[0]), nil
+}
+
+func (r *Registry) collectByKindLocked(jobType string, detect bool, now time.Time) []*WorkerSession {
+	var candidates []*WorkerSession
+	for _, session := range r.sessions {
+		if r.isSessionStaleLocked(session, now) {
+			continue
+		}
+		capability := session.Capabilities[jobType]
+		if capability == nil {
+			continue
+		}
+		if detect && capability.CanDetect {
+			candidates = append(candidates, session)
+		}
+		if !detect && capability.CanExecute {
+			candidates = append(candidates, session)
+		}
+	}
+	return candidates
+}
+
+func (r *Registry) isSessionStaleLocked(session *WorkerSession, now time.Time) bool {
+	if session == nil {
+		return true
+	}
+	if r.staleAfter <= 0 {
+		return false
+	}
+
+	lastSeen := session.LastSeenAt
+	if lastSeen.IsZero() {
+		lastSeen = session.ConnectedAt
+	}
+	if lastSeen.IsZero() {
+		return false
+	}
+	return now.Sub(lastSeen) > r.staleAfter
+}
+
+func sortByKind(candidates []*WorkerSession, jobType string, detect bool) {
+	sort.Slice(candidates, func(i, j int) bool {
+		a := candidates[i]
+		b := candidates[j]
+		ac := a.Capabilities[jobType]
+		bc := b.Capabilities[jobType]
+
+		aSlots := availableSlotsByKind(a, ac, detect)
+		bSlots := availableSlotsByKind(b, bc, detect)
+
+		if aSlots != bSlots {
+			return aSlots > bSlots
+		}
+		return a.WorkerID < b.WorkerID
+	})
+}
+
+func (r *Registry) rotateTopCandidatesLocked(candidates []*WorkerSession, jobType string, detect bool) {
+	if len(candidates) < 2 {
+		return
+	}
+
+	capability := candidates[0].Capabilities[jobType]
+	topSlots := availableSlotsByKind(candidates[0], capability, detect)
+	tieEnd := 1
+	for tieEnd < len(candidates) {
+		nextCapability := candidates[tieEnd].Capabilities[jobType]
+		if availableSlotsByKind(candidates[tieEnd], nextCapability, detect) != topSlots {
+			break
+		}
+		tieEnd++
+	}
+	if tieEnd <= 1 {
+		return
+	}
+
+	cursorKey := strings.TrimSpace(jobType)
+	if cursorKey == "" {
+		cursorKey = "*"
+	}
+
+	var offset int
+	if detect {
+		offset = r.detectorCursor[cursorKey] % tieEnd
+		r.detectorCursor[cursorKey] = (offset + 1) % tieEnd
+	} else {
+		offset = r.executorCursor[cursorKey] % tieEnd
+		r.executorCursor[cursorKey] = (offset + 1) % tieEnd
+	}
+
+	if offset == 0 {
+		return
+	}
+
+	prefix := append([]*WorkerSession(nil), candidates[:tieEnd]...)
+	for i := 0; i < tieEnd; i++ {
+		candidates[i] = prefix[(i+offset)%tieEnd]
+	}
+}
+
+func availableSlotsByKind(
+	session *WorkerSession,
+	capability *plugin_pb.JobTypeCapability,
+	detect bool,
+) int {
+	if detect {
+		return availableDetectionSlots(session, capability)
+	}
+	return availableExecutionSlots(session, capability)
+}
+
+func availableDetectionSlots(session *WorkerSession, capability *plugin_pb.JobTypeCapability) int {
+	if session.Heartbeat != nil && session.Heartbeat.DetectionSlotsTotal > 0 {
+		free := int(session.Heartbeat.DetectionSlotsTotal - session.Heartbeat.DetectionSlotsUsed)
+		if free < 0 {
+			return 0
+		}
+		return free
+	}
+	if capability.MaxDetectionConcurrency > 0 {
+		return int(capability.MaxDetectionConcurrency)
+	}
+	return 1
+}
+
+func availableExecutionSlots(session *WorkerSession, capability *plugin_pb.JobTypeCapability) int {
+	if session.Heartbeat != nil && session.Heartbeat.ExecutionSlotsTotal > 0 {
+		free := int(session.Heartbeat.ExecutionSlotsTotal - session.Heartbeat.ExecutionSlotsUsed)
+		if free < 0 {
+			return 0
+		}
+		return free
+	}
+	if capability.MaxExecutionConcurrency > 0 {
+		return int(capability.MaxExecutionConcurrency)
+	}
+	return 1
+}
+
+func cloneWorkerSession(in *WorkerSession) *WorkerSession {
+	if in == nil {
+		return nil
+	}
+	out := *in
+	out.Capabilities = make(map[string]*plugin_pb.JobTypeCapability, len(in.Capabilities))
+	for jobType, cap := range in.Capabilities {
+		out.Capabilities[jobType] = cloneJobTypeCapability(cap)
+	}
+	out.Heartbeat = cloneWorkerHeartbeat(in.Heartbeat)
+	return &out
+}
+
+func cloneJobTypeCapability(in *plugin_pb.JobTypeCapability) *plugin_pb.JobTypeCapability {
+	if in == nil {
+		return nil
+	}
+	out := *in
+	return &out
+}
+
+func cloneWorkerHeartbeat(in *plugin_pb.WorkerHeartbeat) *plugin_pb.WorkerHeartbeat {
+	if in == nil {
+		return nil
+	}
+	out := *in
+	if in.RunningWork != nil {
+		out.RunningWork = make([]*plugin_pb.RunningWork, 0, len(in.RunningWork))
+		for _, rw := range in.RunningWork {
+			if rw == nil {
+				continue
+			}
+			clone := *rw
+			out.RunningWork = append(out.RunningWork, &clone)
+		}
+	}
+	if in.QueuedJobsByType != nil {
+		out.QueuedJobsByType = make(map[string]int32, len(in.QueuedJobsByType))
+		for k, v := range in.QueuedJobsByType {
+			out.QueuedJobsByType[k] = v
+		}
+	}
+	if in.Metadata != nil {
+		out.Metadata = make(map[string]string, len(in.Metadata))
+		for k, v := range in.Metadata {
+			out.Metadata[k] = v
+		}
+	}
+	return &out
+}

weed/admin/plugin/registry_test.go

@@ -0,0 +1,403 @@
+package plugin
+
+import (
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+func TestRegistryPickDetectorPrefersMoreFreeSlots(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true, CanExecute: true, MaxDetectionConcurrency: 2, MaxExecutionConcurrency: 2},
+		},
+	})
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-b",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true, CanExecute: true, MaxDetectionConcurrency: 4, MaxExecutionConcurrency: 4},
+		},
+	})
+
+	r.UpdateHeartbeat("worker-a", &plugin_pb.WorkerHeartbeat{
+		WorkerId:            "worker-a",
+		DetectionSlotsUsed:  1,
+		DetectionSlotsTotal: 2,
+	})
+	r.UpdateHeartbeat("worker-b", &plugin_pb.WorkerHeartbeat{
+		WorkerId:            "worker-b",
+		DetectionSlotsUsed:  1,
+		DetectionSlotsTotal: 4,
+	})
+
+	picked, err := r.PickDetector("vacuum")
+	if err != nil {
+		t.Fatalf("PickDetector: %v", err)
+	}
+	if picked.WorkerID != "worker-b" {
+		t.Fatalf("unexpected detector picked: got %s want worker-b", picked.WorkerID)
+	}
+}
+
+func TestRegistryPickExecutorAllowsSameWorker(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-x",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanDetect: true, CanExecute: true, MaxDetectionConcurrency: 1, MaxExecutionConcurrency: 1},
+		},
+	})
+
+	detector, err := r.PickDetector("balance")
+	if err != nil {
+		t.Fatalf("PickDetector: %v", err)
+	}
+	executor, err := r.PickExecutor("balance")
+	if err != nil {
+		t.Fatalf("PickExecutor: %v", err)
+	}
+
+	if detector.WorkerID != "worker-x" || executor.WorkerID != "worker-x" {
+		t.Fatalf("expected same worker for detect/execute, got detector=%s executor=%s", detector.WorkerID, executor.WorkerID)
+	}
+}
+
+func TestRegistryDetectableJobTypes(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true, CanExecute: true},
+			{JobType: "balance", CanDetect: false, CanExecute: true},
+		},
+	})
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-b",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "ec", CanDetect: true, CanExecute: false},
+			{JobType: "vacuum", CanDetect: true, CanExecute: false},
+		},
+	})
+
+	got := r.DetectableJobTypes()
+	want := []string{"ec", "vacuum"}
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("unexpected detectable job types: got=%v want=%v", got, want)
+	}
+}
+
+func TestRegistryJobTypes(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true},
+			{JobType: "balance", CanExecute: true},
+		},
+	})
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-b",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "ec", CanDetect: true},
+		},
+	})
+
+	got := r.JobTypes()
+	want := []string{"balance", "ec", "vacuum"}
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("unexpected job types: got=%v want=%v", got, want)
+	}
+}
+
+func TestRegistryListExecutorsSortedBySlots(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 2},
+		},
+	})
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-b",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 4},
+		},
+	})
+
+	r.UpdateHeartbeat("worker-a", &plugin_pb.WorkerHeartbeat{
+		WorkerId:            "worker-a",
+		ExecutionSlotsUsed:  1,
+		ExecutionSlotsTotal: 2,
+	})
+	r.UpdateHeartbeat("worker-b", &plugin_pb.WorkerHeartbeat{
+		WorkerId:            "worker-b",
+		ExecutionSlotsUsed:  1,
+		ExecutionSlotsTotal: 4,
+	})
+
+	executors, err := r.ListExecutors("balance")
+	if err != nil {
+		t.Fatalf("ListExecutors: %v", err)
+	}
+	if len(executors) != 2 {
+		t.Fatalf("unexpected candidate count: got=%d", len(executors))
+	}
+	if executors[0].WorkerID != "worker-b" || executors[1].WorkerID != "worker-a" {
+		t.Fatalf("unexpected executor order: got=%s,%s", executors[0].WorkerID, executors[1].WorkerID)
+	}
+}
+
+func TestRegistryPickExecutorRoundRobinForTopTie(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	for _, workerID := range []string{"worker-a", "worker-b", "worker-c"} {
+		r.UpsertFromHello(&plugin_pb.WorkerHello{
+			WorkerId: workerID,
+			Capabilities: []*plugin_pb.JobTypeCapability{
+				{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 1},
+			},
+		})
+	}
+
+	got := make([]string, 0, 6)
+	for i := 0; i < 6; i++ {
+		executor, err := r.PickExecutor("balance")
+		if err != nil {
+			t.Fatalf("PickExecutor: %v", err)
+		}
+		got = append(got, executor.WorkerID)
+	}
+
+	want := []string{"worker-a", "worker-b", "worker-c", "worker-a", "worker-b", "worker-c"}
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("unexpected pick order: got=%v want=%v", got, want)
+	}
+}
+
+func TestRegistryListExecutorsRoundRobinForTopTie(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 2},
+		},
+	})
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-b",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 2},
+		},
+	})
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-c",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "balance", CanExecute: true, MaxExecutionConcurrency: 1},
+		},
+	})
+
+	r.UpdateHeartbeat("worker-a", &plugin_pb.WorkerHeartbeat{
+		WorkerId:            "worker-a",
+		ExecutionSlotsUsed:  0,
+		ExecutionSlotsTotal: 2,
+	})
+	r.UpdateHeartbeat("worker-b", &plugin_pb.WorkerHeartbeat{
+		WorkerId:            "worker-b",
+		ExecutionSlotsUsed:  0,
+		ExecutionSlotsTotal: 2,
+	})
+	r.UpdateHeartbeat("worker-c", &plugin_pb.WorkerHeartbeat{
+		WorkerId:            "worker-c",
+		ExecutionSlotsUsed:  0,
+		ExecutionSlotsTotal: 1,
+	})
+
+	firstCall, err := r.ListExecutors("balance")
+	if err != nil {
+		t.Fatalf("ListExecutors first call: %v", err)
+	}
+	secondCall, err := r.ListExecutors("balance")
+	if err != nil {
+		t.Fatalf("ListExecutors second call: %v", err)
+	}
+	thirdCall, err := r.ListExecutors("balance")
+	if err != nil {
+		t.Fatalf("ListExecutors third call: %v", err)
+	}
+
+	if firstCall[0].WorkerID != "worker-a" || firstCall[1].WorkerID != "worker-b" || firstCall[2].WorkerID != "worker-c" {
+		t.Fatalf("unexpected first executor order: got=%s,%s,%s", firstCall[0].WorkerID, firstCall[1].WorkerID, firstCall[2].WorkerID)
+	}
+	if secondCall[0].WorkerID != "worker-b" || secondCall[1].WorkerID != "worker-a" || secondCall[2].WorkerID != "worker-c" {
+		t.Fatalf("unexpected second executor order: got=%s,%s,%s", secondCall[0].WorkerID, secondCall[1].WorkerID, secondCall[2].WorkerID)
+	}
+	if thirdCall[0].WorkerID != "worker-a" || thirdCall[1].WorkerID != "worker-b" || thirdCall[2].WorkerID != "worker-c" {
+		t.Fatalf("unexpected third executor order: got=%s,%s,%s", thirdCall[0].WorkerID, thirdCall[1].WorkerID, thirdCall[2].WorkerID)
+	}
+}
+
+func TestRegistrySkipsStaleWorkersForSelectionAndListing(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.staleAfter = 2 * time.Second
+
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-stale",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true, CanExecute: true},
+		},
+	})
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-fresh",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true, CanExecute: true},
+		},
+	})
+
+	r.mu.Lock()
+	r.sessions["worker-stale"].LastSeenAt = time.Now().Add(-10 * time.Second)
+	r.sessions["worker-fresh"].LastSeenAt = time.Now()
+	r.mu.Unlock()
+
+	picked, err := r.PickDetector("vacuum")
+	if err != nil {
+		t.Fatalf("PickDetector: %v", err)
+	}
+	if picked.WorkerID != "worker-fresh" {
+		t.Fatalf("unexpected detector: got=%s want=worker-fresh", picked.WorkerID)
+	}
+
+	if _, ok := r.Get("worker-stale"); ok {
+		t.Fatalf("expected stale worker to be hidden from Get")
+	}
+	if _, ok := r.Get("worker-fresh"); !ok {
+		t.Fatalf("expected fresh worker from Get")
+	}
+
+	listed := r.List()
+	if len(listed) != 1 || listed[0].WorkerID != "worker-fresh" {
+		t.Fatalf("unexpected listed workers: %+v", listed)
+	}
+}
+
+func TestRegistryReturnsNoDetectorWhenAllWorkersStale(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.staleAfter = 2 * time.Second
+
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true},
+		},
+	})
+
+	r.mu.Lock()
+	r.sessions["worker-a"].LastSeenAt = time.Now().Add(-10 * time.Second)
+	r.mu.Unlock()
+
+	if _, err := r.PickDetector("vacuum"); err == nil {
+		t.Fatalf("expected no detector when all workers are stale")
+	}
+}
+
+func TestRegistryHasCapableWorkerWithDetectCapability(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true, CanExecute: false},
+		},
+	})
+
+	if !r.HasCapableWorker("vacuum") {
+		t.Fatalf("expected HasCapableWorker to return true for detect-capable worker")
+	}
+	if r.HasCapableWorker("balance") {
+		t.Fatalf("expected HasCapableWorker to return false for unknown job type")
+	}
+}
+
+func TestRegistryHasCapableWorkerWithExecuteCapability(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: false, CanExecute: true},
+		},
+	})
+
+	if !r.HasCapableWorker("vacuum") {
+		t.Fatalf("expected HasCapableWorker to return true for execute-capable worker")
+	}
+}
+
+func TestRegistryHasCapableWorkerReturnsFalseWhenNoWorkers(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	if r.HasCapableWorker("vacuum") {
+		t.Fatalf("expected HasCapableWorker to return false with no workers")
+	}
+}
+
+func TestRegistryHasCapableWorkerSkipsStaleWorkers(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.staleAfter = 2 * time.Second
+
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-stale",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: true, CanExecute: true},
+		},
+	})
+
+	r.mu.Lock()
+	r.sessions["worker-stale"].LastSeenAt = time.Now().Add(-10 * time.Second)
+	r.mu.Unlock()
+
+	if r.HasCapableWorker("vacuum") {
+		t.Fatalf("expected HasCapableWorker to return false when all workers are stale")
+	}
+}
+
+func TestRegistryHasCapableWorkerIgnoresNilCapability(t *testing.T) {
+	t.Parallel()
+
+	r := NewRegistry()
+	r.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: "vacuum", CanDetect: false, CanExecute: false},
+		},
+	})
+
+	if r.HasCapableWorker("vacuum") {
+		t.Fatalf("expected HasCapableWorker to return false when capability has no detect or execute")
+	}
+}

weed/admin/plugin/scheduler_config.go

@@ -0,0 +1,5 @@
+package plugin
+
+import "time"
+
+const defaultSchedulerIdleSleep = 61 * time.Second

weed/admin/plugin/scheduler_lane.go

@@ -0,0 +1,145 @@
+package plugin
+
+import (
+	"sync"
+	"time"
+)
+
+// SchedulerLane identifies an independent scheduling track. Each lane runs
+// its own goroutine and maintains its own detection timing so that
+// workloads in different lanes never block each other.
+type SchedulerLane string
+
+const (
+	// LaneDefault handles volume management operations (vacuum, balance,
+	// erasure coding) and admin scripts. It is the fallback lane for any
+	// job type that is not explicitly mapped elsewhere.
+	LaneDefault SchedulerLane = "default"
+
+	// LaneIceberg handles table-bucket Iceberg compaction and maintenance.
+	LaneIceberg SchedulerLane = "iceberg"
+
+	// LaneLifecycle handles S3 object store lifecycle management
+	// (expiration, transition, abort incomplete multipart uploads).
+	LaneLifecycle SchedulerLane = "lifecycle"
+
+	// LaneLance handles table-bucket Lance maintenance: fragment compaction,
+	// index optimization and version cleanup.
+	LaneLance SchedulerLane = "lance"
+)
+
+// AllLanes returns every defined scheduler lane in a stable order.
+func AllLanes() []SchedulerLane {
+	return []SchedulerLane{LaneDefault, LaneIceberg, LaneLifecycle, LaneLance}
+}
+
+// laneIdleSleep maps each lane to its default idle sleep duration.
+// Each lane can sleep for a different amount when no work is detected,
+// independent of the per-job-type DetectionInterval.
+var laneIdleSleep = map[SchedulerLane]time.Duration{
+	LaneDefault:   61 * time.Second,
+	LaneIceberg:   61 * time.Second,
+	LaneLifecycle: 5 * time.Minute,
+	LaneLance:     61 * time.Second,
+}
+
+// laneRequiresLock maps each lane to whether its job types must be
+// serialised and run under the shared cluster admin lock. The default
+// lane needs this because volume-management operations share global
+// state; the lock is taken around each detection and each job so manual
+// shell operations can interleave. Other lanes run each job type
+// independently without the lock.
+var laneRequiresLock = map[SchedulerLane]bool{
+	LaneDefault:   true,
+	LaneIceberg:   false,
+	LaneLifecycle: false,
+	// Lance maintenance rewrites files inside one table and shares no global
+	// state, so it has no more need of the cluster admin lock than Iceberg does.
+	LaneLance: false,
+}
+
+// LaneRequiresLock returns true if the given lane serialises its job types
+// and runs them under the cluster admin lock. Unknown lanes default to true.
+func LaneRequiresLock(lane SchedulerLane) bool {
+	if v, ok := laneRequiresLock[lane]; ok {
+		return v
+	}
+	return true
+}
+
+// LaneIdleSleep returns the idle sleep duration for the given lane,
+// falling back to defaultSchedulerIdleSleep if the lane is unknown.
+func LaneIdleSleep(lane SchedulerLane) time.Duration {
+	if d, ok := laneIdleSleep[lane]; ok {
+		return d
+	}
+	return defaultSchedulerIdleSleep
+}
+
+// jobTypeLaneMap is the hardcoded mapping from job type to scheduler lane.
+// Job types not present here are assigned to LaneDefault.
+var jobTypeLaneMap = map[string]SchedulerLane{
+	// Volume management (default lane)
+	"vacuum":         LaneDefault,
+	"volume_balance": LaneDefault,
+	"ec_balance":     LaneDefault,
+	"erasure_coding": LaneDefault,
+	"admin_script":   LaneDefault,
+
+	// Iceberg table maintenance
+	"iceberg_maintenance": LaneIceberg,
+
+	// S3 lifecycle management
+	"s3_lifecycle": LaneLifecycle,
+
+	// Lance table maintenance. Without these the job types fall back to the
+	// default lane, which serialises everything under the cluster admin lock
+	// and would queue a compaction behind volume balancing.
+	"lance_compact":          LaneLance,
+	"lance_optimize_indices": LaneLance,
+	"lance_cleanup_versions": LaneLance,
+}
+
+// JobTypeLane returns the scheduler lane for the given job type.
+// Unknown job types are assigned to LaneDefault.
+func JobTypeLane(jobType string) SchedulerLane {
+	if lane, ok := jobTypeLaneMap[jobType]; ok {
+		return lane
+	}
+	return LaneDefault
+}
+
+// LaneJobTypes returns the set of known job types assigned to the given lane.
+func LaneJobTypes(lane SchedulerLane) []string {
+	var result []string
+	for jobType, l := range jobTypeLaneMap {
+		if l == lane {
+			result = append(result, jobType)
+		}
+	}
+	return result
+}
+
+// schedulerLaneState holds the per-lane runtime state used by the scheduler.
+type schedulerLaneState struct {
+	lane   SchedulerLane
+	wakeCh chan struct{}
+
+	loopMu sync.Mutex
+	loop   schedulerLoopState
+
+	// Per-lane execution reservation pool. Each lane tracks how many
+	// execution slots it has reserved on each worker independently,
+	// so lanes cannot starve each other.
+	execMu  sync.Mutex
+	execRes map[string]int
+}
+
+// newLaneState creates a schedulerLaneState for the given lane.
+func newLaneState(lane SchedulerLane) *schedulerLaneState {
+	return &schedulerLaneState{
+		lane:    lane,
+		wakeCh:  make(chan struct{}, 1),
+		execRes: make(map[string]int),
+	}
+}

weed/admin/plugin/scheduler_lane_test.go

@@ -0,0 +1,73 @@
+package plugin
+
+import (
+	"testing"
+)
+
+func TestJobTypeLaneMapCoversKnownTypes(t *testing.T) {
+	// Every job type in the map must resolve to a lane the scheduler runs.
+	// Checked against AllLanes rather than a list spelled out here, so adding a
+	// lane does not silently leave its job types unscheduled.
+	known := make(map[SchedulerLane]bool, len(AllLanes()))
+	for _, lane := range AllLanes() {
+		known[lane] = true
+	}
+	for jobType, lane := range jobTypeLaneMap {
+		if !known[lane] {
+			t.Errorf("jobTypeLaneMap[%q] = %q, want a lane from AllLanes()", jobType, lane)
+		}
+	}
+}
+
+func TestJobTypeLaneFallsBackToDefault(t *testing.T) {
+	if got := JobTypeLane("unknown_job_type"); got != LaneDefault {
+		t.Errorf("JobTypeLane(unknown) = %q, want %q", got, LaneDefault)
+	}
+}
+
+func TestAllLanesHaveIdleSleep(t *testing.T) {
+	for _, lane := range AllLanes() {
+		if d := LaneIdleSleep(lane); d <= 0 {
+			t.Errorf("LaneIdleSleep(%q) = %v, want > 0", lane, d)
+		}
+	}
+}
+
+func TestLaneRequiresLock(t *testing.T) {
+	tests := []struct {
+		name string
+		lane SchedulerLane
+		want bool
+	}{
+		{"Default", LaneDefault, true},
+		{"Iceberg", LaneIceberg, false},
+		{"Lifecycle", LaneLifecycle, false},
+		{"Unknown", "unknown_lane", true},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := LaneRequiresLock(tt.lane); got != tt.want {
+				t.Errorf("LaneRequiresLock(%q) = %v, want %v", tt.lane, got, tt.want)
+			}
+		})
+	}
+}
+
+func TestKnownJobTypesInMap(t *testing.T) {
+	// Ensure the well-known job types are mapped. This catches drift
+	// if a handler's job type string changes without updating the map.
+	expected := map[string]SchedulerLane{
+		"vacuum":              LaneDefault,
+		"volume_balance":      LaneDefault,
+		"ec_balance":          LaneDefault,
+		"erasure_coding":      LaneDefault,
+		"admin_script":        LaneDefault,
+		"iceberg_maintenance": LaneIceberg,
+		"s3_lifecycle":        LaneLifecycle,
+	}
+	for jobType, wantLane := range expected {
+		if got := JobTypeLane(jobType); got != wantLane {
+			t.Errorf("JobTypeLane(%q) = %q, want %q", jobType, got, wantLane)
+		}
+	}
+}

weed/admin/plugin/scheduler_status.go

@@ -0,0 +1,517 @@
+package plugin
+
+import (
+	"sort"
+	"strings"
+	"time"
+)
+
+type SchedulerStatus struct {
+	Now                  time.Time                `json:"now"`
+	Lane                 string                   `json:"lane,omitempty"`
+	SchedulerTickSeconds int                      `json:"scheduler_tick_seconds"`
+	IdleSleepSeconds     int                      `json:"idle_sleep_seconds,omitempty"`
+	NextDetectionAt      *time.Time               `json:"next_detection_at,omitempty"`
+	CurrentJobType       string                   `json:"current_job_type,omitempty"`
+	CurrentPhase         string                   `json:"current_phase,omitempty"`
+	LastIterationHadJobs bool                     `json:"last_iteration_had_jobs,omitempty"`
+	LastIterationDoneAt  *time.Time               `json:"last_iteration_done_at,omitempty"`
+	Waiting              []SchedulerWaitingStatus `json:"waiting,omitempty"`
+	InProcessJobs        []SchedulerJobStatus     `json:"in_process_jobs,omitempty"`
+	JobTypes             []SchedulerJobTypeStatus `json:"job_types,omitempty"`
+}
+
+type SchedulerWaitingStatus struct {
+	Reason  string                 `json:"reason"`
+	JobType string                 `json:"job_type,omitempty"`
+	Since   *time.Time             `json:"since,omitempty"`
+	Until   *time.Time             `json:"until,omitempty"`
+	Details map[string]interface{} `json:"details,omitempty"`
+}
+
+type SchedulerJobStatus struct {
+	JobID           string     `json:"job_id"`
+	JobType         string     `json:"job_type"`
+	State           string     `json:"state"`
+	Stage           string     `json:"stage,omitempty"`
+	WorkerID        string     `json:"worker_id,omitempty"`
+	Message         string     `json:"message,omitempty"`
+	Progress        float64    `json:"progress,omitempty"`
+	CreatedAt       *time.Time `json:"created_at,omitempty"`
+	UpdatedAt       *time.Time `json:"updated_at,omitempty"`
+	DurationSeconds float64    `json:"duration_seconds,omitempty"`
+}
+
+type SchedulerJobTypeStatus struct {
+	JobType                  string     `json:"job_type"`
+	Enabled                  bool       `json:"enabled"`
+	DetectionInFlight        bool       `json:"detection_in_flight"`
+	NextDetectionAt          *time.Time `json:"next_detection_at,omitempty"`
+	DetectionIntervalMinutes int32      `json:"detection_interval_minutes,omitempty"`
+	LastDetectedAt           *time.Time `json:"last_detected_at,omitempty"`
+	LastDetectedCount        int        `json:"last_detected_count,omitempty"`
+	LastDetectionError       string     `json:"last_detection_error,omitempty"`
+	LastDetectionSkipped     string     `json:"last_detection_skipped,omitempty"`
+}
+
+type schedulerDetectionInfo struct {
+	lastDetectedAt    time.Time
+	lastDetectedCount int
+	lastErrorAt       time.Time
+	lastError         string
+	lastSkippedAt     time.Time
+	lastSkippedReason string
+}
+
+type schedulerRunInfo struct {
+	lastRunStartedAt   time.Time
+	lastRunCompletedAt time.Time
+	lastRunStatus      string
+}
+
+type schedulerLoopState struct {
+	currentJobType         string
+	currentPhase           string
+	lastIterationHadJobs   bool
+	lastIterationCompleted time.Time
+}
+
+func (r *Plugin) recordSchedulerDetectionSuccess(jobType string, count int) {
+	if r == nil {
+		return
+	}
+	r.schedulerDetectionMu.Lock()
+	defer r.schedulerDetectionMu.Unlock()
+	info := r.schedulerDetection[jobType]
+	if info == nil {
+		info = &schedulerDetectionInfo{}
+		r.schedulerDetection[jobType] = info
+	}
+	info.lastDetectedAt = time.Now().UTC()
+	info.lastDetectedCount = count
+	info.lastError = ""
+	info.lastSkippedReason = ""
+}
+
+func (r *Plugin) recordSchedulerDetectionError(jobType string, err error) {
+	if r == nil {
+		return
+	}
+	if err == nil {
+		return
+	}
+	r.schedulerDetectionMu.Lock()
+	defer r.schedulerDetectionMu.Unlock()
+	info := r.schedulerDetection[jobType]
+	if info == nil {
+		info = &schedulerDetectionInfo{}
+		r.schedulerDetection[jobType] = info
+	}
+	info.lastErrorAt = time.Now().UTC()
+	info.lastError = err.Error()
+}
+
+func (r *Plugin) recordSchedulerDetectionSkip(jobType string, reason string) {
+	if r == nil {
+		return
+	}
+	if strings.TrimSpace(reason) == "" {
+		return
+	}
+	r.schedulerDetectionMu.Lock()
+	defer r.schedulerDetectionMu.Unlock()
+	info := r.schedulerDetection[jobType]
+	if info == nil {
+		info = &schedulerDetectionInfo{}
+		r.schedulerDetection[jobType] = info
+	}
+	info.lastSkippedAt = time.Now().UTC()
+	info.lastSkippedReason = reason
+}
+
+func (r *Plugin) snapshotSchedulerDetection(jobType string) schedulerDetectionInfo {
+	if r == nil {
+		return schedulerDetectionInfo{}
+	}
+	r.schedulerDetectionMu.Lock()
+	defer r.schedulerDetectionMu.Unlock()
+	info := r.schedulerDetection[jobType]
+	if info == nil {
+		return schedulerDetectionInfo{}
+	}
+	return *info
+}
+
+func (r *Plugin) recordSchedulerRunStart(jobType string) {
+	if r == nil {
+		return
+	}
+	r.schedulerRunMu.Lock()
+	defer r.schedulerRunMu.Unlock()
+	info := r.schedulerRun[jobType]
+	if info == nil {
+		info = &schedulerRunInfo{}
+		r.schedulerRun[jobType] = info
+	}
+	info.lastRunStartedAt = time.Now().UTC()
+	info.lastRunStatus = ""
+}
+
+func (r *Plugin) recordSchedulerRunComplete(jobType, status string) {
+	if r == nil {
+		return
+	}
+	r.schedulerRunMu.Lock()
+	defer r.schedulerRunMu.Unlock()
+	info := r.schedulerRun[jobType]
+	if info == nil {
+		info = &schedulerRunInfo{}
+		r.schedulerRun[jobType] = info
+	}
+	info.lastRunCompletedAt = time.Now().UTC()
+	info.lastRunStatus = status
+}
+
+func (r *Plugin) snapshotSchedulerRun(jobType string) schedulerRunInfo {
+	if r == nil {
+		return schedulerRunInfo{}
+	}
+	r.schedulerRunMu.Lock()
+	defer r.schedulerRunMu.Unlock()
+	info := r.schedulerRun[jobType]
+	if info == nil {
+		return schedulerRunInfo{}
+	}
+	return *info
+}
+
+func (r *Plugin) setSchedulerLoopState(jobType, phase string) {
+	if r == nil {
+		return
+	}
+	r.schedulerLoopMu.Lock()
+	r.schedulerLoopState.currentJobType = jobType
+	r.schedulerLoopState.currentPhase = phase
+	r.schedulerLoopMu.Unlock()
+}
+
+// setSchedulerLoopStateForJobType keeps the aggregate scheduler state and the
+// owning lane state in sync while a specific job type is active.
+func (r *Plugin) setSchedulerLoopStateForJobType(jobType, phase string) {
+	if r == nil {
+		return
+	}
+	r.setSchedulerLoopState(jobType, phase)
+	if jobType == "" {
+		return
+	}
+	if ls := r.lanes[JobTypeLane(jobType)]; ls != nil {
+		r.setLaneLoopState(ls, jobType, phase)
+	}
+}
+
+func (r *Plugin) snapshotSchedulerLoopState() schedulerLoopState {
+	if r == nil {
+		return schedulerLoopState{}
+	}
+	r.schedulerLoopMu.Lock()
+	defer r.schedulerLoopMu.Unlock()
+	return r.schedulerLoopState
+}
+
+// aggregateLaneLoopStates merges per-lane loop states into a single
+// schedulerLoopState for the aggregate GetSchedulerStatus API. It picks
+// the most recent iteration completion, any currently-active job type,
+// and a phase that reflects whether any lane is actively working.
+func (r *Plugin) aggregateLaneLoopStates() schedulerLoopState {
+	if r == nil || len(r.lanes) == 0 {
+		return r.snapshotSchedulerLoopState()
+	}
+
+	var agg schedulerLoopState
+	for _, ls := range r.lanes {
+		snap := r.snapshotLaneLoopState(ls)
+		if snap.lastIterationCompleted.After(agg.lastIterationCompleted) {
+			agg.lastIterationCompleted = snap.lastIterationCompleted
+		}
+		if snap.lastIterationHadJobs {
+			agg.lastIterationHadJobs = true
+		}
+		// Prefer showing an active phase over idle/sleeping.
+		if snap.currentJobType != "" {
+			agg.currentJobType = snap.currentJobType
+			agg.currentPhase = snap.currentPhase
+		}
+	}
+	// If no lane is actively processing, show the most interesting phase.
+	if agg.currentPhase == "" {
+		for _, ls := range r.lanes {
+			snap := r.snapshotLaneLoopState(ls)
+			if snap.currentPhase != "" {
+				agg.currentPhase = snap.currentPhase
+				break
+			}
+		}
+	}
+	return agg
+}
+
+// --- Per-lane loop state helpers ---
+
+func (r *Plugin) setLaneLoopState(ls *schedulerLaneState, jobType, phase string) {
+	if r == nil || ls == nil {
+		return
+	}
+	ls.loopMu.Lock()
+	ls.loop.currentJobType = jobType
+	ls.loop.currentPhase = phase
+	ls.loopMu.Unlock()
+}
+
+func (r *Plugin) recordLaneIterationComplete(ls *schedulerLaneState, hadJobs bool) {
+	if r == nil || ls == nil {
+		return
+	}
+	ls.loopMu.Lock()
+	ls.loop.lastIterationHadJobs = hadJobs
+	ls.loop.lastIterationCompleted = time.Now().UTC()
+	ls.loopMu.Unlock()
+}
+
+func (r *Plugin) snapshotLaneLoopState(ls *schedulerLaneState) schedulerLoopState {
+	if r == nil || ls == nil {
+		return schedulerLoopState{}
+	}
+	ls.loopMu.Lock()
+	defer ls.loopMu.Unlock()
+	return ls.loop
+}
+
+// GetLaneSchedulerStatus returns scheduler status scoped to a single lane.
+func (r *Plugin) GetLaneSchedulerStatus(lane SchedulerLane) SchedulerStatus {
+	ls := r.lanes[lane]
+	if ls == nil {
+		return SchedulerStatus{Now: time.Now().UTC()}
+	}
+	now := time.Now().UTC()
+	loopState := r.snapshotLaneLoopState(ls)
+	idleSleep := LaneIdleSleep(lane)
+	allInProcess := r.listInProcessJobs(now)
+	laneInProcess := make([]SchedulerJobStatus, 0, len(allInProcess))
+	for _, job := range allInProcess {
+		if JobTypeLane(job.JobType) == lane {
+			laneInProcess = append(laneInProcess, job)
+		}
+	}
+
+	status := SchedulerStatus{
+		Now:                  now,
+		Lane:                 string(lane),
+		SchedulerTickSeconds: int(secondsFromDuration(r.schedulerTick)),
+		InProcessJobs:        laneInProcess,
+		IdleSleepSeconds:     int(idleSleep / time.Second),
+		CurrentJobType:       loopState.currentJobType,
+		CurrentPhase:         loopState.currentPhase,
+		LastIterationHadJobs: loopState.lastIterationHadJobs,
+	}
+	nextDetectionAt := r.earliestLaneDetectionAt(lane)
+	if nextDetectionAt.IsZero() && loopState.currentPhase == "sleeping" && !loopState.lastIterationCompleted.IsZero() {
+		nextDetectionAt = loopState.lastIterationCompleted.Add(idleSleep)
+	}
+	if !nextDetectionAt.IsZero() {
+		at := nextDetectionAt
+		status.NextDetectionAt = &at
+	}
+	if !loopState.lastIterationCompleted.IsZero() {
+		at := loopState.lastIterationCompleted
+		status.LastIterationDoneAt = &at
+	}
+
+	states, err := r.ListSchedulerStates()
+	if err != nil {
+		return status
+	}
+
+	waiting := make([]SchedulerWaitingStatus, 0)
+	jobTypes := make([]SchedulerJobTypeStatus, 0)
+
+	for _, state := range states {
+		if JobTypeLane(state.JobType) != lane {
+			continue
+		}
+		jobType := state.JobType
+		info := r.snapshotSchedulerDetection(jobType)
+
+		jobStatus := SchedulerJobTypeStatus{
+			JobType:                  jobType,
+			Enabled:                  state.Enabled,
+			DetectionInFlight:        state.DetectionInFlight,
+			NextDetectionAt:          state.NextDetectionAt,
+			DetectionIntervalMinutes: state.DetectionIntervalMinutes,
+		}
+		if !info.lastDetectedAt.IsZero() {
+			jobStatus.LastDetectedAt = timeToPtr(info.lastDetectedAt)
+			jobStatus.LastDetectedCount = info.lastDetectedCount
+		}
+		if info.lastError != "" {
+			jobStatus.LastDetectionError = info.lastError
+		}
+		if info.lastSkippedReason != "" {
+			jobStatus.LastDetectionSkipped = info.lastSkippedReason
+		}
+		jobTypes = append(jobTypes, jobStatus)
+
+		if state.DetectionInFlight {
+			waiting = append(waiting, SchedulerWaitingStatus{
+				Reason:  "detection_in_flight",
+				JobType: jobType,
+			})
+		} else if state.Enabled && state.NextDetectionAt != nil && now.Before(*state.NextDetectionAt) {
+			waiting = append(waiting, SchedulerWaitingStatus{
+				Reason:  "next_detection_at",
+				JobType: jobType,
+				Until:   state.NextDetectionAt,
+			})
+		}
+	}
+
+	sort.Slice(jobTypes, func(i, j int) bool {
+		return jobTypes[i].JobType < jobTypes[j].JobType
+	})
+
+	status.Waiting = waiting
+	status.JobTypes = jobTypes
+	return status
+}
+
+func (r *Plugin) GetSchedulerStatus() SchedulerStatus {
+	now := time.Now().UTC()
+
+	// Aggregate loop state across all lanes instead of reading the
+	// legacy single-loop state which is no longer updated.
+	aggregated := r.aggregateLaneLoopStates()
+
+	status := SchedulerStatus{
+		Now:                  now,
+		SchedulerTickSeconds: int(secondsFromDuration(r.schedulerTick)),
+		InProcessJobs:        r.listInProcessJobs(now),
+		IdleSleepSeconds:     int(defaultSchedulerIdleSleep / time.Second),
+		CurrentJobType:       aggregated.currentJobType,
+		CurrentPhase:         aggregated.currentPhase,
+		LastIterationHadJobs: aggregated.lastIterationHadJobs,
+	}
+	nextDetectionAt := r.earliestNextDetectionAt()
+	if nextDetectionAt.IsZero() && aggregated.currentPhase == "sleeping" && !aggregated.lastIterationCompleted.IsZero() {
+		nextDetectionAt = aggregated.lastIterationCompleted.Add(defaultSchedulerIdleSleep)
+	}
+	if !nextDetectionAt.IsZero() {
+		at := nextDetectionAt
+		status.NextDetectionAt = &at
+	}
+	if !aggregated.lastIterationCompleted.IsZero() {
+		at := aggregated.lastIterationCompleted
+		status.LastIterationDoneAt = &at
+	}
+
+	states, err := r.ListSchedulerStates()
+	if err != nil {
+		return status
+	}
+
+	waiting := make([]SchedulerWaitingStatus, 0)
+	jobTypes := make([]SchedulerJobTypeStatus, 0, len(states))
+
+	for _, state := range states {
+		jobType := state.JobType
+		info := r.snapshotSchedulerDetection(jobType)
+
+		jobStatus := SchedulerJobTypeStatus{
+			JobType:                  jobType,
+			Enabled:                  state.Enabled,
+			DetectionInFlight:        state.DetectionInFlight,
+			NextDetectionAt:          state.NextDetectionAt,
+			DetectionIntervalMinutes: state.DetectionIntervalMinutes,
+		}
+		if !info.lastDetectedAt.IsZero() {
+			jobStatus.LastDetectedAt = timeToPtr(info.lastDetectedAt)
+			jobStatus.LastDetectedCount = info.lastDetectedCount
+		}
+		if info.lastError != "" {
+			jobStatus.LastDetectionError = info.lastError
+		}
+		if info.lastSkippedReason != "" {
+			jobStatus.LastDetectionSkipped = info.lastSkippedReason
+		}
+		jobTypes = append(jobTypes, jobStatus)
+
+		if state.DetectionInFlight {
+			waiting = append(waiting, SchedulerWaitingStatus{
+				Reason:  "detection_in_flight",
+				JobType: jobType,
+			})
+		} else if state.Enabled && state.NextDetectionAt != nil && now.Before(*state.NextDetectionAt) {
+			waiting = append(waiting, SchedulerWaitingStatus{
+				Reason:  "next_detection_at",
+				JobType: jobType,
+				Until:   state.NextDetectionAt,
+			})
+		}
+	}
+
+	sort.Slice(jobTypes, func(i, j int) bool {
+		return jobTypes[i].JobType < jobTypes[j].JobType
+	})
+
+	status.Waiting = waiting
+	status.JobTypes = jobTypes
+	return status
+}
+
+func (r *Plugin) listInProcessJobs(now time.Time) []SchedulerJobStatus {
+	active := make([]SchedulerJobStatus, 0)
+	if r == nil {
+		return active
+	}
+
+	r.jobsMu.RLock()
+	for _, job := range r.jobs {
+		if job == nil {
+			continue
+		}
+		if !isActiveTrackedJobState(job.State) {
+			continue
+		}
+		start := timeToPtr(now)
+		if job.CreatedAt != nil && !job.CreatedAt.IsZero() {
+			start = job.CreatedAt
+		} else if job.UpdatedAt != nil && !job.UpdatedAt.IsZero() {
+			start = job.UpdatedAt
+		}
+		durationSeconds := 0.0
+		if start != nil {
+			durationSeconds = now.Sub(*start).Seconds()
+		}
+		active = append(active, SchedulerJobStatus{
+			JobID:           job.JobID,
+			JobType:         job.JobType,
+			State:           strings.ToLower(job.State),
+			Stage:           job.Stage,
+			WorkerID:        job.WorkerID,
+			Message:         job.Message,
+			Progress:        job.Progress,
+			CreatedAt:       job.CreatedAt,
+			UpdatedAt:       job.UpdatedAt,
+			DurationSeconds: durationSeconds,
+		})
+	}
+	r.jobsMu.RUnlock()
+
+	sort.Slice(active, func(i, j int) bool {
+		if active[i].DurationSeconds != active[j].DurationSeconds {
+			return active[i].DurationSeconds > active[j].DurationSeconds
+		}
+		return active[i].JobID < active[j].JobID
+	})
+
+	return active
+}

weed/admin/plugin/scheduler_status_test.go

@@ -0,0 +1,147 @@
+package plugin
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
+)
+
+func TestGetSchedulerStatusIncludesInProcessJobs(t *testing.T) {
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	pluginSvc.trackExecutionStart("req-1", "worker-a", &plugin_pb.JobSpec{
+		JobId:   "job-1",
+		JobType: "vacuum",
+	}, 1)
+
+	status := pluginSvc.GetSchedulerStatus()
+	if len(status.InProcessJobs) != 1 {
+		t.Fatalf("expected one in-process job, got %d", len(status.InProcessJobs))
+	}
+	if status.InProcessJobs[0].JobID != "job-1" {
+		t.Fatalf("unexpected job id: %s", status.InProcessJobs[0].JobID)
+	}
+}
+
+func TestGetSchedulerStatusIncludesLastDetectionCount(t *testing.T) {
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	const jobType = "vacuum"
+	pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
+		WorkerId: "worker-a",
+		Capabilities: []*plugin_pb.JobTypeCapability{
+			{JobType: jobType, CanDetect: true},
+		},
+	})
+
+	pluginSvc.recordSchedulerDetectionSuccess(jobType, 3)
+
+	status := pluginSvc.GetSchedulerStatus()
+	found := false
+	for _, jt := range status.JobTypes {
+		if jt.JobType != jobType {
+			continue
+		}
+		found = true
+		if jt.LastDetectedCount != 3 {
+			t.Fatalf("unexpected last detected count: got=%d want=3", jt.LastDetectedCount)
+		}
+		if jt.LastDetectedAt == nil {
+			t.Fatalf("expected last detected at to be set")
+		}
+	}
+	if !found {
+		t.Fatalf("expected job type status for %s", jobType)
+	}
+}
+
+func TestGetLaneSchedulerStatusShowsActiveConcurrentLaneWork(t *testing.T) {
+	clusterContextStarted := make(chan struct{})
+	releaseClusterContext := make(chan struct{})
+
+	// Create the Plugin without a ClusterContextProvider so no background
+	// scheduler goroutines are started; they would race with the direct
+	// runJobTypeIteration call below.
+	pluginSvc, err := New(Options{})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer pluginSvc.Shutdown()
+
+	// Set the provider after construction so runJobTypeIteration can use it.
+	pluginSvc.clusterContextProvider = func(context.Context) (*plugin_pb.ClusterContext, error) {
+		close(clusterContextStarted)
+		<-releaseClusterContext
+		return nil, context.Canceled
+	}
+
+	const jobType = "s3_lifecycle"
+	err = pluginSvc.SaveJobTypeConfig(&plugin_pb.PersistedJobTypeConfig{
+		JobType: jobType,
+		AdminRuntime: &plugin_pb.AdminRuntimeConfig{
+			Enabled:                  true,
+			DetectionIntervalMinutes: 30,
+			DetectionTimeoutSeconds:  15,
+		},
+	})
+	if err != nil {
+		t.Fatalf("SaveJobTypeConfig: %v", err)
+	}
+
+	policy, enabled, err := pluginSvc.loadSchedulerPolicy(jobType)
+	if err != nil {
+		t.Fatalf("loadSchedulerPolicy: %v", err)
+	}
+	if !enabled {
+		t.Fatalf("expected enabled policy")
+	}
+
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		pluginSvc.runJobTypeIteration(jobType, policy)
+	}()
+
+	select {
+	case <-clusterContextStarted:
+	case <-time.After(time.Second):
+		t.Fatalf("timed out waiting for job type iteration to start")
+	}
+
+	var laneStatus SchedulerStatus
+	var aggregateStatus SchedulerStatus
+	deadline := time.Now().Add(time.Second)
+	for time.Now().Before(deadline) {
+		laneStatus = pluginSvc.GetLaneSchedulerStatus(LaneLifecycle)
+		aggregateStatus = pluginSvc.GetSchedulerStatus()
+		if laneStatus.CurrentJobType == jobType && laneStatus.CurrentPhase == "detecting" &&
+			aggregateStatus.CurrentJobType == jobType && aggregateStatus.CurrentPhase == "detecting" {
+			break
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+
+	if laneStatus.CurrentJobType != jobType || laneStatus.CurrentPhase != "detecting" {
+		t.Fatalf("unexpected lane status while work is active: job=%q phase=%q", laneStatus.CurrentJobType, laneStatus.CurrentPhase)
+	}
+	if aggregateStatus.CurrentJobType != jobType || aggregateStatus.CurrentPhase != "detecting" {
+		t.Fatalf("unexpected aggregate status while work is active: job=%q phase=%q", aggregateStatus.CurrentJobType, aggregateStatus.CurrentPhase)
+	}
+
+	close(releaseClusterContext)
+	select {
+	case <-done:
+	case <-time.After(time.Second):
+		t.Fatalf("timed out waiting for job type iteration to finish")
+	}
+}

weed/admin/plugin/types.go

@@ -0,0 +1,113 @@
+package plugin
+
+import (
+	"errors"
+	"time"
+)
+
+const (
+	// Keep exactly the last 10 successful and last 10 error runs per job type.
+	MaxSuccessfulRunHistory = 10
+	MaxErrorRunHistory      = 10
+)
+
+var ErrJobNotFound = errors.New("job not found")
+
+type RunOutcome string
+
+const (
+	RunOutcomeSuccess RunOutcome = "success"
+	RunOutcomeError   RunOutcome = "error"
+)
+
+type JobRunRecord struct {
+	RunID       string     `json:"run_id"`
+	JobID       string     `json:"job_id"`
+	JobType     string     `json:"job_type"`
+	WorkerID    string     `json:"worker_id"`
+	Outcome     RunOutcome `json:"outcome"`
+	Message     string     `json:"message,omitempty"`
+	DurationMs  int64      `json:"duration_ms,omitempty"`
+	CompletedAt *time.Time `json:"completed_at,omitempty"`
+}
+
+type JobTypeRunHistory struct {
+	JobType         string         `json:"job_type"`
+	SuccessfulRuns  []JobRunRecord `json:"successful_runs"`
+	ErrorRuns       []JobRunRecord `json:"error_runs"`
+	LastUpdatedTime *time.Time     `json:"last_updated_time,omitempty"`
+}
+
+type TrackedJob struct {
+	JobID              string                 `json:"job_id"`
+	JobType            string                 `json:"job_type"`
+	RequestID          string                 `json:"request_id"`
+	WorkerID           string                 `json:"worker_id"`
+	DedupeKey          string                 `json:"dedupe_key,omitempty"`
+	Summary            string                 `json:"summary,omitempty"`
+	Detail             string                 `json:"detail,omitempty"`
+	Parameters         map[string]interface{} `json:"parameters,omitempty"`
+	Labels             map[string]string      `json:"labels,omitempty"`
+	State              string                 `json:"state"`
+	Progress           float64                `json:"progress"`
+	Stage              string                 `json:"stage,omitempty"`
+	Message            string                 `json:"message,omitempty"`
+	Attempt            int32                  `json:"attempt,omitempty"`
+	CreatedAt          *time.Time             `json:"created_at,omitempty"`
+	UpdatedAt          *time.Time             `json:"updated_at,omitempty"`
+	CompletedAt        *time.Time             `json:"completed_at,omitempty"`
+	ErrorMessage       string                 `json:"error_message,omitempty"`
+	ResultSummary      string                 `json:"result_summary,omitempty"`
+	ResultOutputValues map[string]interface{} `json:"result_output_values,omitempty"`
+}
+
+type JobActivity struct {
+	JobID      string                 `json:"job_id"`
+	JobType    string                 `json:"job_type"`
+	RequestID  string                 `json:"request_id,omitempty"`
+	WorkerID   string                 `json:"worker_id,omitempty"`
+	Source     string                 `json:"source"`
+	Message    string                 `json:"message"`
+	Stage      string                 `json:"stage,omitempty"`
+	Details    map[string]interface{} `json:"details,omitempty"`
+	OccurredAt *time.Time             `json:"occurred_at,omitempty"`
+}
+
+type JobDetail struct {
+	Job         *TrackedJob   `json:"job"`
+	RunRecord   *JobRunRecord `json:"run_record,omitempty"`
+	Activities  []JobActivity `json:"activities"`
+	RelatedJobs []TrackedJob  `json:"related_jobs,omitempty"`
+	LastUpdated *time.Time    `json:"last_updated,omitempty"`
+}
+
+type SchedulerJobTypeState struct {
+	JobType                       string     `json:"job_type"`
+	Lane                          string     `json:"lane"`
+	Enabled                       bool       `json:"enabled"`
+	PolicyError                   string     `json:"policy_error,omitempty"`
+	DetectionInFlight             bool       `json:"detection_in_flight"`
+	NextDetectionAt               *time.Time `json:"next_detection_at,omitempty"`
+	DetectionIntervalMinutes      int32      `json:"detection_interval_minutes,omitempty"`
+	DetectionTimeoutSeconds       int32      `json:"detection_timeout_seconds,omitempty"`
+	ExecutionTimeoutSeconds       int32      `json:"execution_timeout_seconds,omitempty"`
+	JobTypeMaxRuntimeSeconds      int32      `json:"job_type_max_runtime_seconds,omitempty"`
+	MaxJobsPerDetection           int32      `json:"max_jobs_per_detection,omitempty"`
+	GlobalExecutionConcurrency    int        `json:"global_execution_concurrency,omitempty"`
+	PerWorkerExecutionConcurrency int        `json:"per_worker_execution_concurrency,omitempty"`
+	RetryLimit                    int        `json:"retry_limit,omitempty"`
+	RetryBackoffSeconds           int32      `json:"retry_backoff_seconds,omitempty"`
+	DetectorAvailable             bool       `json:"detector_available"`
+	DetectorWorkerID              string     `json:"detector_worker_id,omitempty"`
+	ExecutorWorkerCount           int        `json:"executor_worker_count"`
+	LastRunStartedAt              *time.Time `json:"last_run_started_at,omitempty"`
+	LastRunCompletedAt            *time.Time `json:"last_run_completed_at,omitempty"`
+	LastRunStatus                 string     `json:"last_run_status,omitempty"`
+}
+
+func timeToPtr(t time.Time) *time.Time {
+	if t.IsZero() {
+		return nil
+	}
+	return &t
+}