snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

seaweed-worker/README.md

@@ -0,0 +1,110 @@
+# SeaweedFS Rust workers
+
+`weed/pb/plugin.proto` is a language-agnostic contract: a maintenance worker
+connects out to admin, announces the job types it can detect and execute, and
+answers requests on that one stream. `weed worker -admin=host:23646` is the Go
+implementation of it from outside the admin process. This workspace is the Rust
+one.
+
+    crates/core     the contract: stream, handshake, heartbeat, registry, config forms
+    crates/lance    maintenance jobs for Lance tables, and a binary
+
+`core` knows nothing about any job. A second worker is a new crate beside
+`lance` that depends on it, not a fork of the protocol.
+
+## Building
+
+Requires Rust 1.94.1+ (2024 edition), matching `rust-version` in `Cargo.toml`.
+The patch release matters: 1.94.0 does not build. The edition itself only needs
+1.85; the higher floor comes from the dependency tree — lance's `aws` feature
+pulls in the AWS SDK — so it moves with those crates. CI builds on the latest
+stable.
+
+`core` compiles `plugin.proto` with the protoc that protoc-bin-vendored ships,
+the way seaweed-volume does, so it needs no system install.
+
+The lance crates compile protos of their own, in their own build-script
+processes, which nothing our build script sets can reach. They need a protoc of
+their own: either one on PATH — `brew install protobuf`, `apt install
+protobuf-compiler` — or `PROTOC` naming one. CI points it at the vendored
+binary for the runner's platform, resolved from the version in `Cargo.lock`.
+
+## Running
+
+    cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646
+
+The admin's *HTTP* address is what an operator has; the gRPC port is derived
+from it the way the Go side does. Dialling the HTTP port fails as "frame with
+invalid size", which reads like a protocol bug rather than a wrong port.
+
+The binary is `weed-worker`, not `weed-lance-worker`: it is the Rust side of
+`weed worker`, and lance is the first family of jobs it carries rather than the
+only one it ever will.
+
+Released builds do not need a toolchain. The worker ships inside the SeaweedFS
+image, beside the Rust volume server, under the verb that mirrors
+`volume-rust`:
+
+    docker run chrislusf/seaweedfs worker-rust --admin admin:23646
+
+and as `weed-worker_linux_{amd64,arm64}.tar.gz` on each GitHub release. Both are
+linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
+expensive build, and the worker runs beside the cluster it maintains. On an
+architecture without a build the image carries an empty placeholder and the
+entrypoint says so rather than failing as "not found".
+
+## Metrics
+
+    cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646 --metrics-port 9328
+
+Serves `/health`, `/ready` and `/metrics` on that port, the same three the Go
+worker serves under `weed worker -metricsPort`, so one scrape config covers
+workers in either language. Off by default, and bound to loopback unless
+`--metrics-ip` says otherwise, because the endpoint is unauthenticated. 9328
+continues the series the other components use (master 9324, volume 9325, filer
+9326, s3 9327); an IPv6 address works with or without brackets.
+
+Grafana: the "Plugin Workers" row of `other/metrics/grafana_seaweedfs.json`
+graphs these. Its panels filter on `$cluster`, which comes from the scrape job's
+labels, so scrape the worker the way the rest of the cluster is scraped or the
+row stays empty.
+
+Names are `SeaweedFS_worker_*`, matching the Go side's convention. The pair
+worth alerting on is `objects_seen_total` and `objects_skipped_total`: a sweep
+that proposes nothing and a sweep that could read nothing look identical from
+`proposals_total` alone.
+
+    SeaweedFS_worker_connected 1
+    SeaweedFS_worker_objects_seen_total{job_type="lance_compact"} 7
+    SeaweedFS_worker_proposals_total{job_type="lance_compact"} 2
+    SeaweedFS_worker_jobs_total{job_type="lance_compact",result="ok"} 2
+    SeaweedFS_worker_lance_fragments_removed_total 25
+
+`/ready` follows the control stream: a worker whose admin has gone away is
+running but is not going to do anything.
+
+## Credentials
+
+The worker holds none. It asks the namespace to describe a table with
+`vend_credentials` and hands the `storage_options` that come back to lance. A
+gateway without STS configured vends no credentials at all, so `--access-key`
+and `--secret-key` supply a fallback; anything the namespace does vend wins over
+them.
+
+## State
+
+All three jobs are implemented and tested end to end against a live gateway:
+
+    compaction result: 12 fragments became 1
+    reindex result:    512 uncovered rows became 0
+    cleanup result:    removed 14 versions and 24272 bytes
+
+`cargo test -p weed-lance-worker` runs them when `WEED_LANCE_NAMESPACE` names a
+live namespace and skips otherwise, the way the Go integration tests skip
+without Docker. Each test seeds the table it needs, including building a vector
+index and then appending rows outside it, so a run does not depend on what the
+previous one left behind — the first version of these did, and quietly stopped
+testing anything once it had done its job.
+
+The handshake, descriptor exchange and heartbeat work against a live admin,
+which logs the worker connecting and prefetches all three descriptors.