snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)
Some checks failed
go: build dev binaries / cleanup (push) Successful in 4s
Codespell / Check for spelling errors (push) Failing after 3s
docker: build dev containers / build-rust-binaries (amd64, x86_64-unknown-linux-musl) (push) Failing after 3s
docker: build dev containers / build-dev-containers (push) Has been skipped
docker: build dev containers / build-rust-binaries (arm64, true, aarch64-unknown-linux-musl) (push) Failing after 3s
End to End / FUSE Mount (push) Failing after 2s
EC Integration Tests / EC Integration Tests (push) Failing after 30s
FUSE DLM Integration Tests / FUSE DLM Integration Tests (push) Failing after 3s
FUSE Volume Server Failover Tests / FUSE Volume Server Failover (push) Failing after 3s
FUSE Integration Tests / FUSE Integration Testing (push) Failing after 2s
FUSE P2P Peer Chunk Sharing Integration Tests / FUSE P2P Peer Chunk Sharing (push) Failing after 3s
go: build binary / Go Vet (push) Failing after 2s
go: build binary / Go Vet 32-bit (push) Failing after 3s
go: build binary / Build (push) Failing after 4s
go: build binary / Build cross-platform (freebsd/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (darwin/arm64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/arm64) (push) Failing after 3s
go: build binary / Test (push) Failing after 2s
go: build binary / Test 32-bit (push) Failing after 3s
helm: lint and test charts / lint-test (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (11) (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (17) (push) Failing after 3s
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Failing after 2s
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Failing after 3s
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been skipped
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been skipped
Java Client Unit Tests / Java Unit Tests (11) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (17) (push) Failing after 2s
Java Client Unit Tests / Java Unit Tests (21) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (8) (push) Failing after 3s
Kafka Quick Test (Load Test with Schema Registry) / Kafka Client Load Test (Quick) (push) Failing after 3s
Kafka Gateway Tests / Kafka Unit Tests (unit-tests-1) (push) Failing after 11s
Kafka Gateway Tests / Kafka Integration Tests (Critical) (integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka End-to-End Tests (with SMQ) (e2e-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Consumer Group Tests (Highly Isolated) (consumer-group-1) (push) Failing after 2s
Kafka Gateway Tests / Kafka Client Compatibility (with SMQ) (client-compat-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka SMQ Integration Tests (Full Stack) (smq-integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Protocol Tests (Isolated) (protocol-1) (push) Failing after 3s
KMS Tests / KMS Provider Integration Tests (push) Failing after 2s
KMS Tests / S3 KMS End-to-End Tests (push) Failing after 3s
Master Cold Start Tests / Master Cold Start Tests (push) Failing after 4s
Metadata Subscribe Integration Tests / Metadata Subscribe Integration Tests (push) Failing after 3s
Multi-Master Tests / Multi-Master Failover Tests (push) Failing after 3s
Performance / CPU and Heap Profile (push) Failing after 2s
Performance / S3 Read/Write Benchmark (rust volume) (push) Failing after 3s
pjdfstest POSIX Compliance / pjdfstest (push) Failing after 2s
Plugin Worker Integration Tests / Plugin Worker: erasure_coding (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: vacuum (push) Failing after 3s
Rust Volume Server Tests / Rust Integration Tests (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 1) (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 1) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 3) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Distributed Tests (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Performance Tests (push) Failing after 3s
Performance / Throughput Benchmark (go volume) (push) Failing after 3s
Performance / Throughput Benchmark (rust volume) (push) Failing after 2s
Performance / S3 Read/Write Benchmark (go volume) (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: volume_balance (push) Failing after 3s
PostgreSQL Gateway Tests / PostgreSQL Basic Tests (push) Failing after 3s
Rust Volume Server Tests / Rust Unit Tests (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 3) (push) Failing after 3s
Rust Plugin Worker Tests / Rust Plugin Worker Build and Unit Tests (push) Failing after 3s
rust: build dev volume server binaries / cleanup (push) Successful in 3s
S3 ETag and ACL Tests / S3 ETag + ACL Integration Tests (push) Failing after 3s
S3 IAM Integration Tests / IAM Unit Tests (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (advanced) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (basic) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (group) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (policy-enforcement) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (sts) (push) Failing after 4s
S3 Keycloak Integration Tests / S3 Keycloak Integration Tests (push) Failing after 3s
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.11) (push) Failing after 3s
S3 PyArrow Parquet Tests / Go Unit Tests (Implicit Directory) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.12) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-amd64, x86_64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-arm64, aarch64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-linux (linux-amd64, x86_64-unknown-linux-gnu) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.9) (push) Has been cancelled
S3 Proxy Signature Tests / S3 Proxy Signature Verification Tests (push) Has been cancelled
S3 SDK V2 Route Disambiguation Tests / S3 SDK V2 Routing Tests (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (quick) (push) Has been cancelled
S3 SSE Tests / S3 SSE Compatibility Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Metadata Persistence Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Copy Operations Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Multipart Upload Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Performance Test (push) Has been cancelled
S3 SSE Tests / S3 Volume Encryption Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (comprehensive) (push) Has been cancelled
Spark Integration Tests / Spark Integration Tests (push) Failing after 2s
Telemetry Integration Tests / telemetry-integration-test (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 1) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 1) (push) Failing after 3s
Ceph S3 tests / Basic S3 tests (KV store) (push) Has been cancelled
Ceph S3 tests / S3 Versioning & Object Lock tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Unit Tests (push) Failing after 3s
Ceph S3 tests / S3 CORS tests (push) Has been cancelled
Ceph S3 tests / SeaweedFS Custom S3 Copy tests (push) Has been cancelled
Ceph S3 tests / Basic S3 tests (SQL store) (push) Has been cancelled
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
TLS Rotation Integration Tests / TLS Rotation Integration Tests (push) Has been cancelled
Vacuum Integration Tests / Vacuum Integration Tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Variables Integration Tests (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (advanced-policy) (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (basic-policy) (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (contextual-policy) (push) Failing after 4s
S3 Policy Integration Tests / Trusted Proxy Detection Tests (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Shell Integration Tests (push) Failing after 3s
Samba on FUSE Integration / samba-integration (push) Failing after 2s
SFTP Integration Tests / SFTP Integration Testing (push) Failing after 3s
terraform: validate and test modules / fmt, validate, plan-level tests (push) Failing after 3s
terraform: validate and test modules / local cluster smoke test (real weed) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 3) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 2) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 2) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 3) (push) Failing after 5s
mount: windows conformance / WinFsp conformance (push) Has been cancelled
mount: windows / Mount on Windows (push) Has been cancelled

This commit is contained in:
upstream-mirror
2026-09-16 03:23:42 +05:00
commit 89f5be005d
3997 changed files with 961970 additions and 0 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 71 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 110 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 67 KiB

BIN
note/SeaweedFS_XDR.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 62 KiB

66
note/icon.svg Normal file
View File

@@ -0,0 +1,66 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg id="Ebene_2" data-name="Ebene 2" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 512 512">
<defs>
<style>
.cls-1 {
fill: url(#Unbenannter_Verlauf_20);
}
.cls-2 {
fill: url(#Unbenannter_Verlauf_32);
}
.cls-3 {
fill: url(#Unbenannter_Verlauf_42);
}
.cls-4 {
fill: #fff;
}
.cls-5 {
fill: url(#Unbenannter_Verlauf_12);
}
.cls-6 {
fill: url(#Unbenannter_Verlauf_28);
}
</style>
<linearGradient id="Unbenannter_Verlauf_12" data-name="Unbenannter Verlauf 12" x1="19.92" y1="19.92" x2="492.08" y2="492.08" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#0282e9"/>
<stop offset=".48" stop-color="#0162bf"/>
<stop offset=".87" stop-color="#01469b"/>
</linearGradient>
<linearGradient id="Unbenannter_Verlauf_42" data-name="Unbenannter Verlauf 42" x1="180.67" y1="151.78" x2="234.43" y2="384.66" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#23b5e6"/>
<stop offset=".41" stop-color="#138bcf"/>
<stop offset=".86" stop-color="#0158b3"/>
</linearGradient>
<linearGradient id="Unbenannter_Verlauf_28" data-name="Unbenannter Verlauf 28" x1="275.58" y1="46.33" x2="275.58" y2="466.33" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#9fecf6"/>
<stop offset=".49" stop-color="#62dff2"/>
<stop offset="1" stop-color="#0a6cb7"/>
</linearGradient>
<linearGradient id="Unbenannter_Verlauf_20" data-name="Unbenannter Verlauf 20" x1="124.99" y1="226.28" x2="235.06" y2="462.33" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#c9f9fd"/>
<stop offset=".29" stop-color="#9fe7f6"/>
<stop offset=".93" stop-color="#36bbe5"/>
<stop offset="1" stop-color="#2ab6e3"/>
</linearGradient>
<linearGradient id="Unbenannter_Verlauf_32" data-name="Unbenannter Verlauf 32" x1="358.38" y1="251.34" x2="277.45" y2="473.69" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#43cbee"/>
<stop offset=".42" stop-color="#29a1d6"/>
<stop offset="1" stop-color="#0161b2"/>
</linearGradient>
</defs>
<g id="seaweedfs">
<rect id="background" class="cls-5" width="512" height="512" rx="68" ry="68"/>
<circle class="cls-4" cx="366.58" cy="152.44" r="31.33"/>
<circle class="cls-4" cx="386.81" cy="223.33" r="18.67"/>
<circle class="cls-4" cx="383.69" cy="288" r="9.22"/>
<path class="cls-3" d="M177.81,152.44s22.5,8.02,30.17,28.02c7.67,20,3.83,61.31,9.5,81.98s19.33,29,21,45.67-1.67,76-1.67,76c0,0-24-31.89-29.5-64s-21.53-41.06-24.6-65.72c-3.07-24.67.77-32.61,2.1-54.61s-1.84-35.72-7-47.33Z"/>
<path class="cls-6" d="M288.58,46.33c-14.56,7.67-42.22,35.22-42.22,79.11s23.11,54.22,23.11,94.22-38.89,48.11-38.89,95.56,21.56,66,21.56,95.56-11.11,55.56-11.11,55.56c0,0,26.97-17.72,33.33-60s-6.89-54.44,7.22-81.78,39-48.22,39-92.89-36.78-89.22-40.89-108.44-2.33-43.22,8.89-76.89Z"/>
<path class="cls-1" d="M120.58,228.33s31.72,14.27,41.55,36.5c9.83,22.22,3.45,40.17,17.45,61.17s51,41,56,73.33-7.67,66.33-7.67,66.33c0,0-5.67-59-30.33-73.33s-46.33-35.67-54.33-59-1.33-50.67-5.33-69c-4-18.33-7.33-23.67-17.33-36Z"/>
<path class="cls-2" d="M352.25,249.11c-9,5.67-18.87,17.85-21.71,31.2s2.94,19.57-7.06,41.57-22.19,25.28-30.97,49.72.21,35.36-11.33,59.56-23.92,35.17-23.92,35.17c0,0,34.77-8.17,45.69-30.94s5.08-35.22,20.53-52.61,24.67-30.89,27.25-56.72-7.33-26.22,1.53-76.94Z"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 3.5 KiB

BIN
note/keepsec.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.8 KiB

BIN
note/piknik.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.4 KiB

BIN
note/seaweedfs.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 77 KiB

317
note/seaweedfs.svg Normal file

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 39 KiB

31
note/slides/Makefile Normal file
View File

@@ -0,0 +1,31 @@
MARP ?= npx @marp-team/marp-cli
SRC := seaweedfs-production-setup.md
PDF := $(SRC:.md=.pdf)
HTML := $(SRC:.md=.html)
PPTX := $(SRC:.md=.pptx)
FLAGS := --allow-local-files
.PHONY: all pdf html pptx watch clean
all: pdf
pdf: $(PDF)
html: $(HTML)
pptx: $(PPTX)
$(PDF): $(SRC)
$(MARP) $(SRC) --pdf $(FLAGS) -o $@
$(HTML): $(SRC)
$(MARP) $(SRC) --html $(FLAGS) -o $@
$(PPTX): $(SRC)
$(MARP) $(SRC) --pptx $(FLAGS) -o $@
watch:
$(MARP) $(SRC) --watch --html $(FLAGS)
clean:
rm -f $(PDF) $(HTML) $(PPTX)

View File

@@ -0,0 +1,118 @@
---
marp: true
theme: default
paginate: true
---
# SeaweedFS Production Setup
A walkthrough of the recommended deployment topology
- Storage layer
- File access layer
- Backend operations
- Erasure coding for durability
---
![bg contain](object-store-layout.png)
---
# Storage Layer
**3 Masters + Volume Servers**
- **Masters (run 3)**
- Form a Raft quorum; tolerate 1 failure
- Track volume locations and cluster topology
- Lightweight; not on the data path for reads/writes after lookup
- **Volume Servers (scale out)**
- Hold the actual needle data on local disks
- Add more servers for capacity and throughput
- Heartbeat to masters; clients talk to them directly for I/O
Rule of thumb: start with 3 masters, then grow volume servers with data.
---
# File Access Layer
**Filer + DB for metadata**
- Filer serves POSIX-like directory and file metadata
- Metadata store options: PostgreSQL, MySQL, Redis, Cassandra, TiKV, etc.
- Run **3 filers** for HA
- Filers are stateless relative to each other — state lives in the DB
**S3 Servers**
- Expose the S3 API on top of the filer
- Scale the count based on throughput requirements
- Can be co-located with filers or run independently
---
# Backend Operations
**1 Admin Server + a few Workers**
- **Admin server (1 is enough)**
- Coordinates background jobs: balancing, EC encoding, vacuum, replication fixes
- **Not on the data path** — a restart does not affect object store reads/writes
- No HA required for normal operation
- **Workers (a few)**
- Execute long-running tasks issued by the admin server
- Horizontally scalable based on maintenance workload
Keeps housekeeping off the hot path.
---
# Production Topology Summary
| Component | Count | Role |
|----------------|----------------------|-----------------------------------|
| Master | 3 | Raft quorum, topology, volume map |
| Volume Server | N (scale out) | Stores data needles |
| Filer | 3 | Metadata gateway |
| Metadata DB | HA cluster | Persists filer metadata |
| S3 Server | N (by throughput) | S3 API frontend |
| Admin | 1 | Background job coordinator |
| Worker | A few | Executes admin-scheduled tasks |
---
# Erasure Coding for Durability
Prefer **many volume servers** so EC shards spread across failure domains.
**Example: 8 volume servers, EC 5+3**
- Each volume becomes 5 data shards + 3 parity shards
- Shards are placed on 8 distinct volume servers
- **Tolerates up to 3 simultaneous volume server failures**
- Storage overhead: 1.6x vs 3x for full replication
```
Volume -> [D1][D2][D3][D4][D5] + [P1][P2][P3]
| | | | | | | |
VS1 VS2 VS3 VS4 VS5 VS6 VS7 VS8
```
More volume servers = more EC layouts available (e.g. 10+4, 6+3).
---
# Getting Started Checklist
1. Provision 3 master nodes (small boxes are fine)
2. Provision volume servers sized to your data footprint
3. Stand up an HA metadata DB, then 3 filers pointing at it
4. Add S3 servers sized to required throughput
5. Run 1 admin + a few workers for background maintenance
6. Once you have enough volume servers, enable EC (e.g. 5+3 on 8 servers)
Scale each layer independently as usage grows.

BIN
note/sponsor_nodion.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 KiB

BIN
note/sponsor_zyner.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 746 B

1974
note/star_history.svg Normal file

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 53 KiB