snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)
Some checks failed
go: build dev binaries / cleanup (push) Successful in 4s
Codespell / Check for spelling errors (push) Failing after 3s
docker: build dev containers / build-rust-binaries (amd64, x86_64-unknown-linux-musl) (push) Failing after 3s
docker: build dev containers / build-dev-containers (push) Has been skipped
docker: build dev containers / build-rust-binaries (arm64, true, aarch64-unknown-linux-musl) (push) Failing after 3s
End to End / FUSE Mount (push) Failing after 2s
EC Integration Tests / EC Integration Tests (push) Failing after 30s
FUSE DLM Integration Tests / FUSE DLM Integration Tests (push) Failing after 3s
FUSE Volume Server Failover Tests / FUSE Volume Server Failover (push) Failing after 3s
FUSE Integration Tests / FUSE Integration Testing (push) Failing after 2s
FUSE P2P Peer Chunk Sharing Integration Tests / FUSE P2P Peer Chunk Sharing (push) Failing after 3s
go: build binary / Go Vet (push) Failing after 2s
go: build binary / Go Vet 32-bit (push) Failing after 3s
go: build binary / Build (push) Failing after 4s
go: build binary / Build cross-platform (freebsd/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (darwin/arm64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/arm64) (push) Failing after 3s
go: build binary / Test (push) Failing after 2s
go: build binary / Test 32-bit (push) Failing after 3s
helm: lint and test charts / lint-test (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (11) (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (17) (push) Failing after 3s
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Failing after 2s
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Failing after 3s
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been skipped
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been skipped
Java Client Unit Tests / Java Unit Tests (11) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (17) (push) Failing after 2s
Java Client Unit Tests / Java Unit Tests (21) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (8) (push) Failing after 3s
Kafka Quick Test (Load Test with Schema Registry) / Kafka Client Load Test (Quick) (push) Failing after 3s
Kafka Gateway Tests / Kafka Unit Tests (unit-tests-1) (push) Failing after 11s
Kafka Gateway Tests / Kafka Integration Tests (Critical) (integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka End-to-End Tests (with SMQ) (e2e-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Consumer Group Tests (Highly Isolated) (consumer-group-1) (push) Failing after 2s
Kafka Gateway Tests / Kafka Client Compatibility (with SMQ) (client-compat-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka SMQ Integration Tests (Full Stack) (smq-integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Protocol Tests (Isolated) (protocol-1) (push) Failing after 3s
KMS Tests / KMS Provider Integration Tests (push) Failing after 2s
KMS Tests / S3 KMS End-to-End Tests (push) Failing after 3s
Master Cold Start Tests / Master Cold Start Tests (push) Failing after 4s
Metadata Subscribe Integration Tests / Metadata Subscribe Integration Tests (push) Failing after 3s
Multi-Master Tests / Multi-Master Failover Tests (push) Failing after 3s
Performance / CPU and Heap Profile (push) Failing after 2s
Performance / S3 Read/Write Benchmark (rust volume) (push) Failing after 3s
pjdfstest POSIX Compliance / pjdfstest (push) Failing after 2s
Plugin Worker Integration Tests / Plugin Worker: erasure_coding (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: vacuum (push) Failing after 3s
Rust Volume Server Tests / Rust Integration Tests (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 1) (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 1) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 3) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Distributed Tests (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Performance Tests (push) Failing after 3s
Performance / Throughput Benchmark (go volume) (push) Failing after 3s
Performance / Throughput Benchmark (rust volume) (push) Failing after 2s
Performance / S3 Read/Write Benchmark (go volume) (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: volume_balance (push) Failing after 3s
PostgreSQL Gateway Tests / PostgreSQL Basic Tests (push) Failing after 3s
Rust Volume Server Tests / Rust Unit Tests (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 3) (push) Failing after 3s
Rust Plugin Worker Tests / Rust Plugin Worker Build and Unit Tests (push) Failing after 3s
rust: build dev volume server binaries / cleanup (push) Successful in 3s
S3 ETag and ACL Tests / S3 ETag + ACL Integration Tests (push) Failing after 3s
S3 IAM Integration Tests / IAM Unit Tests (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (advanced) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (basic) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (group) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (policy-enforcement) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (sts) (push) Failing after 4s
S3 Keycloak Integration Tests / S3 Keycloak Integration Tests (push) Failing after 3s
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.11) (push) Failing after 3s
S3 PyArrow Parquet Tests / Go Unit Tests (Implicit Directory) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.12) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-amd64, x86_64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-arm64, aarch64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-linux (linux-amd64, x86_64-unknown-linux-gnu) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.9) (push) Has been cancelled
S3 Proxy Signature Tests / S3 Proxy Signature Verification Tests (push) Has been cancelled
S3 SDK V2 Route Disambiguation Tests / S3 SDK V2 Routing Tests (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (quick) (push) Has been cancelled
S3 SSE Tests / S3 SSE Compatibility Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Metadata Persistence Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Copy Operations Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Multipart Upload Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Performance Test (push) Has been cancelled
S3 SSE Tests / S3 Volume Encryption Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (comprehensive) (push) Has been cancelled
Spark Integration Tests / Spark Integration Tests (push) Failing after 2s
Telemetry Integration Tests / telemetry-integration-test (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 1) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 1) (push) Failing after 3s
Ceph S3 tests / Basic S3 tests (KV store) (push) Has been cancelled
Ceph S3 tests / S3 Versioning & Object Lock tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Unit Tests (push) Failing after 3s
Ceph S3 tests / S3 CORS tests (push) Has been cancelled
Ceph S3 tests / SeaweedFS Custom S3 Copy tests (push) Has been cancelled
Ceph S3 tests / Basic S3 tests (SQL store) (push) Has been cancelled
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
TLS Rotation Integration Tests / TLS Rotation Integration Tests (push) Has been cancelled
Vacuum Integration Tests / Vacuum Integration Tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Variables Integration Tests (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (advanced-policy) (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (basic-policy) (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (contextual-policy) (push) Failing after 4s
S3 Policy Integration Tests / Trusted Proxy Detection Tests (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Shell Integration Tests (push) Failing after 3s
Samba on FUSE Integration / samba-integration (push) Failing after 2s
SFTP Integration Tests / SFTP Integration Testing (push) Failing after 3s
terraform: validate and test modules / fmt, validate, plan-level tests (push) Failing after 3s
terraform: validate and test modules / local cluster smoke test (real weed) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 3) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 2) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 2) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 3) (push) Failing after 5s
mount: windows conformance / WinFsp conformance (push) Has been cancelled
mount: windows / Mount on Windows (push) Has been cancelled

This commit is contained in:
upstream-mirror
2026-09-16 03:23:42 +05:00
commit 89f5be005d
3997 changed files with 961970 additions and 0 deletions

View File

@@ -0,0 +1,13 @@
# Lance client container for Lance Namespace REST compatibility testing.
# The point of this image is to exercise the namespace with the real client
# rather than hand-built HTTP: every serious bug in this surface so far looked
# fine to a request we wrote ourselves.
FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir lance-namespace pylance pyarrow
COPY test_lance_namespace.py /app/
CMD ["python3", "/app/test_lance_namespace.py", "--help"]

View File

@@ -0,0 +1,13 @@
# PyIceberg test container for Iceberg REST Catalog compatibility testing
FROM python:3.11-slim
WORKDIR /app
# Install PyIceberg with S3 support and dependencies
RUN pip install --no-cache-dir "pyiceberg[s3fs]" pyarrow pandas
# Copy the test script
COPY test_rest_catalog.py /app/
# Default command
CMD ["python3", "/app/test_rest_catalog.py", "--help"]

View File

@@ -0,0 +1,37 @@
# Makefile for Iceberg REST Catalog Integration Tests
#
# This Makefile provides easy-to-use targets for running integration tests
# and starting a local development environment.
.PHONY: build mini test test-pyiceberg clean
# Root directory of the SeaweedFS project
PROJECT_ROOT = ../../..
WEED_BIN = $(PROJECT_ROOT)/weed/weed
# Build the weed binary
build:
@echo "Building SeaweedFS binary..."
cd $(PROJECT_ROOT)/weed && go build -v
# Start SeaweedFS mini cluster for manual testing
# Uses default ports: S3=8333, Iceberg=8182
mini: build
@echo "Starting SeaweedFS mini cluster..."
$(WEED_BIN) mini -s3.port.iceberg=8182 & echo $$! > weed-mini.pid; wait
# Run all integration tests in this directory
test: build
@echo "Running all Iceberg catalog integration tests..."
go test -v .
# Run only the PyIceberg compatibility tests (requires Docker)
test-pyiceberg: build
@echo "Running PyIceberg compatibility tests..."
go test -v -run TestPyIcebergRestCatalog .
# Clean up temporary data and stop weed mini if running
clean:
@echo "Cleaning up..."
@test -f weed-mini.pid && kill $$(cat weed-mini.pid) && rm weed-mini.pid || true
@rm -rf /tmp/seaweed-iceberg-test-*

View File

@@ -0,0 +1,33 @@
# Iceberg REST Catalog Compatibility Test Docker Compose
#
# This compose file sets up a test environment for validating
# the SeaweedFS Iceberg REST Catalog implementation.
#
# Usage:
# docker compose -f docker-compose.test.yaml up --build
#
# Note: SeaweedFS must be running on the host with the Iceberg REST port exposed.
# Set ICEBERG_CATALOG_URL environment variable to point to your catalog.
services:
# PyIceberg-based REST catalog test
pyiceberg-test:
build:
context: .
dockerfile: Dockerfile.pyiceberg
environment:
- CATALOG_URL=${CATALOG_URL:-http://host.docker.internal:8182}
- WAREHOUSE=${WAREHOUSE:-s3://test-bucket/}
- PREFIX=${PREFIX:-pyiceberg-test}
- AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-test}
- AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-test}
- AWS_REGION=${AWS_REGION:-us-east-1}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- ./test_rest_catalog.py:/app/test_rest_catalog.py:ro
command: >
python3 /app/test_rest_catalog.py
--catalog-url ${CATALOG_URL:-http://host.docker.internal:8182}
--warehouse ${WAREHOUSE:-s3://test-bucket/}
--prefix ${PREFIX:-pyiceberg-test}

View File

@@ -0,0 +1,211 @@
package catalog
import (
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/seaweedfs/seaweedfs/test/testutil"
)
// TestDuckDBDeterministicLocationRead validates that DuckDB's iceberg
// extension can resolve and scan a table that lives at the deterministic
// <bucket>/<namespace>/<tableName> path produced when no client-side UUID
// is appended to the table location. This is the layout produced by:
//
// - iceberg-spark (no concept of UUID-suffixed locations)
// - pyiceberg (no concept of UUID-suffixed locations)
// - Trino with iceberg.unique-table-location=false
//
// Post-#9246, REST clients can resolve table locations under a namespace
// because GetNamespace advertises a default `location` property. This
// test creates a namespace + table via the REST API directly (so no
// client-side UUID is added), then asks DuckDB to ATTACH the catalog and
// describe the table — verifying DuckDB walks the catalog → metadata →
// schema chain successfully against a deterministic on-disk path.
//
// The test does not insert data files — DuckDB scanning an empty table
// is sufficient to validate the catalog/metadata resolution path. If
// DuckDB's iceberg extension is unavailable or the build is too old to
// support the ATTACH syntax, the test skips rather than fails.
func TestDuckDBDeterministicLocationRead(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
if !testutil.HasDocker() {
t.Skip("Docker not available, skipping DuckDB integration test")
}
env := newOAuthTestEnv(t)
defer env.cleanup(t)
env.start(t)
bucketName := "duckdb-detloc-" + randomSuffix()
namespace := "detloc_ns_" + randomSuffix()
tableName := "detloc_tbl_" + randomSuffix()
createTableBucketViaShell(t, env, bucketName)
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
createNamespaceWithToken(t, env, token, bucketName, namespace)
createIcebergTableWithToken(t, env, token, bucketName, namespace, tableName)
// Sanity-check the catalog row exposes a deterministic location.
tableLoc := getTableLocation(t, env, token, bucketName, namespace, tableName)
wantPrefix := fmt.Sprintf("s3://%s/%s/%s", bucketName, namespace, tableName)
if !strings.HasPrefix(tableLoc, wantPrefix) {
t.Fatalf("table location %q should start with %q (deterministic, no UUID suffix)", tableLoc, wantPrefix)
}
if strings.Contains(strings.TrimPrefix(tableLoc, wantPrefix), "-") {
t.Fatalf("table location %q should not have a -<uuid> suffix after %q", tableLoc, wantPrefix)
}
sqlContent := fmt.Sprintf(`
INSTALL iceberg;
LOAD iceberg;
CREATE SECRET iceberg_secret (
TYPE ICEBERG,
ENDPOINT 'http://host.docker.internal:%d',
CLIENT_ID '%s',
CLIENT_SECRET '%s'
);
CREATE SECRET s3_secret (
TYPE S3,
KEY_ID '%s',
SECRET '%s',
ENDPOINT 'host.docker.internal:%d',
URL_STYLE 'path',
USE_SSL false
);
ATTACH 's3://%s/' AS detloc_cat (TYPE 'ICEBERG', SECRET iceberg_secret);
DESCRIBE detloc_cat.%s.%s;
SELECT 'duckdb deterministic-location describe ok' AS marker;
`,
env.icebergPort, env.accessKey, env.secretKey,
env.accessKey, env.secretKey, env.s3Port,
bucketName,
namespace, tableName,
)
sqlFile := filepath.Join(env.dataDir, "duckdb_detloc.sql")
if err := os.WriteFile(sqlFile, []byte(sqlContent), 0644); err != nil {
t.Fatalf("write SQL file: %v", err)
}
cmd := exec.Command("docker", "run", "--rm",
"-v", fmt.Sprintf("%s:/test", env.dataDir),
"--add-host", "host.docker.internal:host-gateway",
"--entrypoint", "duckdb",
"duckdb/duckdb:latest",
"-init", "/test/duckdb_detloc.sql",
"-c", "SELECT 1",
)
output, err := cmd.CombinedOutput()
outputStr := string(output)
t.Logf("DuckDB output:\n%s", outputStr)
if err != nil {
// Tolerate environments where the Iceberg extension/version is too
// limited to exercise this code path. The intent is to gain
// coverage where it's available, not to require it everywhere.
if strings.Contains(outputStr, "iceberg extension is not available") ||
strings.Contains(outputStr, "Failed to load") ||
strings.Contains(outputStr, "syntax error") ||
strings.Contains(outputStr, "Unknown extension") {
t.Skipf("DuckDB image lacks the required iceberg extension or syntax: %v", err)
}
t.Fatalf("DuckDB run failed: %v\nOutput:\n%s", err, outputStr)
}
if !strings.Contains(outputStr, "duckdb deterministic-location describe ok") {
t.Fatalf("DuckDB DESCRIBE on deterministic-location table did not complete successfully:\n%s", outputStr)
}
}
// createIcebergTableWithToken creates a minimal Iceberg table via the REST
// catalog using the provided OAuth bearer token. The table is created
// without an explicit location, so the server picks the deterministic
// <bucket>/<namespace>/<tableName> path.
func createIcebergTableWithToken(t *testing.T, env *oauthTestEnv, token, bucketName, namespace, tableName string) {
t.Helper()
body := fmt.Sprintf(`{
"name": %q,
"schema": {
"type": "struct",
"schema-id": 0,
"fields": [
{"id": 1, "name": "id", "required": true, "type": "long"},
{"id": 2, "name": "label", "required": false, "type": "string"}
]
}
}`, tableName)
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables", env.icebergURL(), bucketName, namespace)
req, err := http.NewRequest(http.MethodPost, url, strings.NewReader(body))
if err != nil {
t.Fatalf("create table request: %v", err)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+token)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("create table: %v", err)
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
t.Fatalf("create table failed: status=%d body=%s", resp.StatusCode, respBody)
}
t.Logf("Created table %s.%s in bucket %s", namespace, tableName, bucketName)
}
// getTableLocation fetches the table's metadata via the REST catalog and
// returns the location field reported in the response.
func getTableLocation(t *testing.T, env *oauthTestEnv, token, bucketName, namespace, tableName string) string {
t.Helper()
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables/%s", env.icebergURL(), bucketName, namespace, tableName)
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
t.Fatalf("load table request: %v", err)
}
req.Header.Set("Authorization", "Bearer "+token)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("load table: %v", err)
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
t.Fatalf("load table failed: status=%d body=%s", resp.StatusCode, respBody)
}
// The metadata.location field is nested. A naive substring match is
// sufficient for asserting the expected path shape.
body := string(respBody)
const key = `"location":"`
idx := strings.Index(body, key)
if idx < 0 {
t.Fatalf("LoadTable response missing location field: %s", body)
}
rest := body[idx+len(key):]
end := strings.Index(rest, `"`)
if end < 0 {
t.Fatalf("malformed location field: %s", body)
}
return rest[:end]
}

View File

@@ -0,0 +1,477 @@
package catalog
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/test/testutil"
)
// oauthTestEnv holds a weed mini instance started with IAM credentials
// so that the OAuth token endpoint is functional.
type oauthTestEnv struct {
seaweedDir string
weedBinary string
dataDir string
bindIP string
s3Port int
s3GrpcPort int
icebergPort int
masterPort int
masterGrpcPort int
filerPort int
filerGrpcPort int
volumePort int
volumeGrpcPort int
webdavPort int
weedProcess *exec.Cmd
weedCancel context.CancelFunc
accessKey string
secretKey string
// s3ExternalURL, when set before start, is the S3 endpoint the catalog
// advertises to clients in LoadTable FileIO config.
s3ExternalURL string
}
func newOAuthTestEnv(t *testing.T) *oauthTestEnv {
t.Helper()
wd, err := os.Getwd()
if err != nil {
t.Fatalf("get working directory: %v", err)
}
seaweedDir := wd
for i := 0; i < 6; i++ {
if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
break
}
seaweedDir = filepath.Dir(seaweedDir)
}
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
if info, err := os.Stat(weedBinary); err == nil && !info.IsDir() {
// Say which binary and how old it is. `make test` rebuilds first, but
// `go test` on its own happily runs a weeks-old binary and reports a
// pass for code that is not being exercised.
t.Logf("using %s, built %s", weedBinary, info.ModTime().Format(time.RFC3339))
} else {
weedBinary = "weed"
if _, err := exec.LookPath(weedBinary); err != nil {
t.Skip("weed binary not found, skipping integration test")
}
}
dataDir, err := os.MkdirTemp("", "seaweed-oauth-test-*")
if err != nil {
t.Fatalf("create temp dir: %v", err)
}
bindIP := testutil.FindBindIP()
ports := testutil.MustAllocatePorts(t, 10)
return &oauthTestEnv{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
dataDir: dataDir,
bindIP: bindIP,
masterPort: ports[0],
masterGrpcPort: ports[1],
volumePort: ports[2],
volumeGrpcPort: ports[3],
filerPort: ports[4],
filerGrpcPort: ports[5],
s3Port: ports[6],
s3GrpcPort: ports[7],
icebergPort: ports[8],
webdavPort: ports[9],
accessKey: "AKIAIOSFODNN7EXAMPLE",
secretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
}
}
func (env *oauthTestEnv) start(t *testing.T) {
t.Helper()
iamConfigPath, err := testutil.WriteIAMConfig(env.dataDir, env.accessKey, env.secretKey)
if err != nil {
t.Fatalf("write IAM config: %v", err)
}
securityToml := filepath.Join(env.dataDir, "security.toml")
if err := os.WriteFile(securityToml, []byte("# Empty security config for testing\n"), 0644); err != nil {
t.Fatalf("write security.toml: %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
env.weedCancel = cancel
args := []string{"mini",
"-master.port", fmt.Sprintf("%d", env.masterPort),
"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
"-volume.port", fmt.Sprintf("%d", env.volumePort),
"-volume.port.grpc", fmt.Sprintf("%d", env.volumeGrpcPort),
"-filer.port", fmt.Sprintf("%d", env.filerPort),
"-filer.port.grpc", fmt.Sprintf("%d", env.filerGrpcPort),
"-s3.port", fmt.Sprintf("%d", env.s3Port),
"-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort),
"-s3.port.iceberg", fmt.Sprintf("%d", env.icebergPort),
"-webdav.port", fmt.Sprintf("%d", env.webdavPort),
"-s3.config", iamConfigPath,
"-ip", env.bindIP,
"-ip.bind", "0.0.0.0",
"-dir", env.dataDir,
}
if env.s3ExternalURL != "" {
args = append(args, "-s3.externalUrl", env.s3ExternalURL)
}
cmd := exec.CommandContext(ctx, env.weedBinary, args...)
cmd.Dir = env.dataDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Env = append(os.Environ(),
"AWS_ACCESS_KEY_ID="+env.accessKey,
"AWS_SECRET_ACCESS_KEY="+env.secretKey,
)
if err := cmd.Start(); err != nil {
cancel()
t.Fatalf("start weed mini: %v", err)
}
env.weedProcess = cmd
icebergURL := fmt.Sprintf("http://%s:%d/v1/config", env.bindIP, env.icebergPort)
if !testutil.WaitForService(icebergURL, 30*time.Second) {
cancel()
cmd.Wait()
t.Fatalf("Iceberg REST API did not become ready at %s", icebergURL)
}
}
func (env *oauthTestEnv) cleanup(t *testing.T) {
t.Helper()
if env.weedCancel != nil {
env.weedCancel()
}
if env.weedProcess != nil {
env.weedProcess.Wait()
}
if env.dataDir != "" {
os.RemoveAll(env.dataDir)
}
}
func (env *oauthTestEnv) icebergURL() string {
return fmt.Sprintf("http://%s:%d", env.bindIP, env.icebergPort)
}
// TestOAuthTokenEndpoint tests the /v1/oauth/tokens endpoint directly via HTTP.
func TestOAuthTokenEndpoint(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := newOAuthTestEnv(t)
defer env.cleanup(t)
env.start(t)
t.Run("valid credentials", func(t *testing.T) {
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
if token == "" {
t.Fatal("expected non-empty token")
}
})
t.Run("invalid secret", func(t *testing.T) {
resp, err := http.PostForm(env.icebergURL()+"/v1/oauth/tokens", url.Values{
"grant_type": {"client_credentials"},
"client_id": {env.accessKey},
"client_secret": {"wrong-secret"},
})
if err != nil {
t.Fatalf("POST /v1/oauth/tokens: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusUnauthorized {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("expected 401, got %d: %s", resp.StatusCode, body)
}
})
t.Run("missing grant_type", func(t *testing.T) {
resp, err := http.PostForm(env.icebergURL()+"/v1/oauth/tokens", url.Values{
"client_id": {env.accessKey},
"client_secret": {env.secretKey},
})
if err != nil {
t.Fatalf("POST /v1/oauth/tokens: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("expected 400, got %d: %s", resp.StatusCode, body)
}
})
t.Run("bearer token auth on catalog endpoint", func(t *testing.T) {
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
// Point the call at a table bucket that exists, so anything but 200 is an
// auth or routing fault rather than the catalog reporting a missing bucket.
bucketName := "oauth-bearer-" + randomSuffix()
createTableBucketViaShell(t, env, bucketName)
req, err := http.NewRequest(http.MethodGet, env.icebergURL()+"/v1/"+bucketName+"/namespaces", nil)
if err != nil {
t.Fatalf("create request: %v", err)
}
req.Header.Set("Authorization", "Bearer "+token)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("GET /v1/%s/namespaces with Bearer: %v", bucketName, err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
t.Fatalf("Bearer auth failed unexpectedly: status=%d body=%s", resp.StatusCode, body)
}
})
}
// TestDuckDBOAuthIntegration tests that DuckDB can connect to the Iceberg REST
// catalog using the OAuth2 client_credentials flow (CREATE SECRET with client_id
// and client_secret). This is the scenario reported in issue #9015.
func TestDuckDBOAuthIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
if !testutil.HasDocker() {
t.Skip("Docker not available, skipping DuckDB OAuth integration test")
}
env := newOAuthTestEnv(t)
defer env.cleanup(t)
env.start(t)
// Create a table bucket and namespace so DuckDB has something to query
bucketName := "duckdb-oauth-" + randomSuffix()
createTableBucketViaShell(t, env, bucketName)
// Create a namespace via the Iceberg REST API using OAuth token
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
createNamespaceWithToken(t, env, token, bucketName, "testns")
sqlContent := fmt.Sprintf(`
INSTALL iceberg;
LOAD iceberg;
CREATE SECRET iceberg_secret (
TYPE ICEBERG,
ENDPOINT 'http://host.docker.internal:%d',
CLIENT_ID '%s',
CLIENT_SECRET '%s',
SCOPE 's3://%s/'
);
CREATE SECRET s3_secret (
TYPE S3,
KEY_ID '%s',
SECRET '%s',
ENDPOINT 'host.docker.internal:%d',
URL_STYLE 'path',
USE_SSL false,
SCOPE 's3://%s/'
);
SELECT 'OAuth token obtained successfully' as status;
-- Try listing namespaces via the Iceberg catalog
SELECT * FROM iceberg_scan('iceberg_secret', ALLOW_MOVED_PATHS => TRUE) LIMIT 0;
`, env.icebergPort, env.accessKey, env.secretKey, bucketName,
env.accessKey, env.secretKey, env.s3Port, bucketName)
sqlFile := filepath.Join(env.dataDir, "duckdb_oauth_test.sql")
if err := os.WriteFile(sqlFile, []byte(sqlContent), 0644); err != nil {
t.Fatalf("write SQL file: %v", err)
}
// Run DuckDB in Docker.
// We use a simple test: get a token, create a secret, and verify the
// Iceberg extension can communicate with the catalog.
// The simpler fallback test just verifies CREATE SECRET succeeds (no 404).
fallbackSQL := fmt.Sprintf(`
INSTALL iceberg;
LOAD iceberg;
CREATE SECRET (
TYPE ICEBERG,
ENDPOINT 'http://host.docker.internal:%d',
CLIENT_ID '%s',
CLIENT_SECRET '%s'
);
SELECT 'DuckDB Iceberg OAuth secret created successfully' as result;
`, env.icebergPort, env.accessKey, env.secretKey)
fallbackFile := filepath.Join(env.dataDir, "duckdb_oauth_fallback.sql")
if err := os.WriteFile(fallbackFile, []byte(fallbackSQL), 0644); err != nil {
t.Fatalf("write fallback SQL file: %v", err)
}
// Try the full SQL script first (CREATE SECRET + catalog access).
// Fall back to the simple CREATE SECRET test if iceberg_scan isn't supported.
cmd := exec.Command("docker", "run", "--rm",
"-v", fmt.Sprintf("%s:/test", env.dataDir),
"--add-host", "host.docker.internal:host-gateway",
"--entrypoint", "duckdb",
"duckdb/duckdb:latest",
"-init", "/test/duckdb_oauth_test.sql",
"-c", "SELECT 1",
)
output, err := cmd.CombinedOutput()
outputStr := string(output)
t.Logf("DuckDB output:\n%s", outputStr)
if err != nil {
if strings.Contains(outputStr, "iceberg extension is not available") ||
strings.Contains(outputStr, "Failed to load") {
t.Skip("Skipping: Iceberg extension not available in DuckDB Docker image")
}
// The key check: the old error was "HTTP NotFound_404" on /v1/oauth/tokens.
// With our fix, this should no longer happen.
if strings.Contains(outputStr, "NotFound_404") && strings.Contains(outputStr, "/v1/oauth/tokens") {
t.Fatal("OAuth token endpoint returned 404 - the fix is not working")
}
// If iceberg_scan failed but CREATE SECRET worked, fall back to simpler test
if strings.Contains(outputStr, "OAuth token obtained successfully") {
t.Logf("Full SQL had partial success (token obtained), iceberg_scan may not be supported. Continuing.")
} else {
// Try the fallback script that only tests CREATE SECRET
t.Logf("Full SQL failed, trying fallback CREATE SECRET test...")
fallbackCmd := exec.Command("docker", "run", "--rm",
"-v", fmt.Sprintf("%s:/test", env.dataDir),
"--add-host", "host.docker.internal:host-gateway",
"--entrypoint", "duckdb",
"duckdb/duckdb:latest",
"-init", "/test/duckdb_oauth_fallback.sql",
"-c", "SELECT 1",
)
fallbackOutput, fallbackErr := fallbackCmd.CombinedOutput()
fallbackStr := string(fallbackOutput)
t.Logf("DuckDB fallback output:\n%s", fallbackStr)
if fallbackErr != nil {
if strings.Contains(fallbackStr, "NotFound_404") && strings.Contains(fallbackStr, "/v1/oauth/tokens") {
t.Fatal("OAuth token endpoint returned 404 - the fix is not working")
}
t.Fatalf("DuckDB fallback also failed: %v\nOutput: %s", fallbackErr, fallbackStr)
}
if !strings.Contains(fallbackStr, "DuckDB Iceberg OAuth secret created successfully") {
t.Errorf("expected success message in fallback output, got:\n%s", fallbackStr)
}
return
}
}
if strings.Contains(outputStr, "OAuth token obtained successfully") {
t.Logf("DuckDB OAuth CREATE SECRET succeeded")
} else {
t.Errorf("expected OAuth success message in output, got:\n%s", outputStr)
}
}
// requestOAuthToken obtains a bearer token from the OAuth endpoint.
func requestOAuthToken(t *testing.T, env *oauthTestEnv, accessKey, secretKey string) string {
t.Helper()
resp, err := http.PostForm(env.icebergURL()+"/v1/oauth/tokens", url.Values{
"grant_type": {"client_credentials"},
"client_id": {accessKey},
"client_secret": {secretKey},
})
if err != nil {
t.Fatalf("POST /v1/oauth/tokens: %v", err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
t.Fatalf("OAuth token request failed: status=%d body=%s", resp.StatusCode, body)
}
var tokenResp struct {
AccessToken string `json:"access_token"`
TokenType string `json:"token_type"`
ExpiresIn int `json:"expires_in"`
}
if err := json.Unmarshal(body, &tokenResp); err != nil {
t.Fatalf("decode token response: %v", err)
}
if tokenResp.AccessToken == "" {
t.Fatal("got empty access_token")
}
if tokenResp.TokenType != "bearer" {
t.Errorf("expected token_type=bearer, got %s", tokenResp.TokenType)
}
return tokenResp.AccessToken
}
// createTableBucketViaShell creates a table bucket using weed shell,
// which bypasses S3 auth. This is the same approach used by the Trino tests.
func createTableBucketViaShell(t *testing.T, env *oauthTestEnv, bucketName string) {
t.Helper()
cmd := exec.Command(env.weedBinary, "shell",
fmt.Sprintf("-master=%s:%d.%d", env.bindIP, env.masterPort, env.masterGrpcPort),
)
cmd.Stdin = strings.NewReader(fmt.Sprintf("s3tables.bucket -create -name %s -account 000000000000\nexit\n", bucketName))
output, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("create table bucket %s via weed shell: %v\nOutput: %s", bucketName, err, output)
}
t.Logf("Created table bucket %s", bucketName)
}
// createNamespaceWithToken creates a namespace using a Bearer token.
func createNamespaceWithToken(t *testing.T, env *oauthTestEnv, token, bucketName, namespace string) {
t.Helper()
path := fmt.Sprintf("/v1/%s/namespaces", bucketName)
body := fmt.Sprintf(`{"namespace":["%s"]}`, namespace)
req, err := http.NewRequest(http.MethodPost, env.icebergURL()+path, strings.NewReader(body))
if err != nil {
t.Fatalf("create request: %v", err)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+token)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("create namespace: %v", err)
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusConflict {
t.Fatalf("create namespace failed: status=%d body=%s", resp.StatusCode, respBody)
}
t.Logf("Created namespace %s in bucket %s", namespace, bucketName)
}

View File

@@ -0,0 +1,215 @@
// Tests for the credential-vending access pattern.
//
// DuckDB attaches a catalog with "X-Iceberg-Access-Delegation: vended-credentials"
// and rebuilds its S3 credential out of whatever LoadTable returns in `config`,
// dropping the S3 secret it was configured with. A catalog that advertises an
// endpoint there but vends no credentials therefore leaves DuckDB sending
// unsigned requests, and every metadata and data file comes back 403.
//
// These run against a weed mini started with -s3.externalUrl, which is what
// makes the catalog advertise an endpoint at all; with a wildcard bind and no
// external URL the config is empty and the bug cannot appear.
package catalog
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/seaweedfs/seaweedfs/test/testutil"
)
// duckDBImage tracks whatever DuckDB users are actually running, matching the
// other DuckDB tests here: a client that changes how it handles vended
// credentials is exactly what this suite exists to catch.
const duckDBImage = "duckdb/duckdb:latest"
// requireDuckDBIceberg skips when the image cannot install the iceberg
// extension -- no egress, or a build without it. Probing separately keeps the
// round trip below free to fail on any error instead of having to guess which
// DuckDB messages mean "no extension" and which mean "the catalog is broken".
func requireDuckDBIceberg(t *testing.T) {
t.Helper()
const ready = "iceberg extension ready"
cmd := exec.Command("docker", "run", "--rm",
"--entrypoint", "duckdb",
duckDBImage,
"-c", fmt.Sprintf("INSTALL iceberg; LOAD iceberg; SELECT '%s' AS marker;", ready),
)
output, err := cmd.CombinedOutput()
if err != nil || !strings.Contains(string(output), ready) {
t.Skipf("DuckDB image cannot load the iceberg extension: %v\n%s", err, output)
}
}
func TestIcebergVendedCredentials(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := newOAuthTestEnv(t)
// The DuckDB container reaches S3 through host.docker.internal, so advertise
// the endpoint under the name the client can actually resolve.
env.s3ExternalURL = fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
defer env.cleanup(t)
env.start(t)
bucketName := "vendcreds-" + randomSuffix()
namespace := "vendcreds_ns_" + randomSuffix()
tableName := "vendcreds_tbl_" + randomSuffix()
createTableBucketViaShell(t, env, bucketName)
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
createNamespaceWithToken(t, env, token, bucketName, namespace)
createTableWithToken(t, env, token, bucketName, namespace, tableName)
t.Run("plain load table advertises the endpoint", func(t *testing.T) {
config := loadTableConfig(t, env, token, bucketName, namespace, tableName, "")
if config["s3.endpoint"] != env.s3ExternalURL {
t.Fatalf("s3.endpoint = %q, want %q (clients that bring their own credentials still need the endpoint)",
config["s3.endpoint"], env.s3ExternalURL)
}
})
t.Run("vended credentials request gets no half-configured storage", func(t *testing.T) {
config := loadTableConfig(t, env, token, bucketName, namespace, tableName, "vended-credentials")
if config["s3.endpoint"] == "" {
return
}
// If the catalog ever does vend credentials, the endpoint may come back
// -- but only alongside the credentials that make it usable.
if config["s3.access-key-id"] == "" || config["s3.secret-access-key"] == "" {
t.Fatalf("LoadTable config = %v: a client that asked for vended credentials drops its own and signs with what it gets, so an endpoint without credentials makes every data file read 403", config)
}
})
t.Run("duckdb writes and reads back through the catalog", func(t *testing.T) {
if !testutil.HasDocker() {
t.Skip("Docker not available, skipping DuckDB round trip")
}
requireDuckDBIceberg(t)
duckDBRoundTrip(t, env, bucketName, namespace)
})
}
// duckDBRoundTrip runs the reporter's flow: attach the catalog, create a table
// from a query, then read it back. Both halves need signed S3 access to the
// table's metadata and data files.
func duckDBRoundTrip(t *testing.T, env *oauthTestEnv, bucketName, namespace string) {
t.Helper()
const marker = "vended-credentials round trip ok"
sql := fmt.Sprintf(`
INSTALL iceberg;
LOAD iceberg;
CREATE SECRET iceberg_secret (
TYPE ICEBERG,
ENDPOINT 'http://host.docker.internal:%d',
CLIENT_ID '%s',
CLIENT_SECRET '%s'
);
CREATE SECRET s3_secret (
TYPE S3,
KEY_ID '%s',
SECRET '%s',
ENDPOINT 'host.docker.internal:%d',
URL_STYLE 'path',
USE_SSL false
);
ATTACH 's3://%s' AS vend_cat (
TYPE ICEBERG,
SECRET iceberg_secret,
ENDPOINT 'http://host.docker.internal:%d',
READ_ONLY false
);
CREATE TABLE vend_cat.%s.round_trip AS SELECT 42 AS answer;
SELECT CASE WHEN sum(answer) = 42 THEN '%s' ELSE 'wrong answer' END AS marker
FROM vend_cat.%s.round_trip;
`,
env.icebergPort, env.accessKey, env.secretKey,
env.accessKey, env.secretKey, env.s3Port,
bucketName, env.icebergPort,
namespace, marker, namespace,
)
sqlFile := filepath.Join(env.dataDir, "duckdb_vended_credentials.sql")
if err := os.WriteFile(sqlFile, []byte(sql), 0644); err != nil {
t.Fatalf("write SQL file: %v", err)
}
cmd := exec.Command("docker", "run", "--rm",
"-v", fmt.Sprintf("%s:/test", env.dataDir),
"--add-host", "host.docker.internal:host-gateway",
"-e", "AWS_REGION=us-east-1",
"--entrypoint", "duckdb",
duckDBImage,
"-init", "/test/duckdb_vended_credentials.sql",
"-c", "SELECT 1",
)
output, err := cmd.CombinedOutput()
outputStr := string(output)
t.Logf("DuckDB output:\n%s", outputStr)
// Nothing below skips: requireDuckDBIceberg already established that the
// extension loads, so any failure from here is the catalog's.
if strings.Contains(outputStr, "trying to refresh secret") {
t.Fatalf("DuckDB fell back to refreshing the credential the catalog vended, which no stage-created table can satisfy:\n%s", outputStr)
}
// Match the phrasings rather than a bare "403", which a random port could
// carry.
for _, denial := range []string{"AccessDenied", "403 Forbidden", "code 403"} {
if strings.Contains(outputStr, denial) {
t.Fatalf("DuckDB signed its S3 requests with the catalog's credential-less config:\n%s", outputStr)
}
}
if !strings.Contains(outputStr, marker) {
t.Fatalf("DuckDB did not write and read the table back (err=%v):\n%s", err, outputStr)
}
}
// loadTableConfig loads a table and returns the FileIO config the catalog
// advertises, optionally asking for an access delegation mechanism.
func loadTableConfig(t *testing.T, env *oauthTestEnv, token, bucketName, namespace, tableName, delegation string) map[string]string {
t.Helper()
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables/%s", env.icebergURL(), bucketName, namespace, tableName)
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
t.Fatalf("create request: %v", err)
}
req.Header.Set("Authorization", "Bearer "+token)
if delegation != "" {
req.Header.Set("X-Iceberg-Access-Delegation", delegation)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("load table: %v", err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
t.Fatalf("load table failed: status=%d body=%s", resp.StatusCode, body)
}
var result struct {
Config map[string]string `json:"config"`
}
if err := json.Unmarshal(body, &result); err != nil {
t.Fatalf("decode LoadTableResult: %v", err)
}
return result.Config
}

View File

@@ -0,0 +1,585 @@
// Package catalog provides integration tests for the Iceberg REST Catalog API.
// These tests use DuckDB running in Docker to verify catalog operations.
package catalog
import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/aws/aws-sdk-go/aws/credentials"
v4 "github.com/aws/aws-sdk-go/aws/signer/v4"
"github.com/seaweedfs/seaweedfs/test/testutil"
)
// sharedEnv is the single TestEnvironment shared across all tests in this package.
var sharedEnv *TestEnvironment
// TestMain starts one weed mini instance for the whole package and tears it down
// after all tests have run.
func TestMain(m *testing.M) {
flag.Parse()
if os.Getenv("SHORT") != "" || testing.Short() {
// Let tests self-skip when run with -short.
os.Exit(m.Run())
}
env, err := newTestEnvironmentForMain()
if err != nil {
fmt.Fprintf(os.Stderr, "SKIP: setup failed: %v\n", err)
os.Exit(0) // Skip all tests rather than fail
}
sharedEnv = env
if startErr := sharedEnv.startSeaweedFSForMain(); startErr != nil {
fmt.Fprintf(os.Stderr, "SKIP: weed mini failed to start: %v\n", startErr)
sharedEnv.cleanupForMain()
os.Exit(0)
}
code := m.Run()
sharedEnv.cleanupForMain()
os.Exit(code)
}
// TestEnvironment contains the test environment configuration
type TestEnvironment struct {
seaweedDir string
weedBinary string
dataDir string
s3Port int
s3GrpcPort int
icebergPort int
lancePort int
masterPort int
masterGrpcPort int
filerPort int
filerGrpcPort int
volumePort int
volumeGrpcPort int
weedProcess *exec.Cmd
weedCancel context.CancelFunc
dockerAvailable bool
}
// newTestEnvironmentForMain creates a TestEnvironment without calling t.Fatalf so it
// can be used from TestMain (which has no *testing.T).
func newTestEnvironmentForMain() (*TestEnvironment, error) {
// Find the SeaweedFS root directory
wd, err := os.Getwd()
if err != nil {
return nil, fmt.Errorf("get working directory: %w", err)
}
seaweedDir := wd
for i := 0; i < 5; i++ {
if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
break
}
seaweedDir = filepath.Dir(seaweedDir)
}
// Check for weed binary
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
if info, statErr := os.Stat(weedBinary); statErr == nil {
// Name the binary and its age. `make test` rebuilds first, but a plain
// `go test` will happily drive a weeks-old binary and report a pass for
// code it never ran.
fmt.Fprintf(os.Stderr, "using %s, built %s\n", weedBinary, info.ModTime().Format(time.RFC3339))
} else if os.IsNotExist(statErr) {
weedBinary = "weed"
if _, err := exec.LookPath(weedBinary); err != nil {
return nil, fmt.Errorf("weed binary not found")
}
}
// Create temporary data directory
dataDir, err := os.MkdirTemp("", "seaweed-iceberg-test-*")
if err != nil {
return nil, fmt.Errorf("create temp dir: %w", err)
}
// Allocate 10 unique ports atomically: s3, iceberg, s3Grpc, master, masterGrpc,
// filer, filerGrpc, volume, volumeGrpc, lance
ports, err := testutil.AllocatePorts(10)
if err != nil {
return nil, fmt.Errorf("allocate ports: %w", err)
}
return &TestEnvironment{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
dataDir: dataDir,
s3Port: ports[0],
s3GrpcPort: ports[1],
icebergPort: ports[2],
masterPort: ports[3],
masterGrpcPort: ports[4],
filerPort: ports[5],
filerGrpcPort: ports[6],
volumePort: ports[7],
volumeGrpcPort: ports[8],
lancePort: ports[9],
dockerAvailable: testutil.HasDocker(),
}, nil
}
// startSeaweedFSForMain starts weed mini without a *testing.T (for use in TestMain).
func (env *TestEnvironment) startSeaweedFSForMain() error {
ctx, cancel := context.WithCancel(context.Background())
env.weedCancel = cancel
masterDir := filepath.Join(env.dataDir, "master")
filerDir := filepath.Join(env.dataDir, "filer")
volumeDir := filepath.Join(env.dataDir, "volume")
for _, dir := range []string{masterDir, filerDir, volumeDir} {
if err := os.MkdirAll(dir, 0755); err != nil {
cancel()
return fmt.Errorf("create directory %s: %w", dir, err)
}
}
cmd := exec.CommandContext(ctx, env.weedBinary, "mini",
"-master.port", fmt.Sprintf("%d", env.masterPort),
"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
"-volume.port", fmt.Sprintf("%d", env.volumePort),
"-volume.port.grpc", fmt.Sprintf("%d", env.volumeGrpcPort),
"-filer.port", fmt.Sprintf("%d", env.filerPort),
"-filer.port.grpc", fmt.Sprintf("%d", env.filerGrpcPort),
"-s3.port", fmt.Sprintf("%d", env.s3Port),
"-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort),
"-s3.port.iceberg", fmt.Sprintf("%d", env.icebergPort),
"-s3.port.lance", fmt.Sprintf("%d", env.lancePort),
"-ip.bind", "0.0.0.0",
"-dir", env.dataDir,
)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
cancel()
return fmt.Errorf("start SeaweedFS: %w", err)
}
env.weedProcess = cmd
if !env.waitForService(fmt.Sprintf("http://127.0.0.1:%d/v1/config", env.icebergPort), 30*time.Second) {
cancel()
cmd.Wait()
return fmt.Errorf("Iceberg REST API did not become ready")
}
return nil
}
// cleanupForMain stops SeaweedFS and cleans up resources (no *testing.T needed).
func (env *TestEnvironment) cleanupForMain() {
if env.weedCancel != nil {
env.weedCancel()
}
if env.weedProcess != nil {
time.Sleep(2 * time.Second)
env.weedProcess.Wait()
}
if env.dataDir != "" {
os.RemoveAll(env.dataDir)
}
}
// waitForService waits for a service to become available
func (env *TestEnvironment) waitForService(url string, timeout time.Duration) bool {
client := &http.Client{Timeout: 2 * time.Second}
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
resp, err := client.Get(url)
if err == nil {
resp.Body.Close()
if resp.StatusCode == http.StatusOK {
return true
}
}
time.Sleep(500 * time.Millisecond)
}
return false
}
// IcebergURL returns the Iceberg REST Catalog URL
func (env *TestEnvironment) IcebergURL() string {
return fmt.Sprintf("http://127.0.0.1:%d", env.icebergPort)
}
// LanceURL returns the Lance Namespace server URL
func (env *TestEnvironment) LanceURL() string {
return fmt.Sprintf("http://127.0.0.1:%d", env.lancePort)
}
// TestIcebergConfig tests the /v1/config endpoint
func TestIcebergConfig(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
// Test GET /v1/config
resp, err := http.Get(env.IcebergURL() + "/v1/config")
if err != nil {
t.Fatalf("Failed to get config: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Errorf("Expected status 200, got %d", resp.StatusCode)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
t.Fatalf("Failed to read response body: %v", err)
}
// Verify response contains required fields
bodyStr := string(body)
if !strings.Contains(bodyStr, "defaults") || !strings.Contains(bodyStr, "overrides") {
t.Errorf("Config response missing required fields: %s", bodyStr)
}
}
// TestIcebergNamespaces tests namespace operations
func TestIcebergNamespaces(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
// Create the default table bucket first via S3
bucketName := "warehouse-ns-" + randomSuffix()
createTableBucket(t, env, bucketName, "")
// Test GET /v1/namespaces (should return empty list initially)
resp, err := http.Get(env.IcebergURL() + icebergPath(bucketName, "/v1/namespaces"))
if err != nil {
t.Fatalf("Failed to list namespaces: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Expected status 200, got %d: %s", resp.StatusCode, body)
}
}
// TestStageCreateAndFinalizeFlow verifies staged create remains invisible until assert-create commit finalizes table creation.
func TestStageCreateAndFinalizeFlow(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucketName := "warehouse-stage-" + randomSuffix()
createTableBucket(t, env, bucketName, "")
namespace := "stage_ns_" + randomSuffix()
tableName := "orders"
status, _, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, "/v1/namespaces"), map[string]any{
"namespace": []string{namespace},
})
if err != nil {
t.Fatalf("Create namespace request failed: %v", err)
}
if status != http.StatusOK && status != http.StatusConflict {
t.Fatalf("Create namespace status = %d, want 200 or 409", status)
}
status, badReqResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables", namespace)), map[string]any{
"stage-create": true,
})
if err != nil {
t.Fatalf("Stage create missing-name request failed: %v", err)
}
if status != http.StatusBadRequest {
t.Fatalf("Stage create missing-name status = %d, want 400", status)
}
errorObj, _ := badReqResp["error"].(map[string]any)
if got := errorObj["type"]; got != "BadRequestException" {
t.Fatalf("error.type = %v, want BadRequestException", got)
}
msg, _ := errorObj["message"].(string)
if !strings.Contains(strings.ToLower(msg), "table name is required") {
t.Fatalf("error.message = %v, want it to include %q", errorObj["message"], "table name is required")
}
status, stageResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables", namespace)), map[string]any{
"name": tableName,
"stage-create": true,
})
if err != nil {
t.Fatalf("Stage create request failed: %v", err)
}
if status != http.StatusOK {
t.Fatalf("Stage create status = %d, want 200", status)
}
stageLocation, _ := stageResp["metadata-location"].(string)
if !strings.HasSuffix(stageLocation, "/metadata/v1.metadata.json") {
t.Fatalf("stage metadata-location = %q, want suffix /metadata/v1.metadata.json", stageLocation)
}
status, _, err = doIcebergJSONRequest(env, http.MethodGet, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), nil)
if err != nil {
t.Fatalf("Load staged table request failed: %v", err)
}
if status != http.StatusNotFound {
t.Fatalf("Load staged table status = %d, want 404", status)
}
status, commitResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), map[string]any{
"requirements": []map[string]any{
{"type": "assert-create"},
},
"updates": []any{},
})
if err != nil {
t.Fatalf("Finalize commit request failed: %v", err)
}
if status != http.StatusOK {
t.Fatalf("Finalize commit status = %d, want 200", status)
}
commitLocation, _ := commitResp["metadata-location"].(string)
if !strings.HasSuffix(commitLocation, "/metadata/v1.metadata.json") {
t.Fatalf("final metadata-location = %q, want suffix /metadata/v1.metadata.json", commitLocation)
}
status, loadResp, err := doIcebergJSONRequest(env, http.MethodGet, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), nil)
if err != nil {
t.Fatalf("Load finalized table request failed: %v", err)
}
if status != http.StatusOK {
t.Fatalf("Load finalized table status = %d, want 200", status)
}
loadLocation, _ := loadResp["metadata-location"].(string)
if loadLocation != commitLocation {
t.Fatalf("loaded metadata-location = %q, want %q", loadLocation, commitLocation)
}
}
// TestCommitMissingTableWithoutAssertCreate ensures missing-table commits still require assert-create for creation.
func TestCommitMissingTableWithoutAssertCreate(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucketName := "warehouse-missing-" + randomSuffix()
createTableBucket(t, env, bucketName, "")
namespace := "stage_missing_assert_ns_" + randomSuffix()
tableName := "missing_table"
status, _, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, "/v1/namespaces"), map[string]any{
"namespace": []string{namespace},
})
if err != nil {
t.Fatalf("Create namespace request failed: %v", err)
}
if status != http.StatusOK && status != http.StatusConflict {
t.Fatalf("Create namespace status = %d, want 200 or 409", status)
}
status, _, err = doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), map[string]any{
"requirements": []any{},
"updates": []any{},
})
if err != nil {
t.Fatalf("Commit missing table request failed: %v", err)
}
if status != http.StatusNotFound {
t.Fatalf("Commit missing table status = %d, want 404", status)
}
}
// doIcebergJSONRequest decodes JSON object responses used by catalog tests.
func doIcebergJSONRequest(env *TestEnvironment, method, path string, payload any) (int, map[string]any, error) {
url := env.IcebergURL() + path
var bodyReader io.Reader
if payload != nil {
data, err := json.Marshal(payload)
if err != nil {
return 0, nil, err
}
bodyReader = bytes.NewReader(data)
}
req, err := http.NewRequest(method, url, bodyReader)
if err != nil {
return 0, nil, err
}
if payload != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return 0, nil, err
}
defer resp.Body.Close()
data, err := io.ReadAll(resp.Body)
if err != nil {
return resp.StatusCode, nil, err
}
if len(data) == 0 {
return resp.StatusCode, nil, nil
}
var decoded map[string]any
if err := json.Unmarshal(data, &decoded); err != nil {
return resp.StatusCode, nil, fmt.Errorf("failed to decode %s %s response: %w body=%s", method, path, err, string(data))
}
return resp.StatusCode, decoded, nil
}
// icebergPath inserts the table bucket prefix into Iceberg REST API paths.
// For example, "/v1/namespaces" with prefix "my-bucket" becomes
// "/v1/my-bucket/namespaces".
func icebergPath(prefix, path string) string {
if prefix == "" {
return path
}
const base = "/v1/"
if !strings.HasPrefix(path, base) {
return path
}
withPrefix := base + prefix + "/" + strings.TrimPrefix(path, base)
return withPrefix
}
// createTableBucket creates a table bucket via the S3Tables REST API.
// The request is AWS V4 signed for SERVICE=s3tables so the S3 Tables
// route matcher accepts it; signing with regular SERVICE=s3 would let
// the request fall through to the S3 CreateBucket handler.
// createTableBucket makes a table bucket of the given format. An empty format
// leaves it to the server, which means ICEBERG.
func createTableBucket(t *testing.T, env *TestEnvironment, bucketName, format string) {
t.Helper()
endpoint := fmt.Sprintf("http://localhost:%d/buckets", env.s3Port)
reqBody := fmt.Sprintf(`{"name":"%s"}`, bucketName)
if format != "" {
reqBody = fmt.Sprintf(`{"name":"%s","format":"%s"}`, bucketName, format)
}
req, err := http.NewRequest(http.MethodPut, endpoint, strings.NewReader(reqBody))
if err != nil {
t.Fatalf("Failed to create request: %v", err)
}
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
signS3TablesRequest(t, req, reqBody)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("Failed to create table bucket %s: %v", bucketName, err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
t.Logf("Create table bucket %s response: status=%d, body=%s", bucketName, resp.StatusCode, string(body))
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusConflict {
t.Fatalf("Failed to create table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
}
t.Logf("Created table bucket %s", bucketName)
}
// signS3TablesRequest signs req with AWS V4 for SERVICE=s3tables. The
// underlying weed mini instance runs in default-allow mode so the
// signature itself is not verified; only the credential scope matters,
// because the S3 Tables route matcher requires SERVICE=s3tables to
// distinguish S3 Tables traffic from regular S3 calls on the same paths.
func signS3TablesRequest(t *testing.T, req *http.Request, body string) {
t.Helper()
creds := credentials.NewStaticCredentials("test-ak", "test-sk", "")
signer := v4.NewSigner(creds)
if _, err := signer.Sign(req, strings.NewReader(body), "s3tables", "us-east-1", time.Now()); err != nil {
t.Fatalf("Failed to sign S3 Tables request: %v", err)
}
}
// randomSuffix returns a short random hex suffix for unique resource naming.
func randomSuffix() string {
return fmt.Sprintf("%x", time.Now().UnixNano()&0xffffffff)
}
// TestDuckDBIntegration tests Iceberg catalog operations using DuckDB
// This test requires Docker to be available
func TestDuckDBIntegration(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping DuckDB integration test")
}
// Create a temporary SQL file for DuckDB to execute
sqlFile := filepath.Join(env.dataDir, "test.sql")
sqlContent := fmt.Sprintf(`
-- Install and load Iceberg extension
INSTALL iceberg;
LOAD iceberg;
-- List namespaces via Iceberg REST catalog (basic connectivity test)
-- Note: Full operations require setting up S3 credentials which is beyond this test
SELECT 'Iceberg extension loaded successfully' as result;
`)
if err := os.WriteFile(sqlFile, []byte(sqlContent), 0644); err != nil {
t.Fatalf("Failed to write SQL file: %v", err)
}
// Run DuckDB in Docker to test Iceberg connectivity
// Use host.docker.internal to connect to the host's Iceberg port
cmd := exec.Command("docker", "run", "--rm",
"-v", fmt.Sprintf("%s:/test", env.dataDir),
"--add-host", "host.docker.internal:host-gateway",
"--entrypoint", "duckdb",
"duckdb/duckdb:latest",
"-init", "/test/test.sql",
"-c", "SELECT 1",
)
output, err := cmd.CombinedOutput()
if err != nil {
t.Logf("DuckDB output: %s", output)
// Check for expected errors in certain CI environments
outputStr := string(output)
if strings.Contains(outputStr, "iceberg extension is not available") ||
strings.Contains(outputStr, "Failed to load") {
t.Skip("Skipping DuckDB test: Iceberg extension not available in Docker image")
}
// Any other error is unexpected
t.Fatalf("DuckDB command failed unexpectedly. Output: %s\nError: %v", output, err)
}
// Verify the test completed successfully
outputStr := string(output)
t.Logf("DuckDB output: %s", outputStr)
if !strings.Contains(outputStr, "Iceberg extension loaded successfully") {
t.Errorf("Expected success message in output, got: %s", outputStr)
}
}

View File

@@ -0,0 +1,283 @@
// Reproduction tests for https://github.com/seaweedfs/seaweedfs/issues/9103
//
// The issue reports two distinct failure modes when using DuckDB against the
// SeaweedFS Iceberg REST catalog:
//
// 1. `ATTACH 's3://test/' AS cat (TYPE 'ICEBERG', ...); SELECT * FROM cat.ovirt.disk;`
// fails with "Table with name 'ovirt.disk' does not exist because schema
// 'ovirt' does not exist." The namespace "ovirt" does exist in the bucket.
//
// 2. `SELECT * FROM iceberg_scan('s3://test/ovirt/disk');` fails with HTTP 403
// AccessDenied when DuckDB tries to glob `s3://test/ovirt/disk/metadata/v*`
// because the LoadTable response does not vend S3 file-io credentials back
// to the client.
//
// These tests reproduce the underlying catalog-level misbehavior at the REST
// protocol layer so they run quickly and deterministically. A DuckDB-based
// end-to-end reproduction is also included (gated on Docker availability).
package catalog
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/seaweedfs/seaweedfs/test/testutil"
)
// TestIssue9103_ConfigDoesNotVendWarehousePrefix reproduces failure mode #2.
//
// Per the Iceberg REST spec, a client attaching with warehouse=s3://<bucket>/
// calls GET /v1/config?warehouse=s3://<bucket>/ and expects the server to
// return an `overrides.prefix` that identifies the catalog namespace for
// subsequent requests. Without it the client falls back to unprefixed paths
// like /v1/namespaces, which on SeaweedFS resolve to the hard-coded default
// bucket ("warehouse") and therefore do not list the user's namespaces.
//
// Currently the handler ignores the warehouse query parameter and returns
// empty defaults/overrides. That is exactly what makes DuckDB's ATTACH flow
// report `schema "ovirt" does not exist`.
func TestIssue9103_ConfigDoesNotVendWarehousePrefix(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucketName := "warehouse-9103cfg-" + randomSuffix()
createTableBucket(t, env, bucketName, "")
warehouse := fmt.Sprintf("s3://%s/", bucketName)
u := fmt.Sprintf("%s/v1/config?warehouse=%s", env.IcebergURL(), url.QueryEscape(warehouse))
status, body, err := doIcebergJSONRequest(env, http.MethodGet, fmt.Sprintf("/v1/config?warehouse=%s", url.QueryEscape(warehouse)), nil)
if err != nil {
t.Fatalf("GET %s failed: %v", u, err)
}
if status != http.StatusOK {
t.Fatalf("GET %s status = %d, want 200", u, status)
}
overrides, _ := body["overrides"].(map[string]any)
gotPrefix, _ := overrides["prefix"].(string)
if gotPrefix != bucketName {
t.Fatalf("GET /v1/config?warehouse=%s: overrides.prefix = %q, want %q (catalog must echo the warehouse's table bucket so clients like DuckDB know which /v1/{prefix}/namespaces to use)",
warehouse, gotPrefix, bucketName)
}
}
// TestIssue9103_BareNamespacesListMissesNamespaceInAttachedBucket
// demonstrates the downstream effect of the /v1/config bug.
//
// The client created the namespace "ovirt" inside bucket "test" (via
// /v1/{bucket}/namespaces), but when it later issues GET /v1/namespaces (no
// prefix, because /v1/config didn't give it one), that request resolves to
// the default bucket and returns no namespaces. DuckDB surfaces this as
// `schema "ovirt" does not exist`.
func TestIssue9103_BareNamespacesListMissesNamespaceInAttachedBucket(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucketName := "warehouse-9103ns-" + randomSuffix()
createTableBucket(t, env, bucketName, "")
namespace := "ovirt"
status, _, err := doIcebergJSONRequest(env, http.MethodPost,
icebergPath(bucketName, "/v1/namespaces"),
map[string]any{"namespace": []string{namespace}})
if err != nil {
t.Fatalf("create namespace: %v", err)
}
if status != http.StatusOK && status != http.StatusConflict {
t.Fatalf("create namespace status = %d, want 200 or 409", status)
}
// Sanity: prefixed listing finds the namespace.
status, prefixedBody, err := doIcebergJSONRequest(env, http.MethodGet,
icebergPath(bucketName, "/v1/namespaces"), nil)
if err != nil {
t.Fatalf("list namespaces with prefix: %v", err)
}
if status != http.StatusOK {
t.Fatalf("prefixed list status = %d, want 200", status)
}
if !containsNamespace(prefixedBody, namespace) {
t.Fatalf("prefixed list missing namespace %q: %v", namespace, prefixedBody)
}
// Simulate what a DuckDB-style client does when /v1/config did not vend a
// prefix: it falls back to the unprefixed listing. It should still be
// able to discover the namespace (e.g. via a warehouse query parameter),
// but today it cannot, which is the user-visible bug.
bareWarehouse := "s3://" + bucketName + "/"
status, bareBody, err := doIcebergJSONRequest(env, http.MethodGet,
fmt.Sprintf("/v1/namespaces?warehouse=%s", url.QueryEscape(bareWarehouse)), nil)
if err != nil {
t.Fatalf("bare list namespaces: %v", err)
}
switch {
case status != http.StatusOK:
t.Fatalf("GET /v1/namespaces?warehouse=%s status = %d, want 200 (server ignores the warehouse query parameter and falls back to the default %q bucket, causing DuckDB to see no schemas)",
bareWarehouse, status, "warehouse")
case !containsNamespace(bareBody, namespace):
t.Fatalf("GET /v1/namespaces?warehouse=%s did not return %q: %v (server must honor the warehouse query parameter here or vend overrides.prefix from /v1/config so the client can route to the right bucket)",
bareWarehouse, namespace, bareBody)
}
}
// TestIssue9103_DuckDBAttachCannotResolveNamespace is the end-to-end
// reproduction of the issue using DuckDB in Docker, mirroring the exact
// sequence of commands from the bug report. It runs under its own weed
// mini with IAM configured so the OAuth2 client_credentials flow that
// DuckDB's iceberg extension requires actually works (the shared env has
// no credentials registered). Gated on Docker availability.
func TestIssue9103_DuckDBAttachCannotResolveNamespace(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
if !testutil.HasDocker() {
t.Skip("Docker not available, skipping DuckDB integration reproduction")
}
env := newOAuthTestEnv(t)
defer env.cleanup(t)
env.start(t)
bucketName := "test9103-" + randomSuffix()
createTableBucketViaShell(t, env, bucketName)
namespace := "ovirt"
tableName := "disk"
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
createNamespaceWithToken(t, env, token, bucketName, namespace)
createTableWithToken(t, env, token, bucketName, namespace, tableName)
sql := fmt.Sprintf(`
INSTALL iceberg;
LOAD iceberg;
CREATE SECRET test_berg (
TYPE ICEBERG,
ENDPOINT 'http://host.docker.internal:%d',
SCOPE 's3://%s/',
CLIENT_ID '%s',
CLIENT_SECRET '%s'
);
CREATE SECRET s3_berg (
TYPE S3,
KEY_ID '%s',
SECRET '%s',
ENDPOINT 'host.docker.internal:%d',
URL_STYLE 'path',
USE_SSL false,
SCOPE 's3://%s/'
);
ATTACH 's3://%s/' AS test_catalog (TYPE 'ICEBERG', secret test_berg);
SELECT * FROM test_catalog.%s.%s LIMIT 0;
`, env.icebergPort, bucketName,
env.accessKey, env.secretKey,
env.accessKey, env.secretKey, env.s3Port, bucketName,
bucketName, namespace, tableName)
sqlFile := filepath.Join(env.dataDir, "issue_9103.sql")
if err := os.WriteFile(sqlFile, []byte(sql), 0644); err != nil {
t.Fatalf("write SQL: %v", err)
}
cmd := exec.Command("docker", "run", "--rm",
"-v", fmt.Sprintf("%s:/test", env.dataDir),
"--add-host", "host.docker.internal:host-gateway",
"-e", "AWS_REGION=us-east-1",
"--entrypoint", "duckdb",
"duckdb/duckdb:latest",
"-init", "/test/issue_9103.sql",
"-c", "SELECT 1",
)
out, runErr := cmd.CombinedOutput()
outStr := string(out)
t.Logf("DuckDB output:\n%s", outStr)
if strings.Contains(outStr, "iceberg extension is not available") ||
strings.Contains(outStr, "Failed to load") ||
strings.Contains(outStr, "could not fetch extension") ||
strings.Contains(outStr, "failed to download") {
t.Skip("Iceberg extension not available in DuckDB Docker image")
}
if runErr != nil {
if strings.Contains(outStr, "does not exist because schema") &&
strings.Contains(outStr, namespace) {
t.Fatalf("reproduced issue #9103: DuckDB cannot see namespace %q after ATTACH 's3://%s/'; see output above",
namespace, bucketName)
}
t.Fatalf("DuckDB run failed for an unexpected reason: %v", runErr)
}
}
// createTableWithToken creates an Iceberg table using an OAuth bearer token.
func createTableWithToken(t *testing.T, env *oauthTestEnv, token, bucketName, namespace, tableName string) {
t.Helper()
payload := map[string]any{
"name": tableName,
"schema": map[string]any{
"type": "struct",
"schema-id": 0,
"fields": []map[string]any{
{"id": 1, "name": "id", "required": true, "type": "long"},
},
},
}
body, err := json.Marshal(payload)
if err != nil {
t.Fatalf("marshal create-table payload: %v", err)
}
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables", env.icebergURL(), bucketName, namespace)
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
if err != nil {
t.Fatalf("create request: %v", err)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+token)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("create table: %v", err)
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
t.Fatalf("create table failed: status=%d body=%s", resp.StatusCode, respBody)
}
t.Logf("Created table %s.%s in bucket %s", namespace, tableName, bucketName)
}
// containsNamespace checks whether an Iceberg REST ListNamespaces response
// contains the given single-level namespace.
func containsNamespace(body map[string]any, name string) bool {
arr, _ := body["namespaces"].([]any)
for _, item := range arr {
parts, _ := item.([]any)
if len(parts) == 1 {
if s, ok := parts[0].(string); ok && s == name {
return true
}
}
}
return false
}

View File

@@ -0,0 +1,63 @@
package catalog
import (
"fmt"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
"os"
"os/exec"
"path/filepath"
"testing"
)
// TestLanceNamespaceRealClient drives the namespace with the Lance client rather
// than hand-built HTTP. The Go tests beside this one cover the catalog surface;
// what this adds is that the location and storage_options the namespace vends
// are actually enough to write and read a dataset, which needs the S3 layout
// guard, the endpoint and the credentials all to be right at once.
//
// To run manually:
//
// cd test/s3tables/catalog
// docker build -t lance-namespace-test -f Dockerfile.lance .
func TestLanceNamespaceRealClient(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping Lance client integration test")
}
bucketName := "lance-client-test-" + randomSuffix()
createTableBucket(t, env, bucketName, s3tables.FormatLance)
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
buildCmd := exec.Command("docker", "build", "-t", "lance-namespace-test", "-f", "Dockerfile.lance", ".")
buildCmd.Dir = testDir
if out, err := buildCmd.CombinedOutput(); err != nil {
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
}
namespaceURL := fmt.Sprintf("http://host.docker.internal:%d", env.lancePort)
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
cmd := exec.Command("docker", "run", "--rm",
"--add-host", "host.docker.internal:host-gateway",
"-v", fmt.Sprintf("%s:/app:ro", testDir),
"lance-namespace-test",
"python3", "/app/test_lance_namespace.py",
"--namespace-url", namespaceURL,
"--s3-endpoint", s3Endpoint,
"--bucket", bucketName,
)
cmd.Dir = testDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
t.Logf("Running Lance client test against %s", namespaceURL)
if err := cmd.Run(); err != nil {
t.Errorf("Lance client test failed: %v", err)
}
}

View File

@@ -0,0 +1,378 @@
// Integration tests for the Lance Namespace REST server, driven against a live
// gateway rather than an in-memory filer. The bugs this surface has produced -
// a deregister that deleted the dataset, an S3 door that refused every Lance
// file - all looked fine against a fake.
package catalog
import (
"bytes"
"encoding/json"
"fmt"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
"io"
"net/http"
"strings"
"sync"
"testing"
)
// lanceCall posts to the Lance namespace and returns the status and body.
func lanceCall(t *testing.T, env *TestEnvironment, method, path, body string) (int, []byte) {
t.Helper()
req, err := http.NewRequest(method, env.LanceURL()+path, strings.NewReader(body))
if err != nil {
t.Fatalf("build request: %v", err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
defer resp.Body.Close()
payload, _ := io.ReadAll(resp.Body)
return resp.StatusCode, payload
}
func lanceMust(t *testing.T, env *TestEnvironment, method, path, body string, want int) []byte {
t.Helper()
status, payload := lanceCall(t, env, method, path, body)
if status != want {
t.Fatalf("%s %s = %d (%s), want %d", method, path, status, payload, want)
}
return payload
}
// filerEntryExists reports whether a path exists on storage, which is how these
// tests tell "the catalog forgot the table" from "the data is gone".
func filerEntryExists(t *testing.T, env *TestEnvironment, path string) bool {
t.Helper()
url := fmt.Sprintf("http://127.0.0.1:%d%s", env.filerPort, path)
resp, err := http.Get(url)
if err != nil {
t.Fatalf("filer GET %s: %v", path, err)
}
defer resp.Body.Close()
io.Copy(io.Discard, resp.Body)
return resp.StatusCode == http.StatusOK
}
// lanceTestBucket makes a uniquely named table bucket of the given format. Most
// tests want LANCE; the one that checks what the Lance surface hides needs an
// Iceberg bucket, because a declared bucket holds one format only.
func lanceTestBucket(t *testing.T, env *TestEnvironment, prefix, format string) string {
t.Helper()
bucket := prefix + "-" + randomSuffix()
createTableBucket(t, env, bucket, format)
return bucket
}
func TestLanceNamespaceLifecycle(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucket := lanceTestBucket(t, env, "lance-ns", s3tables.FormatLance)
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$sales/create", `{}`, http.StatusOK)
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$sales/exists", `{}`, http.StatusOK)
// The root lists table buckets, which is the first namespace level here.
var roots struct {
Namespaces []string `json:"namespaces"`
}
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/$/list", "", http.StatusOK), &roots); err != nil {
t.Fatalf("decode root listing: %v", err)
}
found := false
for _, name := range roots.Namespaces {
if name == bucket {
found = true
}
}
if !found {
t.Fatalf("root listing %v does not include %s", roots.Namespaces, bucket)
}
var children struct {
Namespaces []string `json:"namespaces"`
}
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"/list", "", http.StatusOK), &children); err != nil {
t.Fatalf("decode namespace listing: %v", err)
}
if len(children.Namespaces) != 1 || children.Namespaces[0] != "sales" {
t.Fatalf("namespace listing = %v, want [sales]", children.Namespaces)
}
// A namespace is never created as a side effect of naming one inside a
// bucket that does not exist.
status, _ := lanceCall(t, env, http.MethodPost, "/v1/namespace/nosuchbucket$ns/create", `{}`)
if status != http.StatusNotFound {
t.Fatalf("create under a missing bucket = %d, want 404", status)
}
}
func TestLanceTableLifecyclePreservesData(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucket := lanceTestBucket(t, env, "lance-tbl", s3tables.FormatLance)
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
table := "/v1/table/" + bucket + "$ml$vectors"
var declared struct {
Location string `json:"location"`
StorageOptions map[string]string `json:"storage_options"`
}
if err := json.Unmarshal(lanceMust(t, env, http.MethodPost, table+"/declare", `{}`, http.StatusOK), &declared); err != nil {
t.Fatalf("decode declare: %v", err)
}
want := fmt.Sprintf("s3://%s/ml/vectors", bucket)
if declared.Location != want {
t.Fatalf("declared location = %q, want %q", declared.Location, want)
}
datasetPath := fmt.Sprintf("/buckets/%s/ml/vectors/", bucket)
if !filerEntryExists(t, env, datasetPath) {
t.Fatal("declare did not create the dataset directory")
}
var listed struct {
Tables []string `json:"tables"`
}
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"$ml/table/list", "", http.StatusOK), &listed); err != nil {
t.Fatalf("decode listing: %v", err)
}
if len(listed.Tables) != 1 || listed.Tables[0] != bucket+"$ml$vectors" {
t.Fatalf("table listing = %v, want the full identifier", listed.Tables)
}
// Deregistering forgets the table but keeps every byte. The catalog entry is
// the dataset directory, so a drop of the entry would take the data too.
lanceMust(t, env, http.MethodPost, table+"/deregister", `{}`, http.StatusOK)
lanceMust(t, env, http.MethodPost, table+"/exists", `{}`, http.StatusNotFound)
if !filerEntryExists(t, env, datasetPath) {
t.Fatal("deregister deleted the dataset")
}
registerBody := fmt.Sprintf(`{"location":%q}`, want)
lanceMust(t, env, http.MethodPost, table+"/register", registerBody, http.StatusOK)
lanceMust(t, env, http.MethodPost, table+"/exists", `{}`, http.StatusOK)
// Dropping is the operation that does remove the data.
lanceMust(t, env, http.MethodPost, table+"/drop", `{}`, http.StatusOK)
if filerEntryExists(t, env, datasetPath) {
t.Fatal("drop left the dataset behind")
}
}
// A Lance client must never resolve an Iceberg table's location, or it writes a
// dataset over a table another engine owns.
func TestLanceHidesIcebergTables(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
// An Iceberg bucket, because a bucket that declares LANCE cannot hold an
// Iceberg table at all now. The invariant still matters from this side: the
// Lance surface can be pointed at any bucket, and must not describe or list
// a table whose format it does not serve.
bucket := lanceTestBucket(t, env, "lance-mixed", s3tables.FormatIceberg)
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$mixed/create", `{}`, http.StatusOK)
createIcebergTable(t, env, bucket, "mixed", "ledger")
status, _ := lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$ledger/describe", `{}`)
if status != http.StatusNotFound {
t.Fatalf("describing an iceberg table through lance = %d, want 404", status)
}
status, _ = lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$ledger/declare", `{}`)
if status != http.StatusConflict {
t.Fatalf("declaring over an iceberg table = %d, want 409", status)
}
// And a new Lance table cannot be smuggled in beside it either.
status, _ = lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$vectors/declare", `{}`)
if status != http.StatusConflict {
t.Fatalf("declaring a lance table in an iceberg bucket = %d, want 409", status)
}
var listed struct {
Tables []string `json:"tables"`
}
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"$mixed/table/list", "", http.StatusOK), &listed); err != nil {
t.Fatalf("decode listing: %v", err)
}
if len(listed.Tables) != 0 {
t.Fatalf("lance listing shows iceberg tables: %v", listed.Tables)
}
}
// The S3 door validates every object written into a table bucket. A Lance
// dataset's files have to get through it, or the catalog is decorative.
func TestLanceFilesAreAcceptedByTheS3Door(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucket := lanceTestBucket(t, env, "lance-layout", s3tables.FormatLance)
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
lanceMust(t, env, http.MethodPost, "/v1/table/"+bucket+"$ml$vectors/declare", `{}`, http.StatusOK)
accepted := []string{
"ml/vectors/data/0111111010111000110101116.lance",
"ml/vectors/_versions/18446744073709551614.manifest",
"ml/vectors/_versions/18446744073709551614.manifest-a3a292ad",
"ml/vectors/_transactions/0-ddb27ab7.txn",
"ml/vectors/_indices/85814508-fts/index.idx",
"ml/vectors/.lance-reserved",
}
for _, object := range accepted {
if status := putS3Object(t, env, bucket, object); status != http.StatusOK {
t.Errorf("PUT %s = %d, want 200", object, status)
}
}
// The guard still rejects files that belong to no table layout.
for _, object := range []string{"ml/vectors/random.txt", "ml/vectors/notadir/x.lance"} {
if status := putS3Object(t, env, bucket, object); status == http.StatusOK {
t.Errorf("PUT %s = 200, want a rejection", object)
}
}
}
// A Lance commit is a conditional PUT of the next manifest: object_store turns
// PutMode::Create into If-None-Match: *, and lance treats the refusal as a
// commit conflict and rebases. So the whole safety of concurrent writers rests
// on this store answering that precondition atomically. It does, because the
// gateway reduces the header to a filer WriteCondition evaluated at the
// object's owner under a per-path lock.
//
// This is the reason the namespace does not manage versions itself.
func TestLanceCommitPreconditionAdmitsOneWriter(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucket := lanceTestBucket(t, env, "lance-commit", s3tables.FormatLance)
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
lanceMust(t, env, http.MethodPost, "/v1/table/"+bucket+"$ml$vectors/declare", `{}`, http.StatusOK)
const writers = 8
var wg sync.WaitGroup
statuses := make([]int, writers)
failures := make([]error, writers)
for i := 0; i < writers; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
// Not t.Fatalf: from a goroutine it ends only that goroutine, and
// the status stays zero, which reads as a bogus response later.
statuses[i], failures[i] = putIfAbsent(env, bucket, "ml/vectors/_versions/1.manifest")
}(i)
}
wg.Wait()
for i, err := range failures {
if err != nil {
t.Fatalf("writer %d could not reach the gateway: %v", i, err)
}
}
won, refused := 0, 0
for _, status := range statuses {
switch status {
case http.StatusOK:
won++
case http.StatusPreconditionFailed:
refused++
default:
t.Fatalf("unexpected status %d committing a manifest", status)
}
}
if won != 1 {
t.Fatalf("%d writers committed version 1; exactly one may win, %d were refused", won, refused)
}
}
// The namespace answers managed_versioning=false: the dataset owns its version
// history, and a reader that never goes through this catalog still sees all of
// it.
func TestLanceDoesNotManageVersions(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
bucket := lanceTestBucket(t, env, "lance-ver", s3tables.FormatLance)
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
table := "/v1/table/" + bucket + "$ml$vectors"
var declared struct {
ManagedVersioning bool `json:"managed_versioning"`
}
if err := json.Unmarshal(lanceMust(t, env, http.MethodPost, table+"/declare", `{}`, http.StatusOK), &declared); err != nil {
t.Fatalf("decode declare: %v", err)
}
if declared.ManagedVersioning {
t.Fatal("managed_versioning must be false; the dataset owns its versions")
}
// The version ops answer with the spec's Unsupported code rather than a
// bare 404, so a client that asks learns why.
status, _ := lanceCall(t, env, http.MethodPost, table+"/version/list", `{}`)
if status != http.StatusNotImplemented {
t.Fatalf("version/list = %d, want 501", status)
}
}
// putIfAbsent writes an object only if the key is free, the way a Lance commit
// does. It returns an error rather than failing the test, so it is safe to call
// from the racing goroutines.
func putIfAbsent(env *TestEnvironment, bucket, object string) (int, error) {
url := fmt.Sprintf("http://127.0.0.1:%d/%s/%s", env.s3Port, bucket, object)
req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader([]byte("manifest")))
if err != nil {
return 0, err
}
req.Header.Set("If-None-Match", "*")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return 0, err
}
defer resp.Body.Close()
io.Copy(io.Discard, resp.Body)
return resp.StatusCode, nil
}
// putS3Object writes an object through the S3 gateway and returns the status.
func putS3Object(t *testing.T, env *TestEnvironment, bucket, object string) int {
t.Helper()
url := fmt.Sprintf("http://127.0.0.1:%d/%s/%s", env.s3Port, bucket, object)
req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader([]byte("x")))
if err != nil {
t.Fatalf("build request: %v", err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("PUT %s: %v", object, err)
}
defer resp.Body.Close()
io.Copy(io.Discard, resp.Body)
return resp.StatusCode
}
// createIcebergTable registers an ordinary Iceberg table so the mixed-catalog
// tests have one to be confused by.
func createIcebergTable(t *testing.T, env *TestEnvironment, bucket, namespace, name string) {
t.Helper()
body := fmt.Sprintf(`{"name":%q,"schema":{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}}`, name)
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables", env.IcebergURL(), bucket, namespace)
resp, err := http.Post(url, "application/json", strings.NewReader(body))
if err != nil {
t.Fatalf("create iceberg table: %v", err)
}
defer resp.Body.Close()
payload, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
t.Fatalf("create iceberg table = %d: %s", resp.StatusCode, payload)
}
}

View File

@@ -0,0 +1,137 @@
// Package catalog provides integration tests for the Iceberg REST Catalog API.
// This file adds PyIceberg-based compatibility tests using Docker.
package catalog
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"testing"
)
// TestPyIcebergRestCatalog tests the Iceberg REST Catalog using PyIceberg client in Docker.
// This provides a more comprehensive test than DuckDB as PyIceberg fully exercises the REST API.
//
// Prerequisites:
// - Docker must be available
// - SeaweedFS must be running with Iceberg REST enabled
//
// To run manually:
//
// cd test/s3tables/catalog
// docker compose -f docker-compose.test.yaml up --build
func TestPyIcebergRestCatalog(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping PyIceberg integration test")
}
// Create the test bucket first
bucketName := "pyiceberg-compat-test-" + randomSuffix()
createTableBucket(t, env, bucketName, "")
// Build the test working directory path
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
// Run PyIceberg test using Docker
catalogURL := fmt.Sprintf("http://host.docker.internal:%d", env.icebergPort)
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
warehouse := fmt.Sprintf("s3://%s/", bucketName)
// Build the test image first for faster repeated runs
buildCmd := exec.Command("docker", "build", "-t", "iceberg-rest-test", "-f", "Dockerfile.pyiceberg", ".")
buildCmd.Dir = testDir
if out, err := buildCmd.CombinedOutput(); err != nil {
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
}
cmd := exec.Command("docker", "run", "--rm",
"--add-host", "host.docker.internal:host-gateway",
"-e", fmt.Sprintf("AWS_ENDPOINT_URL=%s", s3Endpoint),
"-v", fmt.Sprintf("%s:/app:ro", testDir),
"iceberg-rest-test",
"python3", "/app/test_rest_catalog.py",
"--catalog-url", catalogURL,
"--warehouse", warehouse,
"--prefix", bucketName,
)
cmd.Dir = testDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
t.Logf("Running PyIceberg REST catalog test...")
t.Logf(" Catalog URL: %s", catalogURL)
t.Logf(" Warehouse: %s", warehouse)
if err := cmd.Run(); err != nil {
t.Errorf("PyIceberg test failed: %v", err)
}
}
// TestPyIcebergRestCatalogAuthenticated tests the Iceberg REST Catalog using PyIceberg with authentication.
// This test uses the default admin credentials that SeaweedFS creates on startup.
func TestPyIcebergRestCatalogAuthenticated(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := sharedEnv
if !env.dockerAvailable {
t.Skip("Docker not available, skipping PyIceberg integration test")
}
// Use default admin credentials
testAccessKey := "admin"
testSecretKey := "admin"
// Create the test bucket first (using unauthenticated request, which works with DefaultAllow)
bucketName := "pyiceberg-auth-test-" + randomSuffix()
createTableBucket(t, env, bucketName, "")
// Build the test working directory path
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
// Run PyIceberg test using Docker with authentication
catalogURL := fmt.Sprintf("http://host.docker.internal:%d", env.icebergPort)
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
warehouse := fmt.Sprintf("s3://%s/", bucketName)
// Build the test image first for faster repeated runs
buildCmd := exec.Command("docker", "build", "-t", "iceberg-rest-test", "-f", "Dockerfile.pyiceberg", ".")
buildCmd.Dir = testDir
if out, err := buildCmd.CombinedOutput(); err != nil {
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
}
cmd := exec.Command("docker", "run", "--rm",
"--add-host", "host.docker.internal:host-gateway",
"-e", fmt.Sprintf("AWS_ENDPOINT_URL=%s", s3Endpoint),
"-v", fmt.Sprintf("%s:/app:ro", testDir),
"iceberg-rest-test",
"python3", "/app/test_rest_catalog_auth.py",
"--catalog-url", catalogURL,
"--warehouse", warehouse,
"--prefix", bucketName,
"--access-key", testAccessKey,
"--secret-key", testSecretKey,
)
cmd.Dir = testDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
t.Logf("Running PyIceberg REST catalog test with authentication...")
t.Logf(" Catalog URL: %s", catalogURL)
t.Logf(" Warehouse: %s", warehouse)
t.Logf(" Access Key: %s", testAccessKey)
if err := cmd.Run(); err != nil {
t.Errorf("PyIceberg authenticated test failed: %v", err)
}
}

View File

@@ -0,0 +1,36 @@
package catalog
import (
"fmt"
"io"
"net/http"
"testing"
)
// verifyTableBucketMetadata verifies that a table bucket was created with proper metadata
func verifyTableBucketMetadata(t *testing.T, env *TestEnvironment, bucketName string) {
t.Helper()
// Use S3Tables REST API to get the bucket
endpoint := fmt.Sprintf("http://localhost:%d/buckets/%s", env.s3Port, bucketName)
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
if err != nil {
t.Fatalf("Failed to create request: %v", err)
}
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("Failed to get table bucket %s: %v", bucketName, err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
t.Logf("Get table bucket %s response: status=%d, body=%s", bucketName, resp.StatusCode, string(body))
if resp.StatusCode != http.StatusOK {
t.Fatalf("Failed to get table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
}
t.Logf("Verified table bucket %s exists with metadata", bucketName)
}

View File

@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""Drive the SeaweedFS Lance Namespace with the real Lance client.
The catalog half can be checked with plain HTTP, and the Go integration tests
do. What only a real client proves is that the location and storage_options it
hands back are enough to write and read a dataset: the layout guard on the S3
door, the endpoint and allow_http options, and the credentials all have to be
right at once, and a hand-built request checks none of that.
"""
import argparse
import sys
import warnings
warnings.filterwarnings("ignore")
import lance
import lance_namespace as ln
import pyarrow as pa
def sample_table():
return pa.table(
{
"id": pa.array([1, 2, 3, 4]),
"vec": pa.array([[1.0, 2.0]] * 4, type=pa.list_(pa.float32(), 2)),
}
)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--namespace-url", required=True, help="Lance namespace REST URL")
parser.add_argument("--s3-endpoint", required=True, help="S3 endpoint the dataset lives behind")
parser.add_argument("--bucket", required=True, help="table bucket to use, already created")
parser.add_argument("--access-key", default="any")
parser.add_argument("--secret-key", default="any")
args = parser.parse_args()
ns = ln.connect("rest", {"uri": args.namespace_url})
ns.create_namespace(ln.CreateNamespaceRequest(id=[args.bucket, "ml"]))
table_id = [args.bucket, "ml", "vectors"]
declared = ns.declare_table(ln.DeclareTableRequest(id=table_id))
print(f"declared {table_id} at {declared.location}")
if not declared.location:
print("FAIL: declare returned no location", file=sys.stderr)
return 1
described = ns.describe_table(ln.DescribeTableRequest(id=table_id))
if described.location != declared.location:
print(
f"FAIL: describe location {described.location} != declare {declared.location}",
file=sys.stderr,
)
return 1
options = dict(described.storage_options or {})
print(f"storage options from the namespace: {sorted(options)}")
# The endpoint is overridden with the container's view of the same gateway:
# what the namespace vends is correct for its own host, and a test harness
# bound to a wildcard address vends nothing at all. That the namespace vends
# a usable endpoint when it can is covered by the unit tests; what matters
# here is that everything else it hands back is enough to reach the data.
options["aws_endpoint"] = args.s3_endpoint
options["allow_http"] = "true"
# A deployment without STS still needs credentials to sign with.
options.setdefault("aws_access_key_id", args.access_key)
options.setdefault("aws_secret_access_key", args.secret_key)
lance.write_dataset(sample_table(), described.location, storage_options=options,
mode="overwrite")
dataset = lance.dataset(described.location, storage_options=options)
rows = dataset.count_rows()
print(f"wrote and read back {rows} rows at version {dataset.version}")
if rows != 4:
print(f"FAIL: read back {rows} rows, want 4", file=sys.stderr)
return 1
# The table is listed, and a client that skips the namespace entirely can
# still open the dataset by URI.
listed = ns.list_tables(ln.ListTablesRequest(id=[args.bucket, "ml"])).tables
if f"{args.bucket}$ml$vectors" not in listed:
print(f"FAIL: {listed} does not contain the table", file=sys.stderr)
return 1
if lance.dataset(declared.location, storage_options=options).count_rows() != 4:
print("FAIL: the dataset is not readable straight off its URI", file=sys.stderr)
return 1
# Deregistering hides the table and keeps the data, which is the difference
# between it and a drop.
ns.deregister_table(ln.DeregisterTableRequest(id=table_id))
if f"{args.bucket}$ml$vectors" in ns.list_tables(ln.ListTablesRequest(id=[args.bucket, "ml"])).tables:
print("FAIL: a deregistered table is still listed", file=sys.stderr)
return 1
if lance.dataset(declared.location, storage_options=options).count_rows() != 4:
print("FAIL: deregister destroyed the dataset", file=sys.stderr)
return 1
print("PASS")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,252 @@
#!/usr/bin/env python3
"""
Iceberg REST Catalog Compatibility Test for SeaweedFS
This script tests the Iceberg REST Catalog API compatibility of the
SeaweedFS Iceberg REST Catalog implementation.
Usage:
python3 test_rest_catalog.py --catalog-url http://localhost:8182
Requirements:
pip install pyiceberg[s3fs]
"""
import argparse
import sys
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import (
IntegerType,
LongType,
StringType,
NestedField,
)
from pyiceberg.exceptions import (
NamespaceAlreadyExistsError,
NoSuchNamespaceError,
TableAlreadyExistsError,
NoSuchTableError,
)
def test_config_endpoint(catalog):
"""Test that the catalog config endpoint returns valid configuration."""
print("Testing /v1/config endpoint...")
# The catalog is already loaded which means config endpoint worked
print(" /v1/config endpoint working")
return True
def test_namespace_operations(catalog, prefix):
"""Test namespace CRUD operations."""
print("Testing namespace operations...")
namespace = (f"{prefix.replace('-', '_')}_test_ns",)
# List initial namespaces
namespaces = catalog.list_namespaces()
print(f" Initial namespaces: {namespaces}")
# Create namespace
try:
catalog.create_namespace(namespace)
print(f" Created namespace: {namespace}")
except NamespaceAlreadyExistsError:
print(f" ! Namespace already exists: {namespace}")
# List namespaces (should include our new one)
namespaces = catalog.list_namespaces()
if namespace in namespaces:
print(" Namespace appears in list")
else:
print(f" Namespace not found in list: {namespaces}")
return False
# Get namespace properties
try:
props = catalog.load_namespace_properties(namespace)
print(f" Loaded namespace properties: {props}")
except NoSuchNamespaceError:
print(f" Failed to load namespace properties")
return False
return True
def test_table_operations(catalog, prefix):
"""Test table CRUD operations."""
print("Testing table operations...")
namespace = (f"{prefix.replace('-', '_')}_test_ns",)
table_name = "test_table"
table_id = namespace + (table_name,)
# Define a simple schema
schema = Schema(
NestedField(field_id=1, name="id", field_type=LongType(), required=True),
NestedField(field_id=2, name="name", field_type=StringType(), required=False),
NestedField(field_id=3, name="age", field_type=IntegerType(), required=False),
)
# Create table
try:
table = catalog.create_table(
identifier=table_id,
schema=schema,
)
print(f" Created table: {table_id}")
except TableAlreadyExistsError:
print(f" ! Table already exists: {table_id}")
_ = catalog.load_table(table_id)
# List tables
tables = catalog.list_tables(namespace)
if table_name in [t[1] for t in tables]:
print(" Table appears in list")
else:
print(f" Table not found in list: {tables}")
return False
# Load table
try:
loaded_table = catalog.load_table(table_id)
print(f" Loaded table: {loaded_table.name()}")
print(f" Schema: {loaded_table.schema()}")
print(f" Location: {loaded_table.location()}")
except NoSuchTableError:
print(f" Failed to load table")
return False
return True
def test_table_update(catalog, prefix):
"""Test table update/commit operations."""
print("Testing table update operations...")
namespace = (f"{prefix.replace('-', '_')}_test_ns",)
table_name = "test_table"
table_id = namespace + (table_name,)
try:
table = catalog.load_table(table_id)
# Update table properties
with table.transaction() as transaction:
transaction.set_properties({"test.property": "test.value"})
print(" Updated table properties")
# Reload and verify
table = catalog.load_table(table_id)
if table.properties.get("test.property") == "test.value":
print(" Property update verified")
else:
print(" ! Property update failed or not persisted")
return False
except Exception as e:
print(f" Table update failed: {e}")
return False
return True
def test_cleanup(catalog, prefix):
"""Test table and namespace deletion."""
print("Testing cleanup operations...")
namespace = (f"{prefix.replace('-', '_')}_test_ns",)
table_id = namespace + ("test_table",)
# Drop table
try:
catalog.drop_table(table_id)
print(f" Dropped table: {table_id}")
except NoSuchTableError:
print(f" ! Table already deleted: {table_id}")
# Drop namespace
try:
catalog.drop_namespace(namespace)
print(f" Dropped namespace: {namespace}")
except NoSuchNamespaceError:
print(f" ! Namespace already deleted: {namespace}")
except Exception as e:
print(f" ? Namespace drop error (may be expected): {e}")
return True
def main():
parser = argparse.ArgumentParser(description="Test Iceberg REST Catalog compatibility")
parser.add_argument("--catalog-url", required=True, help="Iceberg REST Catalog URL (e.g., http://localhost:8182)")
parser.add_argument("--warehouse", default="s3://iceberg-test/", help="Warehouse location")
parser.add_argument("--prefix", required=True, help="Table bucket prefix")
parser.add_argument("--skip-cleanup", action="store_true", help="Skip cleanup at the end")
args = parser.parse_args()
print(f"Connecting to Iceberg REST Catalog at: {args.catalog_url}")
print(f"Warehouse: {args.warehouse}")
print(f"Prefix: {args.prefix}")
print()
# Load the REST catalog with retries to handle possible delay in catalog server readiness
import time
max_retries = 10
catalog = None
for attempt in range(max_retries):
try:
catalog = load_catalog(
"rest",
**{
"type": "rest",
"uri": args.catalog_url,
"warehouse": args.warehouse,
"prefix": args.prefix,
"auth": {"type": "noop"},
"s3.anonymous": "true", # Disable AWS request signing for unauthenticated access
}
)
print(f"Successfully connected to catalog on attempt {attempt + 1}")
break
except Exception as e:
if attempt < max_retries - 1:
print(f" Attempt {attempt + 1} failed, retrying in 2s... ({e})")
time.sleep(2)
else:
print(f" All {max_retries} attempts failed.")
raise e
# Run tests
tests = [
("Config Endpoint", lambda: test_config_endpoint(catalog)),
("Namespace Operations", lambda: test_namespace_operations(catalog, args.prefix)),
("Table Operations", lambda: test_table_operations(catalog, args.prefix)),
("Table Update", lambda: test_table_update(catalog, args.prefix)),
]
if not args.skip_cleanup:
tests.append(("Cleanup", lambda: test_cleanup(catalog, args.prefix)))
passed = 0
failed = 0
for name, test_fn in tests:
print(f"\n{'='*50}")
try:
if test_fn():
passed += 1
print(f"PASSED: {name}")
else:
failed += 1
print(f"FAILED: {name}")
except Exception as e:
failed += 1
print(f"ERROR in {name}: {e}")
print(f"\n{'='*50}")
print(f"Results: {passed} passed, {failed} failed")
return 0 if failed == 0 else 1
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,224 @@
#!/usr/bin/env python3
"""
Iceberg REST Catalog Compatibility Test for SeaweedFS (Authenticated)
This script tests the Iceberg REST Catalog API compatibility with authentication.
Usage:
python3 test_rest_catalog_auth.py --catalog-url http://localhost:8182 \\
--access-key admin --secret-key admin
Requirements:
pip install pyiceberg[s3fs]
"""
import argparse
import sys
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import (
IntegerType,
LongType,
StringType,
NestedField,
)
from pyiceberg.exceptions import (
NamespaceAlreadyExistsError,
NoSuchNamespaceError,
TableAlreadyExistsError,
NoSuchTableError,
)
def test_config_endpoint(catalog):
"""Test that the catalog config endpoint returns valid configuration."""
print("Testing /v1/config endpoint...")
# The catalog is already loaded which means config endpoint worked
print(" /v1/config endpoint working")
return True
def test_namespace_operations(catalog, prefix):
"""Test namespace CRUD operations."""
print("Testing namespace operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
# List initial namespaces
namespaces = catalog.list_namespaces()
print(f" Initial namespaces: {namespaces}")
# Create namespace
try:
catalog.create_namespace(namespace)
print(f" Created namespace: {namespace}")
except NamespaceAlreadyExistsError:
print(f" ! Namespace already exists: {namespace}")
# List namespaces (should include our new one)
namespaces = catalog.list_namespaces()
if namespace in namespaces:
print(" Namespace appears in list")
else:
print(f" Namespace not found in list: {namespaces}")
return False
# Get namespace properties
try:
props = catalog.load_namespace_properties(namespace)
print(f" Loaded namespace properties: {props}")
except NoSuchNamespaceError:
print(f" Failed to load namespace properties")
return False
return True
def test_table_operations(catalog, prefix):
"""Test table CRUD operations."""
print("Testing table operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
table_name = "auth_test_table"
table_id = namespace + (table_name,)
# Define a simple schema
schema = Schema(
NestedField(field_id=1, name="id", field_type=LongType(), required=True),
NestedField(field_id=2, name="name", field_type=StringType(), required=False),
NestedField(field_id=3, name="age", field_type=IntegerType(), required=False),
)
# Create table
try:
table = catalog.create_table(
identifier=table_id,
schema=schema,
)
print(f" Created table: {table_id}")
except TableAlreadyExistsError:
print(f" ! Table already exists: {table_id}")
_ = catalog.load_table(table_id)
# List tables
tables = catalog.list_tables(namespace)
if table_name in [t[1] for t in tables]:
print(" Table appears in list")
else:
print(f" Table not found in list: {tables}")
return False
# Load table
try:
loaded_table = catalog.load_table(table_id)
print(f" Loaded table: {loaded_table.name()}")
print(f" Schema: {loaded_table.schema()}")
print(f" Location: {loaded_table.location()}")
except NoSuchTableError:
print(f" Failed to load table")
return False
return True
def test_cleanup(catalog, prefix):
"""Test table and namespace deletion."""
print("Testing cleanup operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
table_id = namespace + ("auth_test_table",)
# Drop table
try:
catalog.drop_table(table_id)
print(f" Dropped table: {table_id}")
except NoSuchTableError:
print(f" ! Table already deleted: {table_id}")
# Drop namespace
try:
catalog.drop_namespace(namespace)
print(f" Dropped namespace: {namespace}")
except NoSuchNamespaceError:
print(f" ! Namespace already deleted: {namespace}")
except Exception as e:
print(f" ? Namespace drop error (may be expected): {e}")
return True
def main():
parser = argparse.ArgumentParser(description="Test Iceberg REST Catalog with authentication")
parser.add_argument("--catalog-url", required=True, help="Iceberg REST Catalog URL")
parser.add_argument("--warehouse", default="s3://iceberg-test/", help="Warehouse location")
parser.add_argument("--prefix", required=True, help="Table bucket prefix")
parser.add_argument("--access-key", required=True, help="AWS Access Key ID")
parser.add_argument("--secret-key", required=True, help="AWS Secret Access Key")
parser.add_argument("--skip-cleanup", action="store_true", help="Skip cleanup at the end")
args = parser.parse_args()
print(f"Connecting to Iceberg REST Catalog at: {args.catalog_url}")
print(f"Warehouse: {args.warehouse}")
print(f"Prefix: {args.prefix}")
print(f"Using authenticated access with key: {args.access_key}")
print()
# Load the REST catalog with authentication
import time
max_retries = 10
catalog = None
for attempt in range(max_retries):
try:
catalog = load_catalog(
"rest",
**{
"type": "rest",
"uri": args.catalog_url,
"warehouse": args.warehouse,
"prefix": args.prefix,
"auth": {"type": "noop"},
"s3.access-key-id": args.access_key,
"s3.secret-access-key": args.secret_key,
}
)
print(f"Successfully connected to catalog on attempt {attempt + 1}")
break
except Exception as e:
if attempt < max_retries - 1:
print(f" Attempt {attempt + 1} failed, retrying in 2s... ({e})")
time.sleep(2)
else:
print(f" All {max_retries} attempts failed.")
raise e
# Run tests
tests = [
("Config Endpoint", lambda: test_config_endpoint(catalog)),
("Namespace Operations", lambda: test_namespace_operations(catalog, args.prefix)),
("Table Operations", lambda: test_table_operations(catalog, args.prefix)),
]
if not args.skip_cleanup:
tests.append(("Cleanup", lambda: test_cleanup(catalog, args.prefix)))
passed = 0
failed = 0
for name, test_fn in tests:
print(f"\n{'='*50}")
try:
if test_fn():
passed += 1
print(f"PASSED: {name}")
else:
failed += 1
print(f"FAILED: {name}")
except Exception as e:
failed += 1
print(f"ERROR in {name}: {e}")
print(f"\n{'='*50}")
print(f"Results: {passed} passed, {failed} failed")
return 0 if failed == 0 else 1
if __name__ == "__main__":
sys.exit(main())