snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)
Some checks failed
go: build dev binaries / cleanup (push) Successful in 4s
Codespell / Check for spelling errors (push) Failing after 3s
docker: build dev containers / build-rust-binaries (amd64, x86_64-unknown-linux-musl) (push) Failing after 3s
docker: build dev containers / build-dev-containers (push) Has been skipped
docker: build dev containers / build-rust-binaries (arm64, true, aarch64-unknown-linux-musl) (push) Failing after 3s
End to End / FUSE Mount (push) Failing after 2s
EC Integration Tests / EC Integration Tests (push) Failing after 30s
FUSE DLM Integration Tests / FUSE DLM Integration Tests (push) Failing after 3s
FUSE Volume Server Failover Tests / FUSE Volume Server Failover (push) Failing after 3s
FUSE Integration Tests / FUSE Integration Testing (push) Failing after 2s
FUSE P2P Peer Chunk Sharing Integration Tests / FUSE P2P Peer Chunk Sharing (push) Failing after 3s
go: build binary / Go Vet (push) Failing after 2s
go: build binary / Go Vet 32-bit (push) Failing after 3s
go: build binary / Build (push) Failing after 4s
go: build binary / Build cross-platform (freebsd/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (darwin/arm64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/arm64) (push) Failing after 3s
go: build binary / Test (push) Failing after 2s
go: build binary / Test 32-bit (push) Failing after 3s
helm: lint and test charts / lint-test (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (11) (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (17) (push) Failing after 3s
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Failing after 2s
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Failing after 3s
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been skipped
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been skipped
Java Client Unit Tests / Java Unit Tests (11) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (17) (push) Failing after 2s
Java Client Unit Tests / Java Unit Tests (21) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (8) (push) Failing after 3s
Kafka Quick Test (Load Test with Schema Registry) / Kafka Client Load Test (Quick) (push) Failing after 3s
Kafka Gateway Tests / Kafka Unit Tests (unit-tests-1) (push) Failing after 11s
Kafka Gateway Tests / Kafka Integration Tests (Critical) (integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka End-to-End Tests (with SMQ) (e2e-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Consumer Group Tests (Highly Isolated) (consumer-group-1) (push) Failing after 2s
Kafka Gateway Tests / Kafka Client Compatibility (with SMQ) (client-compat-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka SMQ Integration Tests (Full Stack) (smq-integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Protocol Tests (Isolated) (protocol-1) (push) Failing after 3s
KMS Tests / KMS Provider Integration Tests (push) Failing after 2s
KMS Tests / S3 KMS End-to-End Tests (push) Failing after 3s
Master Cold Start Tests / Master Cold Start Tests (push) Failing after 4s
Metadata Subscribe Integration Tests / Metadata Subscribe Integration Tests (push) Failing after 3s
Multi-Master Tests / Multi-Master Failover Tests (push) Failing after 3s
Performance / CPU and Heap Profile (push) Failing after 2s
Performance / S3 Read/Write Benchmark (rust volume) (push) Failing after 3s
pjdfstest POSIX Compliance / pjdfstest (push) Failing after 2s
Plugin Worker Integration Tests / Plugin Worker: erasure_coding (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: vacuum (push) Failing after 3s
Rust Volume Server Tests / Rust Integration Tests (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 1) (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 1) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 3) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Distributed Tests (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Performance Tests (push) Failing after 3s
Performance / Throughput Benchmark (go volume) (push) Failing after 3s
Performance / Throughput Benchmark (rust volume) (push) Failing after 2s
Performance / S3 Read/Write Benchmark (go volume) (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: volume_balance (push) Failing after 3s
PostgreSQL Gateway Tests / PostgreSQL Basic Tests (push) Failing after 3s
Rust Volume Server Tests / Rust Unit Tests (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 3) (push) Failing after 3s
Rust Plugin Worker Tests / Rust Plugin Worker Build and Unit Tests (push) Failing after 3s
rust: build dev volume server binaries / cleanup (push) Successful in 3s
S3 ETag and ACL Tests / S3 ETag + ACL Integration Tests (push) Failing after 3s
S3 IAM Integration Tests / IAM Unit Tests (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (advanced) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (basic) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (group) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (policy-enforcement) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (sts) (push) Failing after 4s
S3 Keycloak Integration Tests / S3 Keycloak Integration Tests (push) Failing after 3s
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.11) (push) Failing after 3s
S3 PyArrow Parquet Tests / Go Unit Tests (Implicit Directory) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.12) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-amd64, x86_64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-arm64, aarch64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-linux (linux-amd64, x86_64-unknown-linux-gnu) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.9) (push) Has been cancelled
S3 Proxy Signature Tests / S3 Proxy Signature Verification Tests (push) Has been cancelled
S3 SDK V2 Route Disambiguation Tests / S3 SDK V2 Routing Tests (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (quick) (push) Has been cancelled
S3 SSE Tests / S3 SSE Compatibility Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Metadata Persistence Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Copy Operations Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Multipart Upload Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Performance Test (push) Has been cancelled
S3 SSE Tests / S3 Volume Encryption Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (comprehensive) (push) Has been cancelled
Spark Integration Tests / Spark Integration Tests (push) Failing after 2s
Telemetry Integration Tests / telemetry-integration-test (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 1) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 1) (push) Failing after 3s
Ceph S3 tests / Basic S3 tests (KV store) (push) Has been cancelled
Ceph S3 tests / S3 Versioning & Object Lock tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Unit Tests (push) Failing after 3s
Ceph S3 tests / S3 CORS tests (push) Has been cancelled
Ceph S3 tests / SeaweedFS Custom S3 Copy tests (push) Has been cancelled
Ceph S3 tests / Basic S3 tests (SQL store) (push) Has been cancelled
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
TLS Rotation Integration Tests / TLS Rotation Integration Tests (push) Has been cancelled
Vacuum Integration Tests / Vacuum Integration Tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Variables Integration Tests (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (advanced-policy) (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (basic-policy) (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (contextual-policy) (push) Failing after 4s
S3 Policy Integration Tests / Trusted Proxy Detection Tests (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Shell Integration Tests (push) Failing after 3s
Samba on FUSE Integration / samba-integration (push) Failing after 2s
SFTP Integration Tests / SFTP Integration Testing (push) Failing after 3s
terraform: validate and test modules / fmt, validate, plan-level tests (push) Failing after 3s
terraform: validate and test modules / local cluster smoke test (real weed) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 3) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 2) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 2) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 3) (push) Failing after 5s
mount: windows conformance / WinFsp conformance (push) Has been cancelled
mount: windows / Mount on Windows (push) Has been cancelled
Some checks failed
go: build dev binaries / cleanup (push) Successful in 4s
Codespell / Check for spelling errors (push) Failing after 3s
docker: build dev containers / build-rust-binaries (amd64, x86_64-unknown-linux-musl) (push) Failing after 3s
docker: build dev containers / build-dev-containers (push) Has been skipped
docker: build dev containers / build-rust-binaries (arm64, true, aarch64-unknown-linux-musl) (push) Failing after 3s
End to End / FUSE Mount (push) Failing after 2s
EC Integration Tests / EC Integration Tests (push) Failing after 30s
FUSE DLM Integration Tests / FUSE DLM Integration Tests (push) Failing after 3s
FUSE Volume Server Failover Tests / FUSE Volume Server Failover (push) Failing after 3s
FUSE Integration Tests / FUSE Integration Testing (push) Failing after 2s
FUSE P2P Peer Chunk Sharing Integration Tests / FUSE P2P Peer Chunk Sharing (push) Failing after 3s
go: build binary / Go Vet (push) Failing after 2s
go: build binary / Go Vet 32-bit (push) Failing after 3s
go: build binary / Build (push) Failing after 4s
go: build binary / Build cross-platform (freebsd/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (darwin/arm64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/arm64) (push) Failing after 3s
go: build binary / Test (push) Failing after 2s
go: build binary / Test 32-bit (push) Failing after 3s
helm: lint and test charts / lint-test (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (11) (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (17) (push) Failing after 3s
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Failing after 2s
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Failing after 3s
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been skipped
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been skipped
Java Client Unit Tests / Java Unit Tests (11) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (17) (push) Failing after 2s
Java Client Unit Tests / Java Unit Tests (21) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (8) (push) Failing after 3s
Kafka Quick Test (Load Test with Schema Registry) / Kafka Client Load Test (Quick) (push) Failing after 3s
Kafka Gateway Tests / Kafka Unit Tests (unit-tests-1) (push) Failing after 11s
Kafka Gateway Tests / Kafka Integration Tests (Critical) (integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka End-to-End Tests (with SMQ) (e2e-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Consumer Group Tests (Highly Isolated) (consumer-group-1) (push) Failing after 2s
Kafka Gateway Tests / Kafka Client Compatibility (with SMQ) (client-compat-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka SMQ Integration Tests (Full Stack) (smq-integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Protocol Tests (Isolated) (protocol-1) (push) Failing after 3s
KMS Tests / KMS Provider Integration Tests (push) Failing after 2s
KMS Tests / S3 KMS End-to-End Tests (push) Failing after 3s
Master Cold Start Tests / Master Cold Start Tests (push) Failing after 4s
Metadata Subscribe Integration Tests / Metadata Subscribe Integration Tests (push) Failing after 3s
Multi-Master Tests / Multi-Master Failover Tests (push) Failing after 3s
Performance / CPU and Heap Profile (push) Failing after 2s
Performance / S3 Read/Write Benchmark (rust volume) (push) Failing after 3s
pjdfstest POSIX Compliance / pjdfstest (push) Failing after 2s
Plugin Worker Integration Tests / Plugin Worker: erasure_coding (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: vacuum (push) Failing after 3s
Rust Volume Server Tests / Rust Integration Tests (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 1) (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 1) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 3) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Distributed Tests (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Performance Tests (push) Failing after 3s
Performance / Throughput Benchmark (go volume) (push) Failing after 3s
Performance / Throughput Benchmark (rust volume) (push) Failing after 2s
Performance / S3 Read/Write Benchmark (go volume) (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: volume_balance (push) Failing after 3s
PostgreSQL Gateway Tests / PostgreSQL Basic Tests (push) Failing after 3s
Rust Volume Server Tests / Rust Unit Tests (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 3) (push) Failing after 3s
Rust Plugin Worker Tests / Rust Plugin Worker Build and Unit Tests (push) Failing after 3s
rust: build dev volume server binaries / cleanup (push) Successful in 3s
S3 ETag and ACL Tests / S3 ETag + ACL Integration Tests (push) Failing after 3s
S3 IAM Integration Tests / IAM Unit Tests (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (advanced) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (basic) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (group) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (policy-enforcement) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (sts) (push) Failing after 4s
S3 Keycloak Integration Tests / S3 Keycloak Integration Tests (push) Failing after 3s
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.11) (push) Failing after 3s
S3 PyArrow Parquet Tests / Go Unit Tests (Implicit Directory) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.12) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-amd64, x86_64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-arm64, aarch64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-linux (linux-amd64, x86_64-unknown-linux-gnu) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.9) (push) Has been cancelled
S3 Proxy Signature Tests / S3 Proxy Signature Verification Tests (push) Has been cancelled
S3 SDK V2 Route Disambiguation Tests / S3 SDK V2 Routing Tests (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (quick) (push) Has been cancelled
S3 SSE Tests / S3 SSE Compatibility Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Metadata Persistence Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Copy Operations Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Multipart Upload Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Performance Test (push) Has been cancelled
S3 SSE Tests / S3 Volume Encryption Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (comprehensive) (push) Has been cancelled
Spark Integration Tests / Spark Integration Tests (push) Failing after 2s
Telemetry Integration Tests / telemetry-integration-test (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 1) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 1) (push) Failing after 3s
Ceph S3 tests / Basic S3 tests (KV store) (push) Has been cancelled
Ceph S3 tests / S3 Versioning & Object Lock tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Unit Tests (push) Failing after 3s
Ceph S3 tests / S3 CORS tests (push) Has been cancelled
Ceph S3 tests / SeaweedFS Custom S3 Copy tests (push) Has been cancelled
Ceph S3 tests / Basic S3 tests (SQL store) (push) Has been cancelled
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
TLS Rotation Integration Tests / TLS Rotation Integration Tests (push) Has been cancelled
Vacuum Integration Tests / Vacuum Integration Tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Variables Integration Tests (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (advanced-policy) (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (basic-policy) (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (contextual-policy) (push) Failing after 4s
S3 Policy Integration Tests / Trusted Proxy Detection Tests (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Shell Integration Tests (push) Failing after 3s
Samba on FUSE Integration / samba-integration (push) Failing after 2s
SFTP Integration Tests / SFTP Integration Testing (push) Failing after 3s
terraform: validate and test modules / fmt, validate, plan-level tests (push) Failing after 3s
terraform: validate and test modules / local cluster smoke test (real weed) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 3) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 2) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 2) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 3) (push) Failing after 5s
mount: windows conformance / WinFsp conformance (push) Has been cancelled
mount: windows / Mount on Windows (push) Has been cancelled
This commit is contained in:
13
test/s3tables/catalog/Dockerfile.lance
Normal file
13
test/s3tables/catalog/Dockerfile.lance
Normal file
@@ -0,0 +1,13 @@
|
||||
# Lance client container for Lance Namespace REST compatibility testing.
|
||||
# The point of this image is to exercise the namespace with the real client
|
||||
# rather than hand-built HTTP: every serious bug in this surface so far looked
|
||||
# fine to a request we wrote ourselves.
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN pip install --no-cache-dir lance-namespace pylance pyarrow
|
||||
|
||||
COPY test_lance_namespace.py /app/
|
||||
|
||||
CMD ["python3", "/app/test_lance_namespace.py", "--help"]
|
||||
13
test/s3tables/catalog/Dockerfile.pyiceberg
Normal file
13
test/s3tables/catalog/Dockerfile.pyiceberg
Normal file
@@ -0,0 +1,13 @@
|
||||
# PyIceberg test container for Iceberg REST Catalog compatibility testing
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install PyIceberg with S3 support and dependencies
|
||||
RUN pip install --no-cache-dir "pyiceberg[s3fs]" pyarrow pandas
|
||||
|
||||
# Copy the test script
|
||||
COPY test_rest_catalog.py /app/
|
||||
|
||||
# Default command
|
||||
CMD ["python3", "/app/test_rest_catalog.py", "--help"]
|
||||
37
test/s3tables/catalog/Makefile
Normal file
37
test/s3tables/catalog/Makefile
Normal file
@@ -0,0 +1,37 @@
|
||||
# Makefile for Iceberg REST Catalog Integration Tests
|
||||
#
|
||||
# This Makefile provides easy-to-use targets for running integration tests
|
||||
# and starting a local development environment.
|
||||
|
||||
.PHONY: build mini test test-pyiceberg clean
|
||||
|
||||
# Root directory of the SeaweedFS project
|
||||
PROJECT_ROOT = ../../..
|
||||
WEED_BIN = $(PROJECT_ROOT)/weed/weed
|
||||
|
||||
# Build the weed binary
|
||||
build:
|
||||
@echo "Building SeaweedFS binary..."
|
||||
cd $(PROJECT_ROOT)/weed && go build -v
|
||||
|
||||
# Start SeaweedFS mini cluster for manual testing
|
||||
# Uses default ports: S3=8333, Iceberg=8182
|
||||
mini: build
|
||||
@echo "Starting SeaweedFS mini cluster..."
|
||||
$(WEED_BIN) mini -s3.port.iceberg=8182 & echo $$! > weed-mini.pid; wait
|
||||
|
||||
# Run all integration tests in this directory
|
||||
test: build
|
||||
@echo "Running all Iceberg catalog integration tests..."
|
||||
go test -v .
|
||||
|
||||
# Run only the PyIceberg compatibility tests (requires Docker)
|
||||
test-pyiceberg: build
|
||||
@echo "Running PyIceberg compatibility tests..."
|
||||
go test -v -run TestPyIcebergRestCatalog .
|
||||
|
||||
# Clean up temporary data and stop weed mini if running
|
||||
clean:
|
||||
@echo "Cleaning up..."
|
||||
@test -f weed-mini.pid && kill $$(cat weed-mini.pid) && rm weed-mini.pid || true
|
||||
@rm -rf /tmp/seaweed-iceberg-test-*
|
||||
33
test/s3tables/catalog/docker-compose.test.yaml
Normal file
33
test/s3tables/catalog/docker-compose.test.yaml
Normal file
@@ -0,0 +1,33 @@
|
||||
# Iceberg REST Catalog Compatibility Test Docker Compose
|
||||
#
|
||||
# This compose file sets up a test environment for validating
|
||||
# the SeaweedFS Iceberg REST Catalog implementation.
|
||||
#
|
||||
# Usage:
|
||||
# docker compose -f docker-compose.test.yaml up --build
|
||||
#
|
||||
# Note: SeaweedFS must be running on the host with the Iceberg REST port exposed.
|
||||
# Set ICEBERG_CATALOG_URL environment variable to point to your catalog.
|
||||
|
||||
services:
|
||||
# PyIceberg-based REST catalog test
|
||||
pyiceberg-test:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.pyiceberg
|
||||
environment:
|
||||
- CATALOG_URL=${CATALOG_URL:-http://host.docker.internal:8182}
|
||||
- WAREHOUSE=${WAREHOUSE:-s3://test-bucket/}
|
||||
- PREFIX=${PREFIX:-pyiceberg-test}
|
||||
- AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-test}
|
||||
- AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-test}
|
||||
- AWS_REGION=${AWS_REGION:-us-east-1}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- ./test_rest_catalog.py:/app/test_rest_catalog.py:ro
|
||||
command: >
|
||||
python3 /app/test_rest_catalog.py
|
||||
--catalog-url ${CATALOG_URL:-http://host.docker.internal:8182}
|
||||
--warehouse ${WAREHOUSE:-s3://test-bucket/}
|
||||
--prefix ${PREFIX:-pyiceberg-test}
|
||||
211
test/s3tables/catalog/duckdb_deterministic_location_test.go
Normal file
211
test/s3tables/catalog/duckdb_deterministic_location_test.go
Normal file
@@ -0,0 +1,211 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/test/testutil"
|
||||
)
|
||||
|
||||
// TestDuckDBDeterministicLocationRead validates that DuckDB's iceberg
|
||||
// extension can resolve and scan a table that lives at the deterministic
|
||||
// <bucket>/<namespace>/<tableName> path produced when no client-side UUID
|
||||
// is appended to the table location. This is the layout produced by:
|
||||
//
|
||||
// - iceberg-spark (no concept of UUID-suffixed locations)
|
||||
// - pyiceberg (no concept of UUID-suffixed locations)
|
||||
// - Trino with iceberg.unique-table-location=false
|
||||
//
|
||||
// Post-#9246, REST clients can resolve table locations under a namespace
|
||||
// because GetNamespace advertises a default `location` property. This
|
||||
// test creates a namespace + table via the REST API directly (so no
|
||||
// client-side UUID is added), then asks DuckDB to ATTACH the catalog and
|
||||
// describe the table — verifying DuckDB walks the catalog → metadata →
|
||||
// schema chain successfully against a deterministic on-disk path.
|
||||
//
|
||||
// The test does not insert data files — DuckDB scanning an empty table
|
||||
// is sufficient to validate the catalog/metadata resolution path. If
|
||||
// DuckDB's iceberg extension is unavailable or the build is too old to
|
||||
// support the ATTACH syntax, the test skips rather than fails.
|
||||
func TestDuckDBDeterministicLocationRead(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
if !testutil.HasDocker() {
|
||||
t.Skip("Docker not available, skipping DuckDB integration test")
|
||||
}
|
||||
|
||||
env := newOAuthTestEnv(t)
|
||||
defer env.cleanup(t)
|
||||
env.start(t)
|
||||
|
||||
bucketName := "duckdb-detloc-" + randomSuffix()
|
||||
namespace := "detloc_ns_" + randomSuffix()
|
||||
tableName := "detloc_tbl_" + randomSuffix()
|
||||
|
||||
createTableBucketViaShell(t, env, bucketName)
|
||||
|
||||
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
|
||||
createNamespaceWithToken(t, env, token, bucketName, namespace)
|
||||
createIcebergTableWithToken(t, env, token, bucketName, namespace, tableName)
|
||||
|
||||
// Sanity-check the catalog row exposes a deterministic location.
|
||||
tableLoc := getTableLocation(t, env, token, bucketName, namespace, tableName)
|
||||
wantPrefix := fmt.Sprintf("s3://%s/%s/%s", bucketName, namespace, tableName)
|
||||
if !strings.HasPrefix(tableLoc, wantPrefix) {
|
||||
t.Fatalf("table location %q should start with %q (deterministic, no UUID suffix)", tableLoc, wantPrefix)
|
||||
}
|
||||
if strings.Contains(strings.TrimPrefix(tableLoc, wantPrefix), "-") {
|
||||
t.Fatalf("table location %q should not have a -<uuid> suffix after %q", tableLoc, wantPrefix)
|
||||
}
|
||||
|
||||
sqlContent := fmt.Sprintf(`
|
||||
INSTALL iceberg;
|
||||
LOAD iceberg;
|
||||
|
||||
CREATE SECRET iceberg_secret (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://host.docker.internal:%d',
|
||||
CLIENT_ID '%s',
|
||||
CLIENT_SECRET '%s'
|
||||
);
|
||||
|
||||
CREATE SECRET s3_secret (
|
||||
TYPE S3,
|
||||
KEY_ID '%s',
|
||||
SECRET '%s',
|
||||
ENDPOINT 'host.docker.internal:%d',
|
||||
URL_STYLE 'path',
|
||||
USE_SSL false
|
||||
);
|
||||
|
||||
ATTACH 's3://%s/' AS detloc_cat (TYPE 'ICEBERG', SECRET iceberg_secret);
|
||||
|
||||
DESCRIBE detloc_cat.%s.%s;
|
||||
|
||||
SELECT 'duckdb deterministic-location describe ok' AS marker;
|
||||
`,
|
||||
env.icebergPort, env.accessKey, env.secretKey,
|
||||
env.accessKey, env.secretKey, env.s3Port,
|
||||
bucketName,
|
||||
namespace, tableName,
|
||||
)
|
||||
|
||||
sqlFile := filepath.Join(env.dataDir, "duckdb_detloc.sql")
|
||||
if err := os.WriteFile(sqlFile, []byte(sqlContent), 0644); err != nil {
|
||||
t.Fatalf("write SQL file: %v", err)
|
||||
}
|
||||
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"-v", fmt.Sprintf("%s:/test", env.dataDir),
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"--entrypoint", "duckdb",
|
||||
"duckdb/duckdb:latest",
|
||||
"-init", "/test/duckdb_detloc.sql",
|
||||
"-c", "SELECT 1",
|
||||
)
|
||||
output, err := cmd.CombinedOutput()
|
||||
outputStr := string(output)
|
||||
t.Logf("DuckDB output:\n%s", outputStr)
|
||||
|
||||
if err != nil {
|
||||
// Tolerate environments where the Iceberg extension/version is too
|
||||
// limited to exercise this code path. The intent is to gain
|
||||
// coverage where it's available, not to require it everywhere.
|
||||
if strings.Contains(outputStr, "iceberg extension is not available") ||
|
||||
strings.Contains(outputStr, "Failed to load") ||
|
||||
strings.Contains(outputStr, "syntax error") ||
|
||||
strings.Contains(outputStr, "Unknown extension") {
|
||||
t.Skipf("DuckDB image lacks the required iceberg extension or syntax: %v", err)
|
||||
}
|
||||
t.Fatalf("DuckDB run failed: %v\nOutput:\n%s", err, outputStr)
|
||||
}
|
||||
|
||||
if !strings.Contains(outputStr, "duckdb deterministic-location describe ok") {
|
||||
t.Fatalf("DuckDB DESCRIBE on deterministic-location table did not complete successfully:\n%s", outputStr)
|
||||
}
|
||||
}
|
||||
|
||||
// createIcebergTableWithToken creates a minimal Iceberg table via the REST
|
||||
// catalog using the provided OAuth bearer token. The table is created
|
||||
// without an explicit location, so the server picks the deterministic
|
||||
// <bucket>/<namespace>/<tableName> path.
|
||||
func createIcebergTableWithToken(t *testing.T, env *oauthTestEnv, token, bucketName, namespace, tableName string) {
|
||||
t.Helper()
|
||||
|
||||
body := fmt.Sprintf(`{
|
||||
"name": %q,
|
||||
"schema": {
|
||||
"type": "struct",
|
||||
"schema-id": 0,
|
||||
"fields": [
|
||||
{"id": 1, "name": "id", "required": true, "type": "long"},
|
||||
{"id": 2, "name": "label", "required": false, "type": "string"}
|
||||
]
|
||||
}
|
||||
}`, tableName)
|
||||
|
||||
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables", env.icebergURL(), bucketName, namespace)
|
||||
req, err := http.NewRequest(http.MethodPost, url, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatalf("create table request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("create table: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
respBody, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("create table failed: status=%d body=%s", resp.StatusCode, respBody)
|
||||
}
|
||||
t.Logf("Created table %s.%s in bucket %s", namespace, tableName, bucketName)
|
||||
}
|
||||
|
||||
// getTableLocation fetches the table's metadata via the REST catalog and
|
||||
// returns the location field reported in the response.
|
||||
func getTableLocation(t *testing.T, env *oauthTestEnv, token, bucketName, namespace, tableName string) string {
|
||||
t.Helper()
|
||||
|
||||
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables/%s", env.icebergURL(), bucketName, namespace, tableName)
|
||||
req, err := http.NewRequest(http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("load table request: %v", err)
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("load table: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
respBody, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("load table failed: status=%d body=%s", resp.StatusCode, respBody)
|
||||
}
|
||||
// The metadata.location field is nested. A naive substring match is
|
||||
// sufficient for asserting the expected path shape.
|
||||
body := string(respBody)
|
||||
const key = `"location":"`
|
||||
idx := strings.Index(body, key)
|
||||
if idx < 0 {
|
||||
t.Fatalf("LoadTable response missing location field: %s", body)
|
||||
}
|
||||
rest := body[idx+len(key):]
|
||||
end := strings.Index(rest, `"`)
|
||||
if end < 0 {
|
||||
t.Fatalf("malformed location field: %s", body)
|
||||
}
|
||||
return rest[:end]
|
||||
}
|
||||
477
test/s3tables/catalog/duckdb_oauth_test.go
Normal file
477
test/s3tables/catalog/duckdb_oauth_test.go
Normal file
@@ -0,0 +1,477 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/test/testutil"
|
||||
)
|
||||
|
||||
// oauthTestEnv holds a weed mini instance started with IAM credentials
|
||||
// so that the OAuth token endpoint is functional.
|
||||
type oauthTestEnv struct {
|
||||
seaweedDir string
|
||||
weedBinary string
|
||||
dataDir string
|
||||
bindIP string
|
||||
s3Port int
|
||||
s3GrpcPort int
|
||||
icebergPort int
|
||||
masterPort int
|
||||
masterGrpcPort int
|
||||
filerPort int
|
||||
filerGrpcPort int
|
||||
volumePort int
|
||||
volumeGrpcPort int
|
||||
webdavPort int
|
||||
weedProcess *exec.Cmd
|
||||
weedCancel context.CancelFunc
|
||||
accessKey string
|
||||
secretKey string
|
||||
// s3ExternalURL, when set before start, is the S3 endpoint the catalog
|
||||
// advertises to clients in LoadTable FileIO config.
|
||||
s3ExternalURL string
|
||||
}
|
||||
|
||||
func newOAuthTestEnv(t *testing.T) *oauthTestEnv {
|
||||
t.Helper()
|
||||
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Fatalf("get working directory: %v", err)
|
||||
}
|
||||
|
||||
seaweedDir := wd
|
||||
for i := 0; i < 6; i++ {
|
||||
if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
|
||||
break
|
||||
}
|
||||
seaweedDir = filepath.Dir(seaweedDir)
|
||||
}
|
||||
|
||||
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
|
||||
if info, err := os.Stat(weedBinary); err == nil && !info.IsDir() {
|
||||
// Say which binary and how old it is. `make test` rebuilds first, but
|
||||
// `go test` on its own happily runs a weeks-old binary and reports a
|
||||
// pass for code that is not being exercised.
|
||||
t.Logf("using %s, built %s", weedBinary, info.ModTime().Format(time.RFC3339))
|
||||
} else {
|
||||
weedBinary = "weed"
|
||||
if _, err := exec.LookPath(weedBinary); err != nil {
|
||||
t.Skip("weed binary not found, skipping integration test")
|
||||
}
|
||||
}
|
||||
|
||||
dataDir, err := os.MkdirTemp("", "seaweed-oauth-test-*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
bindIP := testutil.FindBindIP()
|
||||
ports := testutil.MustAllocatePorts(t, 10)
|
||||
|
||||
return &oauthTestEnv{
|
||||
seaweedDir: seaweedDir,
|
||||
weedBinary: weedBinary,
|
||||
dataDir: dataDir,
|
||||
bindIP: bindIP,
|
||||
masterPort: ports[0],
|
||||
masterGrpcPort: ports[1],
|
||||
volumePort: ports[2],
|
||||
volumeGrpcPort: ports[3],
|
||||
filerPort: ports[4],
|
||||
filerGrpcPort: ports[5],
|
||||
s3Port: ports[6],
|
||||
s3GrpcPort: ports[7],
|
||||
icebergPort: ports[8],
|
||||
webdavPort: ports[9],
|
||||
accessKey: "AKIAIOSFODNN7EXAMPLE",
|
||||
secretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
}
|
||||
}
|
||||
|
||||
func (env *oauthTestEnv) start(t *testing.T) {
|
||||
t.Helper()
|
||||
|
||||
iamConfigPath, err := testutil.WriteIAMConfig(env.dataDir, env.accessKey, env.secretKey)
|
||||
if err != nil {
|
||||
t.Fatalf("write IAM config: %v", err)
|
||||
}
|
||||
|
||||
securityToml := filepath.Join(env.dataDir, "security.toml")
|
||||
if err := os.WriteFile(securityToml, []byte("# Empty security config for testing\n"), 0644); err != nil {
|
||||
t.Fatalf("write security.toml: %v", err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
env.weedCancel = cancel
|
||||
|
||||
args := []string{"mini",
|
||||
"-master.port", fmt.Sprintf("%d", env.masterPort),
|
||||
"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
|
||||
"-volume.port", fmt.Sprintf("%d", env.volumePort),
|
||||
"-volume.port.grpc", fmt.Sprintf("%d", env.volumeGrpcPort),
|
||||
"-filer.port", fmt.Sprintf("%d", env.filerPort),
|
||||
"-filer.port.grpc", fmt.Sprintf("%d", env.filerGrpcPort),
|
||||
"-s3.port", fmt.Sprintf("%d", env.s3Port),
|
||||
"-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort),
|
||||
"-s3.port.iceberg", fmt.Sprintf("%d", env.icebergPort),
|
||||
"-webdav.port", fmt.Sprintf("%d", env.webdavPort),
|
||||
"-s3.config", iamConfigPath,
|
||||
"-ip", env.bindIP,
|
||||
"-ip.bind", "0.0.0.0",
|
||||
"-dir", env.dataDir,
|
||||
}
|
||||
if env.s3ExternalURL != "" {
|
||||
args = append(args, "-s3.externalUrl", env.s3ExternalURL)
|
||||
}
|
||||
|
||||
cmd := exec.CommandContext(ctx, env.weedBinary, args...)
|
||||
cmd.Dir = env.dataDir
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
cmd.Env = append(os.Environ(),
|
||||
"AWS_ACCESS_KEY_ID="+env.accessKey,
|
||||
"AWS_SECRET_ACCESS_KEY="+env.secretKey,
|
||||
)
|
||||
|
||||
if err := cmd.Start(); err != nil {
|
||||
cancel()
|
||||
t.Fatalf("start weed mini: %v", err)
|
||||
}
|
||||
env.weedProcess = cmd
|
||||
|
||||
icebergURL := fmt.Sprintf("http://%s:%d/v1/config", env.bindIP, env.icebergPort)
|
||||
if !testutil.WaitForService(icebergURL, 30*time.Second) {
|
||||
cancel()
|
||||
cmd.Wait()
|
||||
t.Fatalf("Iceberg REST API did not become ready at %s", icebergURL)
|
||||
}
|
||||
}
|
||||
|
||||
func (env *oauthTestEnv) cleanup(t *testing.T) {
|
||||
t.Helper()
|
||||
if env.weedCancel != nil {
|
||||
env.weedCancel()
|
||||
}
|
||||
if env.weedProcess != nil {
|
||||
env.weedProcess.Wait()
|
||||
}
|
||||
if env.dataDir != "" {
|
||||
os.RemoveAll(env.dataDir)
|
||||
}
|
||||
}
|
||||
|
||||
func (env *oauthTestEnv) icebergURL() string {
|
||||
return fmt.Sprintf("http://%s:%d", env.bindIP, env.icebergPort)
|
||||
}
|
||||
|
||||
// TestOAuthTokenEndpoint tests the /v1/oauth/tokens endpoint directly via HTTP.
|
||||
func TestOAuthTokenEndpoint(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := newOAuthTestEnv(t)
|
||||
defer env.cleanup(t)
|
||||
env.start(t)
|
||||
|
||||
t.Run("valid credentials", func(t *testing.T) {
|
||||
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
|
||||
if token == "" {
|
||||
t.Fatal("expected non-empty token")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("invalid secret", func(t *testing.T) {
|
||||
resp, err := http.PostForm(env.icebergURL()+"/v1/oauth/tokens", url.Values{
|
||||
"grant_type": {"client_credentials"},
|
||||
"client_id": {env.accessKey},
|
||||
"client_secret": {"wrong-secret"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("POST /v1/oauth/tokens: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusUnauthorized {
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
t.Fatalf("expected 401, got %d: %s", resp.StatusCode, body)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("missing grant_type", func(t *testing.T) {
|
||||
resp, err := http.PostForm(env.icebergURL()+"/v1/oauth/tokens", url.Values{
|
||||
"client_id": {env.accessKey},
|
||||
"client_secret": {env.secretKey},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("POST /v1/oauth/tokens: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusBadRequest {
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
t.Fatalf("expected 400, got %d: %s", resp.StatusCode, body)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("bearer token auth on catalog endpoint", func(t *testing.T) {
|
||||
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
|
||||
|
||||
// Point the call at a table bucket that exists, so anything but 200 is an
|
||||
// auth or routing fault rather than the catalog reporting a missing bucket.
|
||||
bucketName := "oauth-bearer-" + randomSuffix()
|
||||
createTableBucketViaShell(t, env, bucketName)
|
||||
|
||||
req, err := http.NewRequest(http.MethodGet, env.icebergURL()+"/v1/"+bucketName+"/namespaces", nil)
|
||||
if err != nil {
|
||||
t.Fatalf("create request: %v", err)
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("GET /v1/%s/namespaces with Bearer: %v", bucketName, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("Bearer auth failed unexpectedly: status=%d body=%s", resp.StatusCode, body)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestDuckDBOAuthIntegration tests that DuckDB can connect to the Iceberg REST
|
||||
// catalog using the OAuth2 client_credentials flow (CREATE SECRET with client_id
|
||||
// and client_secret). This is the scenario reported in issue #9015.
|
||||
func TestDuckDBOAuthIntegration(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
if !testutil.HasDocker() {
|
||||
t.Skip("Docker not available, skipping DuckDB OAuth integration test")
|
||||
}
|
||||
|
||||
env := newOAuthTestEnv(t)
|
||||
defer env.cleanup(t)
|
||||
env.start(t)
|
||||
|
||||
// Create a table bucket and namespace so DuckDB has something to query
|
||||
bucketName := "duckdb-oauth-" + randomSuffix()
|
||||
createTableBucketViaShell(t, env, bucketName)
|
||||
|
||||
// Create a namespace via the Iceberg REST API using OAuth token
|
||||
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
|
||||
createNamespaceWithToken(t, env, token, bucketName, "testns")
|
||||
|
||||
sqlContent := fmt.Sprintf(`
|
||||
INSTALL iceberg;
|
||||
LOAD iceberg;
|
||||
|
||||
CREATE SECRET iceberg_secret (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://host.docker.internal:%d',
|
||||
CLIENT_ID '%s',
|
||||
CLIENT_SECRET '%s',
|
||||
SCOPE 's3://%s/'
|
||||
);
|
||||
|
||||
CREATE SECRET s3_secret (
|
||||
TYPE S3,
|
||||
KEY_ID '%s',
|
||||
SECRET '%s',
|
||||
ENDPOINT 'host.docker.internal:%d',
|
||||
URL_STYLE 'path',
|
||||
USE_SSL false,
|
||||
SCOPE 's3://%s/'
|
||||
);
|
||||
|
||||
SELECT 'OAuth token obtained successfully' as status;
|
||||
|
||||
-- Try listing namespaces via the Iceberg catalog
|
||||
SELECT * FROM iceberg_scan('iceberg_secret', ALLOW_MOVED_PATHS => TRUE) LIMIT 0;
|
||||
`, env.icebergPort, env.accessKey, env.secretKey, bucketName,
|
||||
env.accessKey, env.secretKey, env.s3Port, bucketName)
|
||||
|
||||
sqlFile := filepath.Join(env.dataDir, "duckdb_oauth_test.sql")
|
||||
if err := os.WriteFile(sqlFile, []byte(sqlContent), 0644); err != nil {
|
||||
t.Fatalf("write SQL file: %v", err)
|
||||
}
|
||||
|
||||
// Run DuckDB in Docker.
|
||||
// We use a simple test: get a token, create a secret, and verify the
|
||||
// Iceberg extension can communicate with the catalog.
|
||||
// The simpler fallback test just verifies CREATE SECRET succeeds (no 404).
|
||||
fallbackSQL := fmt.Sprintf(`
|
||||
INSTALL iceberg;
|
||||
LOAD iceberg;
|
||||
|
||||
CREATE SECRET (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://host.docker.internal:%d',
|
||||
CLIENT_ID '%s',
|
||||
CLIENT_SECRET '%s'
|
||||
);
|
||||
|
||||
SELECT 'DuckDB Iceberg OAuth secret created successfully' as result;
|
||||
`, env.icebergPort, env.accessKey, env.secretKey)
|
||||
|
||||
fallbackFile := filepath.Join(env.dataDir, "duckdb_oauth_fallback.sql")
|
||||
if err := os.WriteFile(fallbackFile, []byte(fallbackSQL), 0644); err != nil {
|
||||
t.Fatalf("write fallback SQL file: %v", err)
|
||||
}
|
||||
|
||||
// Try the full SQL script first (CREATE SECRET + catalog access).
|
||||
// Fall back to the simple CREATE SECRET test if iceberg_scan isn't supported.
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"-v", fmt.Sprintf("%s:/test", env.dataDir),
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"--entrypoint", "duckdb",
|
||||
"duckdb/duckdb:latest",
|
||||
"-init", "/test/duckdb_oauth_test.sql",
|
||||
"-c", "SELECT 1",
|
||||
)
|
||||
|
||||
output, err := cmd.CombinedOutput()
|
||||
outputStr := string(output)
|
||||
t.Logf("DuckDB output:\n%s", outputStr)
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(outputStr, "iceberg extension is not available") ||
|
||||
strings.Contains(outputStr, "Failed to load") {
|
||||
t.Skip("Skipping: Iceberg extension not available in DuckDB Docker image")
|
||||
}
|
||||
// The key check: the old error was "HTTP NotFound_404" on /v1/oauth/tokens.
|
||||
// With our fix, this should no longer happen.
|
||||
if strings.Contains(outputStr, "NotFound_404") && strings.Contains(outputStr, "/v1/oauth/tokens") {
|
||||
t.Fatal("OAuth token endpoint returned 404 - the fix is not working")
|
||||
}
|
||||
// If iceberg_scan failed but CREATE SECRET worked, fall back to simpler test
|
||||
if strings.Contains(outputStr, "OAuth token obtained successfully") {
|
||||
t.Logf("Full SQL had partial success (token obtained), iceberg_scan may not be supported. Continuing.")
|
||||
} else {
|
||||
// Try the fallback script that only tests CREATE SECRET
|
||||
t.Logf("Full SQL failed, trying fallback CREATE SECRET test...")
|
||||
fallbackCmd := exec.Command("docker", "run", "--rm",
|
||||
"-v", fmt.Sprintf("%s:/test", env.dataDir),
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"--entrypoint", "duckdb",
|
||||
"duckdb/duckdb:latest",
|
||||
"-init", "/test/duckdb_oauth_fallback.sql",
|
||||
"-c", "SELECT 1",
|
||||
)
|
||||
fallbackOutput, fallbackErr := fallbackCmd.CombinedOutput()
|
||||
fallbackStr := string(fallbackOutput)
|
||||
t.Logf("DuckDB fallback output:\n%s", fallbackStr)
|
||||
|
||||
if fallbackErr != nil {
|
||||
if strings.Contains(fallbackStr, "NotFound_404") && strings.Contains(fallbackStr, "/v1/oauth/tokens") {
|
||||
t.Fatal("OAuth token endpoint returned 404 - the fix is not working")
|
||||
}
|
||||
t.Fatalf("DuckDB fallback also failed: %v\nOutput: %s", fallbackErr, fallbackStr)
|
||||
}
|
||||
if !strings.Contains(fallbackStr, "DuckDB Iceberg OAuth secret created successfully") {
|
||||
t.Errorf("expected success message in fallback output, got:\n%s", fallbackStr)
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if strings.Contains(outputStr, "OAuth token obtained successfully") {
|
||||
t.Logf("DuckDB OAuth CREATE SECRET succeeded")
|
||||
} else {
|
||||
t.Errorf("expected OAuth success message in output, got:\n%s", outputStr)
|
||||
}
|
||||
}
|
||||
|
||||
// requestOAuthToken obtains a bearer token from the OAuth endpoint.
|
||||
func requestOAuthToken(t *testing.T, env *oauthTestEnv, accessKey, secretKey string) string {
|
||||
t.Helper()
|
||||
|
||||
resp, err := http.PostForm(env.icebergURL()+"/v1/oauth/tokens", url.Values{
|
||||
"grant_type": {"client_credentials"},
|
||||
"client_id": {accessKey},
|
||||
"client_secret": {secretKey},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("POST /v1/oauth/tokens: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("OAuth token request failed: status=%d body=%s", resp.StatusCode, body)
|
||||
}
|
||||
|
||||
var tokenResp struct {
|
||||
AccessToken string `json:"access_token"`
|
||||
TokenType string `json:"token_type"`
|
||||
ExpiresIn int `json:"expires_in"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &tokenResp); err != nil {
|
||||
t.Fatalf("decode token response: %v", err)
|
||||
}
|
||||
|
||||
if tokenResp.AccessToken == "" {
|
||||
t.Fatal("got empty access_token")
|
||||
}
|
||||
if tokenResp.TokenType != "bearer" {
|
||||
t.Errorf("expected token_type=bearer, got %s", tokenResp.TokenType)
|
||||
}
|
||||
return tokenResp.AccessToken
|
||||
}
|
||||
|
||||
// createTableBucketViaShell creates a table bucket using weed shell,
|
||||
// which bypasses S3 auth. This is the same approach used by the Trino tests.
|
||||
func createTableBucketViaShell(t *testing.T, env *oauthTestEnv, bucketName string) {
|
||||
t.Helper()
|
||||
|
||||
cmd := exec.Command(env.weedBinary, "shell",
|
||||
fmt.Sprintf("-master=%s:%d.%d", env.bindIP, env.masterPort, env.masterGrpcPort),
|
||||
)
|
||||
cmd.Stdin = strings.NewReader(fmt.Sprintf("s3tables.bucket -create -name %s -account 000000000000\nexit\n", bucketName))
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("create table bucket %s via weed shell: %v\nOutput: %s", bucketName, err, output)
|
||||
}
|
||||
t.Logf("Created table bucket %s", bucketName)
|
||||
}
|
||||
|
||||
// createNamespaceWithToken creates a namespace using a Bearer token.
|
||||
func createNamespaceWithToken(t *testing.T, env *oauthTestEnv, token, bucketName, namespace string) {
|
||||
t.Helper()
|
||||
|
||||
path := fmt.Sprintf("/v1/%s/namespaces", bucketName)
|
||||
body := fmt.Sprintf(`{"namespace":["%s"]}`, namespace)
|
||||
req, err := http.NewRequest(http.MethodPost, env.icebergURL()+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatalf("create request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("create namespace: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
respBody, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusConflict {
|
||||
t.Fatalf("create namespace failed: status=%d body=%s", resp.StatusCode, respBody)
|
||||
}
|
||||
t.Logf("Created namespace %s in bucket %s", namespace, bucketName)
|
||||
}
|
||||
215
test/s3tables/catalog/duckdb_vended_credentials_test.go
Normal file
215
test/s3tables/catalog/duckdb_vended_credentials_test.go
Normal file
@@ -0,0 +1,215 @@
|
||||
// Tests for the credential-vending access pattern.
|
||||
//
|
||||
// DuckDB attaches a catalog with "X-Iceberg-Access-Delegation: vended-credentials"
|
||||
// and rebuilds its S3 credential out of whatever LoadTable returns in `config`,
|
||||
// dropping the S3 secret it was configured with. A catalog that advertises an
|
||||
// endpoint there but vends no credentials therefore leaves DuckDB sending
|
||||
// unsigned requests, and every metadata and data file comes back 403.
|
||||
//
|
||||
// These run against a weed mini started with -s3.externalUrl, which is what
|
||||
// makes the catalog advertise an endpoint at all; with a wildcard bind and no
|
||||
// external URL the config is empty and the bug cannot appear.
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/test/testutil"
|
||||
)
|
||||
|
||||
// duckDBImage tracks whatever DuckDB users are actually running, matching the
|
||||
// other DuckDB tests here: a client that changes how it handles vended
|
||||
// credentials is exactly what this suite exists to catch.
|
||||
const duckDBImage = "duckdb/duckdb:latest"
|
||||
|
||||
// requireDuckDBIceberg skips when the image cannot install the iceberg
|
||||
// extension -- no egress, or a build without it. Probing separately keeps the
|
||||
// round trip below free to fail on any error instead of having to guess which
|
||||
// DuckDB messages mean "no extension" and which mean "the catalog is broken".
|
||||
func requireDuckDBIceberg(t *testing.T) {
|
||||
t.Helper()
|
||||
|
||||
const ready = "iceberg extension ready"
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"--entrypoint", "duckdb",
|
||||
duckDBImage,
|
||||
"-c", fmt.Sprintf("INSTALL iceberg; LOAD iceberg; SELECT '%s' AS marker;", ready),
|
||||
)
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil || !strings.Contains(string(output), ready) {
|
||||
t.Skipf("DuckDB image cannot load the iceberg extension: %v\n%s", err, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIcebergVendedCredentials(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := newOAuthTestEnv(t)
|
||||
// The DuckDB container reaches S3 through host.docker.internal, so advertise
|
||||
// the endpoint under the name the client can actually resolve.
|
||||
env.s3ExternalURL = fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
|
||||
defer env.cleanup(t)
|
||||
env.start(t)
|
||||
|
||||
bucketName := "vendcreds-" + randomSuffix()
|
||||
namespace := "vendcreds_ns_" + randomSuffix()
|
||||
tableName := "vendcreds_tbl_" + randomSuffix()
|
||||
|
||||
createTableBucketViaShell(t, env, bucketName)
|
||||
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
|
||||
createNamespaceWithToken(t, env, token, bucketName, namespace)
|
||||
createTableWithToken(t, env, token, bucketName, namespace, tableName)
|
||||
|
||||
t.Run("plain load table advertises the endpoint", func(t *testing.T) {
|
||||
config := loadTableConfig(t, env, token, bucketName, namespace, tableName, "")
|
||||
if config["s3.endpoint"] != env.s3ExternalURL {
|
||||
t.Fatalf("s3.endpoint = %q, want %q (clients that bring their own credentials still need the endpoint)",
|
||||
config["s3.endpoint"], env.s3ExternalURL)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("vended credentials request gets no half-configured storage", func(t *testing.T) {
|
||||
config := loadTableConfig(t, env, token, bucketName, namespace, tableName, "vended-credentials")
|
||||
if config["s3.endpoint"] == "" {
|
||||
return
|
||||
}
|
||||
// If the catalog ever does vend credentials, the endpoint may come back
|
||||
// -- but only alongside the credentials that make it usable.
|
||||
if config["s3.access-key-id"] == "" || config["s3.secret-access-key"] == "" {
|
||||
t.Fatalf("LoadTable config = %v: a client that asked for vended credentials drops its own and signs with what it gets, so an endpoint without credentials makes every data file read 403", config)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("duckdb writes and reads back through the catalog", func(t *testing.T) {
|
||||
if !testutil.HasDocker() {
|
||||
t.Skip("Docker not available, skipping DuckDB round trip")
|
||||
}
|
||||
requireDuckDBIceberg(t)
|
||||
duckDBRoundTrip(t, env, bucketName, namespace)
|
||||
})
|
||||
}
|
||||
|
||||
// duckDBRoundTrip runs the reporter's flow: attach the catalog, create a table
|
||||
// from a query, then read it back. Both halves need signed S3 access to the
|
||||
// table's metadata and data files.
|
||||
func duckDBRoundTrip(t *testing.T, env *oauthTestEnv, bucketName, namespace string) {
|
||||
t.Helper()
|
||||
|
||||
const marker = "vended-credentials round trip ok"
|
||||
sql := fmt.Sprintf(`
|
||||
INSTALL iceberg;
|
||||
LOAD iceberg;
|
||||
|
||||
CREATE SECRET iceberg_secret (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://host.docker.internal:%d',
|
||||
CLIENT_ID '%s',
|
||||
CLIENT_SECRET '%s'
|
||||
);
|
||||
|
||||
CREATE SECRET s3_secret (
|
||||
TYPE S3,
|
||||
KEY_ID '%s',
|
||||
SECRET '%s',
|
||||
ENDPOINT 'host.docker.internal:%d',
|
||||
URL_STYLE 'path',
|
||||
USE_SSL false
|
||||
);
|
||||
|
||||
ATTACH 's3://%s' AS vend_cat (
|
||||
TYPE ICEBERG,
|
||||
SECRET iceberg_secret,
|
||||
ENDPOINT 'http://host.docker.internal:%d',
|
||||
READ_ONLY false
|
||||
);
|
||||
|
||||
CREATE TABLE vend_cat.%s.round_trip AS SELECT 42 AS answer;
|
||||
|
||||
SELECT CASE WHEN sum(answer) = 42 THEN '%s' ELSE 'wrong answer' END AS marker
|
||||
FROM vend_cat.%s.round_trip;
|
||||
`,
|
||||
env.icebergPort, env.accessKey, env.secretKey,
|
||||
env.accessKey, env.secretKey, env.s3Port,
|
||||
bucketName, env.icebergPort,
|
||||
namespace, marker, namespace,
|
||||
)
|
||||
|
||||
sqlFile := filepath.Join(env.dataDir, "duckdb_vended_credentials.sql")
|
||||
if err := os.WriteFile(sqlFile, []byte(sql), 0644); err != nil {
|
||||
t.Fatalf("write SQL file: %v", err)
|
||||
}
|
||||
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"-v", fmt.Sprintf("%s:/test", env.dataDir),
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"-e", "AWS_REGION=us-east-1",
|
||||
"--entrypoint", "duckdb",
|
||||
duckDBImage,
|
||||
"-init", "/test/duckdb_vended_credentials.sql",
|
||||
"-c", "SELECT 1",
|
||||
)
|
||||
output, err := cmd.CombinedOutput()
|
||||
outputStr := string(output)
|
||||
t.Logf("DuckDB output:\n%s", outputStr)
|
||||
|
||||
// Nothing below skips: requireDuckDBIceberg already established that the
|
||||
// extension loads, so any failure from here is the catalog's.
|
||||
if strings.Contains(outputStr, "trying to refresh secret") {
|
||||
t.Fatalf("DuckDB fell back to refreshing the credential the catalog vended, which no stage-created table can satisfy:\n%s", outputStr)
|
||||
}
|
||||
// Match the phrasings rather than a bare "403", which a random port could
|
||||
// carry.
|
||||
for _, denial := range []string{"AccessDenied", "403 Forbidden", "code 403"} {
|
||||
if strings.Contains(outputStr, denial) {
|
||||
t.Fatalf("DuckDB signed its S3 requests with the catalog's credential-less config:\n%s", outputStr)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(outputStr, marker) {
|
||||
t.Fatalf("DuckDB did not write and read the table back (err=%v):\n%s", err, outputStr)
|
||||
}
|
||||
}
|
||||
|
||||
// loadTableConfig loads a table and returns the FileIO config the catalog
|
||||
// advertises, optionally asking for an access delegation mechanism.
|
||||
func loadTableConfig(t *testing.T, env *oauthTestEnv, token, bucketName, namespace, tableName, delegation string) map[string]string {
|
||||
t.Helper()
|
||||
|
||||
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables/%s", env.icebergURL(), bucketName, namespace, tableName)
|
||||
req, err := http.NewRequest(http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("create request: %v", err)
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
if delegation != "" {
|
||||
req.Header.Set("X-Iceberg-Access-Delegation", delegation)
|
||||
}
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("load table: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("load table failed: status=%d body=%s", resp.StatusCode, body)
|
||||
}
|
||||
|
||||
var result struct {
|
||||
Config map[string]string `json:"config"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &result); err != nil {
|
||||
t.Fatalf("decode LoadTableResult: %v", err)
|
||||
}
|
||||
return result.Config
|
||||
}
|
||||
585
test/s3tables/catalog/iceberg_catalog_test.go
Normal file
585
test/s3tables/catalog/iceberg_catalog_test.go
Normal file
@@ -0,0 +1,585 @@
|
||||
// Package catalog provides integration tests for the Iceberg REST Catalog API.
|
||||
// These tests use DuckDB running in Docker to verify catalog operations.
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws/credentials"
|
||||
v4 "github.com/aws/aws-sdk-go/aws/signer/v4"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/test/testutil"
|
||||
)
|
||||
|
||||
// sharedEnv is the single TestEnvironment shared across all tests in this package.
|
||||
var sharedEnv *TestEnvironment
|
||||
|
||||
// TestMain starts one weed mini instance for the whole package and tears it down
|
||||
// after all tests have run.
|
||||
func TestMain(m *testing.M) {
|
||||
flag.Parse()
|
||||
if os.Getenv("SHORT") != "" || testing.Short() {
|
||||
// Let tests self-skip when run with -short.
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
env, err := newTestEnvironmentForMain()
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "SKIP: setup failed: %v\n", err)
|
||||
os.Exit(0) // Skip all tests rather than fail
|
||||
}
|
||||
sharedEnv = env
|
||||
|
||||
if startErr := sharedEnv.startSeaweedFSForMain(); startErr != nil {
|
||||
fmt.Fprintf(os.Stderr, "SKIP: weed mini failed to start: %v\n", startErr)
|
||||
sharedEnv.cleanupForMain()
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
code := m.Run()
|
||||
|
||||
sharedEnv.cleanupForMain()
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
// TestEnvironment contains the test environment configuration
|
||||
type TestEnvironment struct {
|
||||
seaweedDir string
|
||||
weedBinary string
|
||||
dataDir string
|
||||
s3Port int
|
||||
s3GrpcPort int
|
||||
icebergPort int
|
||||
lancePort int
|
||||
masterPort int
|
||||
masterGrpcPort int
|
||||
filerPort int
|
||||
filerGrpcPort int
|
||||
volumePort int
|
||||
volumeGrpcPort int
|
||||
weedProcess *exec.Cmd
|
||||
weedCancel context.CancelFunc
|
||||
dockerAvailable bool
|
||||
}
|
||||
|
||||
// newTestEnvironmentForMain creates a TestEnvironment without calling t.Fatalf so it
|
||||
// can be used from TestMain (which has no *testing.T).
|
||||
func newTestEnvironmentForMain() (*TestEnvironment, error) {
|
||||
// Find the SeaweedFS root directory
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("get working directory: %w", err)
|
||||
}
|
||||
|
||||
seaweedDir := wd
|
||||
for i := 0; i < 5; i++ {
|
||||
if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
|
||||
break
|
||||
}
|
||||
seaweedDir = filepath.Dir(seaweedDir)
|
||||
}
|
||||
|
||||
// Check for weed binary
|
||||
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
|
||||
if info, statErr := os.Stat(weedBinary); statErr == nil {
|
||||
// Name the binary and its age. `make test` rebuilds first, but a plain
|
||||
// `go test` will happily drive a weeks-old binary and report a pass for
|
||||
// code it never ran.
|
||||
fmt.Fprintf(os.Stderr, "using %s, built %s\n", weedBinary, info.ModTime().Format(time.RFC3339))
|
||||
} else if os.IsNotExist(statErr) {
|
||||
weedBinary = "weed"
|
||||
if _, err := exec.LookPath(weedBinary); err != nil {
|
||||
return nil, fmt.Errorf("weed binary not found")
|
||||
}
|
||||
}
|
||||
|
||||
// Create temporary data directory
|
||||
dataDir, err := os.MkdirTemp("", "seaweed-iceberg-test-*")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create temp dir: %w", err)
|
||||
}
|
||||
|
||||
// Allocate 10 unique ports atomically: s3, iceberg, s3Grpc, master, masterGrpc,
|
||||
// filer, filerGrpc, volume, volumeGrpc, lance
|
||||
ports, err := testutil.AllocatePorts(10)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("allocate ports: %w", err)
|
||||
}
|
||||
|
||||
return &TestEnvironment{
|
||||
seaweedDir: seaweedDir,
|
||||
weedBinary: weedBinary,
|
||||
dataDir: dataDir,
|
||||
s3Port: ports[0],
|
||||
s3GrpcPort: ports[1],
|
||||
icebergPort: ports[2],
|
||||
masterPort: ports[3],
|
||||
masterGrpcPort: ports[4],
|
||||
filerPort: ports[5],
|
||||
filerGrpcPort: ports[6],
|
||||
volumePort: ports[7],
|
||||
volumeGrpcPort: ports[8],
|
||||
lancePort: ports[9],
|
||||
dockerAvailable: testutil.HasDocker(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// startSeaweedFSForMain starts weed mini without a *testing.T (for use in TestMain).
|
||||
func (env *TestEnvironment) startSeaweedFSForMain() error {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
env.weedCancel = cancel
|
||||
|
||||
masterDir := filepath.Join(env.dataDir, "master")
|
||||
filerDir := filepath.Join(env.dataDir, "filer")
|
||||
volumeDir := filepath.Join(env.dataDir, "volume")
|
||||
|
||||
for _, dir := range []string{masterDir, filerDir, volumeDir} {
|
||||
if err := os.MkdirAll(dir, 0755); err != nil {
|
||||
cancel()
|
||||
return fmt.Errorf("create directory %s: %w", dir, err)
|
||||
}
|
||||
}
|
||||
|
||||
cmd := exec.CommandContext(ctx, env.weedBinary, "mini",
|
||||
"-master.port", fmt.Sprintf("%d", env.masterPort),
|
||||
"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
|
||||
"-volume.port", fmt.Sprintf("%d", env.volumePort),
|
||||
"-volume.port.grpc", fmt.Sprintf("%d", env.volumeGrpcPort),
|
||||
"-filer.port", fmt.Sprintf("%d", env.filerPort),
|
||||
"-filer.port.grpc", fmt.Sprintf("%d", env.filerGrpcPort),
|
||||
"-s3.port", fmt.Sprintf("%d", env.s3Port),
|
||||
"-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort),
|
||||
"-s3.port.iceberg", fmt.Sprintf("%d", env.icebergPort),
|
||||
"-s3.port.lance", fmt.Sprintf("%d", env.lancePort),
|
||||
"-ip.bind", "0.0.0.0",
|
||||
"-dir", env.dataDir,
|
||||
)
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
|
||||
if err := cmd.Start(); err != nil {
|
||||
cancel()
|
||||
return fmt.Errorf("start SeaweedFS: %w", err)
|
||||
}
|
||||
env.weedProcess = cmd
|
||||
|
||||
if !env.waitForService(fmt.Sprintf("http://127.0.0.1:%d/v1/config", env.icebergPort), 30*time.Second) {
|
||||
cancel()
|
||||
cmd.Wait()
|
||||
return fmt.Errorf("Iceberg REST API did not become ready")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// cleanupForMain stops SeaweedFS and cleans up resources (no *testing.T needed).
|
||||
func (env *TestEnvironment) cleanupForMain() {
|
||||
if env.weedCancel != nil {
|
||||
env.weedCancel()
|
||||
}
|
||||
if env.weedProcess != nil {
|
||||
time.Sleep(2 * time.Second)
|
||||
env.weedProcess.Wait()
|
||||
}
|
||||
if env.dataDir != "" {
|
||||
os.RemoveAll(env.dataDir)
|
||||
}
|
||||
}
|
||||
|
||||
// waitForService waits for a service to become available
|
||||
func (env *TestEnvironment) waitForService(url string, timeout time.Duration) bool {
|
||||
client := &http.Client{Timeout: 2 * time.Second}
|
||||
deadline := time.Now().Add(timeout)
|
||||
for time.Now().Before(deadline) {
|
||||
resp, err := client.Get(url)
|
||||
if err == nil {
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
return true
|
||||
}
|
||||
}
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// IcebergURL returns the Iceberg REST Catalog URL
|
||||
func (env *TestEnvironment) IcebergURL() string {
|
||||
return fmt.Sprintf("http://127.0.0.1:%d", env.icebergPort)
|
||||
}
|
||||
|
||||
// LanceURL returns the Lance Namespace server URL
|
||||
func (env *TestEnvironment) LanceURL() string {
|
||||
return fmt.Sprintf("http://127.0.0.1:%d", env.lancePort)
|
||||
}
|
||||
|
||||
// TestIcebergConfig tests the /v1/config endpoint
|
||||
func TestIcebergConfig(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
|
||||
// Test GET /v1/config
|
||||
resp, err := http.Get(env.IcebergURL() + "/v1/config")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to get config: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Errorf("Expected status 200, got %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read response body: %v", err)
|
||||
}
|
||||
|
||||
// Verify response contains required fields
|
||||
bodyStr := string(body)
|
||||
if !strings.Contains(bodyStr, "defaults") || !strings.Contains(bodyStr, "overrides") {
|
||||
t.Errorf("Config response missing required fields: %s", bodyStr)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIcebergNamespaces tests namespace operations
|
||||
func TestIcebergNamespaces(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
|
||||
// Create the default table bucket first via S3
|
||||
bucketName := "warehouse-ns-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
// Test GET /v1/namespaces (should return empty list initially)
|
||||
resp, err := http.Get(env.IcebergURL() + icebergPath(bucketName, "/v1/namespaces"))
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to list namespaces: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
t.Fatalf("Expected status 200, got %d: %s", resp.StatusCode, body)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStageCreateAndFinalizeFlow verifies staged create remains invisible until assert-create commit finalizes table creation.
|
||||
func TestStageCreateAndFinalizeFlow(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
bucketName := "warehouse-stage-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
namespace := "stage_ns_" + randomSuffix()
|
||||
tableName := "orders"
|
||||
|
||||
status, _, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, "/v1/namespaces"), map[string]any{
|
||||
"namespace": []string{namespace},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Create namespace request failed: %v", err)
|
||||
}
|
||||
if status != http.StatusOK && status != http.StatusConflict {
|
||||
t.Fatalf("Create namespace status = %d, want 200 or 409", status)
|
||||
}
|
||||
|
||||
status, badReqResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables", namespace)), map[string]any{
|
||||
"stage-create": true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Stage create missing-name request failed: %v", err)
|
||||
}
|
||||
if status != http.StatusBadRequest {
|
||||
t.Fatalf("Stage create missing-name status = %d, want 400", status)
|
||||
}
|
||||
errorObj, _ := badReqResp["error"].(map[string]any)
|
||||
if got := errorObj["type"]; got != "BadRequestException" {
|
||||
t.Fatalf("error.type = %v, want BadRequestException", got)
|
||||
}
|
||||
msg, _ := errorObj["message"].(string)
|
||||
if !strings.Contains(strings.ToLower(msg), "table name is required") {
|
||||
t.Fatalf("error.message = %v, want it to include %q", errorObj["message"], "table name is required")
|
||||
}
|
||||
|
||||
status, stageResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables", namespace)), map[string]any{
|
||||
"name": tableName,
|
||||
"stage-create": true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Stage create request failed: %v", err)
|
||||
}
|
||||
if status != http.StatusOK {
|
||||
t.Fatalf("Stage create status = %d, want 200", status)
|
||||
}
|
||||
stageLocation, _ := stageResp["metadata-location"].(string)
|
||||
if !strings.HasSuffix(stageLocation, "/metadata/v1.metadata.json") {
|
||||
t.Fatalf("stage metadata-location = %q, want suffix /metadata/v1.metadata.json", stageLocation)
|
||||
}
|
||||
|
||||
status, _, err = doIcebergJSONRequest(env, http.MethodGet, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Load staged table request failed: %v", err)
|
||||
}
|
||||
if status != http.StatusNotFound {
|
||||
t.Fatalf("Load staged table status = %d, want 404", status)
|
||||
}
|
||||
|
||||
status, commitResp, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), map[string]any{
|
||||
"requirements": []map[string]any{
|
||||
{"type": "assert-create"},
|
||||
},
|
||||
"updates": []any{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Finalize commit request failed: %v", err)
|
||||
}
|
||||
if status != http.StatusOK {
|
||||
t.Fatalf("Finalize commit status = %d, want 200", status)
|
||||
}
|
||||
commitLocation, _ := commitResp["metadata-location"].(string)
|
||||
if !strings.HasSuffix(commitLocation, "/metadata/v1.metadata.json") {
|
||||
t.Fatalf("final metadata-location = %q, want suffix /metadata/v1.metadata.json", commitLocation)
|
||||
}
|
||||
|
||||
status, loadResp, err := doIcebergJSONRequest(env, http.MethodGet, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Load finalized table request failed: %v", err)
|
||||
}
|
||||
if status != http.StatusOK {
|
||||
t.Fatalf("Load finalized table status = %d, want 200", status)
|
||||
}
|
||||
loadLocation, _ := loadResp["metadata-location"].(string)
|
||||
if loadLocation != commitLocation {
|
||||
t.Fatalf("loaded metadata-location = %q, want %q", loadLocation, commitLocation)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCommitMissingTableWithoutAssertCreate ensures missing-table commits still require assert-create for creation.
|
||||
func TestCommitMissingTableWithoutAssertCreate(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
bucketName := "warehouse-missing-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
namespace := "stage_missing_assert_ns_" + randomSuffix()
|
||||
tableName := "missing_table"
|
||||
|
||||
status, _, err := doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, "/v1/namespaces"), map[string]any{
|
||||
"namespace": []string{namespace},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Create namespace request failed: %v", err)
|
||||
}
|
||||
if status != http.StatusOK && status != http.StatusConflict {
|
||||
t.Fatalf("Create namespace status = %d, want 200 or 409", status)
|
||||
}
|
||||
|
||||
status, _, err = doIcebergJSONRequest(env, http.MethodPost, icebergPath(bucketName, fmt.Sprintf("/v1/namespaces/%s/tables/%s", namespace, tableName)), map[string]any{
|
||||
"requirements": []any{},
|
||||
"updates": []any{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Commit missing table request failed: %v", err)
|
||||
}
|
||||
if status != http.StatusNotFound {
|
||||
t.Fatalf("Commit missing table status = %d, want 404", status)
|
||||
}
|
||||
}
|
||||
|
||||
// doIcebergJSONRequest decodes JSON object responses used by catalog tests.
|
||||
func doIcebergJSONRequest(env *TestEnvironment, method, path string, payload any) (int, map[string]any, error) {
|
||||
url := env.IcebergURL() + path
|
||||
|
||||
var bodyReader io.Reader
|
||||
if payload != nil {
|
||||
data, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
bodyReader = bytes.NewReader(data)
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(method, url, bodyReader)
|
||||
if err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
if payload != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
data, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return resp.StatusCode, nil, err
|
||||
}
|
||||
|
||||
if len(data) == 0 {
|
||||
return resp.StatusCode, nil, nil
|
||||
}
|
||||
|
||||
var decoded map[string]any
|
||||
if err := json.Unmarshal(data, &decoded); err != nil {
|
||||
return resp.StatusCode, nil, fmt.Errorf("failed to decode %s %s response: %w body=%s", method, path, err, string(data))
|
||||
}
|
||||
return resp.StatusCode, decoded, nil
|
||||
}
|
||||
|
||||
// icebergPath inserts the table bucket prefix into Iceberg REST API paths.
|
||||
// For example, "/v1/namespaces" with prefix "my-bucket" becomes
|
||||
// "/v1/my-bucket/namespaces".
|
||||
func icebergPath(prefix, path string) string {
|
||||
if prefix == "" {
|
||||
return path
|
||||
}
|
||||
const base = "/v1/"
|
||||
if !strings.HasPrefix(path, base) {
|
||||
return path
|
||||
}
|
||||
withPrefix := base + prefix + "/" + strings.TrimPrefix(path, base)
|
||||
return withPrefix
|
||||
}
|
||||
|
||||
// createTableBucket creates a table bucket via the S3Tables REST API.
|
||||
// The request is AWS V4 signed for SERVICE=s3tables so the S3 Tables
|
||||
// route matcher accepts it; signing with regular SERVICE=s3 would let
|
||||
// the request fall through to the S3 CreateBucket handler.
|
||||
// createTableBucket makes a table bucket of the given format. An empty format
|
||||
// leaves it to the server, which means ICEBERG.
|
||||
func createTableBucket(t *testing.T, env *TestEnvironment, bucketName, format string) {
|
||||
t.Helper()
|
||||
|
||||
endpoint := fmt.Sprintf("http://localhost:%d/buckets", env.s3Port)
|
||||
reqBody := fmt.Sprintf(`{"name":"%s"}`, bucketName)
|
||||
if format != "" {
|
||||
reqBody = fmt.Sprintf(`{"name":"%s","format":"%s"}`, bucketName, format)
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(http.MethodPut, endpoint, strings.NewReader(reqBody))
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
|
||||
signS3TablesRequest(t, req, reqBody)
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create table bucket %s: %v", bucketName, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
t.Logf("Create table bucket %s response: status=%d, body=%s", bucketName, resp.StatusCode, string(body))
|
||||
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusConflict {
|
||||
t.Fatalf("Failed to create table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
|
||||
}
|
||||
t.Logf("Created table bucket %s", bucketName)
|
||||
}
|
||||
|
||||
// signS3TablesRequest signs req with AWS V4 for SERVICE=s3tables. The
|
||||
// underlying weed mini instance runs in default-allow mode so the
|
||||
// signature itself is not verified; only the credential scope matters,
|
||||
// because the S3 Tables route matcher requires SERVICE=s3tables to
|
||||
// distinguish S3 Tables traffic from regular S3 calls on the same paths.
|
||||
func signS3TablesRequest(t *testing.T, req *http.Request, body string) {
|
||||
t.Helper()
|
||||
creds := credentials.NewStaticCredentials("test-ak", "test-sk", "")
|
||||
signer := v4.NewSigner(creds)
|
||||
if _, err := signer.Sign(req, strings.NewReader(body), "s3tables", "us-east-1", time.Now()); err != nil {
|
||||
t.Fatalf("Failed to sign S3 Tables request: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// randomSuffix returns a short random hex suffix for unique resource naming.
|
||||
func randomSuffix() string {
|
||||
return fmt.Sprintf("%x", time.Now().UnixNano()&0xffffffff)
|
||||
}
|
||||
|
||||
// TestDuckDBIntegration tests Iceberg catalog operations using DuckDB
|
||||
// This test requires Docker to be available
|
||||
func TestDuckDBIntegration(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
|
||||
if !env.dockerAvailable {
|
||||
t.Skip("Docker not available, skipping DuckDB integration test")
|
||||
}
|
||||
|
||||
// Create a temporary SQL file for DuckDB to execute
|
||||
sqlFile := filepath.Join(env.dataDir, "test.sql")
|
||||
sqlContent := fmt.Sprintf(`
|
||||
-- Install and load Iceberg extension
|
||||
INSTALL iceberg;
|
||||
LOAD iceberg;
|
||||
|
||||
-- List namespaces via Iceberg REST catalog (basic connectivity test)
|
||||
-- Note: Full operations require setting up S3 credentials which is beyond this test
|
||||
SELECT 'Iceberg extension loaded successfully' as result;
|
||||
`)
|
||||
|
||||
if err := os.WriteFile(sqlFile, []byte(sqlContent), 0644); err != nil {
|
||||
t.Fatalf("Failed to write SQL file: %v", err)
|
||||
}
|
||||
|
||||
// Run DuckDB in Docker to test Iceberg connectivity
|
||||
// Use host.docker.internal to connect to the host's Iceberg port
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"-v", fmt.Sprintf("%s:/test", env.dataDir),
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"--entrypoint", "duckdb",
|
||||
"duckdb/duckdb:latest",
|
||||
"-init", "/test/test.sql",
|
||||
"-c", "SELECT 1",
|
||||
)
|
||||
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Logf("DuckDB output: %s", output)
|
||||
// Check for expected errors in certain CI environments
|
||||
outputStr := string(output)
|
||||
if strings.Contains(outputStr, "iceberg extension is not available") ||
|
||||
strings.Contains(outputStr, "Failed to load") {
|
||||
t.Skip("Skipping DuckDB test: Iceberg extension not available in Docker image")
|
||||
}
|
||||
// Any other error is unexpected
|
||||
t.Fatalf("DuckDB command failed unexpectedly. Output: %s\nError: %v", output, err)
|
||||
}
|
||||
|
||||
// Verify the test completed successfully
|
||||
outputStr := string(output)
|
||||
t.Logf("DuckDB output: %s", outputStr)
|
||||
if !strings.Contains(outputStr, "Iceberg extension loaded successfully") {
|
||||
t.Errorf("Expected success message in output, got: %s", outputStr)
|
||||
}
|
||||
}
|
||||
283
test/s3tables/catalog/issue_9103_test.go
Normal file
283
test/s3tables/catalog/issue_9103_test.go
Normal file
@@ -0,0 +1,283 @@
|
||||
// Reproduction tests for https://github.com/seaweedfs/seaweedfs/issues/9103
|
||||
//
|
||||
// The issue reports two distinct failure modes when using DuckDB against the
|
||||
// SeaweedFS Iceberg REST catalog:
|
||||
//
|
||||
// 1. `ATTACH 's3://test/' AS cat (TYPE 'ICEBERG', ...); SELECT * FROM cat.ovirt.disk;`
|
||||
// fails with "Table with name 'ovirt.disk' does not exist because schema
|
||||
// 'ovirt' does not exist." The namespace "ovirt" does exist in the bucket.
|
||||
//
|
||||
// 2. `SELECT * FROM iceberg_scan('s3://test/ovirt/disk');` fails with HTTP 403
|
||||
// AccessDenied when DuckDB tries to glob `s3://test/ovirt/disk/metadata/v*`
|
||||
// because the LoadTable response does not vend S3 file-io credentials back
|
||||
// to the client.
|
||||
//
|
||||
// These tests reproduce the underlying catalog-level misbehavior at the REST
|
||||
// protocol layer so they run quickly and deterministically. A DuckDB-based
|
||||
// end-to-end reproduction is also included (gated on Docker availability).
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/test/testutil"
|
||||
)
|
||||
|
||||
// TestIssue9103_ConfigDoesNotVendWarehousePrefix reproduces failure mode #2.
|
||||
//
|
||||
// Per the Iceberg REST spec, a client attaching with warehouse=s3://<bucket>/
|
||||
// calls GET /v1/config?warehouse=s3://<bucket>/ and expects the server to
|
||||
// return an `overrides.prefix` that identifies the catalog namespace for
|
||||
// subsequent requests. Without it the client falls back to unprefixed paths
|
||||
// like /v1/namespaces, which on SeaweedFS resolve to the hard-coded default
|
||||
// bucket ("warehouse") and therefore do not list the user's namespaces.
|
||||
//
|
||||
// Currently the handler ignores the warehouse query parameter and returns
|
||||
// empty defaults/overrides. That is exactly what makes DuckDB's ATTACH flow
|
||||
// report `schema "ovirt" does not exist`.
|
||||
func TestIssue9103_ConfigDoesNotVendWarehousePrefix(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
bucketName := "warehouse-9103cfg-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
warehouse := fmt.Sprintf("s3://%s/", bucketName)
|
||||
u := fmt.Sprintf("%s/v1/config?warehouse=%s", env.IcebergURL(), url.QueryEscape(warehouse))
|
||||
|
||||
status, body, err := doIcebergJSONRequest(env, http.MethodGet, fmt.Sprintf("/v1/config?warehouse=%s", url.QueryEscape(warehouse)), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("GET %s failed: %v", u, err)
|
||||
}
|
||||
if status != http.StatusOK {
|
||||
t.Fatalf("GET %s status = %d, want 200", u, status)
|
||||
}
|
||||
|
||||
overrides, _ := body["overrides"].(map[string]any)
|
||||
gotPrefix, _ := overrides["prefix"].(string)
|
||||
if gotPrefix != bucketName {
|
||||
t.Fatalf("GET /v1/config?warehouse=%s: overrides.prefix = %q, want %q (catalog must echo the warehouse's table bucket so clients like DuckDB know which /v1/{prefix}/namespaces to use)",
|
||||
warehouse, gotPrefix, bucketName)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIssue9103_BareNamespacesListMissesNamespaceInAttachedBucket
|
||||
// demonstrates the downstream effect of the /v1/config bug.
|
||||
//
|
||||
// The client created the namespace "ovirt" inside bucket "test" (via
|
||||
// /v1/{bucket}/namespaces), but when it later issues GET /v1/namespaces (no
|
||||
// prefix, because /v1/config didn't give it one), that request resolves to
|
||||
// the default bucket and returns no namespaces. DuckDB surfaces this as
|
||||
// `schema "ovirt" does not exist`.
|
||||
func TestIssue9103_BareNamespacesListMissesNamespaceInAttachedBucket(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
bucketName := "warehouse-9103ns-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
namespace := "ovirt"
|
||||
status, _, err := doIcebergJSONRequest(env, http.MethodPost,
|
||||
icebergPath(bucketName, "/v1/namespaces"),
|
||||
map[string]any{"namespace": []string{namespace}})
|
||||
if err != nil {
|
||||
t.Fatalf("create namespace: %v", err)
|
||||
}
|
||||
if status != http.StatusOK && status != http.StatusConflict {
|
||||
t.Fatalf("create namespace status = %d, want 200 or 409", status)
|
||||
}
|
||||
|
||||
// Sanity: prefixed listing finds the namespace.
|
||||
status, prefixedBody, err := doIcebergJSONRequest(env, http.MethodGet,
|
||||
icebergPath(bucketName, "/v1/namespaces"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("list namespaces with prefix: %v", err)
|
||||
}
|
||||
if status != http.StatusOK {
|
||||
t.Fatalf("prefixed list status = %d, want 200", status)
|
||||
}
|
||||
if !containsNamespace(prefixedBody, namespace) {
|
||||
t.Fatalf("prefixed list missing namespace %q: %v", namespace, prefixedBody)
|
||||
}
|
||||
|
||||
// Simulate what a DuckDB-style client does when /v1/config did not vend a
|
||||
// prefix: it falls back to the unprefixed listing. It should still be
|
||||
// able to discover the namespace (e.g. via a warehouse query parameter),
|
||||
// but today it cannot, which is the user-visible bug.
|
||||
bareWarehouse := "s3://" + bucketName + "/"
|
||||
status, bareBody, err := doIcebergJSONRequest(env, http.MethodGet,
|
||||
fmt.Sprintf("/v1/namespaces?warehouse=%s", url.QueryEscape(bareWarehouse)), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("bare list namespaces: %v", err)
|
||||
}
|
||||
switch {
|
||||
case status != http.StatusOK:
|
||||
t.Fatalf("GET /v1/namespaces?warehouse=%s status = %d, want 200 (server ignores the warehouse query parameter and falls back to the default %q bucket, causing DuckDB to see no schemas)",
|
||||
bareWarehouse, status, "warehouse")
|
||||
case !containsNamespace(bareBody, namespace):
|
||||
t.Fatalf("GET /v1/namespaces?warehouse=%s did not return %q: %v (server must honor the warehouse query parameter here or vend overrides.prefix from /v1/config so the client can route to the right bucket)",
|
||||
bareWarehouse, namespace, bareBody)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIssue9103_DuckDBAttachCannotResolveNamespace is the end-to-end
|
||||
// reproduction of the issue using DuckDB in Docker, mirroring the exact
|
||||
// sequence of commands from the bug report. It runs under its own weed
|
||||
// mini with IAM configured so the OAuth2 client_credentials flow that
|
||||
// DuckDB's iceberg extension requires actually works (the shared env has
|
||||
// no credentials registered). Gated on Docker availability.
|
||||
func TestIssue9103_DuckDBAttachCannotResolveNamespace(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
if !testutil.HasDocker() {
|
||||
t.Skip("Docker not available, skipping DuckDB integration reproduction")
|
||||
}
|
||||
|
||||
env := newOAuthTestEnv(t)
|
||||
defer env.cleanup(t)
|
||||
env.start(t)
|
||||
|
||||
bucketName := "test9103-" + randomSuffix()
|
||||
createTableBucketViaShell(t, env, bucketName)
|
||||
|
||||
namespace := "ovirt"
|
||||
tableName := "disk"
|
||||
|
||||
token := requestOAuthToken(t, env, env.accessKey, env.secretKey)
|
||||
createNamespaceWithToken(t, env, token, bucketName, namespace)
|
||||
createTableWithToken(t, env, token, bucketName, namespace, tableName)
|
||||
|
||||
sql := fmt.Sprintf(`
|
||||
INSTALL iceberg;
|
||||
LOAD iceberg;
|
||||
|
||||
CREATE SECRET test_berg (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://host.docker.internal:%d',
|
||||
SCOPE 's3://%s/',
|
||||
CLIENT_ID '%s',
|
||||
CLIENT_SECRET '%s'
|
||||
);
|
||||
|
||||
CREATE SECRET s3_berg (
|
||||
TYPE S3,
|
||||
KEY_ID '%s',
|
||||
SECRET '%s',
|
||||
ENDPOINT 'host.docker.internal:%d',
|
||||
URL_STYLE 'path',
|
||||
USE_SSL false,
|
||||
SCOPE 's3://%s/'
|
||||
);
|
||||
|
||||
ATTACH 's3://%s/' AS test_catalog (TYPE 'ICEBERG', secret test_berg);
|
||||
SELECT * FROM test_catalog.%s.%s LIMIT 0;
|
||||
`, env.icebergPort, bucketName,
|
||||
env.accessKey, env.secretKey,
|
||||
env.accessKey, env.secretKey, env.s3Port, bucketName,
|
||||
bucketName, namespace, tableName)
|
||||
|
||||
sqlFile := filepath.Join(env.dataDir, "issue_9103.sql")
|
||||
if err := os.WriteFile(sqlFile, []byte(sql), 0644); err != nil {
|
||||
t.Fatalf("write SQL: %v", err)
|
||||
}
|
||||
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"-v", fmt.Sprintf("%s:/test", env.dataDir),
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"-e", "AWS_REGION=us-east-1",
|
||||
"--entrypoint", "duckdb",
|
||||
"duckdb/duckdb:latest",
|
||||
"-init", "/test/issue_9103.sql",
|
||||
"-c", "SELECT 1",
|
||||
)
|
||||
|
||||
out, runErr := cmd.CombinedOutput()
|
||||
outStr := string(out)
|
||||
t.Logf("DuckDB output:\n%s", outStr)
|
||||
|
||||
if strings.Contains(outStr, "iceberg extension is not available") ||
|
||||
strings.Contains(outStr, "Failed to load") ||
|
||||
strings.Contains(outStr, "could not fetch extension") ||
|
||||
strings.Contains(outStr, "failed to download") {
|
||||
t.Skip("Iceberg extension not available in DuckDB Docker image")
|
||||
}
|
||||
|
||||
if runErr != nil {
|
||||
if strings.Contains(outStr, "does not exist because schema") &&
|
||||
strings.Contains(outStr, namespace) {
|
||||
t.Fatalf("reproduced issue #9103: DuckDB cannot see namespace %q after ATTACH 's3://%s/'; see output above",
|
||||
namespace, bucketName)
|
||||
}
|
||||
t.Fatalf("DuckDB run failed for an unexpected reason: %v", runErr)
|
||||
}
|
||||
}
|
||||
|
||||
// createTableWithToken creates an Iceberg table using an OAuth bearer token.
|
||||
func createTableWithToken(t *testing.T, env *oauthTestEnv, token, bucketName, namespace, tableName string) {
|
||||
t.Helper()
|
||||
|
||||
payload := map[string]any{
|
||||
"name": tableName,
|
||||
"schema": map[string]any{
|
||||
"type": "struct",
|
||||
"schema-id": 0,
|
||||
"fields": []map[string]any{
|
||||
{"id": 1, "name": "id", "required": true, "type": "long"},
|
||||
},
|
||||
},
|
||||
}
|
||||
body, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal create-table payload: %v", err)
|
||||
}
|
||||
|
||||
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables", env.icebergURL(), bucketName, namespace)
|
||||
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatalf("create request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("create table: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
respBody, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("create table failed: status=%d body=%s", resp.StatusCode, respBody)
|
||||
}
|
||||
t.Logf("Created table %s.%s in bucket %s", namespace, tableName, bucketName)
|
||||
}
|
||||
|
||||
// containsNamespace checks whether an Iceberg REST ListNamespaces response
|
||||
// contains the given single-level namespace.
|
||||
func containsNamespace(body map[string]any, name string) bool {
|
||||
arr, _ := body["namespaces"].([]any)
|
||||
for _, item := range arr {
|
||||
parts, _ := item.([]any)
|
||||
if len(parts) == 1 {
|
||||
if s, ok := parts[0].(string); ok && s == name {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
63
test/s3tables/catalog/lance_client_test.go
Normal file
63
test/s3tables/catalog/lance_client_test.go
Normal file
@@ -0,0 +1,63 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestLanceNamespaceRealClient drives the namespace with the Lance client rather
|
||||
// than hand-built HTTP. The Go tests beside this one cover the catalog surface;
|
||||
// what this adds is that the location and storage_options the namespace vends
|
||||
// are actually enough to write and read a dataset, which needs the S3 layout
|
||||
// guard, the endpoint and the credentials all to be right at once.
|
||||
//
|
||||
// To run manually:
|
||||
//
|
||||
// cd test/s3tables/catalog
|
||||
// docker build -t lance-namespace-test -f Dockerfile.lance .
|
||||
func TestLanceNamespaceRealClient(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
if !env.dockerAvailable {
|
||||
t.Skip("Docker not available, skipping Lance client integration test")
|
||||
}
|
||||
|
||||
bucketName := "lance-client-test-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, s3tables.FormatLance)
|
||||
|
||||
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
|
||||
|
||||
buildCmd := exec.Command("docker", "build", "-t", "lance-namespace-test", "-f", "Dockerfile.lance", ".")
|
||||
buildCmd.Dir = testDir
|
||||
if out, err := buildCmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
|
||||
}
|
||||
|
||||
namespaceURL := fmt.Sprintf("http://host.docker.internal:%d", env.lancePort)
|
||||
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
|
||||
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"-v", fmt.Sprintf("%s:/app:ro", testDir),
|
||||
"lance-namespace-test",
|
||||
"python3", "/app/test_lance_namespace.py",
|
||||
"--namespace-url", namespaceURL,
|
||||
"--s3-endpoint", s3Endpoint,
|
||||
"--bucket", bucketName,
|
||||
)
|
||||
cmd.Dir = testDir
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
|
||||
t.Logf("Running Lance client test against %s", namespaceURL)
|
||||
if err := cmd.Run(); err != nil {
|
||||
t.Errorf("Lance client test failed: %v", err)
|
||||
}
|
||||
}
|
||||
378
test/s3tables/catalog/lance_namespace_test.go
Normal file
378
test/s3tables/catalog/lance_namespace_test.go
Normal file
@@ -0,0 +1,378 @@
|
||||
// Integration tests for the Lance Namespace REST server, driven against a live
|
||||
// gateway rather than an in-memory filer. The bugs this surface has produced -
|
||||
// a deregister that deleted the dataset, an S3 door that refused every Lance
|
||||
// file - all looked fine against a fake.
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// lanceCall posts to the Lance namespace and returns the status and body.
|
||||
func lanceCall(t *testing.T, env *TestEnvironment, method, path, body string) (int, []byte) {
|
||||
t.Helper()
|
||||
req, err := http.NewRequest(method, env.LanceURL()+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatalf("build request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %s: %v", method, path, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
payload, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, payload
|
||||
}
|
||||
|
||||
func lanceMust(t *testing.T, env *TestEnvironment, method, path, body string, want int) []byte {
|
||||
t.Helper()
|
||||
status, payload := lanceCall(t, env, method, path, body)
|
||||
if status != want {
|
||||
t.Fatalf("%s %s = %d (%s), want %d", method, path, status, payload, want)
|
||||
}
|
||||
return payload
|
||||
}
|
||||
|
||||
// filerEntryExists reports whether a path exists on storage, which is how these
|
||||
// tests tell "the catalog forgot the table" from "the data is gone".
|
||||
func filerEntryExists(t *testing.T, env *TestEnvironment, path string) bool {
|
||||
t.Helper()
|
||||
url := fmt.Sprintf("http://127.0.0.1:%d%s", env.filerPort, path)
|
||||
resp, err := http.Get(url)
|
||||
if err != nil {
|
||||
t.Fatalf("filer GET %s: %v", path, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
return resp.StatusCode == http.StatusOK
|
||||
}
|
||||
|
||||
// lanceTestBucket makes a uniquely named table bucket of the given format. Most
|
||||
// tests want LANCE; the one that checks what the Lance surface hides needs an
|
||||
// Iceberg bucket, because a declared bucket holds one format only.
|
||||
func lanceTestBucket(t *testing.T, env *TestEnvironment, prefix, format string) string {
|
||||
t.Helper()
|
||||
bucket := prefix + "-" + randomSuffix()
|
||||
createTableBucket(t, env, bucket, format)
|
||||
return bucket
|
||||
}
|
||||
|
||||
func TestLanceNamespaceLifecycle(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-ns", s3tables.FormatLance)
|
||||
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$sales/create", `{}`, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$sales/exists", `{}`, http.StatusOK)
|
||||
|
||||
// The root lists table buckets, which is the first namespace level here.
|
||||
var roots struct {
|
||||
Namespaces []string `json:"namespaces"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/$/list", "", http.StatusOK), &roots); err != nil {
|
||||
t.Fatalf("decode root listing: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, name := range roots.Namespaces {
|
||||
if name == bucket {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("root listing %v does not include %s", roots.Namespaces, bucket)
|
||||
}
|
||||
|
||||
var children struct {
|
||||
Namespaces []string `json:"namespaces"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"/list", "", http.StatusOK), &children); err != nil {
|
||||
t.Fatalf("decode namespace listing: %v", err)
|
||||
}
|
||||
if len(children.Namespaces) != 1 || children.Namespaces[0] != "sales" {
|
||||
t.Fatalf("namespace listing = %v, want [sales]", children.Namespaces)
|
||||
}
|
||||
|
||||
// A namespace is never created as a side effect of naming one inside a
|
||||
// bucket that does not exist.
|
||||
status, _ := lanceCall(t, env, http.MethodPost, "/v1/namespace/nosuchbucket$ns/create", `{}`)
|
||||
if status != http.StatusNotFound {
|
||||
t.Fatalf("create under a missing bucket = %d, want 404", status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLanceTableLifecyclePreservesData(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-tbl", s3tables.FormatLance)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
|
||||
|
||||
table := "/v1/table/" + bucket + "$ml$vectors"
|
||||
var declared struct {
|
||||
Location string `json:"location"`
|
||||
StorageOptions map[string]string `json:"storage_options"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodPost, table+"/declare", `{}`, http.StatusOK), &declared); err != nil {
|
||||
t.Fatalf("decode declare: %v", err)
|
||||
}
|
||||
want := fmt.Sprintf("s3://%s/ml/vectors", bucket)
|
||||
if declared.Location != want {
|
||||
t.Fatalf("declared location = %q, want %q", declared.Location, want)
|
||||
}
|
||||
|
||||
datasetPath := fmt.Sprintf("/buckets/%s/ml/vectors/", bucket)
|
||||
if !filerEntryExists(t, env, datasetPath) {
|
||||
t.Fatal("declare did not create the dataset directory")
|
||||
}
|
||||
|
||||
var listed struct {
|
||||
Tables []string `json:"tables"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"$ml/table/list", "", http.StatusOK), &listed); err != nil {
|
||||
t.Fatalf("decode listing: %v", err)
|
||||
}
|
||||
if len(listed.Tables) != 1 || listed.Tables[0] != bucket+"$ml$vectors" {
|
||||
t.Fatalf("table listing = %v, want the full identifier", listed.Tables)
|
||||
}
|
||||
|
||||
// Deregistering forgets the table but keeps every byte. The catalog entry is
|
||||
// the dataset directory, so a drop of the entry would take the data too.
|
||||
lanceMust(t, env, http.MethodPost, table+"/deregister", `{}`, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, table+"/exists", `{}`, http.StatusNotFound)
|
||||
if !filerEntryExists(t, env, datasetPath) {
|
||||
t.Fatal("deregister deleted the dataset")
|
||||
}
|
||||
|
||||
registerBody := fmt.Sprintf(`{"location":%q}`, want)
|
||||
lanceMust(t, env, http.MethodPost, table+"/register", registerBody, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, table+"/exists", `{}`, http.StatusOK)
|
||||
|
||||
// Dropping is the operation that does remove the data.
|
||||
lanceMust(t, env, http.MethodPost, table+"/drop", `{}`, http.StatusOK)
|
||||
if filerEntryExists(t, env, datasetPath) {
|
||||
t.Fatal("drop left the dataset behind")
|
||||
}
|
||||
}
|
||||
|
||||
// A Lance client must never resolve an Iceberg table's location, or it writes a
|
||||
// dataset over a table another engine owns.
|
||||
func TestLanceHidesIcebergTables(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
// An Iceberg bucket, because a bucket that declares LANCE cannot hold an
|
||||
// Iceberg table at all now. The invariant still matters from this side: the
|
||||
// Lance surface can be pointed at any bucket, and must not describe or list
|
||||
// a table whose format it does not serve.
|
||||
bucket := lanceTestBucket(t, env, "lance-mixed", s3tables.FormatIceberg)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$mixed/create", `{}`, http.StatusOK)
|
||||
|
||||
createIcebergTable(t, env, bucket, "mixed", "ledger")
|
||||
|
||||
status, _ := lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$ledger/describe", `{}`)
|
||||
if status != http.StatusNotFound {
|
||||
t.Fatalf("describing an iceberg table through lance = %d, want 404", status)
|
||||
}
|
||||
status, _ = lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$ledger/declare", `{}`)
|
||||
if status != http.StatusConflict {
|
||||
t.Fatalf("declaring over an iceberg table = %d, want 409", status)
|
||||
}
|
||||
// And a new Lance table cannot be smuggled in beside it either.
|
||||
status, _ = lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$vectors/declare", `{}`)
|
||||
if status != http.StatusConflict {
|
||||
t.Fatalf("declaring a lance table in an iceberg bucket = %d, want 409", status)
|
||||
}
|
||||
|
||||
var listed struct {
|
||||
Tables []string `json:"tables"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"$mixed/table/list", "", http.StatusOK), &listed); err != nil {
|
||||
t.Fatalf("decode listing: %v", err)
|
||||
}
|
||||
if len(listed.Tables) != 0 {
|
||||
t.Fatalf("lance listing shows iceberg tables: %v", listed.Tables)
|
||||
}
|
||||
}
|
||||
|
||||
// The S3 door validates every object written into a table bucket. A Lance
|
||||
// dataset's files have to get through it, or the catalog is decorative.
|
||||
func TestLanceFilesAreAcceptedByTheS3Door(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-layout", s3tables.FormatLance)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/table/"+bucket+"$ml$vectors/declare", `{}`, http.StatusOK)
|
||||
|
||||
accepted := []string{
|
||||
"ml/vectors/data/0111111010111000110101116.lance",
|
||||
"ml/vectors/_versions/18446744073709551614.manifest",
|
||||
"ml/vectors/_versions/18446744073709551614.manifest-a3a292ad",
|
||||
"ml/vectors/_transactions/0-ddb27ab7.txn",
|
||||
"ml/vectors/_indices/85814508-fts/index.idx",
|
||||
"ml/vectors/.lance-reserved",
|
||||
}
|
||||
for _, object := range accepted {
|
||||
if status := putS3Object(t, env, bucket, object); status != http.StatusOK {
|
||||
t.Errorf("PUT %s = %d, want 200", object, status)
|
||||
}
|
||||
}
|
||||
|
||||
// The guard still rejects files that belong to no table layout.
|
||||
for _, object := range []string{"ml/vectors/random.txt", "ml/vectors/notadir/x.lance"} {
|
||||
if status := putS3Object(t, env, bucket, object); status == http.StatusOK {
|
||||
t.Errorf("PUT %s = 200, want a rejection", object)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A Lance commit is a conditional PUT of the next manifest: object_store turns
|
||||
// PutMode::Create into If-None-Match: *, and lance treats the refusal as a
|
||||
// commit conflict and rebases. So the whole safety of concurrent writers rests
|
||||
// on this store answering that precondition atomically. It does, because the
|
||||
// gateway reduces the header to a filer WriteCondition evaluated at the
|
||||
// object's owner under a per-path lock.
|
||||
//
|
||||
// This is the reason the namespace does not manage versions itself.
|
||||
func TestLanceCommitPreconditionAdmitsOneWriter(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-commit", s3tables.FormatLance)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/table/"+bucket+"$ml$vectors/declare", `{}`, http.StatusOK)
|
||||
|
||||
const writers = 8
|
||||
var wg sync.WaitGroup
|
||||
statuses := make([]int, writers)
|
||||
failures := make([]error, writers)
|
||||
for i := 0; i < writers; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
// Not t.Fatalf: from a goroutine it ends only that goroutine, and
|
||||
// the status stays zero, which reads as a bogus response later.
|
||||
statuses[i], failures[i] = putIfAbsent(env, bucket, "ml/vectors/_versions/1.manifest")
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
for i, err := range failures {
|
||||
if err != nil {
|
||||
t.Fatalf("writer %d could not reach the gateway: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
won, refused := 0, 0
|
||||
for _, status := range statuses {
|
||||
switch status {
|
||||
case http.StatusOK:
|
||||
won++
|
||||
case http.StatusPreconditionFailed:
|
||||
refused++
|
||||
default:
|
||||
t.Fatalf("unexpected status %d committing a manifest", status)
|
||||
}
|
||||
}
|
||||
if won != 1 {
|
||||
t.Fatalf("%d writers committed version 1; exactly one may win, %d were refused", won, refused)
|
||||
}
|
||||
}
|
||||
|
||||
// The namespace answers managed_versioning=false: the dataset owns its version
|
||||
// history, and a reader that never goes through this catalog still sees all of
|
||||
// it.
|
||||
func TestLanceDoesNotManageVersions(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-ver", s3tables.FormatLance)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
|
||||
table := "/v1/table/" + bucket + "$ml$vectors"
|
||||
|
||||
var declared struct {
|
||||
ManagedVersioning bool `json:"managed_versioning"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodPost, table+"/declare", `{}`, http.StatusOK), &declared); err != nil {
|
||||
t.Fatalf("decode declare: %v", err)
|
||||
}
|
||||
if declared.ManagedVersioning {
|
||||
t.Fatal("managed_versioning must be false; the dataset owns its versions")
|
||||
}
|
||||
|
||||
// The version ops answer with the spec's Unsupported code rather than a
|
||||
// bare 404, so a client that asks learns why.
|
||||
status, _ := lanceCall(t, env, http.MethodPost, table+"/version/list", `{}`)
|
||||
if status != http.StatusNotImplemented {
|
||||
t.Fatalf("version/list = %d, want 501", status)
|
||||
}
|
||||
}
|
||||
|
||||
// putIfAbsent writes an object only if the key is free, the way a Lance commit
|
||||
// does. It returns an error rather than failing the test, so it is safe to call
|
||||
// from the racing goroutines.
|
||||
func putIfAbsent(env *TestEnvironment, bucket, object string) (int, error) {
|
||||
url := fmt.Sprintf("http://127.0.0.1:%d/%s/%s", env.s3Port, bucket, object)
|
||||
req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader([]byte("manifest")))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
req.Header.Set("If-None-Match", "*")
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
return resp.StatusCode, nil
|
||||
}
|
||||
|
||||
// putS3Object writes an object through the S3 gateway and returns the status.
|
||||
func putS3Object(t *testing.T, env *TestEnvironment, bucket, object string) int {
|
||||
t.Helper()
|
||||
url := fmt.Sprintf("http://127.0.0.1:%d/%s/%s", env.s3Port, bucket, object)
|
||||
req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader([]byte("x")))
|
||||
if err != nil {
|
||||
t.Fatalf("build request: %v", err)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("PUT %s: %v", object, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
return resp.StatusCode
|
||||
}
|
||||
|
||||
// createIcebergTable registers an ordinary Iceberg table so the mixed-catalog
|
||||
// tests have one to be confused by.
|
||||
func createIcebergTable(t *testing.T, env *TestEnvironment, bucket, namespace, name string) {
|
||||
t.Helper()
|
||||
body := fmt.Sprintf(`{"name":%q,"schema":{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}}`, name)
|
||||
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables", env.IcebergURL(), bucket, namespace)
|
||||
resp, err := http.Post(url, "application/json", strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatalf("create iceberg table: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
payload, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("create iceberg table = %d: %s", resp.StatusCode, payload)
|
||||
}
|
||||
}
|
||||
137
test/s3tables/catalog/pyiceberg_test.go
Normal file
137
test/s3tables/catalog/pyiceberg_test.go
Normal file
@@ -0,0 +1,137 @@
|
||||
// Package catalog provides integration tests for the Iceberg REST Catalog API.
|
||||
// This file adds PyIceberg-based compatibility tests using Docker.
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestPyIcebergRestCatalog tests the Iceberg REST Catalog using PyIceberg client in Docker.
|
||||
// This provides a more comprehensive test than DuckDB as PyIceberg fully exercises the REST API.
|
||||
//
|
||||
// Prerequisites:
|
||||
// - Docker must be available
|
||||
// - SeaweedFS must be running with Iceberg REST enabled
|
||||
//
|
||||
// To run manually:
|
||||
//
|
||||
// cd test/s3tables/catalog
|
||||
// docker compose -f docker-compose.test.yaml up --build
|
||||
func TestPyIcebergRestCatalog(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
|
||||
if !env.dockerAvailable {
|
||||
t.Skip("Docker not available, skipping PyIceberg integration test")
|
||||
}
|
||||
|
||||
// Create the test bucket first
|
||||
bucketName := "pyiceberg-compat-test-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
// Build the test working directory path
|
||||
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
|
||||
|
||||
// Run PyIceberg test using Docker
|
||||
catalogURL := fmt.Sprintf("http://host.docker.internal:%d", env.icebergPort)
|
||||
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
|
||||
warehouse := fmt.Sprintf("s3://%s/", bucketName)
|
||||
|
||||
// Build the test image first for faster repeated runs
|
||||
buildCmd := exec.Command("docker", "build", "-t", "iceberg-rest-test", "-f", "Dockerfile.pyiceberg", ".")
|
||||
buildCmd.Dir = testDir
|
||||
if out, err := buildCmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
|
||||
}
|
||||
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"-e", fmt.Sprintf("AWS_ENDPOINT_URL=%s", s3Endpoint),
|
||||
"-v", fmt.Sprintf("%s:/app:ro", testDir),
|
||||
"iceberg-rest-test",
|
||||
"python3", "/app/test_rest_catalog.py",
|
||||
"--catalog-url", catalogURL,
|
||||
"--warehouse", warehouse,
|
||||
"--prefix", bucketName,
|
||||
)
|
||||
cmd.Dir = testDir
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
|
||||
t.Logf("Running PyIceberg REST catalog test...")
|
||||
t.Logf(" Catalog URL: %s", catalogURL)
|
||||
t.Logf(" Warehouse: %s", warehouse)
|
||||
|
||||
if err := cmd.Run(); err != nil {
|
||||
t.Errorf("PyIceberg test failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPyIcebergRestCatalogAuthenticated tests the Iceberg REST Catalog using PyIceberg with authentication.
|
||||
// This test uses the default admin credentials that SeaweedFS creates on startup.
|
||||
func TestPyIcebergRestCatalogAuthenticated(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
|
||||
if !env.dockerAvailable {
|
||||
t.Skip("Docker not available, skipping PyIceberg integration test")
|
||||
}
|
||||
|
||||
// Use default admin credentials
|
||||
testAccessKey := "admin"
|
||||
testSecretKey := "admin"
|
||||
|
||||
// Create the test bucket first (using unauthenticated request, which works with DefaultAllow)
|
||||
bucketName := "pyiceberg-auth-test-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
// Build the test working directory path
|
||||
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
|
||||
|
||||
// Run PyIceberg test using Docker with authentication
|
||||
catalogURL := fmt.Sprintf("http://host.docker.internal:%d", env.icebergPort)
|
||||
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
|
||||
warehouse := fmt.Sprintf("s3://%s/", bucketName)
|
||||
|
||||
// Build the test image first for faster repeated runs
|
||||
buildCmd := exec.Command("docker", "build", "-t", "iceberg-rest-test", "-f", "Dockerfile.pyiceberg", ".")
|
||||
buildCmd.Dir = testDir
|
||||
if out, err := buildCmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
|
||||
}
|
||||
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"-e", fmt.Sprintf("AWS_ENDPOINT_URL=%s", s3Endpoint),
|
||||
"-v", fmt.Sprintf("%s:/app:ro", testDir),
|
||||
"iceberg-rest-test",
|
||||
"python3", "/app/test_rest_catalog_auth.py",
|
||||
"--catalog-url", catalogURL,
|
||||
"--warehouse", warehouse,
|
||||
"--prefix", bucketName,
|
||||
"--access-key", testAccessKey,
|
||||
"--secret-key", testSecretKey,
|
||||
)
|
||||
cmd.Dir = testDir
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
|
||||
t.Logf("Running PyIceberg REST catalog test with authentication...")
|
||||
t.Logf(" Catalog URL: %s", catalogURL)
|
||||
t.Logf(" Warehouse: %s", warehouse)
|
||||
t.Logf(" Access Key: %s", testAccessKey)
|
||||
|
||||
if err := cmd.Run(); err != nil {
|
||||
t.Errorf("PyIceberg authenticated test failed: %v", err)
|
||||
}
|
||||
}
|
||||
36
test/s3tables/catalog/pyiceberg_test_helpers_test.go
Normal file
36
test/s3tables/catalog/pyiceberg_test_helpers_test.go
Normal file
@@ -0,0 +1,36 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// verifyTableBucketMetadata verifies that a table bucket was created with proper metadata
|
||||
func verifyTableBucketMetadata(t *testing.T, env *TestEnvironment, bucketName string) {
|
||||
t.Helper()
|
||||
|
||||
// Use S3Tables REST API to get the bucket
|
||||
endpoint := fmt.Sprintf("http://localhost:%d/buckets/%s", env.s3Port, bucketName)
|
||||
|
||||
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to get table bucket %s: %v", bucketName, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
t.Logf("Get table bucket %s response: status=%d, body=%s", bucketName, resp.StatusCode, string(body))
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("Failed to get table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
|
||||
}
|
||||
t.Logf("Verified table bucket %s exists with metadata", bucketName)
|
||||
}
|
||||
106
test/s3tables/catalog/test_lance_namespace.py
Executable file
106
test/s3tables/catalog/test_lance_namespace.py
Executable file
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Drive the SeaweedFS Lance Namespace with the real Lance client.
|
||||
|
||||
The catalog half can be checked with plain HTTP, and the Go integration tests
|
||||
do. What only a real client proves is that the location and storage_options it
|
||||
hands back are enough to write and read a dataset: the layout guard on the S3
|
||||
door, the endpoint and allow_http options, and the credentials all have to be
|
||||
right at once, and a hand-built request checks none of that.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import warnings
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
import lance
|
||||
import lance_namespace as ln
|
||||
import pyarrow as pa
|
||||
|
||||
|
||||
def sample_table():
|
||||
return pa.table(
|
||||
{
|
||||
"id": pa.array([1, 2, 3, 4]),
|
||||
"vec": pa.array([[1.0, 2.0]] * 4, type=pa.list_(pa.float32(), 2)),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--namespace-url", required=True, help="Lance namespace REST URL")
|
||||
parser.add_argument("--s3-endpoint", required=True, help="S3 endpoint the dataset lives behind")
|
||||
parser.add_argument("--bucket", required=True, help="table bucket to use, already created")
|
||||
parser.add_argument("--access-key", default="any")
|
||||
parser.add_argument("--secret-key", default="any")
|
||||
args = parser.parse_args()
|
||||
|
||||
ns = ln.connect("rest", {"uri": args.namespace_url})
|
||||
ns.create_namespace(ln.CreateNamespaceRequest(id=[args.bucket, "ml"]))
|
||||
table_id = [args.bucket, "ml", "vectors"]
|
||||
|
||||
declared = ns.declare_table(ln.DeclareTableRequest(id=table_id))
|
||||
print(f"declared {table_id} at {declared.location}")
|
||||
if not declared.location:
|
||||
print("FAIL: declare returned no location", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
described = ns.describe_table(ln.DescribeTableRequest(id=table_id))
|
||||
if described.location != declared.location:
|
||||
print(
|
||||
f"FAIL: describe location {described.location} != declare {declared.location}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
options = dict(described.storage_options or {})
|
||||
print(f"storage options from the namespace: {sorted(options)}")
|
||||
# The endpoint is overridden with the container's view of the same gateway:
|
||||
# what the namespace vends is correct for its own host, and a test harness
|
||||
# bound to a wildcard address vends nothing at all. That the namespace vends
|
||||
# a usable endpoint when it can is covered by the unit tests; what matters
|
||||
# here is that everything else it hands back is enough to reach the data.
|
||||
options["aws_endpoint"] = args.s3_endpoint
|
||||
options["allow_http"] = "true"
|
||||
# A deployment without STS still needs credentials to sign with.
|
||||
options.setdefault("aws_access_key_id", args.access_key)
|
||||
options.setdefault("aws_secret_access_key", args.secret_key)
|
||||
|
||||
lance.write_dataset(sample_table(), described.location, storage_options=options,
|
||||
mode="overwrite")
|
||||
dataset = lance.dataset(described.location, storage_options=options)
|
||||
rows = dataset.count_rows()
|
||||
print(f"wrote and read back {rows} rows at version {dataset.version}")
|
||||
if rows != 4:
|
||||
print(f"FAIL: read back {rows} rows, want 4", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# The table is listed, and a client that skips the namespace entirely can
|
||||
# still open the dataset by URI.
|
||||
listed = ns.list_tables(ln.ListTablesRequest(id=[args.bucket, "ml"])).tables
|
||||
if f"{args.bucket}$ml$vectors" not in listed:
|
||||
print(f"FAIL: {listed} does not contain the table", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if lance.dataset(declared.location, storage_options=options).count_rows() != 4:
|
||||
print("FAIL: the dataset is not readable straight off its URI", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# Deregistering hides the table and keeps the data, which is the difference
|
||||
# between it and a drop.
|
||||
ns.deregister_table(ln.DeregisterTableRequest(id=table_id))
|
||||
if f"{args.bucket}$ml$vectors" in ns.list_tables(ln.ListTablesRequest(id=[args.bucket, "ml"])).tables:
|
||||
print("FAIL: a deregistered table is still listed", file=sys.stderr)
|
||||
return 1
|
||||
if lance.dataset(declared.location, storage_options=options).count_rows() != 4:
|
||||
print("FAIL: deregister destroyed the dataset", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print("PASS")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
252
test/s3tables/catalog/test_rest_catalog.py
Normal file
252
test/s3tables/catalog/test_rest_catalog.py
Normal file
@@ -0,0 +1,252 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Iceberg REST Catalog Compatibility Test for SeaweedFS
|
||||
|
||||
This script tests the Iceberg REST Catalog API compatibility of the
|
||||
SeaweedFS Iceberg REST Catalog implementation.
|
||||
|
||||
Usage:
|
||||
python3 test_rest_catalog.py --catalog-url http://localhost:8182
|
||||
|
||||
Requirements:
|
||||
pip install pyiceberg[s3fs]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pyiceberg.catalog import load_catalog
|
||||
from pyiceberg.schema import Schema
|
||||
from pyiceberg.types import (
|
||||
IntegerType,
|
||||
LongType,
|
||||
StringType,
|
||||
NestedField,
|
||||
)
|
||||
from pyiceberg.exceptions import (
|
||||
NamespaceAlreadyExistsError,
|
||||
NoSuchNamespaceError,
|
||||
TableAlreadyExistsError,
|
||||
NoSuchTableError,
|
||||
)
|
||||
|
||||
|
||||
def test_config_endpoint(catalog):
|
||||
"""Test that the catalog config endpoint returns valid configuration."""
|
||||
print("Testing /v1/config endpoint...")
|
||||
# The catalog is already loaded which means config endpoint worked
|
||||
print(" /v1/config endpoint working")
|
||||
return True
|
||||
|
||||
|
||||
def test_namespace_operations(catalog, prefix):
|
||||
"""Test namespace CRUD operations."""
|
||||
print("Testing namespace operations...")
|
||||
namespace = (f"{prefix.replace('-', '_')}_test_ns",)
|
||||
|
||||
# List initial namespaces
|
||||
namespaces = catalog.list_namespaces()
|
||||
print(f" Initial namespaces: {namespaces}")
|
||||
|
||||
# Create namespace
|
||||
try:
|
||||
catalog.create_namespace(namespace)
|
||||
print(f" Created namespace: {namespace}")
|
||||
except NamespaceAlreadyExistsError:
|
||||
print(f" ! Namespace already exists: {namespace}")
|
||||
|
||||
# List namespaces (should include our new one)
|
||||
namespaces = catalog.list_namespaces()
|
||||
if namespace in namespaces:
|
||||
print(" Namespace appears in list")
|
||||
else:
|
||||
print(f" Namespace not found in list: {namespaces}")
|
||||
return False
|
||||
|
||||
# Get namespace properties
|
||||
try:
|
||||
props = catalog.load_namespace_properties(namespace)
|
||||
print(f" Loaded namespace properties: {props}")
|
||||
except NoSuchNamespaceError:
|
||||
print(f" Failed to load namespace properties")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def test_table_operations(catalog, prefix):
|
||||
"""Test table CRUD operations."""
|
||||
print("Testing table operations...")
|
||||
namespace = (f"{prefix.replace('-', '_')}_test_ns",)
|
||||
table_name = "test_table"
|
||||
table_id = namespace + (table_name,)
|
||||
|
||||
# Define a simple schema
|
||||
schema = Schema(
|
||||
NestedField(field_id=1, name="id", field_type=LongType(), required=True),
|
||||
NestedField(field_id=2, name="name", field_type=StringType(), required=False),
|
||||
NestedField(field_id=3, name="age", field_type=IntegerType(), required=False),
|
||||
)
|
||||
|
||||
# Create table
|
||||
try:
|
||||
table = catalog.create_table(
|
||||
identifier=table_id,
|
||||
schema=schema,
|
||||
)
|
||||
print(f" Created table: {table_id}")
|
||||
except TableAlreadyExistsError:
|
||||
print(f" ! Table already exists: {table_id}")
|
||||
_ = catalog.load_table(table_id)
|
||||
|
||||
# List tables
|
||||
tables = catalog.list_tables(namespace)
|
||||
if table_name in [t[1] for t in tables]:
|
||||
print(" Table appears in list")
|
||||
else:
|
||||
print(f" Table not found in list: {tables}")
|
||||
return False
|
||||
|
||||
# Load table
|
||||
try:
|
||||
loaded_table = catalog.load_table(table_id)
|
||||
print(f" Loaded table: {loaded_table.name()}")
|
||||
print(f" Schema: {loaded_table.schema()}")
|
||||
print(f" Location: {loaded_table.location()}")
|
||||
except NoSuchTableError:
|
||||
print(f" Failed to load table")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def test_table_update(catalog, prefix):
|
||||
"""Test table update/commit operations."""
|
||||
print("Testing table update operations...")
|
||||
namespace = (f"{prefix.replace('-', '_')}_test_ns",)
|
||||
table_name = "test_table"
|
||||
table_id = namespace + (table_name,)
|
||||
|
||||
try:
|
||||
table = catalog.load_table(table_id)
|
||||
|
||||
# Update table properties
|
||||
with table.transaction() as transaction:
|
||||
transaction.set_properties({"test.property": "test.value"})
|
||||
|
||||
print(" Updated table properties")
|
||||
|
||||
# Reload and verify
|
||||
table = catalog.load_table(table_id)
|
||||
if table.properties.get("test.property") == "test.value":
|
||||
print(" Property update verified")
|
||||
else:
|
||||
print(" ! Property update failed or not persisted")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f" Table update failed: {e}")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def test_cleanup(catalog, prefix):
|
||||
"""Test table and namespace deletion."""
|
||||
print("Testing cleanup operations...")
|
||||
namespace = (f"{prefix.replace('-', '_')}_test_ns",)
|
||||
table_id = namespace + ("test_table",)
|
||||
|
||||
# Drop table
|
||||
try:
|
||||
catalog.drop_table(table_id)
|
||||
print(f" Dropped table: {table_id}")
|
||||
except NoSuchTableError:
|
||||
print(f" ! Table already deleted: {table_id}")
|
||||
|
||||
# Drop namespace
|
||||
try:
|
||||
catalog.drop_namespace(namespace)
|
||||
print(f" Dropped namespace: {namespace}")
|
||||
except NoSuchNamespaceError:
|
||||
print(f" ! Namespace already deleted: {namespace}")
|
||||
except Exception as e:
|
||||
print(f" ? Namespace drop error (may be expected): {e}")
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Test Iceberg REST Catalog compatibility")
|
||||
parser.add_argument("--catalog-url", required=True, help="Iceberg REST Catalog URL (e.g., http://localhost:8182)")
|
||||
parser.add_argument("--warehouse", default="s3://iceberg-test/", help="Warehouse location")
|
||||
parser.add_argument("--prefix", required=True, help="Table bucket prefix")
|
||||
parser.add_argument("--skip-cleanup", action="store_true", help="Skip cleanup at the end")
|
||||
args = parser.parse_args()
|
||||
|
||||
print(f"Connecting to Iceberg REST Catalog at: {args.catalog_url}")
|
||||
print(f"Warehouse: {args.warehouse}")
|
||||
print(f"Prefix: {args.prefix}")
|
||||
print()
|
||||
|
||||
# Load the REST catalog with retries to handle possible delay in catalog server readiness
|
||||
import time
|
||||
max_retries = 10
|
||||
catalog = None
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
catalog = load_catalog(
|
||||
"rest",
|
||||
**{
|
||||
"type": "rest",
|
||||
"uri": args.catalog_url,
|
||||
"warehouse": args.warehouse,
|
||||
"prefix": args.prefix,
|
||||
"auth": {"type": "noop"},
|
||||
"s3.anonymous": "true", # Disable AWS request signing for unauthenticated access
|
||||
}
|
||||
)
|
||||
print(f"Successfully connected to catalog on attempt {attempt + 1}")
|
||||
break
|
||||
except Exception as e:
|
||||
if attempt < max_retries - 1:
|
||||
print(f" Attempt {attempt + 1} failed, retrying in 2s... ({e})")
|
||||
time.sleep(2)
|
||||
else:
|
||||
print(f" All {max_retries} attempts failed.")
|
||||
raise e
|
||||
|
||||
# Run tests
|
||||
tests = [
|
||||
("Config Endpoint", lambda: test_config_endpoint(catalog)),
|
||||
("Namespace Operations", lambda: test_namespace_operations(catalog, args.prefix)),
|
||||
("Table Operations", lambda: test_table_operations(catalog, args.prefix)),
|
||||
("Table Update", lambda: test_table_update(catalog, args.prefix)),
|
||||
]
|
||||
|
||||
if not args.skip_cleanup:
|
||||
tests.append(("Cleanup", lambda: test_cleanup(catalog, args.prefix)))
|
||||
|
||||
passed = 0
|
||||
failed = 0
|
||||
|
||||
for name, test_fn in tests:
|
||||
print(f"\n{'='*50}")
|
||||
try:
|
||||
if test_fn():
|
||||
passed += 1
|
||||
print(f"PASSED: {name}")
|
||||
else:
|
||||
failed += 1
|
||||
print(f"FAILED: {name}")
|
||||
except Exception as e:
|
||||
failed += 1
|
||||
print(f"ERROR in {name}: {e}")
|
||||
|
||||
print(f"\n{'='*50}")
|
||||
print(f"Results: {passed} passed, {failed} failed")
|
||||
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
224
test/s3tables/catalog/test_rest_catalog_auth.py
Normal file
224
test/s3tables/catalog/test_rest_catalog_auth.py
Normal file
@@ -0,0 +1,224 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Iceberg REST Catalog Compatibility Test for SeaweedFS (Authenticated)
|
||||
|
||||
This script tests the Iceberg REST Catalog API compatibility with authentication.
|
||||
|
||||
Usage:
|
||||
python3 test_rest_catalog_auth.py --catalog-url http://localhost:8182 \\
|
||||
--access-key admin --secret-key admin
|
||||
|
||||
Requirements:
|
||||
pip install pyiceberg[s3fs]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pyiceberg.catalog import load_catalog
|
||||
from pyiceberg.schema import Schema
|
||||
from pyiceberg.types import (
|
||||
IntegerType,
|
||||
LongType,
|
||||
StringType,
|
||||
NestedField,
|
||||
)
|
||||
from pyiceberg.exceptions import (
|
||||
NamespaceAlreadyExistsError,
|
||||
NoSuchNamespaceError,
|
||||
TableAlreadyExistsError,
|
||||
NoSuchTableError,
|
||||
)
|
||||
|
||||
|
||||
def test_config_endpoint(catalog):
|
||||
"""Test that the catalog config endpoint returns valid configuration."""
|
||||
print("Testing /v1/config endpoint...")
|
||||
# The catalog is already loaded which means config endpoint worked
|
||||
print(" /v1/config endpoint working")
|
||||
return True
|
||||
|
||||
|
||||
def test_namespace_operations(catalog, prefix):
|
||||
"""Test namespace CRUD operations."""
|
||||
print("Testing namespace operations...")
|
||||
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
|
||||
|
||||
# List initial namespaces
|
||||
namespaces = catalog.list_namespaces()
|
||||
print(f" Initial namespaces: {namespaces}")
|
||||
|
||||
# Create namespace
|
||||
try:
|
||||
catalog.create_namespace(namespace)
|
||||
print(f" Created namespace: {namespace}")
|
||||
except NamespaceAlreadyExistsError:
|
||||
print(f" ! Namespace already exists: {namespace}")
|
||||
|
||||
# List namespaces (should include our new one)
|
||||
namespaces = catalog.list_namespaces()
|
||||
if namespace in namespaces:
|
||||
print(" Namespace appears in list")
|
||||
else:
|
||||
print(f" Namespace not found in list: {namespaces}")
|
||||
return False
|
||||
|
||||
# Get namespace properties
|
||||
try:
|
||||
props = catalog.load_namespace_properties(namespace)
|
||||
print(f" Loaded namespace properties: {props}")
|
||||
except NoSuchNamespaceError:
|
||||
print(f" Failed to load namespace properties")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def test_table_operations(catalog, prefix):
|
||||
"""Test table CRUD operations."""
|
||||
print("Testing table operations...")
|
||||
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
|
||||
table_name = "auth_test_table"
|
||||
table_id = namespace + (table_name,)
|
||||
|
||||
# Define a simple schema
|
||||
schema = Schema(
|
||||
NestedField(field_id=1, name="id", field_type=LongType(), required=True),
|
||||
NestedField(field_id=2, name="name", field_type=StringType(), required=False),
|
||||
NestedField(field_id=3, name="age", field_type=IntegerType(), required=False),
|
||||
)
|
||||
|
||||
# Create table
|
||||
try:
|
||||
table = catalog.create_table(
|
||||
identifier=table_id,
|
||||
schema=schema,
|
||||
)
|
||||
print(f" Created table: {table_id}")
|
||||
except TableAlreadyExistsError:
|
||||
print(f" ! Table already exists: {table_id}")
|
||||
_ = catalog.load_table(table_id)
|
||||
|
||||
# List tables
|
||||
tables = catalog.list_tables(namespace)
|
||||
if table_name in [t[1] for t in tables]:
|
||||
print(" Table appears in list")
|
||||
else:
|
||||
print(f" Table not found in list: {tables}")
|
||||
return False
|
||||
|
||||
# Load table
|
||||
try:
|
||||
loaded_table = catalog.load_table(table_id)
|
||||
print(f" Loaded table: {loaded_table.name()}")
|
||||
print(f" Schema: {loaded_table.schema()}")
|
||||
print(f" Location: {loaded_table.location()}")
|
||||
except NoSuchTableError:
|
||||
print(f" Failed to load table")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def test_cleanup(catalog, prefix):
|
||||
"""Test table and namespace deletion."""
|
||||
print("Testing cleanup operations...")
|
||||
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
|
||||
table_id = namespace + ("auth_test_table",)
|
||||
|
||||
# Drop table
|
||||
try:
|
||||
catalog.drop_table(table_id)
|
||||
print(f" Dropped table: {table_id}")
|
||||
except NoSuchTableError:
|
||||
print(f" ! Table already deleted: {table_id}")
|
||||
|
||||
# Drop namespace
|
||||
try:
|
||||
catalog.drop_namespace(namespace)
|
||||
print(f" Dropped namespace: {namespace}")
|
||||
except NoSuchNamespaceError:
|
||||
print(f" ! Namespace already deleted: {namespace}")
|
||||
except Exception as e:
|
||||
print(f" ? Namespace drop error (may be expected): {e}")
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Test Iceberg REST Catalog with authentication")
|
||||
parser.add_argument("--catalog-url", required=True, help="Iceberg REST Catalog URL")
|
||||
parser.add_argument("--warehouse", default="s3://iceberg-test/", help="Warehouse location")
|
||||
parser.add_argument("--prefix", required=True, help="Table bucket prefix")
|
||||
parser.add_argument("--access-key", required=True, help="AWS Access Key ID")
|
||||
parser.add_argument("--secret-key", required=True, help="AWS Secret Access Key")
|
||||
parser.add_argument("--skip-cleanup", action="store_true", help="Skip cleanup at the end")
|
||||
args = parser.parse_args()
|
||||
|
||||
print(f"Connecting to Iceberg REST Catalog at: {args.catalog_url}")
|
||||
print(f"Warehouse: {args.warehouse}")
|
||||
print(f"Prefix: {args.prefix}")
|
||||
print(f"Using authenticated access with key: {args.access_key}")
|
||||
print()
|
||||
|
||||
# Load the REST catalog with authentication
|
||||
import time
|
||||
max_retries = 10
|
||||
catalog = None
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
catalog = load_catalog(
|
||||
"rest",
|
||||
**{
|
||||
"type": "rest",
|
||||
"uri": args.catalog_url,
|
||||
"warehouse": args.warehouse,
|
||||
"prefix": args.prefix,
|
||||
"auth": {"type": "noop"},
|
||||
"s3.access-key-id": args.access_key,
|
||||
"s3.secret-access-key": args.secret_key,
|
||||
}
|
||||
)
|
||||
print(f"Successfully connected to catalog on attempt {attempt + 1}")
|
||||
break
|
||||
except Exception as e:
|
||||
if attempt < max_retries - 1:
|
||||
print(f" Attempt {attempt + 1} failed, retrying in 2s... ({e})")
|
||||
time.sleep(2)
|
||||
else:
|
||||
print(f" All {max_retries} attempts failed.")
|
||||
raise e
|
||||
|
||||
# Run tests
|
||||
tests = [
|
||||
("Config Endpoint", lambda: test_config_endpoint(catalog)),
|
||||
("Namespace Operations", lambda: test_namespace_operations(catalog, args.prefix)),
|
||||
("Table Operations", lambda: test_table_operations(catalog, args.prefix)),
|
||||
]
|
||||
|
||||
if not args.skip_cleanup:
|
||||
tests.append(("Cleanup", lambda: test_cleanup(catalog, args.prefix)))
|
||||
|
||||
passed = 0
|
||||
failed = 0
|
||||
|
||||
for name, test_fn in tests:
|
||||
print(f"\n{'='*50}")
|
||||
try:
|
||||
if test_fn():
|
||||
passed += 1
|
||||
print(f"PASSED: {name}")
|
||||
else:
|
||||
failed += 1
|
||||
print(f"FAILED: {name}")
|
||||
except Exception as e:
|
||||
failed += 1
|
||||
print(f"ERROR in {name}: {e}")
|
||||
|
||||
print(f"\n{'='*50}")
|
||||
print(f"Results: {passed} passed, {failed} failed")
|
||||
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user