snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)
Some checks failed
go: build dev binaries / cleanup (push) Successful in 4s
Codespell / Check for spelling errors (push) Failing after 3s
docker: build dev containers / build-rust-binaries (amd64, x86_64-unknown-linux-musl) (push) Failing after 3s
docker: build dev containers / build-dev-containers (push) Has been skipped
docker: build dev containers / build-rust-binaries (arm64, true, aarch64-unknown-linux-musl) (push) Failing after 3s
End to End / FUSE Mount (push) Failing after 2s
EC Integration Tests / EC Integration Tests (push) Failing after 30s
FUSE DLM Integration Tests / FUSE DLM Integration Tests (push) Failing after 3s
FUSE Volume Server Failover Tests / FUSE Volume Server Failover (push) Failing after 3s
FUSE Integration Tests / FUSE Integration Testing (push) Failing after 2s
FUSE P2P Peer Chunk Sharing Integration Tests / FUSE P2P Peer Chunk Sharing (push) Failing after 3s
go: build binary / Go Vet (push) Failing after 2s
go: build binary / Go Vet 32-bit (push) Failing after 3s
go: build binary / Build (push) Failing after 4s
go: build binary / Build cross-platform (freebsd/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (darwin/arm64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/arm64) (push) Failing after 3s
go: build binary / Test (push) Failing after 2s
go: build binary / Test 32-bit (push) Failing after 3s
helm: lint and test charts / lint-test (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (11) (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (17) (push) Failing after 3s
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Failing after 2s
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Failing after 3s
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been skipped
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been skipped
Java Client Unit Tests / Java Unit Tests (11) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (17) (push) Failing after 2s
Java Client Unit Tests / Java Unit Tests (21) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (8) (push) Failing after 3s
Kafka Quick Test (Load Test with Schema Registry) / Kafka Client Load Test (Quick) (push) Failing after 3s
Kafka Gateway Tests / Kafka Unit Tests (unit-tests-1) (push) Failing after 11s
Kafka Gateway Tests / Kafka Integration Tests (Critical) (integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka End-to-End Tests (with SMQ) (e2e-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Consumer Group Tests (Highly Isolated) (consumer-group-1) (push) Failing after 2s
Kafka Gateway Tests / Kafka Client Compatibility (with SMQ) (client-compat-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka SMQ Integration Tests (Full Stack) (smq-integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Protocol Tests (Isolated) (protocol-1) (push) Failing after 3s
KMS Tests / KMS Provider Integration Tests (push) Failing after 2s
KMS Tests / S3 KMS End-to-End Tests (push) Failing after 3s
Master Cold Start Tests / Master Cold Start Tests (push) Failing after 4s
Metadata Subscribe Integration Tests / Metadata Subscribe Integration Tests (push) Failing after 3s
Multi-Master Tests / Multi-Master Failover Tests (push) Failing after 3s
Performance / CPU and Heap Profile (push) Failing after 2s
Performance / S3 Read/Write Benchmark (rust volume) (push) Failing after 3s
pjdfstest POSIX Compliance / pjdfstest (push) Failing after 2s
Plugin Worker Integration Tests / Plugin Worker: erasure_coding (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: vacuum (push) Failing after 3s
Rust Volume Server Tests / Rust Integration Tests (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 1) (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 1) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 3) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Distributed Tests (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Performance Tests (push) Failing after 3s
Performance / Throughput Benchmark (go volume) (push) Failing after 3s
Performance / Throughput Benchmark (rust volume) (push) Failing after 2s
Performance / S3 Read/Write Benchmark (go volume) (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: volume_balance (push) Failing after 3s
PostgreSQL Gateway Tests / PostgreSQL Basic Tests (push) Failing after 3s
Rust Volume Server Tests / Rust Unit Tests (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 3) (push) Failing after 3s
Rust Plugin Worker Tests / Rust Plugin Worker Build and Unit Tests (push) Failing after 3s
rust: build dev volume server binaries / cleanup (push) Successful in 3s
S3 ETag and ACL Tests / S3 ETag + ACL Integration Tests (push) Failing after 3s
S3 IAM Integration Tests / IAM Unit Tests (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (advanced) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (basic) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (group) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (policy-enforcement) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (sts) (push) Failing after 4s
S3 Keycloak Integration Tests / S3 Keycloak Integration Tests (push) Failing after 3s
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.11) (push) Failing after 3s
S3 PyArrow Parquet Tests / Go Unit Tests (Implicit Directory) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.12) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-amd64, x86_64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-arm64, aarch64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-linux (linux-amd64, x86_64-unknown-linux-gnu) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.9) (push) Has been cancelled
S3 Proxy Signature Tests / S3 Proxy Signature Verification Tests (push) Has been cancelled
S3 SDK V2 Route Disambiguation Tests / S3 SDK V2 Routing Tests (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (quick) (push) Has been cancelled
S3 SSE Tests / S3 SSE Compatibility Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Metadata Persistence Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Copy Operations Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Multipart Upload Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Performance Test (push) Has been cancelled
S3 SSE Tests / S3 Volume Encryption Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (comprehensive) (push) Has been cancelled
Spark Integration Tests / Spark Integration Tests (push) Failing after 2s
Telemetry Integration Tests / telemetry-integration-test (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 1) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 1) (push) Failing after 3s
Ceph S3 tests / Basic S3 tests (KV store) (push) Has been cancelled
Ceph S3 tests / S3 Versioning & Object Lock tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Unit Tests (push) Failing after 3s
Ceph S3 tests / S3 CORS tests (push) Has been cancelled
Ceph S3 tests / SeaweedFS Custom S3 Copy tests (push) Has been cancelled
Ceph S3 tests / Basic S3 tests (SQL store) (push) Has been cancelled
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
TLS Rotation Integration Tests / TLS Rotation Integration Tests (push) Has been cancelled
Vacuum Integration Tests / Vacuum Integration Tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Variables Integration Tests (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (advanced-policy) (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (basic-policy) (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (contextual-policy) (push) Failing after 4s
S3 Policy Integration Tests / Trusted Proxy Detection Tests (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Shell Integration Tests (push) Failing after 3s
Samba on FUSE Integration / samba-integration (push) Failing after 2s
SFTP Integration Tests / SFTP Integration Testing (push) Failing after 3s
terraform: validate and test modules / fmt, validate, plan-level tests (push) Failing after 3s
terraform: validate and test modules / local cluster smoke test (real weed) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 3) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 2) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 2) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 3) (push) Failing after 5s
mount: windows conformance / WinFsp conformance (push) Has been cancelled
mount: windows / Mount on Windows (push) Has been cancelled

This commit is contained in:
upstream-mirror
2026-09-16 03:23:42 +05:00
commit 89f5be005d
3997 changed files with 961970 additions and 0 deletions

View File

@@ -0,0 +1,15 @@
# Seeds a Lance dataset so DuckDB has something to read. The catalog records
# where a table lives and does not carry its data, so the writing half is
# pylance; DuckDB then reads it over S3 with no catalog involved.
FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir \
"lance-namespace==0.8.6" \
"pylance==10.0.0" \
"pyarrow==25.0.1"
COPY seed_table.py /app/
CMD ["python3", "/app/seed_table.py", "--help"]

View File

@@ -0,0 +1,66 @@
# DuckDB Lance Integration Test
Reads SeaweedFS Lance tables from DuckDB's `lance` core extension, the
counterpart of the DuckDB Iceberg tests in `../catalog/`.
## What this one proves that the others do not
The LanceDB and Spark suites go through the catalog. DuckDB does not: it reaches
the data over S3 with no namespace involved. That exercises the other half of the
design — a table bucket's layout is a valid Lance dataset directory, so a table
stays readable when the catalog is not in the path.
## What it does
`TestDuckDBLance`:
1. Starts a `weed mini` cluster with S3 and the Lance Namespace enabled.
2. Creates a table bucket declared `LANCE`, then declares a table through the
namespace and writes a dataset into it with pylance.
3. Runs `duckdb_lance_ops.sql` in `duckdb/duckdb:latest`.
| Step | What it proves |
| --- | --- |
| `__lance_scan(s3://…)` | DuckDB reads a table this catalog created |
| `DESCRIBE` | the schema survived, vector column included |
| `WHERE id < 5` | the filter path |
| `lance_vector_search` | vector search over data behind SeaweedFS |
| a `.lance` path | the replacement scan works on a suffixed path |
| a suffix-less path | and does **not** see one without the suffix |
## The `.lance` suffix
DuckDB's replacement scan — `SELECT * FROM 's3://…'` — recognises a Lance dataset
by a `.lance` path suffix. Tables created through this catalog deliberately have
none: the catalog entry *is* the dataset directory, table names may not contain a
dot, and a suffix would leak into ARNs and policies.
So from DuckDB, a table in a SeaweedFS table bucket is read with
`__lance_scan('s3://bucket/namespace/table')` rather than the bare `FROM 's3://…'`
form. The test asserts both halves, so if the extension ever recognises a
suffix-less directory the test fails and tells us to update the documentation.
## Credentials
```sql
CREATE SECRET seaweedfs (
TYPE lance,
ACCESS_KEY_ID '…', SECRET_ACCESS_KEY '…',
REGION 'us-east-1',
ENDPOINT 'http://seaweed:8333',
ALLOW_HTTP true,
VIRTUAL_HOSTED_STYLE_REQUEST false
);
```
Those are object_store's key names, the same as everywhere else Lance touches
storage.
## Running it
cd test/s3tables/catalog_duckdb_lance
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestDuckDBLance -v -timeout 30m .
Skipped without Docker, in `-short` mode, and if the DuckDB image cannot load the
extension.

View File

@@ -0,0 +1,48 @@
-- Read a SeaweedFS Lance table from DuckDB.
--
-- The lance extension reaches the data over S3 rather than through the
-- namespace, so what is exercised here is the layout on the S3 door and the
-- credentials, not the catalog protocol. That is the point: this is the client
-- that proves a Lance table stays readable without a catalog at all.
--
-- Placeholders are substituted by the Go harness: __ENDPOINT__, __KEY__,
-- __SECRET__, __TABLE__ (s3://bucket/ns/table), __SUFFIXED__ (the same data at a
-- path ending in .lance).
INSTALL lance;
LOAD lance;
CREATE SECRET seaweedfs (
TYPE lance,
ACCESS_KEY_ID '__KEY__',
SECRET_ACCESS_KEY '__SECRET__',
REGION 'us-east-1',
ENDPOINT '__ENDPOINT__',
ALLOW_HTTP true,
VIRTUAL_HOSTED_STYLE_REQUEST false
);
-- 1. The table the catalog created, read by URI. A table bucket's layout is a
-- valid Lance dataset directory, which is what makes this possible.
SELECT 'scan_rows=' || count(*) FROM __lance_scan('__TABLE__');
-- 2. Its schema survived, vector column included. Ordered, because string_agg
-- over an unordered relation is free to return the names in any order.
SELECT 'scan_columns=' || string_agg(column_name, ',' ORDER BY column_name)
FROM (DESCRIBE SELECT * FROM __lance_scan('__TABLE__'));
-- 3. A filter, so it is not only a full scan.
SELECT 'filtered_rows=' || count(*) FROM __lance_scan('__TABLE__') WHERE id < 5;
-- 4. Vector search, which is what the format is for. No index is built here, so
-- this is a brute-force search; the ids nearest the query are what matters.
SELECT 'nearest=' || string_agg(id::VARCHAR, ',' ORDER BY _distance ASC, id ASC)
FROM lance_vector_search('__TABLE__', 'vector',
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0], k := 3);
-- 5. DuckDB's replacement scan recognises a path by its .lance suffix. Tables
-- this catalog creates deliberately have no suffix - the name is the table's
-- name, and a suffix would leak into ARNs and policies - so the bare
-- SELECT ... FROM 's3://…' form does not see them, and __lance_scan is the
-- way in. This asserts both halves so that a change upstream is noticed.
SELECT 'suffixed_rows=' || count(*) FROM '__SUFFIXED__';

View File

@@ -0,0 +1,390 @@
// Package duckdblance reads SeaweedFS Lance tables from DuckDB's lance
// extension, the counterpart of the DuckDB Iceberg tests next door.
//
// DuckDB reaches the data over S3 rather than through the namespace, so what
// this proves is the other half of the design: a table bucket's layout is a
// valid Lance dataset directory, and a table stays readable with no catalog in
// the path at all. It also pins the one place that costs us - DuckDB's
// replacement scan recognises a dataset by its .lance suffix, which tables
// created through this catalog deliberately do not have.
package duckdblance
import (
"context"
"fmt"
"math/rand"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/test/testutil"
)
const (
// Tracks whatever DuckDB users are actually running; the lance extension is
// a core extension, so there is nothing to pin beyond the image.
duckDBImage = "duckdb/duckdb:latest"
// Seeds a dataset for DuckDB to read.
seedImage = "seaweedfs-lance-seed"
startupTimeout = 60 * time.Second
clientTimeout = 15 * time.Minute
seededRows = 128
)
// TestDuckDBLance reads a table this catalog created, from DuckDB.
func TestDuckDBLance(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
if !hasDocker() {
t.Skip("Docker not available, skipping DuckDB Lance integration test")
}
env := newEnvironment(t)
defer env.cleanup()
env.start(t)
bucket := "duckdb-" + randomSuffix()
env.createTableBucket(t, bucket)
requireDuckDBLance(t)
buildSeedImage(t)
env.seedTable(t, bucket)
env.runDuckDB(t, bucket)
}
type environment struct {
weedBinary string
dataDir string
bindIP string
masterPort int
masterGrpcPort int
volumePort int
volumeGrpcPort int
filerPort int
filerGrpcPort int
s3Port int
s3GrpcPort int
lancePort int
accessKey string
secretKey string
weedCancel context.CancelFunc
weedCmd *exec.Cmd
}
func newEnvironment(t *testing.T) *environment {
t.Helper()
wd, err := os.Getwd()
if err != nil {
t.Fatalf("get working directory: %v", err)
}
seaweedDir := wd
for i := 0; i < 5; i++ {
if _, err := os.Stat(filepath.Join(seaweedDir, "go.mod")); err == nil {
break
}
seaweedDir = filepath.Dir(seaweedDir)
}
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
if info, statErr := os.Stat(weedBinary); statErr == nil && !info.IsDir() {
// `make test` builds first; a plain `go test` will otherwise drive a
// binary from days ago and report a pass for code it never ran.
t.Logf("using %s, built %s", weedBinary, info.ModTime().Format(time.RFC3339))
} else {
weedBinary = "weed"
if _, err := exec.LookPath(weedBinary); err != nil {
t.Skip("weed binary not found, skipping integration test")
}
}
dataDir, err := os.MkdirTemp("", "seaweed-duckdb-lance-test-*")
if err != nil {
t.Fatalf("create temp dir: %v", err)
}
ports := testutil.MustAllocatePorts(t, 9)
return &environment{
weedBinary: weedBinary,
dataDir: dataDir,
bindIP: testutil.FindBindIP(),
masterPort: ports[0],
masterGrpcPort: ports[1],
volumePort: ports[2],
volumeGrpcPort: ports[3],
filerPort: ports[4],
filerGrpcPort: ports[5],
s3Port: ports[6],
s3GrpcPort: ports[7],
lancePort: ports[8],
accessKey: "AKIAIOSFODNN7EXAMPLE",
secretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
}
}
func (env *environment) start(t *testing.T) {
t.Helper()
iamConfigPath, err := testutil.WriteIAMConfig(env.dataDir, env.accessKey, env.secretKey)
if err != nil {
t.Fatalf("write IAM config: %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
env.weedCancel = cancel
cmd := exec.CommandContext(ctx, env.weedBinary, "mini",
"-master.port", fmt.Sprintf("%d", env.masterPort),
"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
"-volume.port", fmt.Sprintf("%d", env.volumePort),
"-volume.port.grpc", fmt.Sprintf("%d", env.volumeGrpcPort),
"-filer.port", fmt.Sprintf("%d", env.filerPort),
"-filer.port.grpc", fmt.Sprintf("%d", env.filerGrpcPort),
"-s3.port", fmt.Sprintf("%d", env.s3Port),
"-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort),
"-s3.port.lance", fmt.Sprintf("%d", env.lancePort),
"-s3.config", iamConfigPath,
"-ip", env.bindIP,
"-ip.bind", "0.0.0.0",
"-dir", env.dataDir,
)
cmd.Dir = env.dataDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Env = append(os.Environ(),
"AWS_ACCESS_KEY_ID="+env.accessKey,
"AWS_SECRET_ACCESS_KEY="+env.secretKey,
)
if err := cmd.Start(); err != nil {
t.Fatalf("start SeaweedFS: %v", err)
}
env.weedCmd = cmd
// The namespace answers /v1/table once it is serving, which is a cheaper
// readiness check than waiting on a bucket that does not exist yet.
url := fmt.Sprintf("http://%s:%d/v1/table", env.bindIP, env.lancePort)
if !waitForHTTP(url, startupTimeout) {
t.Fatalf("the Lance namespace did not become ready at %s", url)
}
}
// waitForHTTP polls until the URL answers at all. An auth refusal counts: it
// means the server is up, which is the only thing being waited on.
func waitForHTTP(url string, timeout time.Duration) bool {
client := &http.Client{Timeout: 2 * time.Second}
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
resp, err := client.Get(url)
if err != nil {
time.Sleep(500 * time.Millisecond)
continue
}
status := resp.StatusCode
resp.Body.Close()
if status < 500 {
return true
}
time.Sleep(500 * time.Millisecond)
}
return false
}
func (env *environment) cleanup() {
if env.weedCancel != nil {
env.weedCancel()
}
if env.weedCmd != nil {
_ = env.weedCmd.Wait()
}
if env.dataDir != "" {
_ = os.RemoveAll(env.dataDir)
}
}
// createTableBucket makes the bucket LanceDB will read through, declared LANCE
// so the catalog refuses anything of another format in it.
func (env *environment) createTableBucket(t *testing.T, bucket string) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, env.weedBinary, "shell",
fmt.Sprintf("-master=%s:%d.%d", env.bindIP, env.masterPort, env.masterGrpcPort),
)
cmd.Stdin = strings.NewReader(fmt.Sprintf(
"s3tables.bucket -create -name %s -format LANCE -account 000000000000\nexit\n", bucket))
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("create table bucket %s: %v\n%s", bucket, err, out)
}
// weed shell reports a command's own failure on stdout and still exits 0, so
// the exit code alone would let a missing bucket through and turn a setup
// failure into a confusing engine failure later.
if !env.tableBucketExists(t, bucket) {
t.Fatalf("table bucket %s was not created:\n%s", bucket, out)
}
t.Logf("created LANCE table bucket %s", bucket)
}
// tableBucketExists asks the namespace, which lists table buckets at its root.
func (env *environment) tableBucketExists(t *testing.T, bucket string) bool {
t.Helper()
url := fmt.Sprintf("http://%s:%d/v1/namespace/%s/exists", env.bindIP, env.lancePort, bucket)
resp, err := http.Post(url, "application/json", strings.NewReader("{}"))
if err != nil {
t.Fatalf("ask the namespace whether %s exists: %v", bucket, err)
}
defer resp.Body.Close()
return resp.StatusCode == http.StatusOK
}
// runSpark runs the SQL driver inside the stock Spark image. The connector is
// pulled from Maven at submit time, the way the Iceberg Spark suite pulls its
// runtime, so nothing has to be built here.
// requireDuckDBLance skips rather than fails when the image cannot load the
// extension, so a DuckDB build without it reads as "not applicable" rather than
// as a broken catalog.
func requireDuckDBLance(t *testing.T) {
t.Helper()
const ready = "lance extension ready"
cmd := exec.Command("docker", "run", "--rm", "--entrypoint", "duckdb", duckDBImage,
"-c", fmt.Sprintf("INSTALL lance; LOAD lance; SELECT '%s' AS marker;", ready))
out, err := cmd.CombinedOutput()
if err != nil || !strings.Contains(string(out), ready) {
t.Skipf("DuckDB image cannot load the lance extension: %v\n%s", err, out)
}
}
func buildSeedImage(t *testing.T) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
cmd := exec.CommandContext(ctx, "docker", "build", "-t", seedImage, "-f", "Dockerfile.seed", ".")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("build the seed image: %v\n%s", err, out)
}
}
// seedTable declares a table through the namespace and writes a dataset into
// it, which is the split the catalog serves: it records where a table lives and
// does not carry its data.
func (env *environment) seedTable(t *testing.T, bucket string) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), clientTimeout)
defer cancel()
cmd := exec.CommandContext(ctx, "docker", "run", "--rm",
"--add-host", "host.docker.internal:host-gateway",
"-e", "AWS_ACCESS_KEY_ID="+env.accessKey,
"-e", "AWS_SECRET_ACCESS_KEY="+env.secretKey,
"-e", "AWS_REGION=us-east-1",
seedImage,
"python3", "/app/seed_table.py",
"--namespace-url", fmt.Sprintf("http://host.docker.internal:%d", env.lancePort),
"--s3-endpoint", fmt.Sprintf("http://host.docker.internal:%d", env.s3Port),
"--bucket", bucket,
"--rows", fmt.Sprintf("%d", seededRows),
"--access-key", env.accessKey,
"--secret-key", env.secretKey,
)
out, err := cmd.CombinedOutput()
t.Logf("seeder:\n%s", out)
if err != nil {
t.Fatalf("seeding the table failed: %v", err)
}
}
// runDuckDB substitutes the endpoint and paths into the SQL and runs it.
func (env *environment) runDuckDB(t *testing.T, bucket string) {
t.Helper()
sqlBytes, err := os.ReadFile("duckdb_lance_ops.sql")
if err != nil {
t.Fatalf("read the SQL: %v", err)
}
table := fmt.Sprintf("s3://%s/ml/embeddings", bucket)
sql := strings.NewReplacer(
"__ENDPOINT__", fmt.Sprintf("http://host.docker.internal:%d", env.s3Port),
"__KEY__", env.accessKey,
"__SECRET__", env.secretKey,
"__TABLE__", table,
"__SUFFIXED__", table+"-direct.lance",
).Replace(string(sqlBytes))
ctx, cancel := context.WithTimeout(context.Background(), clientTimeout)
defer cancel()
cmd := exec.CommandContext(ctx, "docker", "run", "--rm", "-i",
"--add-host", "host.docker.internal:host-gateway",
"--entrypoint", "duckdb", duckDBImage, "-c", sql)
out, err := cmd.CombinedOutput()
t.Logf("DuckDB output:\n%s", out)
if err != nil {
t.Fatalf("the DuckDB query failed: %v", err)
}
output := string(out)
for _, want := range []string{
fmt.Sprintf("scan_rows=%d", seededRows),
"scan_columns=id,title,vector",
"filtered_rows=5",
"nearest=1,0,2",
fmt.Sprintf("suffixed_rows=%d", seededRows),
} {
if !strings.Contains(output, want) {
t.Fatalf("DuckDB did not report %q", want)
}
}
// The other half of the suffix rule: a table this catalog created has no
// .lance suffix, so DuckDB's replacement scan does not see it and
// __lance_scan is the way in. If this ever starts working, the docs saying
// otherwise are wrong.
bare := exec.CommandContext(ctx, "docker", "run", "--rm", "-i",
"--add-host", "host.docker.internal:host-gateway",
"--entrypoint", "duckdb", duckDBImage,
"-c", fmt.Sprintf("INSTALL lance; LOAD lance; SELECT count(*) FROM '%s';", table))
bareOut, bareErr := bare.CombinedOutput()
// DuckDB exits nonzero for any error, so the exit status alone does not tell
// "the replacement scan refused the path" from "the query never ran": require
// the catalog error either way.
if !strings.Contains(string(bareOut), "does not exist") {
t.Fatalf("a suffix-less path did not fail the way the docs say it does (%v); "+
"if the replacement scan now reads it, update them:\n%s", bareErr, bareOut)
}
t.Logf("a suffix-less path is not seen by the replacement scan, as expected")
}
// hasDocker reports whether a Docker daemon answers. Bounded, because an
// unhealthy daemon makes `docker version` hang, and this runs before the test
// has a timeout of its own: better to skip than to eat the whole budget.
func hasDocker() bool {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
return exec.CommandContext(ctx, "docker", "version").Run() == nil
}
func randomSuffix() string {
const charset = "abcdefghijklmnopqrstuvwxyz0123456789"
suffix := make([]byte, 8)
for i := range suffix {
suffix[i] = charset[rand.Intn(len(charset))]
}
return string(suffix)
}

View File

@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""Declare a table through the namespace and write a dataset into it.
Also writes the same rows to a second path whose name ends in `.lance`, because
DuckDB's replacement scan recognises a dataset by that suffix and tables created
through this catalog do not have one. The test asserts both behaviours so a
change upstream is noticed rather than silently making the docs wrong.
"""
import argparse
import sys
import warnings
warnings.filterwarnings("ignore")
import lance
import lance_namespace as ln
import pyarrow as pa
DIM = 8
def rows(count):
return pa.table(
{
"id": pa.array(list(range(count)), type=pa.int64()),
"title": pa.array([f"row-{i}" for i in range(count)]),
"vector": pa.array(
[[float(i) + d for d in range(DIM)] for i in range(count)],
type=pa.list_(pa.float32(), DIM),
),
}
)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--namespace-url", required=True)
parser.add_argument("--s3-endpoint", required=True)
parser.add_argument("--bucket", required=True)
parser.add_argument("--namespace", default="ml")
parser.add_argument("--table", default="embeddings")
parser.add_argument("--rows", type=int, default=128)
parser.add_argument("--access-key", default="any")
parser.add_argument("--secret-key", default="any")
args = parser.parse_args()
storage = {
"aws_endpoint": args.s3_endpoint,
"allow_http": "true",
"aws_access_key_id": args.access_key,
"aws_secret_access_key": args.secret_key,
"aws_region": "us-east-1",
}
ns = ln.connect("rest", {"uri": args.namespace_url})
ns.create_namespace(ln.CreateNamespaceRequest(id=[args.bucket], mode="EXIST_OK"))
ns.create_namespace(
ln.CreateNamespaceRequest(id=[args.bucket, args.namespace], mode="EXIST_OK")
)
declared = ns.declare_table(
ln.DeclareTableRequest(id=[args.bucket, args.namespace, args.table])
)
lance.write_dataset(rows(args.rows), declared.location, storage_options=storage,
mode="overwrite")
print(f"seeded {args.rows} rows at {declared.location}")
# The same data under a name DuckDB's replacement scan recognises. Written
# directly rather than declared, because a table name containing a dot is
# not a valid catalog name.
suffixed = f"s3://{args.bucket}/{args.namespace}/{args.table}-direct.lance"
lance.write_dataset(rows(args.rows), suffixed, storage_options=storage,
mode="overwrite")
print(f"seeded {args.rows} rows at {suffixed}")
return 0
if __name__ == "__main__":
sys.exit(main())