upstream-seaweedfs/test/s3tables/catalog_spark_lance/README.md

Spark Lance Integration Test

Drives the SeaweedFS Lance Namespace with Spark, through the Lance Spark connector's DSV2 catalog. The Lance counterpart of catalog_spark, which does the same for the Iceberg REST catalog.

Why Spark

Spark is the engine most likely to be pointed at a lakehouse, and it reaches the catalog over the same Lance Namespace routes as every other client. Between this and catalog_lancedb, the catalog is exercised by the two clients people actually use, rather than only by the protocol's reference implementation.

What it does

TestSparkLanceNamespace:

1. Starts a weed mini cluster with S3 and the Lance Namespace enabled.
2. Creates a table bucket declared LANCE.
3. Runs spark_lance_ops.py inside the stock apache/spark:3.5.1 image, with the connector pulled from Maven at submit time.

Inside Spark:

Step	What it proves
CREATE NAMESPACE / SHOW NAMESPACES	the bucket is the first level of the path, and the catalog is writable
CREATE TABLE ... USING lance	the connector declares through the namespace and writes the data itself
INSERT / SELECT count(*)	a write lands and reads back through the catalog
schema check	the vector column survived the round trip
WHERE id >= 2	the filter path, not only a full scan
a second INSERT	the commit a store that cannot order commits fails on

Configuration

The connector's catalog properties. The host names below are placeholders — the test passes dynamically allocated host.docker.internal ports, and a real deployment uses whatever address the gateway answers on:

spark.sql.catalog.lance                        org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.lance.impl                   rest
spark.sql.catalog.lance.uri                    http://seaweed:9101
spark.sql.catalog.lance.storage.aws_endpoint   http://seaweed:8333
spark.sql.catalog.lance.storage.allow_http     true
spark.sql.catalog.lance.storage.aws_access_key_id      …
spark.sql.catalog.lance.storage.aws_secret_access_key  …

Anything under storage. is handed to lance as object_store options, so those are object_store's key names rather than Spark's s3a ones. Credentials belong there because a gateway without STS vends none — see the note in ../catalog_lancedb/README.md.

Running it

cd test/s3tables/catalog_spark_lance
(cd ../../../weed && go build .)     # the harness runs this binary
go test -run TestSparkLanceNamespace -v -timeout 40m .

Skipped without Docker, and in -short mode. The first run downloads the Spark image and the connector bundle, which is a few hundred megabytes; later runs reuse both.