snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

2026-09-16 03:23:42 +05:00
commit 89f5be005d
3997 changed files with 961970 additions and 0 deletions
--- a/test/s3tables/catalog_spark_lance/README.md
+++ b/test/s3tables/catalog_spark_lance/README.md
@@ -0,0 +1,63 @@
+# Spark Lance Integration Test
+
+Drives the SeaweedFS Lance Namespace with Spark, through the Lance Spark
+connector's DSV2 catalog. The Lance counterpart of `catalog_spark`, which does
+the same for the Iceberg REST catalog.
+
+## Why Spark
+
+Spark is the engine most likely to be pointed at a lakehouse, and it reaches the
+catalog over the same Lance Namespace routes as every other client. Between this
+and `catalog_lancedb`, the catalog is exercised by the two clients people
+actually use, rather than only by the protocol's reference implementation.
+
+## What it does
+
+`TestSparkLanceNamespace`:
+
+1. Starts a `weed mini` cluster with S3 and the Lance Namespace enabled.
+2. Creates a table bucket declared `LANCE`.
+3. Runs `spark_lance_ops.py` inside the stock `apache/spark:3.5.1` image, with
+   the connector pulled from Maven at submit time.
+
+Inside Spark:
+
+| Step | What it proves |
+| --- | --- |
+| `CREATE NAMESPACE` / `SHOW NAMESPACES` | the bucket is the first level of the path, and the catalog is writable |
+| `CREATE TABLE ... USING lance` | the connector declares through the namespace and writes the data itself |
+| `INSERT` / `SELECT count(*)` | a write lands and reads back through the catalog |
+| schema check | the vector column survived the round trip |
+| `WHERE id >= 2` | the filter path, not only a full scan |
+| a second `INSERT` | the commit a store that cannot order commits fails on |
+
+## Configuration
+
+The connector's catalog properties. The host names below are placeholders — the
+test passes dynamically allocated `host.docker.internal` ports, and a real
+deployment uses whatever address the gateway answers on:
+
+```
+spark.sql.catalog.lance                        org.lance.spark.LanceNamespaceSparkCatalog
+spark.sql.catalog.lance.impl                   rest
+spark.sql.catalog.lance.uri                    http://seaweed:9101
+spark.sql.catalog.lance.storage.aws_endpoint   http://seaweed:8333
+spark.sql.catalog.lance.storage.allow_http     true
+spark.sql.catalog.lance.storage.aws_access_key_id      …
+spark.sql.catalog.lance.storage.aws_secret_access_key  …
+```
+
+Anything under `storage.` is handed to lance as object_store options, so those
+are object_store's key names rather than Spark's `s3a` ones. Credentials belong
+there because a gateway without STS vends none — see the note in
+`../catalog_lancedb/README.md`.
+
+## Running it
+
+    cd test/s3tables/catalog_spark_lance
+    (cd ../../../weed && go build .)     # the harness runs this binary
+    go test -run TestSparkLanceNamespace -v -timeout 40m .
+
+Skipped without Docker, and in `-short` mode. The first run downloads the Spark
+image and the connector bundle, which is a few hundred megabytes; later runs
+reuse both.