snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

test/s3tables/catalog_dremio/Dockerfile.writer

@@ -0,0 +1,12 @@
+# Iceberg writer used by the Dremio integration test to populate a table with
+# data files via PyIceberg, so the downstream Dremio SELECT verifies the read
+# path against non-empty results rather than just COUNT(*) = 0.
+FROM python:3.11-slim
+
+WORKDIR /app
+
+RUN pip install --no-cache-dir "pyiceberg[s3fs]" pyarrow
+
+COPY append_rows.py /app/
+
+ENTRYPOINT ["python3", "/app/append_rows.py"]

test/s3tables/catalog_dremio/README.md

@@ -0,0 +1,55 @@
+# Dremio Iceberg Catalog Integration Test
+
+This directory contains a Dremio integration smoke test for SeaweedFS's Iceberg REST Catalog implementation.
+
+## What It Tests
+
+`TestDremioIcebergCatalog` verifies the Dremio path end to end:
+
+1. Starts a local SeaweedFS mini cluster with S3 Tables and Iceberg REST enabled.
+2. Creates a SeaweedFS table bucket.
+3. Creates an Iceberg namespace and empty table through the SeaweedFS REST catalog OAuth flow.
+4. Starts `dremio/dremio-oss:25.2.0`.
+5. Bootstraps a Dremio admin user and logs in.
+6. Creates a Dremio `RESTCATALOG` source that points at the SeaweedFS catalog.
+7. Submits Dremio SQL through `/api/v3/sql`, polls the job API, and reads job results.
+8. Runs subtests against the SeaweedFS-backed Iceberg table:
+   - `BasicSelect`: Dremio is alive and answering SQL.
+   - `CountEmptyTable`: catalog-to-table resolution and a scan of an empty table.
+   - `ColumnProjection`: `SELECT id, label` succeeds and the response schema reports both columns. Failure here means Dremio could not parse the schema returned by the SeaweedFS catalog.
+   - `InformationSchemaColumns`: the table's columns are exposed through Dremio's metadata layer with the expected ordinal order.
+   - `InformationSchemaTables`: the table is registered in Dremio's `INFORMATION_SCHEMA`.
+   - `MultiLevelNamespace`: a 2-level Iceberg namespace (created via the REST API) is exposed by Dremio as nested folders, and a table inside it is queryable with dot-separated identifiers.
+   - `ReadWrittenDataCount` and `ReadWrittenDataValues`: a separate table is populated with three rows by a PyIceberg writer container (`Dockerfile.writer` + `append_rows.py`) before Dremio bootstraps; Dremio reads the data back and the values are verified. This exercises the actual data path, not just metadata.
+
+The PyIceberg writer image is built on demand via Docker layer caching. The first build pulls `python:3.11-slim` and pip-installs PyIceberg + PyArrow (~1-2 min in CI); subsequent invocations are cheap.
+
+## Running Locally
+
+Build or install `weed`, then run:
+
+```bash
+cd test/s3tables/catalog_dremio
+go test -v -timeout 20m .
+```
+
+The test requires Docker. The GitHub Actions job runs on `ubuntu-22.04` and executes the test for pull requests.
+
+## Configuration
+
+The test uses these fixed credentials for the local SeaweedFS IAM config:
+
+- S3 access key: `AKIAIOSFODNN7EXAMPLE`
+- S3 secret key: `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`
+- Region: `us-west-2`
+- Warehouse bucket: `iceberg-tables`
+
+The Dremio source is configured via `POST /api/v3/catalog`; it is not configured in `dremio.conf`.
+The Dremio container starts with the `plugins.restcatalog.enabled` support key enabled, which is required for the Iceberg REST Catalog source in Dremio OSS 25.2.
+
+## Troubleshooting
+
+- Ensure Docker is running: `docker version`
+- Ensure `weed` is built or available on `PATH`
+- Check host-gateway routing if Dremio cannot reach SeaweedFS: `docker run --add-host host.docker.internal:host-gateway --rm alpine getent hosts host.docker.internal`
+- Check Dremio logs from the failed test output; the harness prints the Dremio container tail on Dremio startup, source setup, or job failures.

test/s3tables/catalog_dremio/append_rows.py

@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Append rows to an existing Iceberg table via the SeaweedFS REST catalog.
+
+Used by the Dremio integration test to materialize data files so a downstream
+SELECT from Dremio verifies the read path against non-empty results.
+
+Usage:
+    python3 append_rows.py \\
+        --catalog-url http://localhost:8181 \\
+        --warehouse s3://my-bucket \\
+        --prefix my-bucket \\
+        --s3-endpoint http://localhost:8333 \\
+        --access-key AKIA... --secret-key wJalr... \\
+        --namespace foo --namespace bar \\
+        --table events
+"""
+
+import argparse
+import sys
+
+import pyarrow as pa
+from pyiceberg.catalog import load_catalog
+
+
+def main() -> int:
+    p = argparse.ArgumentParser()
+    p.add_argument("--catalog-url", required=True)
+    p.add_argument("--warehouse", required=True, help="s3://<bucket-name>")
+    p.add_argument("--prefix", required=True, help="REST catalog prefix (table bucket name)")
+    p.add_argument("--s3-endpoint", required=True, help="http://host:port")
+    p.add_argument("--access-key", required=True)
+    p.add_argument("--secret-key", required=True)
+    p.add_argument("--region", default="us-east-1")
+    p.add_argument(
+        "--namespace",
+        action="append",
+        required=True,
+        help="One per level (e.g. --namespace foo --namespace bar for foo.bar).",
+    )
+    p.add_argument("--table", required=True)
+    args = p.parse_args()
+
+    # `credential` triggers OAuth2 client_credentials against
+    # <catalog_uri>/v1/oauth/tokens, matching the helper Go test uses for
+    # REST-API table creation. The s3.* keys are needed for parquet writes.
+    catalog = load_catalog(
+        "rest",
+        **{
+            "type": "rest",
+            "uri": args.catalog_url,
+            "warehouse": args.warehouse,
+            "prefix": args.prefix,
+            "credential": f"{args.access_key}:{args.secret_key}",
+            "s3.access-key-id": args.access_key,
+            "s3.secret-access-key": args.secret_key,
+            "s3.endpoint": args.s3_endpoint,
+            "s3.region": args.region,
+            "s3.path-style-access": "true",
+        },
+    )
+
+    table_id = tuple(args.namespace) + (args.table,)
+    table = catalog.load_table(table_id)
+
+    # Match the Iceberg table schema: id is `required long`, label is
+    # `optional string`. Default pyarrow columns are nullable, which fails
+    # PyIceberg's required-field compatibility check.
+    arrow_schema = pa.schema(
+        [
+            pa.field("id", pa.int64(), nullable=False),
+            pa.field("label", pa.string(), nullable=True),
+        ]
+    )
+    arrow_table = pa.Table.from_pydict(
+        {
+            "id": [1, 2, 3],
+            "label": ["one", "two", "three"],
+        },
+        schema=arrow_schema,
+    )
+    table.append(arrow_table)
+    print(f"appended {arrow_table.num_rows} rows to {'.'.join(table_id)}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())