One table, all the way through: created in the catalog, filled by a real client,
maintained, read again, dropped. Once for Iceberg and once for Lance. The
Iceberg half always maintains through the worker; the Lance half maintains
through the Rust worker or through the lance library, depending on what the
environment has - see below.
Why this suite exists
#10853 was a compaction
that rewrote every dictionary-encoded column onto a single value. It shipped.
The maintenance tests we had were thorough about the bookkeeping - sequence
numbers, added and deleted manifest entries, metadata versions, the manifest
list - and every one of them passed, because not one of them opened the parquet
file the worker had just written.
So the assertion here is the dull one nothing else was making: tally the table
before maintenance, tally it again after, and require the two to be equal. The
tally is a row count, the cardinality of each dictionary-encoded column, and an
md5 over whole rows. The cardinalities name the failure that happened; the
digest catches a rewrite that keeps every column's cardinality and hands the
values to the wrong rows.
The same shape covers Lance, because the exposure is the same: a compaction that
merges fragments can hand back a table that reads without complaint and answers
wrongly.
What runs
TestIcebergTableLifecycle starts a weed mini cluster, declares an ICEBERG
table bucket, and runs two clients against it:
Client
Why both
DuckDB
the client the bug was reported against, and the only one here that writes the deprecated PLAIN_DICTIONARY encoding - parquet-go normalizes it away on write, so a Go writer cannot produce it
PyIceberg
writes RLE_DICTIONARY, the modern spelling, so between the two the merge is checked against both dictionary encodings in the spec
Between the write and the read, the test runs the worker's whole maintenance
cycle in-process against the live filer: compact, expire snapshots, remove
orphans, rewrite manifests. A compaction that merged nothing fails the test
rather than passing it - otherwise the read afterwards is checking a file the
worker never wrote.
TestLanceTableLifecycle does the same against a LANCE bucket: declare
through the namespace, write a fragment per append, maintain, read, drop.
Maintenance goes through the Rust worker's own handlers where cargo is
installed, and through the two lance calls those handlers wrap where it is not.
WEED_LANCE_MAINTENANCE=library|worker picks one instead of letting the test
guess; CI sets library, because a cold build of the lance crate costs more
than the layer it would be checking.
Both tests finish by dropping the table and checking the data actually left the
filer, which is the half of a lifecycle a catalog test never reaches.
Running it
cd test/s3tables/lifecycle
(cd ../../../weed && go build .) # the harness runs this binary
go test -v -timeout 40m .
Skipped without Docker and in -short mode. The first run builds the two client
images and pulls duckdb/duckdb:latest; later runs reuse them. The DuckDB half
skips itself, rather than failing, on an image whose iceberg extension cannot
write through a REST catalog.
To watch it catch the bug it was written for, pin parquet-go back to the version
that had it:
go mod edit -require=github.com/parquet-go/parquet-go@v0.30.1 && go mod tidy
go test -run TestIcebergTableLifecycle/DuckDB -v .
maintenance collapsed the category column: 7 distinct values -> 1
maintenance collapsed the value column: 13 distinct values -> 1
The PyIceberg half still passes there, which is the reason both clients are in
this directory.
89f5be005d