snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)
Some checks failed
go: build dev binaries / cleanup (push) Successful in 4s
Codespell / Check for spelling errors (push) Failing after 3s
docker: build dev containers / build-rust-binaries (amd64, x86_64-unknown-linux-musl) (push) Failing after 3s
docker: build dev containers / build-dev-containers (push) Has been skipped
docker: build dev containers / build-rust-binaries (arm64, true, aarch64-unknown-linux-musl) (push) Failing after 3s
End to End / FUSE Mount (push) Failing after 2s
EC Integration Tests / EC Integration Tests (push) Failing after 30s
FUSE DLM Integration Tests / FUSE DLM Integration Tests (push) Failing after 3s
FUSE Volume Server Failover Tests / FUSE Volume Server Failover (push) Failing after 3s
FUSE Integration Tests / FUSE Integration Testing (push) Failing after 2s
FUSE P2P Peer Chunk Sharing Integration Tests / FUSE P2P Peer Chunk Sharing (push) Failing after 3s
go: build binary / Go Vet (push) Failing after 2s
go: build binary / Go Vet 32-bit (push) Failing after 3s
go: build binary / Build (push) Failing after 4s
go: build binary / Build cross-platform (freebsd/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (darwin/arm64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/arm64) (push) Failing after 3s
go: build binary / Test (push) Failing after 2s
go: build binary / Test 32-bit (push) Failing after 3s
helm: lint and test charts / lint-test (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (11) (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (17) (push) Failing after 3s
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Failing after 2s
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Failing after 3s
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been skipped
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been skipped
Java Client Unit Tests / Java Unit Tests (11) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (17) (push) Failing after 2s
Java Client Unit Tests / Java Unit Tests (21) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (8) (push) Failing after 3s
Kafka Quick Test (Load Test with Schema Registry) / Kafka Client Load Test (Quick) (push) Failing after 3s
Kafka Gateway Tests / Kafka Unit Tests (unit-tests-1) (push) Failing after 11s
Kafka Gateway Tests / Kafka Integration Tests (Critical) (integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka End-to-End Tests (with SMQ) (e2e-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Consumer Group Tests (Highly Isolated) (consumer-group-1) (push) Failing after 2s
Kafka Gateway Tests / Kafka Client Compatibility (with SMQ) (client-compat-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka SMQ Integration Tests (Full Stack) (smq-integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Protocol Tests (Isolated) (protocol-1) (push) Failing after 3s
KMS Tests / KMS Provider Integration Tests (push) Failing after 2s
KMS Tests / S3 KMS End-to-End Tests (push) Failing after 3s
Master Cold Start Tests / Master Cold Start Tests (push) Failing after 4s
Metadata Subscribe Integration Tests / Metadata Subscribe Integration Tests (push) Failing after 3s
Multi-Master Tests / Multi-Master Failover Tests (push) Failing after 3s
Performance / CPU and Heap Profile (push) Failing after 2s
Performance / S3 Read/Write Benchmark (rust volume) (push) Failing after 3s
pjdfstest POSIX Compliance / pjdfstest (push) Failing after 2s
Plugin Worker Integration Tests / Plugin Worker: erasure_coding (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: vacuum (push) Failing after 3s
Rust Volume Server Tests / Rust Integration Tests (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 1) (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 1) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 3) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Distributed Tests (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Performance Tests (push) Failing after 3s
Performance / Throughput Benchmark (go volume) (push) Failing after 3s
Performance / Throughput Benchmark (rust volume) (push) Failing after 2s
Performance / S3 Read/Write Benchmark (go volume) (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: volume_balance (push) Failing after 3s
PostgreSQL Gateway Tests / PostgreSQL Basic Tests (push) Failing after 3s
Rust Volume Server Tests / Rust Unit Tests (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 3) (push) Failing after 3s
Rust Plugin Worker Tests / Rust Plugin Worker Build and Unit Tests (push) Failing after 3s
rust: build dev volume server binaries / cleanup (push) Successful in 3s
S3 ETag and ACL Tests / S3 ETag + ACL Integration Tests (push) Failing after 3s
S3 IAM Integration Tests / IAM Unit Tests (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (advanced) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (basic) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (group) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (policy-enforcement) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (sts) (push) Failing after 4s
S3 Keycloak Integration Tests / S3 Keycloak Integration Tests (push) Failing after 3s
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.11) (push) Failing after 3s
S3 PyArrow Parquet Tests / Go Unit Tests (Implicit Directory) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.12) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-amd64, x86_64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-arm64, aarch64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-linux (linux-amd64, x86_64-unknown-linux-gnu) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.9) (push) Has been cancelled
S3 Proxy Signature Tests / S3 Proxy Signature Verification Tests (push) Has been cancelled
S3 SDK V2 Route Disambiguation Tests / S3 SDK V2 Routing Tests (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (quick) (push) Has been cancelled
S3 SSE Tests / S3 SSE Compatibility Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Metadata Persistence Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Copy Operations Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Multipart Upload Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Performance Test (push) Has been cancelled
S3 SSE Tests / S3 Volume Encryption Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (comprehensive) (push) Has been cancelled
Spark Integration Tests / Spark Integration Tests (push) Failing after 2s
Telemetry Integration Tests / telemetry-integration-test (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 1) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 1) (push) Failing after 3s
Ceph S3 tests / Basic S3 tests (KV store) (push) Has been cancelled
Ceph S3 tests / S3 Versioning & Object Lock tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Unit Tests (push) Failing after 3s
Ceph S3 tests / S3 CORS tests (push) Has been cancelled
Ceph S3 tests / SeaweedFS Custom S3 Copy tests (push) Has been cancelled
Ceph S3 tests / Basic S3 tests (SQL store) (push) Has been cancelled
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
TLS Rotation Integration Tests / TLS Rotation Integration Tests (push) Has been cancelled
Vacuum Integration Tests / Vacuum Integration Tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Variables Integration Tests (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (advanced-policy) (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (basic-policy) (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (contextual-policy) (push) Failing after 4s
S3 Policy Integration Tests / Trusted Proxy Detection Tests (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Shell Integration Tests (push) Failing after 3s
Samba on FUSE Integration / samba-integration (push) Failing after 2s
SFTP Integration Tests / SFTP Integration Testing (push) Failing after 3s
terraform: validate and test modules / fmt, validate, plan-level tests (push) Failing after 3s
terraform: validate and test modules / local cluster smoke test (real weed) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 3) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 2) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 2) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 3) (push) Failing after 5s
mount: windows conformance / WinFsp conformance (push) Has been cancelled
mount: windows / Mount on Windows (push) Has been cancelled

This commit is contained in:
upstream-mirror
2026-09-16 03:23:42 +05:00
commit 89f5be005d
3997 changed files with 961970 additions and 0 deletions

View File

@@ -0,0 +1,254 @@
package iceberg
import (
"context"
"encoding/json"
"errors"
"fmt"
"math/rand/v2"
"net/http"
"strings"
"time"
"github.com/apache/iceberg-go/table"
"github.com/google/uuid"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
const requirementAssertCreate = "assert-create"
// sleepBeforeCommitRetry backs off between commit attempts, jittered so that
// writers that collided do not line up again on the next try.
func sleepBeforeCommitRetry(attempt int) {
jitter := time.Duration(rand.Int64N(int64(25 * time.Millisecond)))
time.Sleep(time.Duration(50*attempt)*time.Millisecond + jitter)
}
// stageCommitMetadata writes the metadata file a commit will point the catalog
// at, returning the name and location it landed on. The exclusive create keeps
// one writer from overwriting another's file; when the name is already taken
// the metadata goes to a unique one instead of failing, because the file there
// may be an orphan left by an interrupted commit, and refusing would wedge the
// table until an orphan sweep ran. The catalog pointer, not the file name,
// decides which commit won. This mirrors how the maintenance worker stages its
// own metadata.
func (s *Server) stageCommitMetadata(ctx context.Context, metadataBucket, metadataPath, location, metadataFileName string, metadataBytes []byte) (string, string, error) {
err := s.saveMetadataFile(ctx, metadataBucket, metadataPath, metadataFileName, metadataBytes, true)
if errors.Is(err, filer_pb.ErrEntryAlreadyExists) {
// v{N}-{uuid}, a form both the catalog and the maintenance worker
// still read the version from.
metadataFileName = fmt.Sprintf("%s-%s.metadata.json", strings.TrimSuffix(metadataFileName, ".metadata.json"), uuid.NewString())
glog.V(1).Infof("Iceberg: metadata file already staged for %s, using %s", location, metadataFileName)
err = s.saveMetadataFile(ctx, metadataBucket, metadataPath, metadataFileName, metadataBytes, true)
}
if err != nil {
return "", "", err
}
return metadataFileName, fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(location, "/"), metadataFileName), nil
}
type icebergRequestError struct {
status int
errType string
message string
}
type createOnCommitInput struct {
bucketARN string
markerBucket string
namespace []string
tableName string
identityName string
location string
tableUUID uuid.UUID
baseMetadata table.Metadata
baseMetadataLoc string
baseMetadataVer int
updates table.Updates
statisticsUpdates []statisticsUpdate
}
func isS3TablesConflict(err error) bool {
if err == nil {
return false
}
if errors.Is(err, s3tables.ErrVersionTokenMismatch) {
return true
}
var tableErr *s3tables.S3TablesError
return errors.As(err, &tableErr) && tableErr.Type == s3tables.ErrCodeConflict
}
func isS3TablesNotFound(err error) bool {
if err == nil {
return false
}
if strings.Contains(strings.ToLower(err.Error()), "not found") {
return true
}
var tableErr *s3tables.S3TablesError
return errors.As(err, &tableErr) &&
(tableErr.Type == s3tables.ErrCodeNoSuchTable || tableErr.Type == s3tables.ErrCodeNoSuchNamespace || strings.Contains(strings.ToLower(tableErr.Message), "not found"))
}
func hasAssertCreateRequirement(requirements table.Requirements) bool {
for _, requirement := range requirements {
if requirement.GetType() == requirementAssertCreate {
return true
}
}
return false
}
func isS3TablesAlreadyExists(err error) bool {
if err == nil {
return false
}
if strings.Contains(strings.ToLower(err.Error()), "already exists") {
return true
}
var tableErr *s3tables.S3TablesError
return errors.As(err, &tableErr) &&
(tableErr.Type == s3tables.ErrCodeTableAlreadyExists || tableErr.Type == s3tables.ErrCodeNamespaceAlreadyExists || strings.Contains(strings.ToLower(tableErr.Message), "already exists"))
}
func (s *Server) finalizeCreateOnCommit(ctx context.Context, input createOnCommitInput) (*CommitTableResponse, *icebergRequestError) {
builder, err := table.MetadataBuilderFromBase(input.baseMetadata, input.baseMetadataLoc)
if err != nil {
return nil, &icebergRequestError{
status: http.StatusInternalServerError,
errType: "InternalServerError",
message: "Failed to create metadata builder: " + err.Error(),
}
}
for _, update := range input.updates {
if err := update.Apply(builder); err != nil {
return nil, &icebergRequestError{
status: http.StatusBadRequest,
errType: "BadRequestException",
message: "Failed to apply update: " + err.Error(),
}
}
}
newMetadata, err := builder.Build()
if err != nil {
return nil, &icebergRequestError{
status: http.StatusBadRequest,
errType: "BadRequestException",
message: "Failed to build new metadata: " + err.Error(),
}
}
metadataVersion := input.baseMetadataVer + 1
if metadataVersion <= 0 {
metadataVersion = 1
}
metadataFileName := fmt.Sprintf("v%d.metadata.json", metadataVersion)
newMetadataLocation := fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(input.location, "/"), metadataFileName)
metadataBytes, err := json.Marshal(newMetadata)
if err != nil {
return nil, &icebergRequestError{
status: http.StatusInternalServerError,
errType: "InternalServerError",
message: "Failed to serialize metadata: " + err.Error(),
}
}
metadataBytes, err = applyStatisticsUpdates(metadataBytes, input.statisticsUpdates)
if err != nil {
return nil, &icebergRequestError{
status: http.StatusBadRequest,
errType: "BadRequestException",
message: "Failed to apply statistics updates: " + err.Error(),
}
}
metadataBytes = refreshDefaultNameMapping(metadataBytes, newMetadata)
// Same spec-compliance fixup we apply on create-table; ensures
// v{N}.metadata.json files written through this create-on-commit path are
// also readable by strict Iceberg clients reading directly from S3.
metadataBytes = ensureMetadataSpecCompliance(metadataBytes)
newMetadata, err = table.ParseMetadataBytes(metadataBytes)
if err != nil {
return nil, &icebergRequestError{
status: http.StatusInternalServerError,
errType: "InternalServerError",
message: "Failed to parse committed metadata: " + err.Error(),
}
}
metadataBucket, metadataPath, err := parseS3Location(input.location)
if err != nil {
return nil, &icebergRequestError{
status: http.StatusInternalServerError,
errType: "InternalServerError",
message: "Invalid table location: " + err.Error(),
}
}
if err := confineMetadataLocation(metadataBucket, metadataPath, input.markerBucket); err != nil {
return nil, &icebergRequestError{
status: http.StatusBadRequest,
errType: "BadRequestException",
message: err.Error(),
}
}
if err := s.saveMetadataFile(ctx, metadataBucket, metadataPath, metadataFileName, metadataBytes, false); err != nil {
return nil, &icebergRequestError{
status: http.StatusInternalServerError,
errType: "InternalServerError",
message: "Failed to save metadata file: " + err.Error(),
}
}
createReq := &s3tables.CreateTableRequest{
TableBucketARN: input.bucketARN,
Namespace: input.namespace,
Name: input.tableName,
Format: "ICEBERG",
Metadata: &s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{
TableUUID: input.tableUUID.String(),
},
FullMetadata: metadataBytes,
},
MetadataVersion: metadataVersion,
MetadataLocation: newMetadataLocation,
}
createErr := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(ctx, mgrClient, "CreateTable", createReq, nil, input.identityName)
})
if createErr != nil {
if cleanupErr := s.deleteMetadataFile(ctx, metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
glog.V(1).Infof("Iceberg: failed to cleanup metadata file %s after create-on-commit failure: %v", newMetadataLocation, cleanupErr)
}
if isS3TablesConflict(createErr) || isS3TablesAlreadyExists(createErr) {
return nil, &icebergRequestError{
status: http.StatusConflict,
errType: "CommitFailedException",
message: "Table was created concurrently",
}
}
glog.Errorf("Iceberg: CommitTable CreateTable error: %v", createErr)
return nil, &icebergRequestError{
status: http.StatusInternalServerError,
errType: "InternalServerError",
message: "Failed to commit table creation: " + createErr.Error(),
}
}
markerBucket := input.markerBucket
if markerBucket == "" {
markerBucket = metadataBucket
}
if markerErr := s.deleteStageCreateMarkers(ctx, markerBucket, input.namespace, input.tableName); markerErr != nil {
glog.V(1).Infof("Iceberg: failed to cleanup stage-create markers for %s.%s after finalize: %v", flattenNamespacePath(input.namespace), input.tableName, markerErr)
}
return &CommitTableResponse{
MetadataLocation: newMetadataLocation,
Metadata: newMetadata,
}, nil
}

View File

@@ -0,0 +1,173 @@
package iceberg
import (
"encoding/json"
"errors"
"github.com/apache/iceberg-go/table"
)
type statisticsUpdate struct {
set *table.StatisticsFile
remove *int64
}
var ErrIncompleteSetStatistics = errors.New("set-statistics requires snapshot-id, statistics-path, file-size-in-bytes, and file-footer-size-in-bytes")
type commitAction struct {
Action string `json:"action"`
}
type setStatisticsUpdate struct {
Action string `json:"action"`
SnapshotID *int64 `json:"snapshot-id,omitempty"`
StatisticsPath string `json:"statistics-path,omitempty"`
FileSizeInBytes *int64 `json:"file-size-in-bytes,omitempty"`
FileFooterSizeInBytes *int64 `json:"file-footer-size-in-bytes,omitempty"`
KeyMetadata *string `json:"key-metadata,omitempty"`
BlobMetadata []table.BlobMetadata `json:"blob-metadata,omitempty"`
Statistics *table.StatisticsFile `json:"statistics,omitempty"`
}
func (u *setStatisticsUpdate) asStatisticsFile() (*table.StatisticsFile, error) {
if u.Statistics != nil {
if u.Statistics.BlobMetadata == nil {
u.Statistics.BlobMetadata = []table.BlobMetadata{}
}
return u.Statistics, nil
}
if u.SnapshotID == nil || u.StatisticsPath == "" || u.FileSizeInBytes == nil || u.FileFooterSizeInBytes == nil {
return nil, ErrIncompleteSetStatistics
}
stats := &table.StatisticsFile{
SnapshotID: *u.SnapshotID,
StatisticsPath: u.StatisticsPath,
FileSizeInBytes: *u.FileSizeInBytes,
FileFooterSizeInBytes: *u.FileFooterSizeInBytes,
KeyMetadata: u.KeyMetadata,
BlobMetadata: u.BlobMetadata,
}
if stats.BlobMetadata == nil {
stats.BlobMetadata = []table.BlobMetadata{}
}
return stats, nil
}
type removeStatisticsUpdate struct {
Action string `json:"action"`
SnapshotID int64 `json:"snapshot-id"`
}
func parseCommitUpdates(rawUpdates []json.RawMessage) (table.Updates, []statisticsUpdate, error) {
filtered := make([]json.RawMessage, 0, len(rawUpdates))
statisticsUpdates := make([]statisticsUpdate, 0)
for _, raw := range rawUpdates {
var action commitAction
if err := json.Unmarshal(raw, &action); err != nil {
return nil, nil, err
}
switch action.Action {
case "set-statistics":
var setUpdate setStatisticsUpdate
if err := json.Unmarshal(raw, &setUpdate); err != nil {
return nil, nil, err
}
stats, err := setUpdate.asStatisticsFile()
if err != nil {
return nil, nil, err
}
statisticsUpdates = append(statisticsUpdates, statisticsUpdate{set: stats})
case "remove-statistics":
var removeUpdate removeStatisticsUpdate
if err := json.Unmarshal(raw, &removeUpdate); err != nil {
return nil, nil, err
}
snapshotID := removeUpdate.SnapshotID
statisticsUpdates = append(statisticsUpdates, statisticsUpdate{remove: &snapshotID})
default:
filtered = append(filtered, raw)
}
}
if len(filtered) == 0 {
return nil, statisticsUpdates, nil
}
data, err := json.Marshal(filtered)
if err != nil {
return nil, nil, err
}
var updates table.Updates
if err := json.Unmarshal(data, &updates); err != nil {
return nil, nil, err
}
return updates, statisticsUpdates, nil
}
func applyStatisticsUpdates(metadataBytes []byte, updates []statisticsUpdate) ([]byte, error) {
if len(updates) == 0 {
return metadataBytes, nil
}
var metadata map[string]json.RawMessage
if err := json.Unmarshal(metadataBytes, &metadata); err != nil {
return nil, err
}
var statistics []table.StatisticsFile
if rawStatistics, ok := metadata["statistics"]; ok && len(rawStatistics) > 0 {
if err := json.Unmarshal(rawStatistics, &statistics); err != nil {
return nil, err
}
}
statisticsBySnapshot := make(map[int64]table.StatisticsFile, len(statistics))
orderedSnapshotIDs := make([]int64, 0, len(statistics))
inOrder := make(map[int64]bool, len(statistics))
for _, stat := range statistics {
statisticsBySnapshot[stat.SnapshotID] = stat
if !inOrder[stat.SnapshotID] {
orderedSnapshotIDs = append(orderedSnapshotIDs, stat.SnapshotID)
inOrder[stat.SnapshotID] = true
}
}
for _, update := range updates {
if update.set != nil {
statisticsBySnapshot[update.set.SnapshotID] = *update.set
if !inOrder[update.set.SnapshotID] {
orderedSnapshotIDs = append(orderedSnapshotIDs, update.set.SnapshotID)
inOrder[update.set.SnapshotID] = true
}
continue
}
if update.remove != nil {
delete(statisticsBySnapshot, *update.remove)
}
}
statistics = make([]table.StatisticsFile, 0, len(statisticsBySnapshot))
for _, snapshotID := range orderedSnapshotIDs {
stat, ok := statisticsBySnapshot[snapshotID]
if !ok {
continue
}
statistics = append(statistics, stat)
}
if len(statistics) == 0 {
delete(metadata, "statistics")
} else {
data, err := json.Marshal(statistics)
if err != nil {
return nil, err
}
metadata["statistics"] = data
}
return json.Marshal(metadata)
}

View File

@@ -0,0 +1,404 @@
package iceberg
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"path"
"strings"
"github.com/apache/iceberg-go/table"
"github.com/google/uuid"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// handleUpdateTable commits updates to a table.
// Implements the Iceberg REST Catalog commit protocol.
func (s *Server) handleUpdateTable(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
tableName := vars["table"]
if len(namespace) == 0 || tableName == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and table name are required")
return
}
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
// Extract identity from context
identityName := s3_constants.GetIdentityNameFromContext(r)
// Parse commit request and keep statistics updates separate because iceberg-go v0.4.0
// does not decode set/remove-statistics update actions yet.
var raw struct {
Identifier *TableIdentifier `json:"identifier,omitempty"`
Requirements json.RawMessage `json:"requirements"`
Updates []json.RawMessage `json:"updates"`
}
if err := json.NewDecoder(r.Body).Decode(&raw); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body: "+err.Error())
return
}
var req CommitTableRequest
req.Identifier = raw.Identifier
var statisticsUpdates []statisticsUpdate
if len(raw.Requirements) > 0 {
normalized, err := normalizeRequirements(raw.Requirements)
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid requirements: "+err.Error())
return
}
if err := json.Unmarshal(normalized, &req.Requirements); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid requirements: "+err.Error())
return
}
}
if len(raw.Updates) > 0 {
var err error
req.Updates, statisticsUpdates, err = parseCommitUpdates(raw.Updates)
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid updates: "+err.Error())
return
}
}
// Manifest repair runs once, as soon as the table location is known; on
// commit retries the updates already reference the repaired files. Repair
// is best effort end to end: the originals parsed already, so a repair
// that fails to re-parse is discarded rather than failing the commit.
manifestsRepaired := false
repairManifests := func(location string) {
if manifestsRepaired {
return
}
manifestsRepaired = true
repaired, changed := s.repairAddSnapshotManifests(r.Context(), location, raw.Updates)
if !changed {
return
}
repairedUpdates, repairedStatistics, err := parseCommitUpdates(repaired)
if err != nil {
glog.Warningf("Iceberg: repaired updates failed to parse, keeping originals: %v", err)
return
}
raw.Updates = repaired
req.Updates = repairedUpdates
statisticsUpdates = repairedStatistics
}
maxCommitAttempts := 3
generatedLegacyUUID := uuid.New()
stageCreateEnabled := isStageCreateEnabled()
for attempt := 1; attempt <= maxCommitAttempts; attempt++ {
getReq := &s3tables.GetTableRequest{
TableBucketARN: bucketARN,
Namespace: namespace,
Name: tableName,
}
var getResp s3tables.GetTableResponse
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "GetTable", getReq, &getResp, identityName)
})
if err != nil {
if isS3TablesNotFound(err) {
location := fmt.Sprintf("s3://%s/%s", bucketName, path.Join(flattenNamespacePath(namespace), tableName))
tableUUID := generatedLegacyUUID
baseMetadataVersion := 0
baseMetadataLocation := ""
var baseMetadata table.Metadata
var latestMarker *stageCreateMarker
if stageCreateEnabled {
var markerErr error
latestMarker, markerErr = s.loadLatestStageCreateMarker(r.Context(), bucketName, namespace, tableName)
if markerErr != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to load stage-create marker: "+markerErr.Error())
return
}
}
if latestMarker != nil {
if latestMarker.Location != "" {
location = strings.TrimSuffix(latestMarker.Location, "/")
}
if latestMarker.TableUUID != "" {
if parsedUUID, parseErr := uuid.Parse(latestMarker.TableUUID); parseErr == nil {
tableUUID = parsedUUID
}
}
stagedMetadataLocation := latestMarker.StagedMetadataLocation
if stagedMetadataLocation == "" {
stagedMetadataLocation = fmt.Sprintf("%s/metadata/v1.metadata.json", strings.TrimSuffix(location, "/"))
}
stagedLocation := tableLocationFromMetadataLocation(stagedMetadataLocation)
stagedFileName := path.Base(stagedMetadataLocation)
stagedBucket, stagedPath, parseLocationErr := parseS3Location(stagedLocation)
if parseLocationErr != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid staged metadata location: "+parseLocationErr.Error())
return
}
if err := confineMetadataLocation(stagedBucket, stagedPath, bucketName); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
return
}
stagedMetadataBytes, loadErr := s.loadMetadataFile(r.Context(), stagedBucket, stagedPath, stagedFileName)
if loadErr != nil {
if !errors.Is(loadErr, filer_pb.ErrNotFound) {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to load staged metadata: "+loadErr.Error())
return
}
} else if len(stagedMetadataBytes) > 0 {
stagedMetadata, parseErr := table.ParseMetadataBytes(stagedMetadataBytes)
if parseErr != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to parse staged metadata: "+parseErr.Error())
return
}
// Staged metadata is only a template for table creation; commit starts from version 1.
baseMetadata = stagedMetadata
baseMetadataLocation = ""
baseMetadataVersion = 0
if stagedMetadata.TableUUID() != uuid.Nil {
tableUUID = stagedMetadata.TableUUID()
}
}
}
hasAssertCreate := hasAssertCreateRequirement(req.Requirements)
hasStagedTemplate := baseMetadata != nil
if !(stageCreateEnabled && (hasAssertCreate || hasStagedTemplate)) {
writeError(w, http.StatusNotFound, "NoSuchTableException", fmt.Sprintf("Table does not exist: %s", tableName))
return
}
// From here the commit creates the table, writing its metadata
// file before the create that authorizes it.
if authErr := s.authorizeCreateTable(r.Context(), bucketARN, namespace, tableName, identityName); authErr != nil {
writeManagerError(w, authErr)
return
}
for _, requirement := range req.Requirements {
validateAgainst := table.Metadata(nil)
if hasStagedTemplate && requirement.GetType() != requirementAssertCreate {
validateAgainst = baseMetadata
}
if requirementErr := requirement.Validate(validateAgainst); requirementErr != nil {
writeError(w, http.StatusConflict, "CommitFailedException", "Requirement failed: "+requirementErr.Error())
return
}
}
if baseMetadata == nil {
var buildErr error
if baseMetadata, buildErr = newTableMetadata(tableUUID, location, nil, nil, nil, nil); buildErr != nil {
glog.Errorf("Iceberg: CommitTable placeholder metadata for %s: %v", tableName, buildErr)
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to build current metadata")
return
}
}
createBucket, createPath, createLocErr := parseS3Location(location)
if createLocErr != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid table location: "+createLocErr.Error())
return
}
if err := confineMetadataLocation(createBucket, createPath, bucketName); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
return
}
repairManifests(location)
result, reqErr := s.finalizeCreateOnCommit(r.Context(), createOnCommitInput{
bucketARN: bucketARN,
markerBucket: bucketName,
namespace: namespace,
tableName: tableName,
identityName: identityName,
location: location,
tableUUID: tableUUID,
baseMetadata: baseMetadata,
baseMetadataLoc: baseMetadataLocation,
baseMetadataVer: baseMetadataVersion,
updates: req.Updates,
statisticsUpdates: statisticsUpdates,
})
if reqErr != nil {
writeError(w, reqErr.status, reqErr.errType, reqErr.message)
return
}
writeJSON(w, http.StatusOK, result)
return
}
glog.V(1).Infof("Iceberg: CommitTable GetTable error: %v", err)
writeManagerError(w, err)
return
}
location := tableLocationFromMetadataLocation(getResp.MetadataLocation)
if location == "" {
location = fmt.Sprintf("s3://%s/%s", bucketName, path.Join(flattenNamespacePath(namespace), tableName))
}
locBucket, locPath, locErr := parseS3Location(location)
if locErr != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid table location: "+locErr.Error())
return
}
if err := confineMetadataLocation(locBucket, locPath, bucketName); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
return
}
tableUUID := uuid.Nil
if getResp.Metadata != nil && getResp.Metadata.Iceberg != nil && getResp.Metadata.Iceberg.TableUUID != "" {
if parsed, parseErr := uuid.Parse(getResp.Metadata.Iceberg.TableUUID); parseErr == nil {
tableUUID = parsed
}
}
if tableUUID == uuid.Nil {
tableUUID = generatedLegacyUUID
}
var currentMetadata table.Metadata
if getResp.Metadata != nil && len(getResp.Metadata.FullMetadata) > 0 {
currentMetadata, err = table.ParseMetadataBytes(getResp.Metadata.FullMetadata)
if err != nil {
glog.Errorf("Iceberg: Failed to parse current metadata for %s: %v", tableName, err)
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to parse current metadata")
return
}
} else {
currentMetadata, err = newTableMetadata(tableUUID, location, nil, nil, nil, nil)
if err != nil {
glog.Errorf("Iceberg: CommitTable placeholder metadata for %s: %v", tableName, err)
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to build current metadata")
return
}
}
for _, requirement := range req.Requirements {
if err := requirement.Validate(currentMetadata); err != nil {
writeError(w, http.StatusConflict, "CommitFailedException", "Requirement failed: "+err.Error())
return
}
}
repairManifests(location)
builder, err := table.MetadataBuilderFromBase(currentMetadata, getResp.MetadataLocation)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to create metadata builder: "+err.Error())
return
}
for _, update := range req.Updates {
if err := update.Apply(builder); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to apply update: "+err.Error())
return
}
}
newMetadata, err := builder.Build()
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to build new metadata: "+err.Error())
return
}
metadataVersion := getResp.MetadataVersion + 1
metadataFileName := fmt.Sprintf("v%d.metadata.json", metadataVersion)
newMetadataLocation := fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(location, "/"), metadataFileName)
metadataBytes, err := json.Marshal(newMetadata)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to serialize metadata: "+err.Error())
return
}
// iceberg-go does not currently support set/remove-statistics updates in MetadataBuilder.
// Patch the encoded metadata JSON and parse it back to keep the response object consistent.
metadataBytes, err = applyStatisticsUpdates(metadataBytes, statisticsUpdates)
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to apply statistics updates: "+err.Error())
return
}
metadataBytes = refreshDefaultNameMapping(metadataBytes, newMetadata)
// Same spec-compliance fixup we apply on create-table; ensures
// v{N}.metadata.json files written during commit are also readable by
// strict Iceberg clients reading directly from S3, and that the
// FullMetadata persisted in S3Tables stays consistent.
metadataBytes = ensureMetadataSpecCompliance(metadataBytes)
newMetadata, err = table.ParseMetadataBytes(metadataBytes)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to parse committed metadata: "+err.Error())
return
}
metadataBucket, metadataPath, err := parseS3Location(location)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid table location: "+err.Error())
return
}
if err := confineMetadataLocation(metadataBucket, metadataPath, bucketName); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
return
}
metadataFileName, newMetadataLocation, err = s.stageCommitMetadata(r.Context(), metadataBucket, metadataPath, location, metadataFileName, metadataBytes)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to save metadata file: "+err.Error())
return
}
updateReq := &s3tables.UpdateTableRequest{
TableBucketARN: bucketARN,
Namespace: namespace,
Name: tableName,
VersionToken: getResp.VersionToken,
Metadata: &s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{
TableUUID: tableUUID.String(),
},
FullMetadata: metadataBytes,
},
MetadataVersion: metadataVersion,
MetadataLocation: newMetadataLocation,
}
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "UpdateTable", updateReq, nil, identityName)
})
if err == nil {
result := CommitTableResponse{
MetadataLocation: newMetadataLocation,
Metadata: newMetadata,
}
writeJSON(w, http.StatusOK, result)
return
}
if isS3TablesConflict(err) {
if cleanupErr := s.deleteMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
glog.V(1).Infof("Iceberg: failed to cleanup metadata file %s on conflict: %v", newMetadataLocation, cleanupErr)
}
if attempt < maxCommitAttempts {
glog.V(1).Infof("Iceberg: CommitTable conflict for %s (attempt %d/%d), retrying", tableName, attempt, maxCommitAttempts)
sleepBeforeCommitRetry(attempt)
continue
}
writeError(w, http.StatusConflict, "CommitFailedException", "Version token mismatch")
return
}
if cleanupErr := s.deleteMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
glog.V(1).Infof("Iceberg: failed to cleanup metadata file %s after update failure: %v", newMetadataLocation, cleanupErr)
}
glog.Errorf("Iceberg: CommitTable UpdateTable error: %v", err)
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to commit table update: "+err.Error())
return
}
}

View File

@@ -0,0 +1,58 @@
package iceberg
import (
"encoding/json"
"io"
"net/http"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/glog"
)
// Reports carry per-file scan metrics and grow with the table. The content is
// discarded either way, so anything past this is accepted without being read
// into memory rather than rejected.
const maxMetricsReportBytes = 1 << 20
// handleReportMetrics accepts the scan and commit reports engines send after
// planning or committing. The catalog keeps no metrics store, but the endpoint
// has to exist: clients that get a 404 here log an error per query, and some
// treat repeated failures as a broken catalog.
func (s *Server) handleReportMetrics(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
tableName := vars["table"]
if len(namespace) == 0 || tableName == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and table name are required")
return
}
// One byte past the limit distinguishes a report that fits from one that
// was cut short, which would otherwise fail to parse and look malformed.
body, err := io.ReadAll(io.LimitReader(r.Body, maxMetricsReportBytes+1))
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
return
}
if len(body) > maxMetricsReportBytes {
glog.V(3).Infof("Iceberg: discarding oversized metrics report for %s.%s", flattenNamespacePath(namespace), tableName)
w.WriteHeader(http.StatusNoContent)
return
}
var report struct {
ReportType string `json:"report-type"`
}
if err := json.Unmarshal(body, &report); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body: "+err.Error())
return
}
if report.ReportType == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "report-type is required")
return
}
glog.V(3).Infof("Iceberg: metrics report %q for %s.%s", report.ReportType, flattenNamespacePath(namespace), tableName)
w.WriteHeader(http.StatusNoContent)
}

View File

@@ -0,0 +1,63 @@
package iceberg
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/gorilla/mux"
)
func postMetrics(t *testing.T, body string) *httptest.ResponseRecorder {
t.Helper()
r := httptest.NewRequest(http.MethodPost, "/v1/namespaces/ns/tables/t/metrics", strings.NewReader(body))
r = mux.SetURLVars(r, map[string]string{"namespace": "ns", "table": "t"})
w := httptest.NewRecorder()
(&Server{}).handleReportMetrics(w, r)
return w
}
func TestHandleReportMetricsAcceptsScanReport(t *testing.T) {
w := postMetrics(t, `{"report-type":"scan-report","table-name":"t","snapshot-id":1}`)
if w.Code != http.StatusNoContent {
t.Errorf("status = %d, want %d: %s", w.Code, http.StatusNoContent, w.Body.String())
}
}
func TestHandleReportMetricsRejectsGarbage(t *testing.T) {
w := postMetrics(t, `not json`)
if w.Code != http.StatusBadRequest {
t.Errorf("status = %d, want %d", w.Code, http.StatusBadRequest)
}
}
// The spec makes the body and its report-type required.
func TestHandleReportMetricsRequiresReportType(t *testing.T) {
for _, body := range []string{``, `{}`, `{"report-type":""}`} {
if w := postMetrics(t, body); w.Code != http.StatusBadRequest {
t.Errorf("body %q: status = %d, want %d", body, w.Code, http.StatusBadRequest)
}
}
}
// A report larger than the read limit is still a valid report; truncating it
// and then failing to parse would hand the client an error for a query that
// worked.
func TestHandleReportMetricsAcceptsOversizedReport(t *testing.T) {
padding := strings.Repeat("x", maxMetricsReportBytes)
w := postMetrics(t, `{"report-type":"scan-report","padding":"`+padding+`"}`)
if w.Code != http.StatusNoContent {
t.Errorf("status = %d, want %d: %s", w.Code, http.StatusNoContent, w.Body.String())
}
}
func TestHandleReportMetricsRequiresTable(t *testing.T) {
r := httptest.NewRequest(http.MethodPost, "/v1/namespaces/ns/tables//metrics", strings.NewReader(`{}`))
r = mux.SetURLVars(r, map[string]string{"namespace": "ns"})
w := httptest.NewRecorder()
(&Server{}).handleReportMetrics(w, r)
if w.Code != http.StatusBadRequest {
t.Errorf("status = %d, want %d", w.Code, http.StatusBadRequest)
}
}

View File

@@ -0,0 +1,389 @@
package iceberg
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// handleConfig returns catalog configuration.
//
// When a client passes ?warehouse=, the Iceberg REST spec expects the server
// to echo a catalog identifier back as overrides.prefix so subsequent calls
// use /v1/{prefix}/... and land on the right table bucket. Without this,
// clients like DuckDB's ATTACH flow fall back to an unprefixed path that
// resolves to the wrong bucket and report phantom "schema does not exist"
// errors.
func (s *Server) handleConfig(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
config := CatalogConfig{
Defaults: map[string]string{},
Overrides: map[string]string{},
}
if warehouse := strings.TrimSpace(r.URL.Query().Get("warehouse")); warehouse != "" {
// Only the bucket portion of the warehouse is meaningful today —
// SeaweedFS table-bucket routing is bucket-scoped, so any sub-path
// (e.g. s3://bucket/prefix/) is ignored here and clients that try to
// scope a catalog under a sub-prefix will still land on the bucket.
if bucket := resolveWarehouseBucket(warehouse); bucket != "" {
config.Overrides["prefix"] = bucket
// Echo the location form regardless of how the client spelled the
// warehouse: clients derive default table locations from it.
config.Defaults["warehouse"] = "s3://" + bucket
}
}
if err := json.NewEncoder(w).Encode(config); err != nil {
glog.Warningf("handleConfig: Failed to encode config: %v", err)
}
}
// handleListNamespaces lists namespaces in a catalog.
func (s *Server) handleListNamespaces(w http.ResponseWriter, r *http.Request) {
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
// Extract identity from context
identityName := s3_constants.GetIdentityNameFromContext(r)
pageToken, pageSize, err := parsePagination(r)
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
return
}
// The Iceberg REST spec allows a "parent" query parameter for hierarchical
// namespace listing. Convert it to the dot-separated prefix used by S3 Tables.
var prefix string
if parent := r.URL.Query().Get("parent"); parent != "" {
parentParts := parseNamespace(parent)
if len(parentParts) > 0 {
prefix = flattenNamespacePath(parentParts) + "."
}
}
// Use S3 Tables manager to list namespaces
var resp s3tables.ListNamespacesResponse
req := &s3tables.ListNamespacesRequest{
TableBucketARN: bucketARN,
Prefix: prefix,
ContinuationToken: pageToken,
MaxNamespaces: pageSize,
}
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "ListNamespaces", req, &resp, identityName)
})
if err != nil {
glog.Infof("Iceberg: ListNamespaces error: %v", err)
writeManagerError(w, err)
return
}
// Convert to Iceberg format
namespaces := make([]Namespace, 0, len(resp.Namespaces))
for _, ns := range resp.Namespaces {
namespaces = append(namespaces, Namespace(ns.Namespace))
}
result := ListNamespacesResponse{
NextPageToken: resp.ContinuationToken,
Namespaces: namespaces,
}
writeJSON(w, http.StatusOK, result)
}
// handleCreateNamespace creates a new namespace.
func (s *Server) handleCreateNamespace(w http.ResponseWriter, r *http.Request) {
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
// Extract identity from context
identityName := s3_constants.GetIdentityNameFromContext(r)
var req CreateNamespaceRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
return
}
if len(req.Namespace) == 0 {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
return
}
// Use S3 Tables manager to create namespace
createReq := &s3tables.CreateNamespaceRequest{
TableBucketARN: bucketARN,
Namespace: req.Namespace,
Properties: normalizeNamespaceProperties(req.Properties),
}
var createResp s3tables.CreateNamespaceResponse
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
glog.V(2).Infof("Iceberg: handleCreateNamespace calling Execute with identityName=%s", identityName)
return s.tablesManager.Execute(r.Context(), mgrClient, "CreateNamespace", createReq, &createResp, identityName)
})
if err != nil {
if strings.Contains(err.Error(), "already exists") {
writeError(w, http.StatusConflict, "AlreadyExistsException", err.Error())
return
}
glog.Errorf("Iceberg: CreateNamespace error: %v", err)
writeManagerError(w, err)
return
}
result := CreateNamespaceResponse{
Namespace: Namespace(createResp.Namespace),
Properties: withDefaultNamespaceLocation(createResp.Properties, bucketName, createResp.Namespace),
}
writeJSON(w, http.StatusOK, result)
}
// handleGetNamespace gets namespace metadata.
func (s *Server) handleGetNamespace(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
if len(namespace) == 0 {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
return
}
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
// Extract identity from context
identityName := s3_constants.GetIdentityNameFromContext(r)
// Use S3 Tables manager to get namespace
getReq := &s3tables.GetNamespaceRequest{
TableBucketARN: bucketARN,
Namespace: namespace,
}
var getResp s3tables.GetNamespaceResponse
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "GetNamespace", getReq, &getResp, identityName)
})
if err != nil {
if strings.Contains(err.Error(), "not found") {
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
return
}
glog.V(1).Infof("Iceberg: GetNamespace error: %v", err)
writeManagerError(w, err)
return
}
result := GetNamespaceResponse{
Namespace: Namespace(getResp.Namespace),
Properties: withDefaultNamespaceLocation(getResp.Properties, bucketName, getResp.Namespace),
}
writeJSON(w, http.StatusOK, result)
}
// applyNamespacePropertyUpdates applies removals and updates to a copy of
// current and returns the merged property map alongside the Iceberg REST
// summary of which keys were removed, updated, or were missing.
func applyNamespacePropertyUpdates(current map[string]string, removals []string, updates map[string]string) (map[string]string, UpdateNamespacePropertiesResponse) {
properties := make(map[string]string, len(current))
for k, v := range current {
properties[k] = v
}
summary := UpdateNamespacePropertiesResponse{Removed: []string{}, Updated: []string{}, Missing: []string{}}
seen := make(map[string]struct{}, len(removals))
for _, key := range removals {
if _, dup := seen[key]; dup {
continue
}
seen[key] = struct{}{}
if _, ok := properties[key]; ok {
delete(properties, key)
summary.Removed = append(summary.Removed, key)
} else {
summary.Missing = append(summary.Missing, key)
}
}
for key, value := range updates {
properties[key] = value
summary.Updated = append(summary.Updated, key)
}
return properties, summary
}
// handleUpdateNamespaceProperties applies a set of removals and updates to a
// namespace's properties and returns which keys were removed, updated, or
// missing, per the Iceberg REST spec.
func (s *Server) handleUpdateNamespaceProperties(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
if len(namespace) == 0 {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
return
}
var req UpdateNamespacePropertiesRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
return
}
// A key cannot be both removed and updated in the same request.
for _, key := range req.Removals {
if _, ok := req.Updates[key]; ok {
writeError(w, http.StatusUnprocessableEntity, "UnprocessableEntityException",
fmt.Sprintf("Cannot remove and update the same key: %s", key))
return
}
}
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
identityName := s3_constants.GetIdentityNameFromContext(r)
// Load the current properties so we can compute the summary and the merged map.
getReq := &s3tables.GetNamespaceRequest{TableBucketARN: bucketARN, Namespace: namespace}
var getResp s3tables.GetNamespaceResponse
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "GetNamespace", getReq, &getResp, identityName)
})
if err != nil {
writeNamespaceManagerError(w, err, namespace)
return
}
properties, summary := applyNamespacePropertyUpdates(getResp.Properties, req.Removals, req.Updates)
updReq := &s3tables.UpdateNamespaceRequest{TableBucketARN: bucketARN, Namespace: namespace, Properties: properties}
var updResp s3tables.UpdateNamespaceResponse
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "UpdateNamespace", updReq, &updResp, identityName)
})
if err != nil {
writeNamespaceManagerError(w, err, namespace)
return
}
writeJSON(w, http.StatusOK, summary)
}
// writeNamespaceManagerError maps an s3tables manager error to the matching
// Iceberg REST response. A namespace dropped between read and write surfaces
// as 404 and a denied caller as 403; anything else is a 500.
func writeNamespaceManagerError(w http.ResponseWriter, err error, namespace Namespace) {
var s3Err *s3tables.S3TablesError
if errors.As(err, &s3Err) {
switch s3Err.Type {
case s3tables.ErrCodeNoSuchNamespace:
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
return
case s3tables.ErrCodeAccessDenied:
writeError(w, http.StatusForbidden, "ForbiddenException", "Not authorized to update namespace properties")
return
}
}
glog.V(1).Infof("Iceberg: UpdateNamespaceProperties error: %v", err)
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
}
// handleNamespaceExists checks if a namespace exists.
func (s *Server) handleNamespaceExists(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
if len(namespace) == 0 {
w.WriteHeader(http.StatusBadRequest)
return
}
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
// Extract identity from context
identityName := s3_constants.GetIdentityNameFromContext(r)
getReq := &s3tables.GetNamespaceRequest{
TableBucketARN: bucketARN,
Namespace: namespace,
}
var getResp s3tables.GetNamespaceResponse
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "GetNamespace", getReq, &getResp, identityName)
})
if err != nil {
if strings.Contains(err.Error(), "not found") {
w.WriteHeader(http.StatusNotFound)
return
}
if nameValidationError(err) {
w.WriteHeader(http.StatusBadRequest)
return
}
glog.V(1).Infof("Iceberg: NamespaceExists error: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleDropNamespace deletes a namespace.
func (s *Server) handleDropNamespace(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
if len(namespace) == 0 {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
return
}
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
// Extract identity from context
identityName := s3_constants.GetIdentityNameFromContext(r)
deleteReq := &s3tables.DeleteNamespaceRequest{
TableBucketARN: bucketARN,
Namespace: namespace,
}
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "DeleteNamespace", deleteReq, nil, identityName)
})
if err != nil {
if strings.Contains(err.Error(), "not found") {
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
return
}
if strings.Contains(err.Error(), "not empty") {
writeError(w, http.StatusConflict, "NamespaceNotEmptyException", "Namespace is not empty")
return
}
glog.V(1).Infof("Iceberg: DropNamespace error: %v", err)
writeManagerError(w, err)
return
}
w.WriteHeader(http.StatusNoContent)
}

View File

@@ -0,0 +1,397 @@
package iceberg
import (
"crypto/hmac"
"crypto/sha256"
"fmt"
"net/http"
"os"
"strconv"
"strings"
"time"
jwt "github.com/golang-jwt/jwt/v5"
"github.com/seaweedfs/seaweedfs/weed/glog"
)
// OAuthTokenResponse is the response for POST /v1/oauth/tokens.
type OAuthTokenResponse struct {
AccessToken string `json:"access_token"`
TokenType string `json:"token_type"`
ExpiresIn int `json:"expires_in"`
Scope string `json:"scope,omitempty"`
// IssuedTokenType is set to the access-token URN on token-exchange
// responses, where RFC 8693 requires it.
IssuedTokenType string `json:"issued_token_type,omitempty"`
}
// OAuthErrorResponse is the error response for the OAuth endpoint.
type OAuthErrorResponse struct {
Error string `json:"error"`
Description string `json:"error_description,omitempty"`
}
// IcebergClaims are JWT claims for Iceberg catalog OAuth tokens.
type IcebergClaims struct {
IdentityName string `json:"identity_name"`
AccessKey string `json:"access_key"`
jwt.RegisteredClaims
}
const defaultOauthTokenExpiry = 3600
// maxOauthTokenExpiry bounds the configured TTL so seconds-to-Duration
// conversions cannot overflow into already-expired tokens.
const maxOauthTokenExpiry = 365 * 24 * 3600
// accessTokenTokenType is the RFC 8693 token-type URN for access tokens,
// required in issued_token_type on token-exchange responses.
const accessTokenTokenType = "urn:ietf:params:oauth:token-type:access_token"
// grant types accepted by POST /v1/oauth/tokens. Iceberg Java 1.10.x
// refreshes tokens via token-exchange (exchangeEnabled defaults true), so a
// server that only accepts client_credentials leaves those clients unable
// to refresh.
const (
grantTypeClientCredentials = "client_credentials"
grantTypeTokenExchange = "urn:ietf:params:oauth:grant-type:token-exchange"
)
// oauthExpirySeconds returns the OAuth token TTL. Deployments whose
// clients cannot refresh tokens on 401 can raise this to survive client
// restart cycles instead of dying every hour.
func oauthExpirySeconds() int {
if v := os.Getenv("ICEBERG_OAUTH_TOKEN_EXPIRY"); v != "" {
// Parse in 64-bit space: on 32-bit platforms Atoi overflows and
// errors on oversized values, which would silently fall back to
// the default instead of clamping.
if n, err := strconv.ParseInt(v, 10, 64); err == nil && n > 0 {
if n > maxOauthTokenExpiry {
return maxOauthTokenExpiry
}
return int(n)
}
}
return defaultOauthTokenExpiry
}
// tokenExchangeGrace is how long an expired subject token may still be
// exchanged for a fresh access token. Signature verification is the real
// gate; the grace exists so a client holding a token that expired while the
// exchange grant was unsupported (or during a server outage) recovers
// without a process restart.
func tokenExchangeGrace() time.Duration {
grace := 2 * oauthExpirySeconds()
if grace < 3600 {
grace = 3600
}
if grace > 86400 {
grace = 86400
}
return time.Duration(grace) * time.Second
}
// handleOAuthTokens implements the OAuth2 client_credentials flow.
// POST /v1/oauth/tokens
func (s *Server) handleOAuthTokens(w http.ResponseWriter, r *http.Request) {
if err := r.ParseForm(); err != nil {
writeOAuthError(w, http.StatusBadRequest, "invalid_request", "Could not parse form body")
return
}
// Reject credentials in query string to prevent leaking secrets into logs and caches.
if r.URL.Query().Get("client_secret") != "" {
writeOAuthError(w, http.StatusBadRequest, "invalid_request", "client_secret must not be sent in the URL")
return
}
grantType := r.PostFormValue("grant_type")
switch grantType {
case grantTypeClientCredentials:
// handled below
case grantTypeTokenExchange, "token_exchange":
s.handleTokenExchange(w, r)
return
default:
writeOAuthError(w, http.StatusBadRequest, "unsupported_grant_type",
fmt.Sprintf("Unsupported grant_type: %s", grantType))
return
}
clientID := r.PostFormValue("client_id")
clientSecret := r.PostFormValue("client_secret")
// Also support HTTP Basic auth per OAuth2 spec
if clientID == "" && clientSecret == "" {
var ok bool
clientID, clientSecret, ok = r.BasicAuth()
if !ok {
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "Missing client credentials")
return
}
}
if clientID == "" || clientSecret == "" {
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "Missing client_id or client_secret")
return
}
if s.credentialValidator == nil {
writeOAuthError(w, http.StatusInternalServerError, "server_error", "Credential validation not configured")
return
}
identityName, _, err := s.credentialValidator.ValidateS3Credential(clientID, clientSecret)
if err != nil {
glog.V(2).Infof("Iceberg OAuth: credential validation failed for client_id=%s: %v", clientID, err)
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "Invalid client credentials")
return
}
// Generate a JWT signed with a key derived from the client secret.
// Include the access key in claims so we can look up the exact credential for verification.
tokenString, err := mintIcebergToken(identityName, clientID, clientSecret, oauthExpirySeconds())
if err != nil {
glog.Errorf("Iceberg OAuth: failed to sign token: %v", err)
writeOAuthError(w, http.StatusInternalServerError, "server_error", "Failed to generate token")
return
}
scope := r.PostFormValue("scope")
resp := OAuthTokenResponse{
AccessToken: tokenString,
TokenType: "bearer",
ExpiresIn: oauthExpirySeconds(),
Scope: scope,
}
w.Header().Set("Cache-Control", "no-store")
writeJSON(w, http.StatusOK, resp)
}
// handleTokenExchange implements RFC 8693 token exchange for Iceberg REST
// clients: a previously issued access token (subject_token) is exchanged for
// a fresh one. Iceberg Java's OAuth2Manager refreshes via this grant
// (exchangeEnabled defaults true), so supporting it lets those clients
// self-heal before their token expires — no client restart needed.
//
// Client authentication (RFC 8693 §2.1) is accepted but not required:
// Iceberg Java's refreshExpiredToken sends Basic credentials with the
// exchange, while its proactive scheduled refresh sends only the Bearer
// session headers. An expired subject_token is only exchangeable with valid
// client credentials, and an unauthenticated exchange never extends the
// token's lifetime beyond the subject_token's own expiry.
func (s *Server) handleTokenExchange(w http.ResponseWriter, r *http.Request) {
if s.credentialValidator == nil {
writeOAuthError(w, http.StatusInternalServerError, "server_error", "Credential validation not configured")
return
}
subjectToken := r.PostFormValue("subject_token")
if subjectToken == "" {
writeOAuthError(w, http.StatusBadRequest, "invalid_request", "Missing subject_token")
return
}
// Verify the subject token by signature (exp checked separately against
// the recovery grace).
unverified := &IcebergClaims{}
parser := jwt.NewParser(jwt.WithoutClaimsValidation())
if _, _, err := parser.ParseUnverified(subjectToken, unverified); err != nil {
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
return
}
if unverified.AccessKey == "" || unverified.Issuer != "seaweedfs-iceberg" {
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
return
}
identityName, _, secretKey, err := s.credentialValidator.GetCredentialByAccessKey(unverified.AccessKey)
if err != nil {
glog.V(2).Infof("Iceberg OAuth: token exchange failed to get credential for access key: %v", err)
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
return
}
// Optional client authentication. When present, the client must
// authenticate with the same credential that issued the subject_token.
clientID, clientSecret, hasBasic := r.BasicAuth()
if !hasBasic {
clientID = r.PostFormValue("client_id")
clientSecret = r.PostFormValue("client_secret")
hasBasic = clientID != "" && clientSecret != ""
}
clientAuthenticated := false
if hasBasic {
if _, _, err := s.credentialValidator.ValidateS3Credential(clientID, clientSecret); err != nil {
glog.V(2).Infof("Iceberg OAuth: token exchange client authentication failed for client_id=%s: %v", clientID, err)
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "Invalid client credentials")
return
}
if clientID != unverified.AccessKey {
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "subject_token was issued to a different client")
return
}
clientAuthenticated = true
}
signingKey := deriveSigningKey(unverified.AccessKey, secretKey)
claims := &IcebergClaims{}
parsed, err := jwt.ParseWithClaims(subjectToken, claims, func(token *jwt.Token) (interface{}, error) {
if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
}
return signingKey, nil
}, jwt.WithoutClaimsValidation())
if err != nil || !parsed.Valid {
glog.V(2).Infof("Iceberg OAuth: token exchange signature verification failed: %v", err)
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
return
}
if claims.Issuer != "seaweedfs-iceberg" {
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
return
}
if claims.ExpiresAt == nil {
// Tokens minted by this server always carry an expiry; a subject
// token without one must not be exchangeable forever.
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "subject_token has no expiry")
return
}
ttlSeconds := oauthExpirySeconds()
if time.Now().After(claims.ExpiresAt.Time) {
// Expired subject tokens are only exchangeable by an authenticated
// client, within the recovery grace.
if !clientAuthenticated {
writeOAuthError(w, http.StatusBadRequest, "invalid_grant",
"subject_token is expired; client authentication is required to exchange it")
return
}
if time.Since(claims.ExpiresAt.Time) > tokenExchangeGrace() {
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "subject_token expired beyond the exchange grace window")
return
}
} else if remaining := int(time.Until(claims.ExpiresAt.Time).Seconds()); !clientAuthenticated && remaining < ttlSeconds {
// Unauthenticated exchange (proactive refresh carrying only the
// Bearer) must not extend the lifetime past the subject's expiry;
// otherwise a leaked token could chain-refresh forever. An
// authenticated client gets a fresh full TTL — it could mint one
// via client_credentials anyway.
ttlSeconds = remaining
}
if ttlSeconds <= 0 {
// A subject token with under a second left would otherwise be
// exchanged for a token that is born expired.
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "subject_token has no remaining lifetime")
return
}
tokenString, err := mintIcebergToken(identityName, unverified.AccessKey, secretKey, ttlSeconds)
if err != nil {
glog.Errorf("Iceberg OAuth: failed to sign exchanged token: %v", err)
writeOAuthError(w, http.StatusInternalServerError, "server_error", "Failed to generate token")
return
}
resp := OAuthTokenResponse{
AccessToken: tokenString,
TokenType: "bearer",
ExpiresIn: ttlSeconds,
Scope: r.PostFormValue("scope"),
IssuedTokenType: accessTokenTokenType,
}
w.Header().Set("Cache-Control", "no-store")
writeJSON(w, http.StatusOK, resp)
}
// mintIcebergToken issues a signed access token for the given identity and
// credential, valid for ttlSeconds.
func mintIcebergToken(identityName, accessKey, secret string, ttlSeconds int) (string, error) {
signingKey := deriveSigningKey(accessKey, secret)
now := time.Now()
claims := IcebergClaims{
IdentityName: identityName,
AccessKey: accessKey,
RegisteredClaims: jwt.RegisteredClaims{
IssuedAt: jwt.NewNumericDate(now),
ExpiresAt: jwt.NewNumericDate(now.Add(time.Duration(ttlSeconds) * time.Second)),
Issuer: "seaweedfs-iceberg",
},
}
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
return token.SignedString(signingKey)
}
// authenticateBearer validates a Bearer token from the Authorization header.
// Returns the identity name, identity object, and whether auth succeeded.
func (s *Server) authenticateBearer(r *http.Request) (string, interface{}, bool) {
auth := r.Header.Get("Authorization")
if auth == "" {
return "", nil, false
}
if !strings.HasPrefix(strings.ToLower(auth), "bearer ") {
return "", nil, false
}
tokenString := strings.TrimSpace(auth[7:])
if tokenString == "" {
return "", nil, false
}
if s.credentialValidator == nil {
return "", nil, false
}
// Parse the token without verification first to get the access key,
// then look up the exact credential to verify the signature.
parser := jwt.NewParser(jwt.WithoutClaimsValidation())
unverified := &IcebergClaims{}
_, _, err := parser.ParseUnverified(tokenString, unverified)
if err != nil {
glog.V(2).Infof("Iceberg OAuth: failed to parse token: %v", err)
return "", nil, false
}
if unverified.AccessKey == "" {
return "", nil, false
}
// Look up the credential by access key to get the signing key for verification
identityName, identity, secretKey, err := s.credentialValidator.GetCredentialByAccessKey(unverified.AccessKey)
if err != nil {
glog.V(2).Infof("Iceberg OAuth: failed to get credential for access key: %v", err)
return "", nil, false
}
signingKey := deriveSigningKey(unverified.AccessKey, secretKey)
claims := &IcebergClaims{}
verified, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
}
return signingKey, nil
})
if err != nil || !verified.Valid {
glog.V(2).Infof("Iceberg OAuth: token verification failed: %v", err)
return "", nil, false
}
return identityName, identity, true
}
// deriveSigningKey derives a signing key from the access key and secret using HMAC-SHA256.
// Including the access key prevents cross-credential token forgery when two
// credentials happen to share the same secret.
func deriveSigningKey(accessKey, secret string) []byte {
h := hmac.New(sha256.New, []byte("seaweedfs-iceberg-oauth"))
h.Write([]byte(accessKey))
h.Write([]byte{0}) // null separator
h.Write([]byte(secret))
return h.Sum(nil)
}
func writeOAuthError(w http.ResponseWriter, status int, errCode, description string) {
resp := OAuthErrorResponse{
Error: errCode,
Description: description,
}
writeJSON(w, status, resp)
}

View File

@@ -0,0 +1,410 @@
package iceberg
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
jwt "github.com/golang-jwt/jwt/v5"
)
type mockCredentialValidator struct {
credentials map[string]string // accessKey -> secretKey
identities map[string]string // accessKey -> identityName
}
func (m *mockCredentialValidator) ValidateS3Credential(accessKey, secretKey string) (string, interface{}, error) {
expected, ok := m.credentials[accessKey]
if !ok {
return "", nil, fmt.Errorf("access key not found")
}
if expected != secretKey {
return "", nil, fmt.Errorf("invalid secret key")
}
return m.identities[accessKey], nil, nil
}
func (m *mockCredentialValidator) GetCredentialByAccessKey(accessKey string) (string, interface{}, string, error) {
secret, ok := m.credentials[accessKey]
if !ok {
return "", nil, "", fmt.Errorf("access key not found")
}
return m.identities[accessKey], nil, secret, nil
}
func newTestServerWithOAuth() *Server {
cv := &mockCredentialValidator{
credentials: map[string]string{"AKID123": "secret456"},
identities: map[string]string{"AKID123": "testuser"},
}
s := &Server{
credentialValidator: cv,
}
return s
}
func TestHandleOAuthTokens_Success(t *testing.T) {
s := newTestServerWithOAuth()
body := "grant_type=client_credentials&client_id=AKID123&client_secret=secret456"
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
w := httptest.NewRecorder()
s.handleOAuthTokens(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp OAuthTokenResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
if resp.TokenType != "bearer" {
t.Errorf("expected token_type=bearer, got %s", resp.TokenType)
}
if resp.AccessToken == "" {
t.Error("expected non-empty access_token")
}
if resp.ExpiresIn != oauthExpirySeconds() {
t.Errorf("expected expires_in=%d, got %d", oauthExpirySeconds(), resp.ExpiresIn)
}
}
func TestHandleOAuthTokens_InvalidCredentials(t *testing.T) {
s := newTestServerWithOAuth()
body := "grant_type=client_credentials&client_id=AKID123&client_secret=wrongsecret"
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
w := httptest.NewRecorder()
s.handleOAuthTokens(w, req)
if w.Code != http.StatusUnauthorized {
t.Fatalf("expected 401, got %d: %s", w.Code, w.Body.String())
}
}
func TestHandleOAuthTokens_UnsupportedGrantType(t *testing.T) {
s := newTestServerWithOAuth()
body := "grant_type=authorization_code&client_id=AKID123&client_secret=secret456"
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
w := httptest.NewRecorder()
s.handleOAuthTokens(w, req)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
}
}
func TestBearerTokenRoundTrip(t *testing.T) {
s := newTestServerWithOAuth()
// Get a token
body := "grant_type=client_credentials&client_id=AKID123&client_secret=secret456"
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
w := httptest.NewRecorder()
s.handleOAuthTokens(w, req)
var resp OAuthTokenResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
// Use the token for Bearer auth
authReq := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
authReq.Header.Set("Authorization", "Bearer "+resp.AccessToken)
identityName, _, ok := s.authenticateBearer(authReq)
if !ok {
t.Fatal("expected Bearer auth to succeed")
}
if identityName != "testuser" {
t.Errorf("expected identity 'testuser', got '%s'", identityName)
}
}
func TestBearerTokenInvalid(t *testing.T) {
s := newTestServerWithOAuth()
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
req.Header.Set("Authorization", "Bearer invalid-token")
_, _, ok := s.authenticateBearer(req)
if ok {
t.Error("expected Bearer auth to fail with invalid token")
}
}
func TestBearerTokenNone(t *testing.T) {
s := newTestServerWithOAuth()
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
_, _, ok := s.authenticateBearer(req)
if ok {
t.Error("expected Bearer auth to fail with no token")
}
}
// TestOauthExpirySecondsEnvOverride pins the TTL knob: the token TTL must
// be configurable so clients that cannot refresh on 401 can be given
// longer-lived tokens instead of dying every hour.
func TestOauthExpirySecondsEnvOverride(t *testing.T) {
if got := oauthExpirySeconds(); got != defaultOauthTokenExpiry {
t.Fatalf("default TTL = %d, want %d", got, defaultOauthTokenExpiry)
}
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "86400")
if got := oauthExpirySeconds(); got != 86400 {
t.Fatalf("env TTL = %d, want 86400", got)
}
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "-5")
if got := oauthExpirySeconds(); got != defaultOauthTokenExpiry {
t.Fatalf("negative TTL must fall back to default, got %d", got)
}
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "garbage")
if got := oauthExpirySeconds(); got != defaultOauthTokenExpiry {
t.Fatalf("invalid TTL must fall back to default, got %d", got)
}
// oversized values are clamped so Duration math cannot overflow
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "99999999999999999")
if got := oauthExpirySeconds(); got != maxOauthTokenExpiry {
t.Fatalf("oversized TTL must clamp to %d, got %d", maxOauthTokenExpiry, got)
}
// a token minted under an override carries the override's expiry
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "7200")
s := newTestServerWithOAuth()
body := "grant_type=client_credentials&client_id=AKID123&client_secret=secret456"
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
w := httptest.NewRecorder()
s.handleOAuthTokens(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
var resp OAuthTokenResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
if resp.ExpiresIn != 7200 {
t.Fatalf("minted token expires_in = %d, want 7200", resp.ExpiresIn)
}
}
func exchangeRequest(t *testing.T, subjectToken string) *httptest.ResponseRecorder {
return exchangeRequestWithBasic(t, subjectToken, "", "")
}
// exchangeRequestWithBasic mirrors Iceberg Java: refreshExpiredToken sends
// Basic client credentials with the token-exchange grant, the proactive
// scheduled refresh sends only the form body.
func exchangeRequestWithBasic(t *testing.T, subjectToken, basicUser, basicPass string) *httptest.ResponseRecorder {
t.Helper()
s := newTestServerWithOAuth()
body := "grant_type=urn:ietf:params:oauth:grant-type:token-exchange&subject_token=" + url.QueryEscape(subjectToken)
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
if basicUser != "" || basicPass != "" {
req.SetBasicAuth(basicUser, basicPass)
}
w := httptest.NewRecorder()
s.handleOAuthTokens(w, req)
return w
}
// TestTokenExchangeLiveSubject: the proactive refresh path Iceberg Java
// 1.10.x uses (exchangeEnabled defaults true, Bearer-only headers) must
// mint a fresh working token.
func TestTokenExchangeLiveSubject(t *testing.T) {
s := newTestServerWithOAuth()
now := time.Now()
live := mintTestToken(t, "AKID123", "secret456", now, now.Add(30*time.Minute))
w := exchangeRequest(t, live)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp OAuthTokenResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
if resp.TokenType != "bearer" || resp.AccessToken == "" {
t.Fatalf("bad token response: %+v", resp)
}
if resp.IssuedTokenType != accessTokenTokenType {
t.Fatalf("issued_token_type = %q, want %s", resp.IssuedTokenType, accessTokenTokenType)
}
// Unauthenticated exchange must not outlive the subject token.
if resp.ExpiresIn > 30*60 {
t.Fatalf("expires_in = %d, must be bounded by the subject's remaining lifetime (<= 1800)", resp.ExpiresIn)
}
// the exchanged token must authenticate like a normal Bearer
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
req.Header.Set("Authorization", "Bearer "+resp.AccessToken)
if _, _, ok := s.authenticateBearer(req); !ok {
t.Fatalf("exchanged token must pass authenticateBearer")
}
}
// TestTokenExchangeExpiredWithinGrace: a client whose token expired while
// exchange was unsupported must recover without a restart. This is the
// refreshExpiredToken path, which authenticates with Basic credentials.
func TestTokenExchangeExpiredWithinGrace(t *testing.T) {
now := time.Now()
expiredRecently := mintTestToken(t, "AKID123", "secret456", now.Add(-20*time.Minute), now.Add(-10*time.Minute))
w := exchangeRequestWithBasic(t, expiredRecently, "AKID123", "secret456")
if w.Code != http.StatusOK {
t.Fatalf("expected 200 within grace, got %d: %s", w.Code, w.Body.String())
}
var resp OAuthTokenResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
// Authenticated refresh gets the full configured TTL.
if resp.ExpiresIn != oauthExpirySeconds() {
t.Fatalf("expires_in = %d, want %d", resp.ExpiresIn, oauthExpirySeconds())
}
}
// TestTokenExchangeAuthenticatedLiveSubjectFullTTL: an authenticated client
// exchanging a live token renews the session — the TTL cap only applies to
// unauthenticated (Bearer-only) exchanges.
func TestTokenExchangeAuthenticatedLiveSubjectFullTTL(t *testing.T) {
now := time.Now()
live := mintTestToken(t, "AKID123", "secret456", now, now.Add(30*time.Minute))
w := exchangeRequestWithBasic(t, live, "AKID123", "secret456")
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp OAuthTokenResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
if resp.ExpiresIn != oauthExpirySeconds() {
t.Fatalf("authenticated live exchange expires_in = %d, want full TTL %d", resp.ExpiresIn, oauthExpirySeconds())
}
}
// TestTokenExchangeSubjectNearlyExpired: a Bearer-only exchange when the
// subject token has under a second left must be rejected, not minted into
// an already-expired token (expires_in: 0).
func TestTokenExchangeSubjectNearlyExpired(t *testing.T) {
// jwt/v5 serializes exp at one-second precision, so pin the expiry to
// the next whole-second boundary: the token is always live when minted
// and exchanged, yet has under a second of remaining lifetime.
exp := time.Now().Truncate(time.Second).Add(time.Second)
nearlyDead := mintTestToken(t, "AKID123", "secret456", exp.Add(-time.Hour), exp)
w := exchangeRequest(t, nearlyDead)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
}
if !strings.Contains(w.Body.String(), "invalid_grant") {
t.Fatalf("expected invalid_grant, got: %s", w.Body.String())
}
}
// TestTokenExchangeExpiredWithoutClientAuth: a leaked expired token must
// not be exchangeable without client credentials.
func TestTokenExchangeExpiredWithoutClientAuth(t *testing.T) {
now := time.Now()
expiredRecently := mintTestToken(t, "AKID123", "secret456", now.Add(-20*time.Minute), now.Add(-10*time.Minute))
w := exchangeRequest(t, expiredRecently)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
}
if !strings.Contains(w.Body.String(), "invalid_grant") {
t.Fatalf("expected invalid_grant, got: %s", w.Body.String())
}
}
// TestTokenExchangeExpiredBeyondGrace: stale tokens far past expiry must not
// act as eternal credentials, even with client auth.
func TestTokenExchangeExpiredBeyondGrace(t *testing.T) {
now := time.Now()
longDead := mintTestToken(t, "AKID123", "secret456", now.Add(-48*time.Hour), now.Add(-47*time.Hour))
w := exchangeRequestWithBasic(t, longDead, "AKID123", "secret456")
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400 beyond grace, got %d: %s", w.Code, w.Body.String())
}
if !strings.Contains(w.Body.String(), "invalid_grant") {
t.Fatalf("expected invalid_grant, got: %s", w.Body.String())
}
}
// TestTokenExchangeGarbageSubject: malformed subject tokens are rejected
// with 400 invalid_grant per RFC 6749 §5.2.
func TestTokenExchangeGarbageSubject(t *testing.T) {
w := exchangeRequest(t, "not-a-jwt")
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d", w.Code)
}
}
// TestTokenExchangeWrongClientCredentials: valid subject token, but Basic
// auth with wrong credentials or a different client is rejected.
func TestTokenExchangeWrongClientCredentials(t *testing.T) {
now := time.Now()
live := mintTestToken(t, "AKID123", "secret456", now, now.Add(30*time.Minute))
// wrong secret → invalid_client 401
w := exchangeRequestWithBasic(t, live, "AKID123", "wrongsecret")
if w.Code != http.StatusUnauthorized {
t.Fatalf("wrong secret: expected 401, got %d", w.Code)
}
// a different valid client must not exchange someone else's token
s := &Server{credentialValidator: &mockCredentialValidator{
credentials: map[string]string{"AKID123": "secret456", "AKID999": "other-secret"},
identities: map[string]string{"AKID123": "testuser", "AKID999": "otheruser"},
}}
body := "grant_type=urn:ietf:params:oauth:grant-type:token-exchange&subject_token=" + url.QueryEscape(live)
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.SetBasicAuth("AKID999", "other-secret")
rec := httptest.NewRecorder()
s.handleOAuthTokens(rec, req)
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "invalid_grant") {
t.Fatalf("cross-client exchange: expected 400 invalid_grant, got %d: %s", rec.Code, rec.Body.String())
}
}
// TestTokenExchangeSubjectWithoutExpiry: a correctly signed subject token
// without an exp claim must be rejected — it would otherwise refresh forever.
func TestTokenExchangeSubjectWithoutExpiry(t *testing.T) {
claims := IcebergClaims{
IdentityName: "testuser",
AccessKey: "AKID123",
RegisteredClaims: jwt.RegisteredClaims{
IssuedAt: jwt.NewNumericDate(time.Now()),
Issuer: "seaweedfs-iceberg",
// no ExpiresAt
},
}
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
signed, err := token.SignedString(deriveSigningKey("AKID123", "secret456"))
if err != nil {
t.Fatal(err)
}
w := exchangeRequestWithBasic(t, signed, "AKID123", "secret456")
if w.Code != http.StatusBadRequest || !strings.Contains(w.Body.String(), "invalid_grant") {
t.Fatalf("expected 400 invalid_grant for nil-exp subject, got %d: %s", w.Code, w.Body.String())
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,364 @@
package iceberg
import (
"context"
"encoding/json"
"fmt"
"net/http"
"path"
"strings"
"github.com/apache/iceberg-go/table"
"github.com/google/uuid"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// CommitTransactionRequest is sent to POST /v1/transactions/commit.
type CommitTransactionRequest struct {
TableChanges []tableChangeRequest `json:"table-changes"`
}
type tableChangeRequest struct {
Identifier *TableIdentifier `json:"identifier"`
Requirements json.RawMessage `json:"requirements"`
Updates []json.RawMessage `json:"updates"`
}
// preparedTableCommit holds the per-table work resolved during validation so
// pointer flips (and their rollback) can run after every table is validated.
type preparedTableCommit struct {
namespace []string
tableName string
tableUUID uuid.UUID
versionToken string
metadataBucket string
metadataPath string
location string
metadataFileName string
metadataBytes []byte
metadataVersion int
newMetadataLoc string
// prior table state captured before the flip, so a rollback can revert
// every field handleUpdateTable would have mutated (not just the location).
prevMetadataLoc string
prevMetadataVersion int
prevMetadata *s3tables.TableMetadata
}
// handleCommitTransaction commits changes to multiple tables in one request.
// Validation is atomic (all requirements evaluated before any write); pointer
// flips are best-effort with rollback, so this is not crash-atomic.
func (s *Server) handleCommitTransaction(w http.ResponseWriter, r *http.Request) {
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
identityName := s3_constants.GetIdentityNameFromContext(r)
var req CommitTransactionRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body: "+err.Error())
return
}
if len(req.TableChanges) == 0 {
w.WriteHeader(http.StatusNoContent)
return
}
// Phase 1: resolve, load, and validate every table; build new metadata bytes.
prepared := make([]preparedTableCommit, 0, len(req.TableChanges))
for _, change := range req.TableChanges {
if change.Identifier == nil || len(change.Identifier.Namespace) == 0 || change.Identifier.Name == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "Each table change requires identifier namespace and name")
return
}
pc, reqErr := s.prepareTableCommit(r.Context(), bucketName, bucketARN, identityName, change)
if reqErr != nil {
writeError(w, reqErr.status, reqErr.errType, reqErr.message)
return
}
prepared = append(prepared, *pc)
}
// Phase 2: write each new metadata.json object. Staging is exclusive for the
// same reason a single-table commit stages exclusively: a transaction racing
// another writer on one of its tables would otherwise overwrite that
// writer's metadata, and the pointer flip below decides who won.
for i := range prepared {
pc := &prepared[i]
fileName, location, err := s.stageCommitMetadata(r.Context(), pc.metadataBucket, pc.metadataPath, pc.location, pc.metadataFileName, pc.metadataBytes)
if err != nil {
// No pointer flipped yet, so every written file is safe to delete.
s.cleanupPreparedMetadata(r.Context(), prepared[:i+1], nil)
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to save metadata file: "+err.Error())
return
}
pc.metadataFileName = fileName
pc.newMetadataLoc = location
}
// Phase 3: flip each table's pointer xattr; on failure roll back prior flips.
for i := range prepared {
pc := &prepared[i]
if err := s.flipTablePointer(r.Context(), bucketARN, identityName, pc); err != nil {
rolledBack := s.rollbackTablePointers(r.Context(), bucketARN, identityName, prepared[:i])
s.cleanupPreparedMetadata(r.Context(), prepared, cleanupSafeMetadata(prepared, i, rolledBack))
if isS3TablesConflict(err) {
writeError(w, http.StatusConflict, "CommitFailedException", "Version token mismatch")
return
}
glog.Errorf("Iceberg: CommitTransaction UpdateTable error: %v", err)
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to commit table update: "+err.Error())
return
}
}
w.WriteHeader(http.StatusNoContent)
}
func (s *Server) prepareTableCommit(ctx context.Context, bucketName, bucketARN, identityName string, change tableChangeRequest) (*preparedTableCommit, *icebergRequestError) {
namespace := []string(change.Identifier.Namespace)
tableName := change.Identifier.Name
requirements, updates, statisticsUpdates, reqErr := parseTableChange(change)
if reqErr != nil {
return nil, reqErr
}
getReq := &s3tables.GetTableRequest{TableBucketARN: bucketARN, Namespace: namespace, Name: tableName}
var getResp s3tables.GetTableResponse
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(ctx, mgrClient, "GetTable", getReq, &getResp, identityName)
})
if err != nil {
if isS3TablesNotFound(err) {
return nil, &icebergRequestError{http.StatusNotFound, "NoSuchTableException", fmt.Sprintf("Table does not exist: %s", tableName)}
}
glog.V(1).Infof("Iceberg: CommitTransaction GetTable error: %v", err)
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", err.Error()}
}
location := tableLocationFromMetadataLocation(getResp.MetadataLocation)
if location == "" {
location = fmt.Sprintf("s3://%s/%s", bucketName, path.Join(flattenNamespacePath(namespace), tableName))
}
tableUUID := uuid.Nil
if getResp.Metadata != nil && getResp.Metadata.Iceberg != nil && getResp.Metadata.Iceberg.TableUUID != "" {
if parsed, parseErr := uuid.Parse(getResp.Metadata.Iceberg.TableUUID); parseErr == nil {
tableUUID = parsed
}
}
if tableUUID == uuid.Nil {
tableUUID = uuid.New()
}
var currentMetadata table.Metadata
if getResp.Metadata != nil && len(getResp.Metadata.FullMetadata) > 0 {
currentMetadata, err = table.ParseMetadataBytes(getResp.Metadata.FullMetadata)
if err != nil {
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Failed to parse current metadata"}
}
} else {
currentMetadata, err = newTableMetadata(tableUUID, location, nil, nil, nil, nil)
if err != nil {
glog.Errorf("Iceberg: CommitTransaction placeholder metadata for %s: %v", tableName, err)
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Failed to build current metadata"}
}
}
for _, requirement := range requirements {
if err := requirement.Validate(currentMetadata); err != nil {
return nil, &icebergRequestError{http.StatusConflict, "CommitFailedException", "Requirement failed: " + err.Error()}
}
}
builder, err := table.MetadataBuilderFromBase(currentMetadata, getResp.MetadataLocation)
if err != nil {
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Failed to create metadata builder: " + err.Error()}
}
for _, update := range updates {
if err := update.Apply(builder); err != nil {
return nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Failed to apply update: " + err.Error()}
}
}
newMetadata, err := builder.Build()
if err != nil {
return nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Failed to build new metadata: " + err.Error()}
}
metadataVersion := getResp.MetadataVersion + 1
metadataFileName := fmt.Sprintf("v%d.metadata.json", metadataVersion)
newMetadataLocation := fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(location, "/"), metadataFileName)
metadataBytes, err := json.Marshal(newMetadata)
if err != nil {
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Failed to serialize metadata: " + err.Error()}
}
metadataBytes, err = applyStatisticsUpdates(metadataBytes, statisticsUpdates)
if err != nil {
return nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Failed to apply statistics updates: " + err.Error()}
}
metadataBytes = ensureMetadataSpecCompliance(metadataBytes)
metadataBucket, metadataPath, err := parseS3Location(location)
if err != nil {
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Invalid table location: " + err.Error()}
}
if err := confineMetadataLocation(metadataBucket, metadataPath, bucketName); err != nil {
return nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", err.Error()}
}
return &preparedTableCommit{
namespace: namespace,
tableName: tableName,
tableUUID: tableUUID,
versionToken: getResp.VersionToken,
metadataBucket: metadataBucket,
metadataPath: metadataPath,
location: location,
metadataFileName: metadataFileName,
metadataBytes: metadataBytes,
metadataVersion: metadataVersion,
newMetadataLoc: newMetadataLocation,
prevMetadataLoc: getResp.MetadataLocation,
prevMetadataVersion: getResp.MetadataVersion,
prevMetadata: cloneTableMetadata(getResp.Metadata),
}, nil
}
// cloneTableMetadata deep-copies the prior table metadata so a later rollback
// restores the exact pre-transaction bytes without aliasing the get response.
func cloneTableMetadata(m *s3tables.TableMetadata) *s3tables.TableMetadata {
if m == nil {
return nil
}
clone := &s3tables.TableMetadata{}
if m.Iceberg != nil {
iceberg := *m.Iceberg
clone.Iceberg = &iceberg
}
if len(m.FullMetadata) > 0 {
clone.FullMetadata = append(json.RawMessage(nil), m.FullMetadata...)
}
return clone
}
func parseTableChange(change tableChangeRequest) (table.Requirements, table.Updates, []statisticsUpdate, *icebergRequestError) {
var requirements table.Requirements
if len(change.Requirements) > 0 {
normalized, err := normalizeRequirements(change.Requirements)
if err != nil {
return nil, nil, nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Invalid requirements: " + err.Error()}
}
if err := json.Unmarshal(normalized, &requirements); err != nil {
return nil, nil, nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Invalid requirements: " + err.Error()}
}
}
var updates table.Updates
var statisticsUpdates []statisticsUpdate
if len(change.Updates) > 0 {
var err error
updates, statisticsUpdates, err = parseCommitUpdates(change.Updates)
if err != nil {
return nil, nil, nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Invalid updates: " + err.Error()}
}
}
return requirements, updates, statisticsUpdates, nil
}
func (s *Server) flipTablePointer(ctx context.Context, bucketARN, identityName string, pc *preparedTableCommit) error {
updateReq := &s3tables.UpdateTableRequest{
TableBucketARN: bucketARN,
Namespace: pc.namespace,
Name: pc.tableName,
VersionToken: pc.versionToken,
Metadata: &s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{TableUUID: pc.tableUUID.String()},
FullMetadata: pc.metadataBytes,
},
MetadataVersion: pc.metadataVersion,
MetadataLocation: pc.newMetadataLoc,
}
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(ctx, mgrClient, "UpdateTable", updateReq, nil, identityName)
})
}
// rollbackTablePointers restores already-flipped tables to their full prior
// state. handleUpdateTable applies partial field updates, so the restore must
// re-send every field the flip changed (location, version, and full metadata),
// not just the location. Best-effort: a failed rollback is logged, not surfaced.
// The returned slice flags, per flipped table, whether the restore succeeded so
// the caller can avoid deleting metadata a still-flipped pointer references.
func (s *Server) rollbackTablePointers(ctx context.Context, bucketARN, identityName string, flipped []preparedTableCommit) []bool {
restored := make([]bool, len(flipped))
for i := range flipped {
pc := &flipped[i]
updateReq := buildTableRestoreRequest(bucketARN, pc)
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(ctx, mgrClient, "UpdateTable", updateReq, nil, identityName)
})
if err != nil {
glog.Errorf("Iceberg: CommitTransaction rollback of %s failed: %v", pc.tableName, err)
continue
}
restored[i] = true
}
return restored
}
// buildTableRestoreRequest reconstructs the UpdateTableRequest that reverts a
// flipped table to its captured prior state. Every field handleUpdateTable
// would have mutated on the flip is re-supplied here so the partial update
// fully restores it (ModifiedAt and VersionToken are always regenerated by the
// handler and cannot be pinned back).
func buildTableRestoreRequest(bucketARN string, pc *preparedTableCommit) *s3tables.UpdateTableRequest {
return &s3tables.UpdateTableRequest{
TableBucketARN: bucketARN,
Namespace: pc.namespace,
Name: pc.tableName,
Metadata: cloneTableMetadata(pc.prevMetadata),
MetadataVersion: pc.prevMetadataVersion,
MetadataLocation: pc.prevMetadataLoc,
}
}
// cleanupSafeMetadata reports, per prepared table, whether deleting its newly
// written metadata file is safe after a flip failed at failedIndex. A file is
// safe to delete only when its table's pointer no longer references it: tables
// at or past failedIndex were never flipped, and earlier tables are safe only
// if their rollback succeeded. A table whose rollback failed still points at the
// new metadata, so deleting it would strand the pointer on a missing file.
func cleanupSafeMetadata(prepared []preparedTableCommit, failedIndex int, rolledBack []bool) []bool {
safe := make([]bool, len(prepared))
for i := range prepared {
switch {
case i >= failedIndex:
safe[i] = true
case i < len(rolledBack):
safe[i] = rolledBack[i]
}
}
return safe
}
// cleanupPreparedMetadata deletes the newly written metadata file for each
// prepared table flagged safe; unflagged tables are left in place because a
// still-flipped pointer references them.
func (s *Server) cleanupPreparedMetadata(ctx context.Context, prepared []preparedTableCommit, safe []bool) {
for i := range prepared {
pc := &prepared[i]
if i < len(safe) && !safe[i] {
glog.Errorf("Iceberg: CommitTransaction keeping metadata %s; %s rollback failed and still references it", pc.newMetadataLoc, pc.tableName)
continue
}
if cleanupErr := s.deleteMetadataFile(ctx, pc.metadataBucket, pc.metadataPath, pc.metadataFileName); cleanupErr != nil {
glog.V(1).Infof("Iceberg: failed to cleanup metadata file %s: %v", pc.newMetadataLoc, cleanupErr)
}
}
}

View File

@@ -0,0 +1,449 @@
package iceberg
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"path"
"strings"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/view"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// handleListViews lists views in a namespace.
func (s *Server) handleListViews(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
if len(namespace) == 0 {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
return
}
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
identityName := s3_constants.GetIdentityNameFromContext(r)
pageToken, pageSize, err := parsePagination(r)
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
return
}
listReq := &s3tables.ListViewsRequest{
TableBucketARN: bucketARN,
Namespace: namespace,
ContinuationToken: pageToken,
MaxViews: pageSize,
}
var listResp s3tables.ListViewsResponse
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "ListViews", listReq, &listResp, identityName)
})
if err != nil {
if strings.Contains(err.Error(), "not found") {
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
return
}
glog.V(1).Infof("Iceberg: ListViews error: %v", err)
writeManagerError(w, err)
return
}
identifiers := make([]TableIdentifier, 0, len(listResp.Views))
for _, v := range listResp.Views {
identifiers = append(identifiers, TableIdentifier{Namespace: namespace, Name: v.Name})
}
writeJSON(w, http.StatusOK, ListViewsResponse{
NextPageToken: listResp.ContinuationToken,
Identifiers: identifiers,
})
}
// handleCreateView creates a new view and writes its v1 metadata.json.
func (s *Server) handleCreateView(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
if len(namespace) == 0 {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
return
}
var req CreateViewRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
return
}
if req.Name == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "view name is required")
return
}
if req.Schema == nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "view schema is required")
return
}
if req.ViewVersion == nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "view-version is required")
return
}
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
identityName := s3_constants.GetIdentityNameFromContext(r)
viewPath := path.Join(flattenNamespacePath(namespace), req.Name)
location := strings.TrimSuffix(req.Location, "/")
if location == "" {
if req.Properties != nil {
if warehouse := strings.TrimSuffix(req.Properties["warehouse"], "/"); warehouse != "" {
location = fmt.Sprintf("%s/%s", warehouse, viewPath)
}
}
if location == "" {
if warehouse := strings.TrimSuffix(os.Getenv("ICEBERG_WAREHOUSE"), "/"); warehouse != "" {
location = fmt.Sprintf("%s/%s", warehouse, viewPath)
}
}
if location == "" {
location = fmt.Sprintf("s3://%s/%s", bucketName, viewPath)
}
} else {
parsedBucket, parsedPath, err := parseS3Location(location)
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid view location: "+err.Error())
return
}
if parsedPath == "" {
location = fmt.Sprintf("s3://%s/%s", parsedBucket, viewPath)
}
}
metadataBucket, metadataPath, err := parseS3Location(location)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid view location: "+err.Error())
return
}
if metadataBucket != bucketName {
writeError(w, http.StatusBadRequest, "BadRequestException", "view location must be within bucket "+bucketName)
return
}
if !isValidTablePath(metadataPath) {
writeError(w, http.StatusBadRequest, "BadRequestException", "invalid view location path")
return
}
metadata, err := view.NewMetadata(req.ViewVersion, req.Schema, location, viewProperties(req.Properties))
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to build view metadata: "+err.Error())
return
}
metadataFileName := "v1.metadata.json"
metadataLocation := fmt.Sprintf("%s/metadata/%s", location, metadataFileName)
metadataBytes, err := json.Marshal(metadata)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to serialize view metadata: "+err.Error())
return
}
// Authoritative existence check before touching storage: a registered view
// short-circuits with its stored definition (idempotent CreateView) so we
// never overwrite the persisted metadata of an existing view.
if existsResp, existsErr := s.getView(r, namespace, req.Name); existsErr == nil {
result, buildErr := s.buildViewResponse(r, existsResp, bucketName, namespace, req.Name)
if buildErr != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", buildErr.Error())
return
}
writeLoadResult(w, http.StatusOK, result)
return
} else if !isViewNotFound(existsErr) {
glog.V(1).Infof("Iceberg: CreateView existence check failed for %s.%s: %v", flattenNamespacePath(namespace), req.Name, existsErr)
writeManagerError(w, existsErr)
return
}
createReq := &s3tables.CreateViewRequest{
TableBucketARN: bucketARN,
Namespace: namespace,
Name: req.Name,
Metadata: &s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{TableUUID: metadata.ViewUUID().String()},
FullMetadata: metadataBytes,
},
MetadataLocation: metadataLocation,
MetadataVersion: 1,
}
var createResp s3tables.CreateViewResponse
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "CreateView", createReq, &createResp, identityName)
})
if err != nil {
if isViewAlreadyExists(err) {
writeError(w, http.StatusConflict, "AlreadyExistsException", err.Error())
return
}
var viewErr *s3tables.S3TablesError
if errors.As(err, &viewErr) && viewErr.Type == s3tables.ErrCodeNoSuchNamespace {
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
return
}
glog.V(1).Infof("Iceberg: CreateView error: %v", err)
writeManagerError(w, err)
return
}
// Persist the metadata file only after the catalog registers the view, so a
// missing namespace or name collision fails before any bytes hit storage.
if err := s.saveMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName, metadataBytes, false); err != nil {
// Roll back the registered view so it doesn't linger pointing at metadata
// that was never written.
if dropErr := s.dropView(r, namespace, req.Name); dropErr != nil {
glog.V(1).Infof("Iceberg: failed to roll back view %s.%s after metadata write error: %v", flattenNamespacePath(namespace), req.Name, dropErr)
}
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to save view metadata file: "+err.Error())
return
}
finalLocation := createResp.MetadataLocation
if finalLocation == "" {
finalLocation = metadataLocation
}
config, _ := s.buildFileIOConfig(r, location)
writeLoadResult(w, http.StatusOK, ViewResponse{
MetadataLocation: finalLocation,
Metadata: metadata,
Config: config,
})
}
// handleLoadView loads view metadata.
func (s *Server) handleLoadView(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
viewName := vars["view"]
if len(namespace) == 0 || viewName == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and view name are required")
return
}
getResp, err := s.getView(r, namespace, viewName)
if err != nil {
if isViewNotFound(err) {
writeError(w, http.StatusNotFound, "NoSuchViewException", fmt.Sprintf("View does not exist: %s", viewName))
return
}
glog.V(1).Infof("Iceberg: LoadView error: %v", err)
writeManagerError(w, err)
return
}
result, err := s.buildViewResponse(r, getResp, getBucketFromPrefix(r), namespace, viewName)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
return
}
writeLoadResult(w, http.StatusOK, result)
}
// handleViewExists checks if a view exists.
func (s *Server) handleViewExists(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
viewName := vars["view"]
if len(namespace) == 0 || viewName == "" {
w.WriteHeader(http.StatusBadRequest)
return
}
if _, err := s.getView(r, namespace, viewName); err != nil {
w.WriteHeader(http.StatusNotFound)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleDropView deletes a view.
func (s *Server) handleDropView(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
viewName := vars["view"]
if len(namespace) == 0 || viewName == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and view name are required")
return
}
var storedMetadataLocation string
if getResp, err := s.getView(r, namespace, viewName); err == nil {
storedMetadataLocation = getResp.MetadataLocation
}
err := s.dropView(r, namespace, viewName)
if err != nil {
if isViewNotFound(err) {
writeError(w, http.StatusNotFound, "NoSuchViewException", fmt.Sprintf("View does not exist: %s", viewName))
return
}
glog.V(1).Infof("Iceberg: DropView error: %v", err)
writeManagerError(w, err)
return
}
if storedMetadataLocation != "" {
if viewLoc := tableLocationFromMetadataLocation(storedMetadataLocation); viewLoc != "" {
if dataBucket, dataPath, parseErr := parseS3Location(viewLoc); parseErr == nil {
if cleanupErr := s.cleanupStaleTableLocation(r.Context(), dataBucket, dataPath); cleanupErr != nil {
glog.V(1).Infof("Iceberg: failed to purge dropped view location s3://%s/%s: %v", dataBucket, dataPath, cleanupErr)
}
}
}
}
w.WriteHeader(http.StatusNoContent)
}
// dropView removes a view registration from the catalog.
func (s *Server) dropView(r *http.Request, namespace []string, viewName string) error {
deleteReq := &s3tables.DeleteViewRequest{
TableBucketARN: buildTableBucketARN(getBucketFromPrefix(r)),
Namespace: namespace,
Name: viewName,
}
identityName := s3_constants.GetIdentityNameFromContext(r)
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "DeleteView", deleteReq, nil, identityName)
})
}
// getView fetches a view's stored metadata pointer from the catalog.
func (s *Server) getView(r *http.Request, namespace []string, viewName string) (s3tables.GetViewResponse, error) {
getReq := &s3tables.GetViewRequest{
TableBucketARN: buildTableBucketARN(getBucketFromPrefix(r)),
Namespace: namespace,
Name: viewName,
}
var getResp s3tables.GetViewResponse
identityName := s3_constants.GetIdentityNameFromContext(r)
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "GetView", getReq, &getResp, identityName)
})
return getResp, err
}
// buildViewResponse parses the stored view metadata into a ViewResponse.
func (s *Server) buildViewResponse(r *http.Request, getResp s3tables.GetViewResponse, bucketName string, namespace []string, viewName string) (ViewResponse, error) {
if getResp.Metadata == nil || len(getResp.Metadata.FullMetadata) == 0 {
return ViewResponse{}, fmt.Errorf("view %s has no metadata", viewName)
}
metadata, err := view.ParseMetadataBytes(getResp.Metadata.FullMetadata)
if err != nil {
return ViewResponse{}, fmt.Errorf("failed to parse view metadata: %w", err)
}
config, _ := s.buildFileIOConfig(r, tableLocationFromMetadataLocation(getResp.MetadataLocation))
return ViewResponse{
MetadataLocation: getResp.MetadataLocation,
Metadata: metadata,
Config: config,
}, nil
}
// viewProperties returns a non-nil copy of props for view metadata construction.
func viewProperties(props iceberg.Properties) iceberg.Properties {
out := make(iceberg.Properties, len(props))
for k, v := range props {
out[k] = v
}
return out
}
func isViewNotFound(err error) bool {
if err == nil {
return false
}
var viewErr *s3tables.S3TablesError
if errors.As(err, &viewErr) {
if viewErr.Type == s3tables.ErrCodeNoSuchView || viewErr.Type == s3tables.ErrCodeNoSuchNamespace {
return true
}
}
return strings.Contains(strings.ToLower(err.Error()), "not found")
}
func isViewAlreadyExists(err error) bool {
if err == nil {
return false
}
var viewErr *s3tables.S3TablesError
if errors.As(err, &viewErr) && viewErr.Type == s3tables.ErrCodeViewAlreadyExists {
return true
}
return strings.Contains(strings.ToLower(err.Error()), "already exists")
}
// handleRenameView moves a view's catalog pointer to a new namespace/name,
// the view counterpart of POST /v1/{prefix}/tables/rename.
func (s *Server) handleRenameView(w http.ResponseWriter, r *http.Request) {
var req RenameTableRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
return
}
source := parseNamespace(encodeNamespace(req.Source.Namespace))
dest := parseNamespace(encodeNamespace(req.Destination.Namespace))
if len(source) == 0 || req.Source.Name == "" || len(dest) == 0 || req.Destination.Name == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "source and destination namespace and name are required")
return
}
bucketName := getBucketFromPrefix(r)
identityName := s3_constants.GetIdentityNameFromContext(r)
renameReq := &s3tables.RenameTableRequest{
TableBucketARN: buildTableBucketARN(bucketName),
SourceNamespace: source,
SourceName: req.Source.Name,
DestNamespace: dest,
DestName: req.Destination.Name,
}
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "RenameView", renameReq, nil, identityName)
})
if err != nil {
var viewErr *s3tables.S3TablesError
if errors.As(err, &viewErr) {
switch viewErr.Type {
case s3tables.ErrCodeNoSuchView:
writeError(w, http.StatusNotFound, "NoSuchViewException", fmt.Sprintf("View does not exist: %s", req.Source.Name))
return
case s3tables.ErrCodeNoSuchNamespace:
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", dest))
return
case s3tables.ErrCodeViewAlreadyExists:
writeError(w, http.StatusConflict, "AlreadyExistsException", fmt.Sprintf("View already exists: %s", req.Destination.Name))
return
case s3tables.ErrCodeInvalidRequest:
writeError(w, http.StatusBadRequest, "BadRequestException", viewErr.Message)
return
}
}
glog.V(1).Infof("Iceberg: RenameView error: %v", err)
writeManagerError(w, err)
return
}
w.WriteHeader(http.StatusNoContent)
}

View File

@@ -0,0 +1,165 @@
package iceberg
import (
"encoding/json"
"fmt"
"net/http"
"strings"
"github.com/apache/iceberg-go/view"
"github.com/google/uuid"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// handleUpdateView applies requirements and updates to a view, writes a new
// metadata.json, and flips the stored pointer. Mirrors the table commit flow.
func (s *Server) handleUpdateView(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
namespace := parseNamespace(vars["namespace"])
viewName := vars["view"]
if len(namespace) == 0 || viewName == "" {
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and view name are required")
return
}
var req UpdateViewRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body: "+err.Error())
return
}
bucketName := getBucketFromPrefix(r)
bucketARN := buildTableBucketARN(bucketName)
identityName := s3_constants.GetIdentityNameFromContext(r)
const maxCommitAttempts = 3
for attempt := 1; attempt <= maxCommitAttempts; attempt++ {
getResp, err := s.getView(r, namespace, viewName)
if err != nil {
if isViewNotFound(err) {
writeError(w, http.StatusNotFound, "NoSuchViewException", fmt.Sprintf("View does not exist: %s", viewName))
return
}
glog.V(1).Infof("Iceberg: UpdateView GetView error: %v", err)
writeManagerError(w, err)
return
}
if getResp.Metadata == nil || len(getResp.Metadata.FullMetadata) == 0 {
writeError(w, http.StatusInternalServerError, "InternalServerError", "view has no metadata")
return
}
currentMetadata, err := view.ParseMetadataBytes(getResp.Metadata.FullMetadata)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to parse current view metadata: "+err.Error())
return
}
for _, requirement := range req.Requirements {
if err := requirement.Validate(currentMetadata); err != nil {
writeError(w, http.StatusConflict, "CommitFailedException", "Requirement failed: "+err.Error())
return
}
}
builder, err := view.MetadataBuilderFromBase(currentMetadata)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to create view metadata builder: "+err.Error())
return
}
for _, update := range req.Updates {
if err := update.Apply(builder); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to apply update: "+err.Error())
return
}
}
newMetadata, err := builder.Build()
if err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to build new view metadata: "+err.Error())
return
}
location := tableLocationFromMetadataLocation(getResp.MetadataLocation)
if location == "" {
location = newMetadata.Location()
}
metadataVersion := getResp.MetadataVersion + 1
metadataFileName := fmt.Sprintf("v%d.metadata.json", metadataVersion)
newMetadataLocation := fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(location, "/"), metadataFileName)
metadataBytes, err := json.Marshal(newMetadata)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to serialize view metadata: "+err.Error())
return
}
metadataBucket, metadataPath, err := parseS3Location(location)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid view location: "+err.Error())
return
}
if err := confineMetadataLocation(metadataBucket, metadataPath, bucketName); err != nil {
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
return
}
metadataFileName, newMetadataLocation, err = s.stageCommitMetadata(r.Context(), metadataBucket, metadataPath, location, metadataFileName, metadataBytes)
if err != nil {
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to save view metadata file: "+err.Error())
return
}
tableUUID := newMetadata.ViewUUID()
if tableUUID == uuid.Nil {
tableUUID = currentMetadata.ViewUUID()
}
updateReq := &s3tables.UpdateViewRequest{
TableBucketARN: bucketARN,
Namespace: namespace,
Name: viewName,
VersionToken: getResp.VersionToken,
Metadata: &s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{TableUUID: tableUUID.String()},
FullMetadata: metadataBytes,
},
MetadataVersion: metadataVersion,
MetadataLocation: newMetadataLocation,
}
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
mgrClient := s3tables.NewManagerClient(client)
return s.tablesManager.Execute(r.Context(), mgrClient, "UpdateView", updateReq, nil, identityName)
})
if err == nil {
config, _ := s.buildFileIOConfig(r, location)
writeLoadResult(w, http.StatusOK, ViewResponse{
MetadataLocation: newMetadataLocation,
Metadata: newMetadata,
Config: config,
})
return
}
if isS3TablesConflict(err) {
if cleanupErr := s.deleteMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
glog.V(1).Infof("Iceberg: failed to cleanup view metadata file %s on conflict: %v", newMetadataLocation, cleanupErr)
}
if attempt < maxCommitAttempts {
glog.V(1).Infof("Iceberg: UpdateView conflict for %s (attempt %d/%d), retrying", viewName, attempt, maxCommitAttempts)
sleepBeforeCommitRetry(attempt)
continue
}
writeError(w, http.StatusConflict, "CommitFailedException", "Version token mismatch")
return
}
if cleanupErr := s.deleteMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
glog.V(1).Infof("Iceberg: failed to cleanup view metadata file %s after update failure: %v", newMetadataLocation, cleanupErr)
}
glog.Errorf("Iceberg: UpdateView error: %v", err)
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to commit view update: "+err.Error())
return
}
}

View File

@@ -0,0 +1,4 @@
// Package iceberg provides Iceberg REST Catalog API support.
// It implements the Apache Iceberg REST Catalog specification
// backed by S3 Tables metadata storage.
package iceberg

View File

@@ -0,0 +1,79 @@
package iceberg
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
func TestWantsVendedCredentials(t *testing.T) {
tests := []struct {
name string
values []string
want bool
}{
{name: "no header", want: false},
{name: "vended credentials", values: []string{"vended-credentials"}, want: true},
{name: "mechanism list", values: []string{"remote-signing,vended-credentials"}, want: true},
{name: "spaced and mixed case", values: []string{"Remote-Signing, Vended-Credentials"}, want: true},
{name: "repeated header", values: []string{"remote-signing", "vended-credentials"}, want: true},
{name: "remote signing only", values: []string{"remote-signing"}, want: false},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
r := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
for _, v := range tc.values {
r.Header.Add(accessDelegationHeader, v)
}
if got := wantsVendedCredentials(r); got != tc.want {
t.Fatalf("wantsVendedCredentials(%v) = %v, want %v", tc.values, got, tc.want)
}
})
}
}
// A client that asked for vended credentials replaces its own storage
// credentials with whatever the catalog returns. With no vending configured
// there are none, so an endpoint by itself would leave the client signing
// nothing and every data file read and write would come back 403.
func TestBuildFileIOConfigWithholdsEndpointFromCredentialVendingClients(t *testing.T) {
s := &Server{s3Endpoint: "http://seaweed.example:8333"}
r := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
r.Header.Set(accessDelegationHeader, "vended-credentials")
if got, _ := s.buildFileIOConfig(r, "s3://warehouse/ns/t"); len(got) != 0 {
t.Fatalf("buildFileIOConfig() = %v, want empty for a vended-credentials request", got)
}
plain := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
if got, _ := s.buildFileIOConfig(plain, "s3://warehouse/ns/t"); got["s3.endpoint"] != s.s3Endpoint {
t.Fatalf("s3.endpoint = %q, want %q", got["s3.endpoint"], s.s3Endpoint)
}
}
// The body of a load response depends on the delegation header, so a cache in
// front of the catalog must not key on the URL alone.
func TestWriteLoadResultVariesOnTheDelegationHeader(t *testing.T) {
rec := httptest.NewRecorder()
writeLoadResult(rec, http.StatusOK, LoadTableResult{})
if got := rec.Header().Get("Vary"); got != accessDelegationHeader {
t.Fatalf("Vary = %q, want %q", got, accessDelegationHeader)
}
}
func TestLoadTableResultOmitsConfigForCredentialVendingClients(t *testing.T) {
s := &Server{s3Endpoint: "http://seaweed.example:8333"}
r := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
r.Header.Set(accessDelegationHeader, "vended-credentials")
getResp := s3tables.GetTableResponse{MetadataLocation: "s3://bkt/ns/t/metadata/v1.metadata.json"}
result, err := s.buildLoadTableResult(r, getResp, "bkt", []string{"ns"}, "t")
if err != nil {
t.Fatalf("buildLoadTableResult() error = %v", err)
}
if len(result.Config) != 0 {
t.Fatalf("LoadTableResult.Config = %v, want empty", result.Config)
}
}

View File

@@ -0,0 +1,97 @@
package iceberg
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// seedPoisonedTable registers a table entry whose stored metadataLocation
// points outside its own bucket via a ".." segment, simulating a value
// persisted verbatim by the raw S3Tables UpdateTable API.
func seedPoisonedTable(t *testing.T, fc *memFiler, bucket, namespace, tableName, metadataLocation string) {
t.Helper()
meta := s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{TableUUID: "00000000-0000-0000-0000-000000000001"},
}
internal := map[string]any{
"name": tableName,
"namespace": namespace,
"format": "ICEBERG",
"ownerAccountId": s3_constants.AccountAdminId,
"versionToken": "v1",
"metadataVersion": 1,
"metadataLocation": metadataLocation,
"metadata": meta,
}
metaBytes, _ := json.Marshal(internal)
fc.seed(s3tables.GetTablePath(bucket, namespace, tableName), &filer_pb.Entry{
Name: tableName,
IsDirectory: true,
Extended: map[string][]byte{s3tables.ExtendedKeyMetadata: metaBytes},
})
}
func TestCommitTableRejectsStoredTraversalLocation(t *testing.T) {
const attacker = "attacker"
const victim = "victim"
fc := newMemFiler()
seedNamespace(fc, attacker, "ns", s3_constants.AccountAdminId)
seedPoisonedTable(t, fc, attacker, "ns", "t",
"s3://attacker/../victim/planted/metadata/v1.metadata.json")
s := NewServer(fc, nil)
r := httptest.NewRequest(http.MethodPost, "/v1/"+attacker+"/namespaces/ns/tables/t",
strings.NewReader(`{"requirements":[],"updates":[]}`))
r = mux.SetURLVars(r, map[string]string{"prefix": attacker, "namespace": "ns", "table": "t"})
r = r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), s3_constants.AccountAdminId))
w := httptest.NewRecorder()
s.handleUpdateTable(w, r)
if w.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d (body=%s)", w.Code, http.StatusBadRequest, w.Body.String())
}
for p := range fc.entries {
if strings.Contains(p, "/"+victim+"/") && !strings.Contains(p, "/"+attacker+"/") {
t.Fatalf("cross-tenant write escaped into victim tree at %s (status=%d body=%s)", p, w.Code, w.Body.String())
}
}
}
func TestCommitTransactionRejectsStoredTraversalLocation(t *testing.T) {
const attacker = "attacker"
const victim = "victim"
fc := newMemFiler()
seedNamespace(fc, attacker, "ns", s3_constants.AccountAdminId)
seedPoisonedTable(t, fc, attacker, "ns", "t",
"s3://attacker/../victim/planted/metadata/v1.metadata.json")
s := NewServer(fc, nil)
body := `{"table-changes":[{"identifier":{"namespace":["ns"],"name":"t"},"requirements":[],"updates":[]}]}`
r := httptest.NewRequest(http.MethodPost, "/v1/"+attacker+"/transactions/commit",
strings.NewReader(body))
r = mux.SetURLVars(r, map[string]string{"prefix": attacker})
r = r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), s3_constants.AccountAdminId))
w := httptest.NewRecorder()
s.handleCommitTransaction(w, r)
if w.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d (body=%s)", w.Code, http.StatusBadRequest, w.Body.String())
}
for p := range fc.entries {
if strings.Contains(p, "/"+victim+"/") && !strings.Contains(p, "/"+attacker+"/") {
t.Fatalf("cross-tenant transaction write escaped into victim tree at %s (status=%d body=%s)", p, w.Code, w.Body.String())
}
}
}

View File

@@ -0,0 +1,134 @@
package iceberg
import (
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/apache/iceberg-go/table"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
func TestParseCommitUpdatesSeparatesStatistics(t *testing.T) {
raw := []json.RawMessage{
json.RawMessage(`{"action":"set-statistics","snapshot-id":10,"statistics-path":"s3://bucket/table/metadata/stats.puffin","file-size-in-bytes":100,"file-footer-size-in-bytes":20,"blob-metadata":[]}`),
json.RawMessage(`{"action":"set-properties","updates":{"k":"v"}}`),
}
updates, stats, err := parseCommitUpdates(raw)
if err != nil {
t.Fatalf("parseCommitUpdates() error = %v", err)
}
if len(stats) != 1 {
t.Fatalf("statistics updates = %d, want 1", len(stats))
}
if stats[0].set == nil || stats[0].set.SnapshotID != 10 {
t.Fatalf("unexpected statistics update: %#v", stats[0])
}
if len(updates) != 1 {
t.Fatalf("decoded updates = %d, want 1", len(updates))
}
}
func TestParseCommitUpdatesRejectsIncompleteSetStatistics(t *testing.T) {
raw := []json.RawMessage{
json.RawMessage(`{"action":"set-statistics","snapshot-id":10}`),
}
_, _, err := parseCommitUpdates(raw)
if err == nil {
t.Fatalf("parseCommitUpdates() expected error")
}
if !errors.Is(err, ErrIncompleteSetStatistics) {
t.Fatalf("parseCommitUpdates() error = %v, want ErrIncompleteSetStatistics", err)
}
}
func TestApplyStatisticsUpdatesUpsertAndRemove(t *testing.T) {
metadata := []byte(`{"format-version":2,"statistics":[{"snapshot-id":1,"statistics-path":"s3://bucket/stats-1.puffin","file-size-in-bytes":10,"file-footer-size-in-bytes":1,"blob-metadata":[]},{"snapshot-id":2,"statistics-path":"s3://bucket/stats-2.puffin","file-size-in-bytes":20,"file-footer-size-in-bytes":2,"blob-metadata":[]}]} `)
snapshotID := int64(2)
setUpdates := []statisticsUpdate{
{
set: &statisticsFileForTest,
},
{
remove: &snapshotID,
},
}
updated, err := applyStatisticsUpdates(metadata, setUpdates)
if err != nil {
t.Fatalf("applyStatisticsUpdates() error = %v", err)
}
var decoded map[string]json.RawMessage
if err := json.Unmarshal(updated, &decoded); err != nil {
t.Fatalf("json.Unmarshal(updated) error = %v", err)
}
var stats []map[string]any
if err := json.Unmarshal(decoded["statistics"], &stats); err != nil {
t.Fatalf("json.Unmarshal(statistics) error = %v", err)
}
if len(stats) != 1 {
t.Fatalf("statistics length = %d, want 1", len(stats))
}
if got := int64(stats[0]["snapshot-id"].(float64)); got != 1 {
t.Fatalf("remaining snapshot-id = %d, want 1", got)
}
if got := int64(stats[0]["file-size-in-bytes"].(float64)); got != 11 {
t.Fatalf("remaining file-size-in-bytes = %d, want 11", got)
}
}
var statisticsFileForTest = table.StatisticsFile{
SnapshotID: 1,
StatisticsPath: "s3://bucket/stats-1.puffin",
FileSizeInBytes: 11,
FileFooterSizeInBytes: 2,
BlobMetadata: []table.BlobMetadata{},
}
func TestIsS3TablesConflict(t *testing.T) {
if !isS3TablesConflict(s3tables.ErrVersionTokenMismatch) {
t.Fatalf("expected ErrVersionTokenMismatch to be conflict")
}
if !isS3TablesConflict(&s3tables.S3TablesError{Type: s3tables.ErrCodeConflict, Message: "Version token mismatch"}) {
t.Fatalf("expected S3Tables conflict error to be conflict")
}
if isS3TablesConflict(errors.New("other")) {
t.Fatalf("unexpected conflict for non-conflict error")
}
}
// A create-on-commit is a create, and it writes the new table's metadata file
// before the registration that authorizes it.
func TestCreateOnCommitDeniedBeforeAnyWrite(t *testing.T) {
const bucket = "warehouse"
fc := newMemFiler()
seedNamespace(fc, bucket, "finance", "alice")
s := NewServer(fc, nil)
r := httptest.NewRequest(http.MethodPost, "/v1/"+bucket+"/namespaces/finance/tables/quarterly_reports",
strings.NewReader(`{"requirements":[{"type":"assert-create"}],"updates":[]}`))
r = mux.SetURLVars(r, map[string]string{"prefix": bucket, "namespace": "finance", "table": "quarterly_reports"})
r = r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), "mallory"))
w := httptest.NewRecorder()
s.handleUpdateTable(w, r)
if w.Code != http.StatusForbidden {
t.Errorf("status = %d, want %d (body: %s)", w.Code, http.StatusForbidden, w.Body.String())
}
for p, entry := range fc.entries {
if !entry.IsDirectory && strings.Contains(p, "quarterly_reports") {
t.Errorf("unauthorized caller wrote %s", p)
}
}
}

View File

@@ -0,0 +1,200 @@
package iceberg
import (
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/apache/iceberg-go"
"github.com/google/uuid"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
func TestValidateCreateTableRequestRequiresName(t *testing.T) {
err := validateCreateTableRequest(CreateTableRequest{})
if !errors.Is(err, errTableNameRequired) {
t.Fatalf("validateCreateTableRequest() error = %v, want errTableNameRequired", err)
}
}
func TestValidateCreateTableRequestAcceptsWithName(t *testing.T) {
err := validateCreateTableRequest(CreateTableRequest{Name: "orders"})
if err != nil {
t.Fatalf("validateCreateTableRequest() error = %v, want nil", err)
}
}
func TestIsStageCreateEnabledDefaultsToTrue(t *testing.T) {
t.Setenv("ICEBERG_ENABLE_STAGE_CREATE", "")
if !isStageCreateEnabled() {
t.Fatalf("isStageCreateEnabled() = false, want true")
}
}
func TestIsStageCreateEnabledFalseValues(t *testing.T) {
falseValues := []string{"0", "false", "FALSE", "no", "off"}
for _, value := range falseValues {
t.Setenv("ICEBERG_ENABLE_STAGE_CREATE", value)
if isStageCreateEnabled() {
t.Fatalf("isStageCreateEnabled() = true for value %q, want false", value)
}
}
}
func mustParseSchema(t *testing.T, raw string) *iceberg.Schema {
t.Helper()
var schema iceberg.Schema
if err := json.Unmarshal([]byte(raw), &schema); err != nil {
t.Fatalf("parse schema: %v", err)
}
return &schema
}
const variantSchema = `{"type":"struct","schema-id":0,"fields":[
{"id":1,"name":"id","required":true,"type":"long"},
{"id":2,"name":"payload","required":false,"type":"variant"}]}`
// A v3-only column type without format-version 3 is the client's mistake, and
// the reason has to travel back to them rather than only into the server log.
func TestNewTableMetadataRejectsV3TypeBelowV3(t *testing.T) {
_, err := newTableMetadata(uuid.New(), "s3://bkt/ns/t", mustParseSchema(t, variantSchema), nil, nil, nil)
if err == nil {
t.Fatal("newTableMetadata() error = nil, want invalid schema")
}
if !errors.Is(err, iceberg.ErrInvalidSchema) {
t.Fatalf("newTableMetadata() error = %v, want ErrInvalidSchema", err)
}
// writeManagerError turns this into a 400; see TestWriteManagerError.
if !strings.Contains(err.Error(), "variant is not supported until v3") {
t.Errorf("error = %q, want the underlying reason", err.Error())
}
}
func TestNewTableMetadataAcceptsV3TypeAtV3(t *testing.T) {
metadata, err := newTableMetadata(uuid.New(), "s3://bkt/ns/t", mustParseSchema(t, variantSchema), nil, nil,
iceberg.Properties{"format-version": "3"})
if err != nil {
t.Fatalf("newTableMetadata() error = %v, want nil", err)
}
if got := metadata.Version(); got != 3 {
t.Fatalf("metadata.Version() = %d, want 3", got)
}
if _, found := metadata.CurrentSchema().FindFieldByName("payload"); !found {
t.Error("variant field missing from stored schema")
}
}
// A LoadTable response must never carry nil metadata: it serializes as
// "metadata":null under HTTP 200, which no Iceberg client can parse.
func TestBuildLoadTableResultNeverReturnsNilMetadata(t *testing.T) {
cases := map[string]s3tables.GetTableResponse{
"no stored metadata": {MetadataLocation: "s3://bkt/ns/t/metadata/v1.metadata.json"},
"empty full metadata": {Metadata: &s3tables.TableMetadata{}},
"unparseable metadata": {Metadata: &s3tables.TableMetadata{FullMetadata: json.RawMessage(`{"nope":`)}},
}
for name, getResp := range cases {
t.Run(name, func(t *testing.T) {
r := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
result, err := (&Server{}).buildLoadTableResult(r, getResp, "bkt", []string{"ns"}, "t")
if err != nil {
t.Fatalf("buildLoadTableResult() error = %v, want nil", err)
}
if result.Metadata == nil {
t.Fatal("buildLoadTableResult() returned nil metadata with no error")
}
})
}
}
func newCreateTableRequest(t *testing.T, bucket, namespace, body, identity string) *http.Request {
t.Helper()
r := httptest.NewRequest(http.MethodPost, "/v1/"+bucket+"/namespaces/"+namespace+"/tables", strings.NewReader(body))
r = mux.SetURLVars(r, map[string]string{"prefix": bucket, "namespace": namespace})
return r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), identity))
}
// A caller who may not create the table must be refused before anything is
// written: both branches of CreateTable write to the bucket, and stage-create
// never reaches the registration that carries the authorization.
func TestCreateTableDeniedBeforeAnyWrite(t *testing.T) {
const bucket = "warehouse"
for name, body := range map[string]string{
"stage-create": `{"name":"quarterly_reports","stage-create":true}`,
"create": `{"name":"quarterly_reports"}`,
} {
t.Run(name, func(t *testing.T) {
fc := newMemFiler()
seedNamespace(fc, bucket, "finance", "alice")
s := NewServer(fc, nil)
w := httptest.NewRecorder()
s.handleCreateTable(w, newCreateTableRequest(t, bucket, "finance", body, "mallory"))
if w.Code != http.StatusForbidden {
t.Errorf("status = %d, want %d (body: %s)", w.Code, http.StatusForbidden, w.Body.String())
}
for p, entry := range fc.entries {
if !entry.IsDirectory && strings.Contains(p, "quarterly_reports") {
t.Errorf("unauthorized caller wrote %s", p)
}
}
})
}
}
// The namespace owner still gets the staged metadata and marker stage-create
// exists to leave behind.
func TestStageCreateWritesStagedFilesForOwner(t *testing.T) {
const bucket = "warehouse"
fc := newMemFiler()
seedNamespace(fc, bucket, "finance", "alice")
s := NewServer(fc, nil)
w := httptest.NewRecorder()
s.handleCreateTable(w, newCreateTableRequest(t, bucket, "finance", `{"name":"quarterly_reports","stage-create":true}`, "alice"))
if w.Code != http.StatusOK {
t.Fatalf("status = %d, want %d (body: %s)", w.Code, http.StatusOK, w.Body.String())
}
staged := 0
for p, entry := range fc.entries {
if !entry.IsDirectory && strings.Contains(p, stageCreateMarkerDirName) {
staged++
}
}
if staged != 2 {
t.Fatalf("staged files = %d, want the metadata file and its marker", staged)
}
if _, ok := fc.entries[s3tables.GetTablePath(bucket, "finance", "quarterly_reports")]; ok {
t.Error("stage-create registered the table")
}
}
// A caller allowed by the actions on its own identity, rather than by ownership
// or a resource policy, has to survive the hop into the s3tables manager: the
// gate must not refuse what the create behind it would allow.
func TestCreateTableAllowedByIdentityActions(t *testing.T) {
const bucket = "warehouse"
fc := newMemFiler()
seedNamespace(fc, bucket, "finance", "alice")
s := NewServer(fc, nil)
r := newCreateTableRequest(t, bucket, "finance", `{"name":"quarterly_reports","stage-create":true}`, "mallory")
r = r.WithContext(s3_constants.SetIdentityInContext(r.Context(), &struct {
Name string
Actions []string
}{Name: "mallory", Actions: []string{"s3tables:CreateTable"}}))
w := httptest.NewRecorder()
s.handleCreateTable(w, r)
if w.Code != http.StatusOK {
t.Fatalf("status = %d, want %d (body: %s)", w.Code, http.StatusOK, w.Body.String())
}
}

View File

@@ -0,0 +1,233 @@
package iceberg
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"net/http"
"net/http/httptest"
"path"
"sort"
"strings"
"testing"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/table"
"github.com/apache/iceberg-go/view"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
"google.golang.org/grpc"
)
// memFiler is a minimal in-memory SeaweedFilerClient backing the view handlers
// end-to-end through the s3tables manager. Only the entry operations the create
// path touches are implemented; anything else panics so a missing dependency is
// loud rather than silently passing.
type memFiler struct {
filer_pb.SeaweedFilerClient
entries map[string]*filer_pb.Entry
// failFileCreate, when set, makes CreateEntry fail for non-directory entries,
// simulating a metadata-file write error.
failFileCreate error
}
func newMemFiler() *memFiler {
return &memFiler{entries: map[string]*filer_pb.Entry{}}
}
func (m *memFiler) WithFilerClient(_ bool, fn func(client filer_pb.SeaweedFilerClient) error) error {
return fn(m)
}
func (m *memFiler) seed(p string, entry *filer_pb.Entry) {
m.entries[p] = entry
}
func (m *memFiler) LookupDirectoryEntry(_ context.Context, in *filer_pb.LookupDirectoryEntryRequest, _ ...grpc.CallOption) (*filer_pb.LookupDirectoryEntryResponse, error) {
entry, ok := m.entries[path.Join(in.Directory, in.Name)]
if !ok {
return nil, filer_pb.ErrNotFound
}
return &filer_pb.LookupDirectoryEntryResponse{Entry: entry}, nil
}
func (m *memFiler) CreateEntry(_ context.Context, in *filer_pb.CreateEntryRequest, _ ...grpc.CallOption) (*filer_pb.CreateEntryResponse, error) {
if m.failFileCreate != nil && !in.Entry.IsDirectory {
return nil, m.failFileCreate
}
m.entries[path.Join(in.Directory, in.Entry.Name)] = in.Entry
return &filer_pb.CreateEntryResponse{}, nil
}
func (m *memFiler) DeleteEntry(_ context.Context, in *filer_pb.DeleteEntryRequest, _ ...grpc.CallOption) (*filer_pb.DeleteEntryResponse, error) {
prefix := path.Join(in.Directory, in.Name)
for p := range m.entries {
if p == prefix || strings.HasPrefix(p, prefix+"/") {
delete(m.entries, p)
}
}
return &filer_pb.DeleteEntryResponse{}, nil
}
func (m *memFiler) UpdateEntry(_ context.Context, in *filer_pb.UpdateEntryRequest, _ ...grpc.CallOption) (*filer_pb.UpdateEntryResponse, error) {
m.entries[path.Join(in.Directory, in.Entry.Name)] = in.Entry
return &filer_pb.UpdateEntryResponse{}, nil
}
func (m *memFiler) ListEntries(_ context.Context, in *filer_pb.ListEntriesRequest, _ ...grpc.CallOption) (filer_pb.SeaweedFiler_ListEntriesClient, error) {
var names []string
for p := range m.entries {
if path.Dir(p) == in.Directory {
names = append(names, p)
}
}
sort.Strings(names)
stream := &memListEntries{}
for _, name := range names {
stream.entries = append(stream.entries, m.entries[name])
}
return stream, nil
}
type memListEntries struct {
grpc.ClientStream
entries []*filer_pb.Entry
}
func (m *memListEntries) Recv() (*filer_pb.ListEntriesResponse, error) {
if len(m.entries) == 0 {
return nil, io.EOF
}
entry := m.entries[0]
m.entries = m.entries[1:]
return &filer_pb.ListEntriesResponse{Entry: entry}, nil
}
func newCreateViewRequest(t *testing.T, namespace, name, sql string) *http.Request {
t.Helper()
schema := iceberg.NewSchemaWithIdentifiers(0, nil,
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int32, Required: true},
)
ver, err := view.NewVersionFromSQL(1, schema.ID, sql, table.Identifier{namespace})
if err != nil {
t.Fatal(err)
}
body, err := json.Marshal(CreateViewRequest{Name: name, Schema: schema, ViewVersion: ver})
if err != nil {
t.Fatal(err)
}
r := httptest.NewRequest(http.MethodPost, "/v1/namespaces/"+namespace+"/views", bytes.NewReader(body))
r = mux.SetURLVars(r, map[string]string{"namespace": namespace})
r = r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), s3_constants.AccountAdminId))
return r
}
// seedNamespace registers a bucket and namespace owned by owner so the s3tables
// existence and auth-context lookups pass.
func seedNamespace(fc *memFiler, bucket, namespace, owner string) {
fc.seed(s3tables.GetTableBucketPath(bucket), &filer_pb.Entry{Name: bucket, IsDirectory: true})
meta, _ := json.Marshal(map[string]any{"namespace": []string{namespace}, "ownerAccountId": owner})
fc.seed(s3tables.GetNamespacePath(bucket, namespace), &filer_pb.Entry{
Name: namespace,
IsDirectory: true,
Extended: map[string][]byte{s3tables.ExtendedKeyMetadata: meta},
})
}
func TestCreateViewMissingNamespaceReturns404(t *testing.T) {
fc := newMemFiler()
s := NewServer(fc, nil)
w := httptest.NewRecorder()
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 1"))
if w.Code != http.StatusNotFound {
t.Fatalf("status = %d, want %d (body: %s)", w.Code, http.StatusNotFound, w.Body.String())
}
var errResp ErrorResponse
if err := json.Unmarshal(w.Body.Bytes(), &errResp); err != nil {
t.Fatalf("unmarshal error body: %v", err)
}
if errResp.Error.Type != "NoSuchNamespaceException" {
t.Fatalf("error type = %q, want NoSuchNamespaceException", errResp.Error.Type)
}
}
func TestCreateViewTagsEntryAsView(t *testing.T) {
const bucket = "warehouse"
fc := newMemFiler()
seedNamespace(fc, bucket, "ns", s3_constants.AccountAdminId)
s := NewServer(fc, nil)
w := httptest.NewRecorder()
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 1"))
if w.Code != http.StatusOK {
t.Fatalf("create status = %d, want 200 (body: %s)", w.Code, w.Body.String())
}
entry, ok := fc.entries[s3tables.GetTablePath(bucket, "ns", "v")]
if !ok {
t.Fatalf("view entry not created")
}
if got := string(entry.Extended[s3tables.ExtendedKeyEntryType]); got != s3tables.EntryTypeView {
t.Fatalf("entryType = %q, want %q", got, s3tables.EntryTypeView)
}
if _, ok := entry.Extended[s3tables.ExtendedKeyMetadata]; !ok {
t.Fatalf("view entry missing metadata attribute")
}
}
func TestCreateViewDuplicateDoesNotClobberMetadata(t *testing.T) {
const bucket = "warehouse"
fc := newMemFiler()
seedNamespace(fc, bucket, "ns", s3_constants.AccountAdminId)
s := NewServer(fc, nil)
w := httptest.NewRecorder()
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 1"))
if w.Code != http.StatusOK {
t.Fatalf("first create status = %d, want 200 (body: %s)", w.Code, w.Body.String())
}
metadataKey := path.Join(s3tables.GetTablePath(bucket, "ns", "v"), "metadata", "v1.metadata.json")
first, ok := fc.entries[metadataKey]
if !ok {
t.Fatalf("metadata file not written at %s", metadataKey)
}
original := append([]byte(nil), first.Content...)
// Re-create the same view with different SQL: the existence pre-check must
// short-circuit so the persisted v1.metadata.json stays byte-for-byte intact.
w = httptest.NewRecorder()
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 2"))
if w.Code != http.StatusOK {
t.Fatalf("duplicate create status = %d, want 200 (body: %s)", w.Code, w.Body.String())
}
if got := fc.entries[metadataKey].Content; !bytes.Equal(got, original) {
t.Fatalf("duplicate create clobbered stored metadata:\n got %s\nwant %s", got, original)
}
}
func TestCreateViewRollsBackEntryWhenMetadataWriteFails(t *testing.T) {
const bucket = "warehouse"
fc := newMemFiler()
seedNamespace(fc, bucket, "ns", s3_constants.AccountAdminId)
fc.failFileCreate = errors.New("disk full")
s := NewServer(fc, nil)
w := httptest.NewRecorder()
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 1"))
if w.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want %d (body: %s)", w.Code, http.StatusInternalServerError, w.Body.String())
}
// The registered view must be rolled back so it doesn't linger pointing at
// metadata that was never written.
if _, ok := fc.entries[s3tables.GetTablePath(bucket, "ns", "v")]; ok {
t.Fatalf("view entry left behind after metadata write failure")
}
}

View File

@@ -0,0 +1,111 @@
package iceberg
import (
"context"
"errors"
"net/http/httptest"
"testing"
"time"
)
type stubVendor struct {
credentials *VendedCredentials
err error
gotBucket string
gotPrefix string
gotcaller string
}
func (v *stubVendor) VendTableCredentials(_ context.Context, principal, bucket, prefix string) (*VendedCredentials, error) {
v.gotcaller, v.gotBucket, v.gotPrefix = principal, bucket, prefix
return v.credentials, v.err
}
func vendingServer(vendor CredentialVendor) *Server {
s := &Server{s3Endpoint: "http://s3.example:8333"}
s.SetCredentialVendor(vendor)
return s
}
func TestBuildFileIOConfigVendingVendsScopedCredentials(t *testing.T) {
expiry := time.Now().Add(time.Hour).Truncate(time.Millisecond)
vendor := &stubVendor{credentials: &VendedCredentials{
AccessKeyID: "ASIAEXAMPLE",
SecretAccessKey: "secret",
SessionToken: "token",
Expiration: expiry,
}}
s := vendingServer(vendor)
r := httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t", nil)
r.Header.Set(accessDelegationHeader, "vended-credentials")
config, storageCredentials := s.buildFileIOConfig(r, "s3://warehouse/ns/t")
if vendor.gotBucket != "warehouse" || vendor.gotPrefix != "ns/t" {
t.Errorf("vendor called with bucket=%q prefix=%q, want warehouse and ns/t", vendor.gotBucket, vendor.gotPrefix)
}
if config["s3.access-key-id"] != "ASIAEXAMPLE" || config["s3.secret-access-key"] != "secret" || config["s3.session-token"] != "token" {
t.Errorf("config missing vended credentials: %v", config)
}
if config["s3.endpoint"] != "http://s3.example:8333" {
t.Errorf("config missing the endpoint the credentials are for: %v", config)
}
if len(storageCredentials) != 1 {
t.Fatalf("storage-credentials = %d entries, want 1", len(storageCredentials))
}
if storageCredentials[0].Prefix != "s3://warehouse/ns/t" {
t.Errorf("storage credential prefix = %q", storageCredentials[0].Prefix)
}
if storageCredentials[0].Config["s3.session-token"] != "token" {
t.Errorf("storage credential is missing the session token: %v", storageCredentials[0].Config)
}
}
// Without a vendor the response must stay silent about the endpoint too: a
// client that asked for vended credentials would otherwise drop its own and
// start sending unsigned requests.
func TestBuildFileIOConfigVendingWithoutVendorSaysNothing(t *testing.T) {
s := &Server{s3Endpoint: "http://s3.example:8333"}
r := httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t", nil)
r.Header.Set(accessDelegationHeader, "vended-credentials")
config, storageCredentials := s.buildFileIOConfig(r, "s3://warehouse/ns/t")
if len(config) != 0 || storageCredentials != nil {
t.Errorf("config = %v, storage-credentials = %v, want both empty", config, storageCredentials)
}
}
func TestBuildFileIOConfigVendingFallsBackWhenVendingFails(t *testing.T) {
s := vendingServer(&stubVendor{err: errors.New("no role configured")})
r := httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t", nil)
r.Header.Set(accessDelegationHeader, "vended-credentials")
config, storageCredentials := s.buildFileIOConfig(r, "s3://warehouse/ns/t")
if len(config) != 0 || storageCredentials != nil {
t.Errorf("config = %v, storage-credentials = %v, want both empty", config, storageCredentials)
}
}
// A client that did not ask for delegation keeps the plain endpoint config and
// must never be handed credentials.
func TestBuildFileIOConfigVendingWithoutDelegationHeader(t *testing.T) {
vendor := &stubVendor{credentials: &VendedCredentials{AccessKeyID: "ASIAEXAMPLE"}}
s := vendingServer(vendor)
r := httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t", nil)
config, storageCredentials := s.buildFileIOConfig(r, "s3://warehouse/ns/t")
if _, vended := config["s3.access-key-id"]; vended {
t.Errorf("credentials vended without the delegation header: %v", config)
}
if config["s3.endpoint"] != "http://s3.example:8333" {
t.Errorf("config = %v, want the endpoint", config)
}
if storageCredentials != nil {
t.Errorf("storage-credentials = %v, want none", storageCredentials)
}
if vendor.gotBucket != "" {
t.Errorf("vendor was called for a request that did not ask for delegation")
}
}

View File

@@ -0,0 +1,87 @@
package iceberg
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/apache/iceberg-go"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
func TestNameValidationError(t *testing.T) {
cases := []struct {
err error
want bool
}{
{nil, false},
// Wrapped exactly as the s3tables manager surfaces it.
{fmt.Errorf("all filers failed, last error: invalid namespace name: only 'a-z', '0-9', and '_' are allowed"), true},
{fmt.Errorf("invalid table name: only 'a-z', '0-9', and '_' are allowed"), true},
{fmt.Errorf("namespace name must start with a letter or digit"), true},
{fmt.Errorf("namespace name cannot start with reserved prefix 'aws'"), true},
{fmt.Errorf("table name must be between 1 and 255 characters"), true},
{fmt.Errorf("namespace not found"), false},
{fmt.Errorf("all filers failed, last error: rpc timeout"), false},
// Unrelated faults that merely mention a name must stay 500.
{fmt.Errorf("failed to resolve table name from index"), false},
{fmt.Errorf("error fetching namespace name mapping"), false},
}
for _, c := range cases {
if got := nameValidationError(c.err); got != c.want {
t.Errorf("nameValidationError(%v) = %v, want %v", c.err, got, c.want)
}
}
}
func TestWriteManagerError(t *testing.T) {
cases := []struct {
name string
err error
wantCode int
wantType string
wantMessage []string
}{
{name: "invalid name is a client error", err: fmt.Errorf("invalid namespace name: only 'a-z', '0-9', and '_' are allowed"), wantCode: http.StatusBadRequest, wantType: "BadRequestException"},
{name: "rejected schema is a client error", err: fmt.Errorf("%w: for v2: variant is not supported until v3", iceberg.ErrInvalidSchema), wantCode: http.StatusBadRequest, wantType: "BadRequestException"},
{name: "bad format version is a client error", err: fmt.Errorf("%w: 4", iceberg.ErrInvalidFormatVersion), wantCode: http.StatusBadRequest, wantType: "BadRequestException"},
{
name: "missing table bucket is a client error",
err: fmt.Errorf("all filers failed, last error: %w", &s3tables.S3TablesError{Type: s3tables.ErrCodeNoSuchBucket, Message: "table bucket warehouse not found"}),
wantCode: http.StatusNotFound,
wantType: "NoSuchNamespaceException",
// The guidance is the point of the mapping: the bucket the request
// resolved to is one the client never named, so the response has to
// say how to name a real one.
wantMessage: []string{"table bucket warehouse not found", "warehouse=s3://<table-bucket>/", "/v1/<table-bucket>/"},
},
{name: "everything else is a server fault", err: fmt.Errorf("all filers failed, last error: connection refused"), wantCode: http.StatusInternalServerError, wantType: "InternalServerError"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
rec := httptest.NewRecorder()
writeManagerError(rec, c.err)
if rec.Code != c.wantCode {
t.Fatalf("status = %d, want %d", rec.Code, c.wantCode)
}
var resp ErrorResponse
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode response: %v", err)
}
if resp.Error.Type != c.wantType {
t.Fatalf("error type = %q, want %q", resp.Error.Type, c.wantType)
}
if resp.Error.Code != c.wantCode {
t.Fatalf("error code = %d, want %d", resp.Error.Code, c.wantCode)
}
for _, want := range c.wantMessage {
if !strings.Contains(resp.Error.Message, want) {
t.Errorf("message %q does not contain %q", resp.Error.Message, want)
}
}
})
}
}

View File

@@ -0,0 +1,105 @@
package iceberg
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestGetBucketFromPrefix_WarehouseQueryFallback(t *testing.T) {
tests := []struct {
name string
url string
want string
}{
{
name: "warehouse query routes to its bucket when no prefix in path",
url: "/v1/namespaces?warehouse=s3%3A%2F%2Fmyblkt%2F",
want: "myblkt",
},
{
name: "warehouse query with sub-path still picks the bucket",
url: "/v1/namespaces?warehouse=s3%3A%2F%2Fanother%2Fextra",
want: "another",
},
{
name: "bare bucket name is taken as the table bucket",
url: "/v1/namespaces?warehouse=not-a-url",
want: "not-a-url",
},
{
name: "table bucket ARN routes to its bucket",
url: "/v1/namespaces?warehouse=arn%3Aaws%3As3tables%3Aus-east-1%3Aadmin%3Abucket%2Fseaweed-iceberg",
want: "seaweed-iceberg",
},
{
name: "unusable warehouse value falls through to default",
url: "/v1/namespaces?warehouse=file%3A%2F%2F%2Ftmp%2Fwh",
want: "warehouse",
},
{
name: "no warehouse and no prefix returns default",
url: "/v1/namespaces",
want: "warehouse",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
r := httptest.NewRequest("GET", tc.url, nil)
got := getBucketFromPrefix(r)
if got != tc.want {
t.Fatalf("getBucketFromPrefix(%q) = %q, want %q", tc.url, got, tc.want)
}
})
}
}
func TestBuildFileIOConfig(t *testing.T) {
loadTable := func() *http.Request {
return httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
}
t.Run("no endpoint configured yields empty config", func(t *testing.T) {
s := &Server{}
got, _ := s.buildFileIOConfig(loadTable(), "s3://warehouse/ns/t")
if len(got) != 0 {
t.Fatalf("buildFileIOConfig() = %v, want empty", got)
}
})
t.Run("endpoint is advertised with path-style-access and region", func(t *testing.T) {
s := &Server{s3Endpoint: "http://seaweed.example:8333"}
got, _ := s.buildFileIOConfig(loadTable(), "s3://warehouse/ns/t")
if got["s3.endpoint"] != "http://seaweed.example:8333" {
t.Fatalf("s3.endpoint = %q, want %q", got["s3.endpoint"], "http://seaweed.example:8333")
}
if got["s3.path-style-access"] != "true" {
t.Fatalf("s3.path-style-access = %q, want %q", got["s3.path-style-access"], "true")
}
if got["s3.region"] == "" {
t.Fatalf("s3.region was empty, want a non-empty default so clients like DuckDB do not require AWS_REGION")
}
})
}
func TestGetBucketFromPrefix_TableBucketEnvFallback(t *testing.T) {
r := httptest.NewRequest("GET", "/v1/namespaces", nil)
t.Setenv("S3_TABLE_BUCKET", " ,analytics, other")
if got := getBucketFromPrefix(r); got != "analytics" {
t.Fatalf("first S3_TABLE_BUCKET entry: got %q, want analytics", got)
}
// The explicit default still wins over the table bucket list.
t.Setenv("S3TABLES_DEFAULT_BUCKET", "explicit")
if got := getBucketFromPrefix(r); got != "explicit" {
t.Fatalf("S3TABLES_DEFAULT_BUCKET override: got %q, want explicit", got)
}
// A prefixless value with neither env set falls through to the default.
t.Setenv("S3TABLES_DEFAULT_BUCKET", "")
t.Setenv("S3_TABLE_BUCKET", " , ")
if got := getBucketFromPrefix(r); got != "warehouse" {
t.Fatalf("empty specs: got %q, want warehouse", got)
}
}

View File

@@ -0,0 +1,29 @@
package iceberg
import "testing"
func TestNormalizeNamespacePropertiesNil(t *testing.T) {
properties := normalizeNamespaceProperties(nil)
if properties == nil {
t.Fatalf("normalizeNamespaceProperties(nil) returned nil map")
}
if len(properties) != 0 {
t.Fatalf("normalizeNamespaceProperties(nil) length = %d, want 0", len(properties))
}
}
func TestNormalizeNamespacePropertiesReturnsInputWhenSet(t *testing.T) {
input := map[string]string{
"owner": "analytics",
}
properties := normalizeNamespaceProperties(input)
if properties["owner"] != "analytics" {
t.Fatalf("normalized properties value = %q, want %q", properties["owner"], "analytics")
}
input["owner"] = "updated"
if properties["owner"] != "updated" {
t.Fatalf("normalizeNamespaceProperties should reuse the input map when non-nil")
}
}

View File

@@ -0,0 +1,69 @@
package iceberg
import (
"net/http/httptest"
"testing"
)
func TestParsePaginationDefaultValues(t *testing.T) {
req := httptest.NewRequest("GET", "/v1/namespaces", nil)
pageToken, pageSize, err := parsePagination(req)
if err != nil {
t.Fatalf("parsePagination() error = %v", err)
}
if pageToken != "" {
t.Fatalf("pageToken = %q, want empty", pageToken)
}
if pageSize != defaultListPageSize {
t.Fatalf("pageSize = %d, want %d", pageSize, defaultListPageSize)
}
}
func TestParsePaginationUsesCamelCaseParameters(t *testing.T) {
req := httptest.NewRequest("GET", "/v1/namespaces?pageToken=abc&pageSize=25", nil)
pageToken, pageSize, err := parsePagination(req)
if err != nil {
t.Fatalf("parsePagination() error = %v", err)
}
if pageToken != "abc" {
t.Fatalf("pageToken = %q, want %q", pageToken, "abc")
}
if pageSize != 25 {
t.Fatalf("pageSize = %d, want %d", pageSize, 25)
}
}
func TestParsePaginationSupportsHyphenatedFallback(t *testing.T) {
req := httptest.NewRequest("GET", "/v1/namespaces?page-token=abc&page-size=17", nil)
pageToken, pageSize, err := parsePagination(req)
if err != nil {
t.Fatalf("parsePagination() error = %v", err)
}
if pageToken != "abc" {
t.Fatalf("pageToken = %q, want %q", pageToken, "abc")
}
if pageSize != 17 {
t.Fatalf("pageSize = %d, want %d", pageSize, 17)
}
}
func TestParsePaginationRejectsInvalidPageSize(t *testing.T) {
testCases := []string{
"/v1/namespaces?pageSize=0",
"/v1/namespaces?pageSize=-1",
"/v1/namespaces?pageSize=foo",
"/v1/namespaces?pageSize=1001",
}
for _, rawURL := range testCases {
t.Run(rawURL, func(t *testing.T) {
req := httptest.NewRequest("GET", rawURL, nil)
if _, _, err := parsePagination(req); err == nil {
t.Fatalf("parsePagination() expected error for url %q", rawURL)
}
})
}
}

View File

@@ -0,0 +1,61 @@
package iceberg
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/gorilla/mux"
)
// TestRegisterTableRequestWireShape pins the field names to the iceberg-go REST
// client payload: {"name", "metadata-location"}.
func TestRegisterTableRequestWireShape(t *testing.T) {
body := `{"name":"orders","metadata-location":"s3://bucket/ns/orders/metadata/v1.metadata.json"}`
var req RegisterTableRequest
if err := json.Unmarshal([]byte(body), &req); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if req.Name != "orders" {
t.Errorf("Name = %q, want orders", req.Name)
}
if req.MetadataLocation != "s3://bucket/ns/orders/metadata/v1.metadata.json" {
t.Errorf("MetadataLocation = %q", req.MetadataLocation)
}
}
// TestRegisterTableRejectsOutOfBoundsLocation confirms a metadata-location that
// points outside the request's table bucket, or contains a traversal segment, is
// rejected with 400 before any metadata read.
func TestRegisterTableRejectsOutOfBoundsLocation(t *testing.T) {
cases := []struct {
name string
metadataLocation string
}{
{"cross bucket", "s3://other/ns/orders/metadata/v1.metadata.json"},
{"traversal", "s3://bucket/ns/../../etc/metadata/v1.metadata.json"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
body := `{"name":"orders","metadata-location":"` + c.metadataLocation + `"}`
r := httptest.NewRequest(http.MethodPost, "/v1/bucket/namespaces/ns/register", strings.NewReader(body))
r = mux.SetURLVars(r, map[string]string{"prefix": "bucket", "namespace": "ns"})
rec := httptest.NewRecorder()
(&Server{}).handleRegisterTable(rec, r)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d", rec.Code, http.StatusBadRequest)
}
var resp ErrorResponse
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode response: %v", err)
}
if resp.Error.Type != "BadRequestException" {
t.Fatalf("error type = %q, want BadRequestException", resp.Error.Type)
}
})
}
}

View File

@@ -0,0 +1,32 @@
package iceberg
import (
"net/http/httptest"
"strings"
"testing"
)
func TestHandleRenameTableRejectsIncompleteIdentifiers(t *testing.T) {
cases := []struct {
name string
body string
}{
{"missing source name", `{"source":{"namespace":["ns"]},"destination":{"namespace":["ns"],"name":"t2"}}`},
{"missing source namespace", `{"source":{"name":"t"},"destination":{"namespace":["ns"],"name":"t2"}}`},
{"missing destination name", `{"source":{"namespace":["ns"],"name":"t"},"destination":{"namespace":["ns"]}}`},
{"missing destination namespace", `{"source":{"namespace":["ns"],"name":"t"},"destination":{"name":"t2"}}`},
{"empty body", `{}`},
{"malformed json", `{`},
}
s := &Server{}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
r := httptest.NewRequest("POST", "/v1/tables/rename", strings.NewReader(tc.body))
w := httptest.NewRecorder()
s.handleRenameTable(w, r)
if w.Code != 400 {
t.Fatalf("handleRenameTable() status = %d, want 400", w.Code)
}
})
}
}

View File

@@ -0,0 +1,93 @@
package iceberg
import (
"net/http/httptest"
"testing"
"time"
"github.com/apache/iceberg-go/table"
"github.com/google/uuid"
)
// metadataWithSnapshots builds metadata whose main branch is on the last
// snapshot, optionally tagging one of the earlier ones.
func metadataWithSnapshots(t *testing.T, ids []int64, tagged int64) table.Metadata {
t.Helper()
base, err := newTableMetadata(uuid.New(), "s3://bkt/ns/t", nil, nil, nil, nil)
if err != nil {
t.Fatalf("newTableMetadata() error = %v", err)
}
builder, err := table.MetadataBuilderFromBase(base, "")
if err != nil {
t.Fatalf("MetadataBuilderFromBase() error = %v", err)
}
now := time.Now().UnixMilli()
for i, id := range ids {
snapshot := table.Snapshot{
SnapshotID: id,
TimestampMs: now + int64(i),
ManifestList: "metadata/snap-" + string(rune('0'+i)) + ".avro",
}
if err := builder.AddSnapshot(&snapshot); err != nil {
t.Fatalf("AddSnapshot(%d) error = %v", id, err)
}
}
if err := builder.SetSnapshotRef(table.MainBranch, ids[len(ids)-1], table.BranchRef); err != nil {
t.Fatalf("SetSnapshotRef(main) error = %v", err)
}
if tagged != 0 {
if err := builder.SetSnapshotRef("release", tagged, table.TagRef); err != nil {
t.Fatalf("SetSnapshotRef(release) error = %v", err)
}
}
metadata, err := builder.Build()
if err != nil {
t.Fatalf("Build() error = %v", err)
}
return metadata
}
func snapshotIDs(metadata table.Metadata) map[int64]bool {
ids := map[int64]bool{}
for _, snapshot := range metadata.Snapshots() {
ids[snapshot.SnapshotID] = true
}
return ids
}
func TestApplySnapshotsParamRefsKeepsOnlyReferenced(t *testing.T) {
metadata := metadataWithSnapshots(t, []int64{1, 2, 3}, 1)
got, err := applySnapshotsParam(httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t?snapshots=refs", nil), metadata)
if err != nil {
t.Fatalf("applySnapshotsParam() error = %v", err)
}
ids := snapshotIDs(got)
if !ids[1] {
t.Error("tagged snapshot 1 was dropped")
}
if !ids[3] {
t.Error("current snapshot 3 was dropped")
}
if ids[2] {
t.Error("unreferenced snapshot 2 was returned")
}
}
func TestApplySnapshotsParamDefaultsToAll(t *testing.T) {
metadata := metadataWithSnapshots(t, []int64{1, 2, 3}, 1)
for _, target := range []string{"/v1/namespaces/ns/tables/t", "/v1/namespaces/ns/tables/t?snapshots=all"} {
got, err := applySnapshotsParam(httptest.NewRequest("GET", target, nil), metadata)
if err != nil {
t.Fatalf("applySnapshotsParam(%s) error = %v", target, err)
}
if len(got.Snapshots()) != 3 {
t.Errorf("applySnapshotsParam(%s) kept %d snapshots, want 3", target, len(got.Snapshots()))
}
}
}

View File

@@ -0,0 +1,254 @@
package iceberg
import (
"bytes"
"encoding/json"
"testing"
"github.com/google/uuid"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
func TestDecodeCommitTransactionRequest(t *testing.T) {
body := `{"table-changes":[
{"identifier":{"namespace":["ns"],"name":"t1"},"requirements":[{"type":"assert-table-uuid","uuid":"00000000-0000-0000-0000-000000000001"}],"updates":[{"action":"set-properties","updates":{"k":"v"}}]},
{"identifier":{"namespace":["ns","sub"],"name":"t2"},"requirements":[],"updates":[]}
]}`
var req CommitTransactionRequest
if err := json.Unmarshal([]byte(body), &req); err != nil {
t.Fatalf("Unmarshal() error = %v", err)
}
if len(req.TableChanges) != 2 {
t.Fatalf("table-changes = %d, want 2", len(req.TableChanges))
}
if req.TableChanges[0].Identifier.Name != "t1" {
t.Fatalf("change[0] name = %q, want t1", req.TableChanges[0].Identifier.Name)
}
if got := []string(req.TableChanges[1].Identifier.Namespace); len(got) != 2 || got[1] != "sub" {
t.Fatalf("change[1] namespace = %v, want [ns sub]", got)
}
}
func TestParseTableChangeSeparatesStatistics(t *testing.T) {
change := tableChangeRequest{
Requirements: json.RawMessage(`[{"type":"assert-table-uuid","uuid":"00000000-0000-0000-0000-000000000001"}]`),
Updates: []json.RawMessage{
json.RawMessage(`{"action":"set-properties","updates":{"k":"v"}}`),
json.RawMessage(`{"action":"set-statistics","snapshot-id":10,"statistics-path":"s3://bucket/table/metadata/stats.puffin","file-size-in-bytes":100,"file-footer-size-in-bytes":20,"blob-metadata":[]}`),
},
}
requirements, updates, statistics, reqErr := parseTableChange(change)
if reqErr != nil {
t.Fatalf("parseTableChange() error = %+v", reqErr)
}
if len(requirements) != 1 {
t.Fatalf("requirements = %d, want 1", len(requirements))
}
if len(updates) != 1 {
t.Fatalf("updates = %d, want 1", len(updates))
}
if len(statistics) != 1 || statistics[0].set == nil || statistics[0].set.SnapshotID != 10 {
t.Fatalf("unexpected statistics updates: %#v", statistics)
}
}
func TestParseTableChangeRejectsInvalidRequirements(t *testing.T) {
change := tableChangeRequest{Requirements: json.RawMessage(`{not json}`)}
_, _, _, reqErr := parseTableChange(change)
if reqErr == nil {
t.Fatalf("parseTableChange() expected error for invalid requirements")
}
if reqErr.status != 400 {
t.Fatalf("status = %d, want 400", reqErr.status)
}
}
func TestCloneTableMetadataIsIndependent(t *testing.T) {
orig := &s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{TableUUID: "uuid-1"},
FullMetadata: json.RawMessage(`{"v":1}`),
}
clone := cloneTableMetadata(orig)
if clone == orig || clone.Iceberg == orig.Iceberg {
t.Fatalf("cloneTableMetadata returned aliased pointers")
}
orig.Iceberg.TableUUID = "mutated"
orig.FullMetadata[0] = 'X'
if clone.Iceberg.TableUUID != "uuid-1" {
t.Fatalf("clone Iceberg.TableUUID = %q, want uuid-1", clone.Iceberg.TableUUID)
}
if string(clone.FullMetadata) != `{"v":1}` {
t.Fatalf("clone FullMetadata = %q, mutated with source", clone.FullMetadata)
}
if cloneTableMetadata(nil) != nil {
t.Fatalf("cloneTableMetadata(nil) = non-nil")
}
}
func TestBuildTableRestoreRequestCarriesPriorState(t *testing.T) {
pc := &preparedTableCommit{
namespace: []string{"ns", "sub"},
tableName: "t1",
prevMetadataLoc: "s3://bucket/ns/t1/metadata/v3.metadata.json",
prevMetadataVersion: 3,
prevMetadata: &s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{TableUUID: uuid.NewString()},
FullMetadata: json.RawMessage(`{"format-version":2,"last-updated-ms":3}`),
},
}
req := buildTableRestoreRequest("arn:bucket", pc)
if req.TableBucketARN != "arn:bucket" || req.Name != "t1" {
t.Fatalf("restore target = %s/%s, want arn:bucket/t1", req.TableBucketARN, req.Name)
}
if req.MetadataLocation != pc.prevMetadataLoc {
t.Fatalf("restore MetadataLocation = %q, want %q", req.MetadataLocation, pc.prevMetadataLoc)
}
if req.MetadataVersion != pc.prevMetadataVersion {
t.Fatalf("restore MetadataVersion = %d, want %d", req.MetadataVersion, pc.prevMetadataVersion)
}
if req.Metadata == nil || !bytes.Equal(req.Metadata.FullMetadata, pc.prevMetadata.FullMetadata) {
t.Fatalf("restore FullMetadata = %q, want %q", req.Metadata.FullMetadata, pc.prevMetadata.FullMetadata)
}
if req.Metadata.Iceberg == nil || req.Metadata.Iceberg.TableUUID != pc.prevMetadata.Iceberg.TableUUID {
t.Fatalf("restore TableUUID mismatch")
}
}
// TestRollbackRevertsEveryFlippedField mirrors handleUpdateTable's partial-field
// update semantics to prove that after a flip + rollback the table is back to its
// pre-transaction state. With the old location-only rollback the table kept the
// new full metadata and version; the restore request must revert all of them.
func TestRollbackRevertsEveryFlippedField(t *testing.T) {
// prior on-disk state of an already-committed table.
prior := tableEntry{
metadataLocation: "s3://bucket/ns/t1/metadata/v3.metadata.json",
metadataVersion: 3,
metadata: &s3tables.TableMetadata{
Iceberg: &s3tables.IcebergMetadata{TableUUID: "uuid-1"},
FullMetadata: json.RawMessage(`{"format-version":2,"v":3}`),
},
}
pc := &preparedTableCommit{
namespace: []string{"ns"},
tableName: "t1",
tableUUID: uuid.MustParse("00000000-0000-0000-0000-000000000001"),
metadataBytes: json.RawMessage(`{"format-version":2,"v":4}`),
metadataVersion: 4,
newMetadataLoc: "s3://bucket/ns/t1/metadata/v4.metadata.json",
prevMetadataLoc: prior.metadataLocation,
prevMetadataVersion: prior.metadataVersion,
prevMetadata: cloneTableMetadata(prior.metadata),
}
// flip: apply the new commit, then roll it back via the restore request.
state := prior.clone()
state.applyUpdate(&s3tables.UpdateTableRequest{
Metadata: &s3tables.TableMetadata{Iceberg: &s3tables.IcebergMetadata{TableUUID: pc.tableUUID.String()}, FullMetadata: pc.metadataBytes},
MetadataVersion: pc.metadataVersion,
MetadataLocation: pc.newMetadataLoc,
})
state.applyUpdate(buildTableRestoreRequest("arn:bucket", pc))
if state.metadataLocation != prior.metadataLocation {
t.Fatalf("after rollback location = %q, want %q", state.metadataLocation, prior.metadataLocation)
}
if state.metadataVersion != prior.metadataVersion {
t.Fatalf("after rollback version = %d, want %d", state.metadataVersion, prior.metadataVersion)
}
if !bytes.Equal(state.metadata.FullMetadata, prior.metadata.FullMetadata) {
t.Fatalf("after rollback FullMetadata = %q, want %q", state.metadata.FullMetadata, prior.metadata.FullMetadata)
}
if state.metadata.Iceberg.TableUUID != prior.metadata.Iceberg.TableUUID {
t.Fatalf("after rollback TableUUID = %q, want %q", state.metadata.Iceberg.TableUUID, prior.metadata.Iceberg.TableUUID)
}
}
// TestCleanupSafeMetadataKeepsFailedRollback proves a flip failure at index 2
// where table 0's rollback also fails leaves table 0's new metadata in place
// (its pointer still references it), while the rolled-back table 1 and the
// never-flipped tables 2 and 3 are safe to delete.
func TestCleanupSafeMetadataKeepsFailedRollback(t *testing.T) {
prepared := make([]preparedTableCommit, 4)
rolledBack := []bool{false, true} // tables 0 and 1 were flipped; 0's rollback failed
safe := cleanupSafeMetadata(prepared, 2, rolledBack)
want := []bool{false, true, true, true}
for i := range want {
if safe[i] != want[i] {
t.Fatalf("safe[%d] = %v, want %v (full %v)", i, safe[i], want[i], safe)
}
}
}
// TestCleanupSafeMetadataFirstFlipFails covers a flip failure at the first table:
// nothing was flipped, so every written file is safe to delete.
func TestCleanupSafeMetadataFirstFlipFails(t *testing.T) {
prepared := make([]preparedTableCommit, 3)
safe := cleanupSafeMetadata(prepared, 0, nil)
for i, ok := range safe {
if !ok {
t.Fatalf("safe[%d] = false, want true (full %v)", i, safe)
}
}
}
// TestCleanupSafeMetadataAllRolledBack covers the common case where every prior
// flip rolls back cleanly, so all metadata files are safe to delete.
func TestCleanupSafeMetadataAllRolledBack(t *testing.T) {
prepared := make([]preparedTableCommit, 3)
safe := cleanupSafeMetadata(prepared, 2, []bool{true, true})
for i, ok := range safe {
if !ok {
t.Fatalf("safe[%d] = false, want true (full %v)", i, safe)
}
}
}
// tableEntry models the persisted table fields and the partial-update rules
// handleUpdateTable applies, so rollback can be exercised without a live filer.
type tableEntry struct {
metadataLocation string
metadataVersion int
metadata *s3tables.TableMetadata
}
func (e tableEntry) clone() *tableEntry {
return &tableEntry{
metadataLocation: e.metadataLocation,
metadataVersion: e.metadataVersion,
metadata: cloneTableMetadata(e.metadata),
}
}
func (e *tableEntry) applyUpdate(req *s3tables.UpdateTableRequest) {
if req.Metadata != nil {
if e.metadata == nil {
e.metadata = &s3tables.TableMetadata{}
}
if req.Metadata.Iceberg != nil {
if e.metadata.Iceberg == nil {
e.metadata.Iceberg = &s3tables.IcebergMetadata{}
}
if req.Metadata.Iceberg.TableUUID != "" {
e.metadata.Iceberg.TableUUID = req.Metadata.Iceberg.TableUUID
}
}
if len(req.Metadata.FullMetadata) > 0 {
e.metadata.FullMetadata = req.Metadata.FullMetadata
}
}
if req.MetadataLocation != "" {
e.metadataLocation = req.MetadataLocation
}
if req.MetadataVersion > 0 {
e.metadataVersion = req.MetadataVersion
}
}

View File

@@ -0,0 +1,87 @@
package iceberg
import (
"encoding/json"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"testing"
"github.com/gorilla/mux"
)
func TestApplyNamespacePropertyUpdates(t *testing.T) {
current := map[string]string{"foo": "bar", "prop": "yes"}
props, summary := applyNamespacePropertyUpdates(current, []string{"abc"}, map[string]string{"prop": "no"})
// Removal of a missing key reports missing, not removed.
if !reflect.DeepEqual(summary.Removed, []string{}) {
t.Errorf("Removed = %v, want []", summary.Removed)
}
if !reflect.DeepEqual(summary.Updated, []string{"prop"}) {
t.Errorf("Updated = %v, want [prop]", summary.Updated)
}
if !reflect.DeepEqual(summary.Missing, []string{"abc"}) {
t.Errorf("Missing = %v, want [abc]", summary.Missing)
}
if props["prop"] != "no" || props["foo"] != "bar" {
t.Errorf("merged properties = %v, want foo=bar prop=no", props)
}
// Input map must not be mutated.
if current["prop"] != "yes" {
t.Errorf("source map mutated: prop = %q, want yes", current["prop"])
}
}
func TestApplyNamespacePropertyUpdatesRemoveExisting(t *testing.T) {
props, summary := applyNamespacePropertyUpdates(
map[string]string{"a": "1", "b": "2"}, []string{"a"}, nil)
if !reflect.DeepEqual(summary.Removed, []string{"a"}) {
t.Errorf("Removed = %v, want [a]", summary.Removed)
}
if _, ok := props["a"]; ok {
t.Errorf("key a should have been removed: %v", props)
}
if len(summary.Missing) != 0 || len(summary.Updated) != 0 {
t.Errorf("unexpected summary: %+v", summary)
}
}
func TestApplyNamespacePropertyUpdatesDuplicateRemovals(t *testing.T) {
// A repeated removal key must not be reported as both removed and missing.
_, summary := applyNamespacePropertyUpdates(
map[string]string{"a": "1"}, []string{"a", "a"}, nil)
if !reflect.DeepEqual(summary.Removed, []string{"a"}) {
t.Errorf("Removed = %v, want [a]", summary.Removed)
}
if len(summary.Missing) != 0 {
t.Errorf("Missing = %v, want []", summary.Missing)
}
}
// A key in both removals and updates is rejected before any backend call, so
// the conflict path is reachable without a filer.
func TestHandleUpdateNamespacePropertiesConflict(t *testing.T) {
req := httptest.NewRequest(http.MethodPost, "/v1/namespaces/ns/properties",
strings.NewReader(`{"removals":["x"],"updates":{"x":"1"}}`))
req = mux.SetURLVars(req, map[string]string{"namespace": "ns"})
rec := httptest.NewRecorder()
(&Server{}).handleUpdateNamespaceProperties(rec, req)
if rec.Code != http.StatusUnprocessableEntity {
t.Fatalf("status = %d, want 422", rec.Code)
}
var resp ErrorResponse
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode: %v", err)
}
if resp.Error.Type != "UnprocessableEntityException" {
t.Fatalf("error type = %q, want UnprocessableEntityException", resp.Error.Type)
}
}

View File

@@ -0,0 +1,115 @@
package iceberg
import (
"encoding/json"
"errors"
"testing"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/table"
"github.com/apache/iceberg-go/view"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
func TestIsViewNotFound(t *testing.T) {
cases := []struct {
err error
want bool
}{
{nil, false},
{&s3tables.S3TablesError{Type: s3tables.ErrCodeNoSuchView, Message: "view x not found"}, true},
{&s3tables.S3TablesError{Type: s3tables.ErrCodeNoSuchNamespace, Message: "namespace x not found"}, true},
{errors.New("view x not found"), true},
{&s3tables.S3TablesError{Type: s3tables.ErrCodeViewAlreadyExists, Message: "already exists"}, false},
{errors.New("connection refused"), false},
}
for _, c := range cases {
if got := isViewNotFound(c.err); got != c.want {
t.Errorf("isViewNotFound(%v) = %v, want %v", c.err, got, c.want)
}
}
}
func TestIsViewAlreadyExists(t *testing.T) {
cases := []struct {
err error
want bool
}{
{nil, false},
{&s3tables.S3TablesError{Type: s3tables.ErrCodeViewAlreadyExists, Message: "already exists"}, true},
{errors.New("view already exists"), true},
{&s3tables.S3TablesError{Type: s3tables.ErrCodeNoSuchView, Message: "not found"}, false},
{errors.New("connection refused"), false},
}
for _, c := range cases {
if got := isViewAlreadyExists(c.err); got != c.want {
t.Errorf("isViewAlreadyExists(%v) = %v, want %v", c.err, got, c.want)
}
}
}
func TestViewPropertiesCopiesAndIsNonNil(t *testing.T) {
if got := viewProperties(nil); got == nil {
t.Fatal("viewProperties(nil) = nil, want non-nil")
}
src := iceberg.Properties{"k": "v"}
out := viewProperties(src)
out["k2"] = "v2"
if _, ok := src["k2"]; ok {
t.Fatal("viewProperties did not copy: mutation leaked into source")
}
if out["k"] != "v" {
t.Fatalf("viewProperties dropped key: got %q", out["k"])
}
}
// TestViewResponseRoundTrip verifies a built view metadata marshals into the
// ViewResponse wire shape and parses back via the iceberg-go view parser.
func TestViewResponseRoundTrip(t *testing.T) {
schema := iceberg.NewSchemaWithIdentifiers(0, nil,
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int32, Required: true},
)
ver, err := view.NewVersionFromSQL(1, schema.ID, "SELECT 1", table.Identifier{"ns"})
if err != nil {
t.Fatal(err)
}
md, err := view.NewMetadata(ver, schema, "s3://bucket/ns/v", iceberg.Properties{})
if err != nil {
t.Fatal(err)
}
resp := ViewResponse{
MetadataLocation: "s3://bucket/ns/v/metadata/v1.metadata.json",
Metadata: md,
Config: iceberg.Properties{},
}
data, err := json.Marshal(resp)
if err != nil {
t.Fatalf("marshal: %v", err)
}
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
t.Fatalf("unmarshal envelope: %v", err)
}
if _, ok := raw["metadata-location"]; !ok {
t.Fatal("response missing metadata-location")
}
if _, ok := raw["metadata"]; !ok {
t.Fatal("response missing metadata")
}
var back ViewResponse
if err := json.Unmarshal(data, &back); err != nil {
t.Fatalf("unmarshal back: %v", err)
}
if back.Metadata == nil {
t.Fatal("metadata nil after round-trip")
}
if back.Metadata.CurrentVersion().VersionID != 1 {
t.Fatalf("current version = %d, want 1", back.Metadata.CurrentVersion().VersionID)
}
if back.MetadataLocation != resp.MetadataLocation {
t.Fatalf("metadata-location = %q, want %q", back.MetadataLocation, resp.MetadataLocation)
}
}

View File

@@ -0,0 +1,71 @@
package iceberg
import (
"encoding/json"
"net/http/httptest"
"net/url"
"testing"
)
// The admin console tells users that every table bucket is its own Iceberg
// catalog, so the /v1/config handshake must turn whichever spelling of that
// bucket the client sends into overrides.prefix. PyIceberg only sends the
// warehouse on this one call, so a dropped value leaves it talking to the
// default bucket for the rest of the session.
func TestHandleConfigWarehouseSpellings(t *testing.T) {
tests := []struct {
name string
warehouse string
wantPrefix string
wantWarehouse string
}{
{
name: "s3 location",
warehouse: "s3://seaweed-iceberg/",
wantPrefix: "seaweed-iceberg",
wantWarehouse: "s3://seaweed-iceberg",
},
{
name: "bare bucket name",
warehouse: "seaweed-iceberg",
wantPrefix: "seaweed-iceberg",
wantWarehouse: "s3://seaweed-iceberg",
},
{
name: "table bucket ARN",
warehouse: "arn:aws:s3tables:us-east-1:admin:bucket/seaweed-iceberg",
wantPrefix: "seaweed-iceberg",
wantWarehouse: "s3://seaweed-iceberg",
},
{
name: "no warehouse",
warehouse: "",
},
{
name: "unusable value",
warehouse: "file:///tmp/wh",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
target := "/v1/config"
if tc.warehouse != "" {
target += "?" + url.Values{"warehouse": {tc.warehouse}}.Encode()
}
r := httptest.NewRequest("GET", target, nil)
rec := httptest.NewRecorder()
(&Server{}).handleConfig(rec, r)
var config CatalogConfig
if err := json.Unmarshal(rec.Body.Bytes(), &config); err != nil {
t.Fatalf("decode config: %v", err)
}
if got := config.Overrides["prefix"]; got != tc.wantPrefix {
t.Errorf("overrides.prefix = %q, want %q", got, tc.wantPrefix)
}
if got := config.Defaults["warehouse"]; got != tc.wantWarehouse {
t.Errorf("defaults.warehouse = %q, want %q", got, tc.wantWarehouse)
}
})
}
}

View File

@@ -0,0 +1,618 @@
package iceberg
import (
"bytes"
"context"
"encoding/json"
"fmt"
"strings"
"github.com/linkedin/goavro/v2"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/wdclient"
)
// Some engines (ClickHouse's experimental Iceberg writes among them) commit
// snapshots whose manifest avro files omit the spec's field-id annotations and
// reference files by bucket-relative path. Both make the snapshot unreadable
// for other engines even though the data itself is fine. Since the catalog is
// the commit chokepoint, add-snapshot updates are repaired here: the manifest
// list and manifests are transcoded with the annotations added and the paths
// absolutized, and the snapshot is pointed at the repaired copies.
// avroFieldIDs describes the spec-mandated ids for one avro field and the
// fields nested beneath it (through unions, arrays, and records).
type avroFieldIDs struct {
id int
elementID int
children map[string]avroFieldIDs
}
// manifestListFieldIDs returns the fixed v2 manifest_file schema ids.
func manifestListFieldIDs() map[string]avroFieldIDs {
return map[string]avroFieldIDs{
"manifest_path": {id: 500},
"manifest_length": {id: 501},
"partition_spec_id": {id: 502},
"content": {id: 517},
"sequence_number": {id: 515},
"min_sequence_number": {id: 516},
"added_snapshot_id": {id: 503},
"added_files_count": {id: 504},
"existing_files_count": {id: 505},
"deleted_files_count": {id: 506},
"added_rows_count": {id: 512},
"existing_rows_count": {id: 513},
"deleted_rows_count": {id: 514},
"partitions": {id: 507, elementID: 508, children: map[string]avroFieldIDs{
"contains_null": {id: 509},
"contains_nan": {id: 518},
"lower_bound": {id: 510},
"upper_bound": {id: 511},
}},
"key_metadata": {id: 519},
"first_row_id": {id: 520},
}
}
// manifestEntryFieldIDs returns the fixed v2 manifest_entry schema ids. The
// partition struct is table-specific; its ids come from the partition-spec
// JSON embedded in the manifest's OCF metadata.
func manifestEntryFieldIDs(partitionIDs map[string]avroFieldIDs) map[string]avroFieldIDs {
return map[string]avroFieldIDs{
"status": {id: 0},
"snapshot_id": {id: 1},
"sequence_number": {id: 3},
"file_sequence_number": {id: 4},
"data_file": {id: 2, children: map[string]avroFieldIDs{
"content": {id: 134},
"file_path": {id: 100},
"file_format": {id: 101},
"partition": {id: 102, children: partitionIDs},
"record_count": {id: 103},
"file_size_in_bytes": {id: 104},
"column_sizes": {id: 108, children: kvFieldIDs(117, 118)},
"value_counts": {id: 109, children: kvFieldIDs(119, 120)},
"null_value_counts": {id: 110, children: kvFieldIDs(121, 122)},
"nan_value_counts": {id: 137, children: kvFieldIDs(138, 139)},
"lower_bounds": {id: 125, children: kvFieldIDs(126, 127)},
"upper_bounds": {id: 128, children: kvFieldIDs(129, 130)},
"key_metadata": {id: 131},
"split_offsets": {id: 132, elementID: 133},
"equality_ids": {id: 135, elementID: 136},
"sort_order_id": {id: 140},
"first_row_id": {id: 142},
"referenced_data_file": {id: 143},
"content_offset": {id: 144},
"content_size_in_bytes": {id: 145},
}},
}
}
func kvFieldIDs(keyID, valueID int) map[string]avroFieldIDs {
return map[string]avroFieldIDs{
"key": {id: keyID},
"value": {id: valueID},
}
}
// partitionFieldIDsFromSpec extracts name -> field-id from the partition-spec
// JSON carried in a manifest's OCF metadata.
func partitionFieldIDsFromSpec(specJSON []byte) (map[string]avroFieldIDs, error) {
if len(specJSON) == 0 {
return map[string]avroFieldIDs{}, nil
}
var fields []struct {
Name string `json:"name"`
FieldID int `json:"field-id"`
}
if err := json.Unmarshal(specJSON, &fields); err != nil {
return nil, fmt.Errorf("parse partition-spec metadata: %w", err)
}
ids := make(map[string]avroFieldIDs, len(fields))
for _, f := range fields {
ids[f.Name] = avroFieldIDs{id: f.FieldID}
}
return ids, nil
}
// annotateAvroRecordFields adds "field-id" to every field of a parsed avro
// record schema, recursing into nested types. Returns whether anything was
// added, and ok=false when a field has no known id (the schema is left for an
// unknown writer dialect rather than half-annotated).
func annotateAvroRecordFields(record map[string]any, ids map[string]avroFieldIDs) (changed, ok bool) {
fields, _ := record["fields"].([]any)
if len(fields) == 0 {
return false, true
}
for _, f := range fields {
field, isMap := f.(map[string]any)
if !isMap {
return changed, false
}
name, _ := field["name"].(string)
spec, known := ids[name]
if !known {
return changed, false
}
if _, has := field["field-id"]; !has {
field["field-id"] = spec.id
changed = true
}
typeChanged, typeOK := annotateAvroType(field["type"], spec)
changed = changed || typeChanged
if !typeOK {
return changed, false
}
}
return changed, true
}
// annotateAvroType annotates the schema node of a single field: unions are
// followed through their non-null branches, arrays get "element-id", and
// nested records recurse with the field's child ids.
func annotateAvroType(node any, spec avroFieldIDs) (changed, ok bool) {
switch t := node.(type) {
case []any: // union
for _, branch := range t {
branchChanged, branchOK := annotateAvroType(branch, spec)
changed = changed || branchChanged
if !branchOK {
return changed, false
}
}
return changed, true
case map[string]any:
switch t["type"] {
case "record":
childIDs := spec.children
if childIDs == nil {
childIDs = map[string]avroFieldIDs{}
}
return annotateAvroRecordFields(t, childIDs)
case "array":
if spec.elementID != 0 {
if _, has := t["element-id"]; !has {
t["element-id"] = spec.elementID
changed = true
}
}
itemChanged, itemOK := annotateAvroType(t["items"], spec)
return changed || itemChanged, itemOK
case "map":
// The spec wants key-id/value-id on genuine avro maps; this repair
// only knows the array-of-key_value encoding, so leave the file to
// its writer rather than half-annotating it.
return changed, false
default:
return false, true
}
default: // primitive name or named-type reference
return false, true
}
}
// transcodeManifestAvro decodes an avro OCF file, annotates its schema with
// the given ids, applies mutate to each datum and fixMeta to the OCF metadata,
// and re-encodes. When nothing changes, the original bytes are returned
// unchanged.
func transcodeManifestAvro(raw []byte, ids map[string]avroFieldIDs, mutate func(map[string]any) bool, fixMeta func(map[string][]byte) bool) ([]byte, bool, error) {
reader, err := goavro.NewOCFReader(bytes.NewReader(raw))
if err != nil {
return nil, false, fmt.Errorf("open avro: %w", err)
}
meta := reader.MetaData()
outMeta := make(map[string][]byte, len(meta))
for k, v := range meta {
if k == "avro.schema" || k == "avro.codec" {
continue
}
outMeta[k] = v
}
metaChanged := false
if fixMeta != nil {
metaChanged = fixMeta(outMeta)
}
var schema map[string]any
if err := json.Unmarshal(meta["avro.schema"], &schema); err != nil {
return nil, false, fmt.Errorf("parse avro schema: %w", err)
}
schemaChanged, ok := annotateAvroRecordFields(schema, ids)
if !ok {
return nil, false, fmt.Errorf("avro schema has fields outside the Iceberg manifest spec")
}
datumChanged := false
var datums []any
for reader.Scan() {
datum, err := reader.Read()
if err != nil {
return nil, false, fmt.Errorf("read avro datum: %w", err)
}
if record, isMap := datum.(map[string]any); isMap && mutate != nil {
if mutate(record) {
datumChanged = true
}
}
datums = append(datums, datum)
}
if err := reader.Err(); err != nil {
return nil, false, fmt.Errorf("scan avro: %w", err)
}
if !schemaChanged && !datumChanged && !metaChanged {
return raw, false, nil
}
annotatedSchema, err := json.Marshal(schema)
if err != nil {
return nil, false, err
}
var buf bytes.Buffer
writer, err := goavro.NewOCFWriter(goavro.OCFConfig{
W: &buf,
Schema: string(annotatedSchema),
MetaData: outMeta,
})
if err != nil {
return nil, false, fmt.Errorf("create avro writer: %w", err)
}
if err := writer.Append(datums); err != nil {
return nil, false, fmt.Errorf("write avro datums: %w", err)
}
return buf.Bytes(), true, nil
}
// manifestStore abstracts reading and writing files in a table's metadata
// directory so the repair logic is testable without a filer.
type manifestStore interface {
loadFile(ctx context.Context, location string) ([]byte, error)
saveFile(ctx context.Context, location string, data []byte) error
}
// absolutizeLocation resolves a bucket-relative path against the bucket of the
// table location. Locations that already carry a scheme pass through.
func absolutizeLocation(location, tableLocation string) string {
if location == "" || strings.Contains(location, "://") {
return location
}
bucket, _, err := parseS3Location(tableLocation)
if err != nil {
return location
}
return "s3://" + bucket + "/" + strings.TrimPrefix(location, "/")
}
// metadataFileName returns the file name when location is a direct child of
// the table's metadata directory, which is the only place repair will read
// from or write to.
func metadataFileName(tableLocation, location string) (string, bool) {
prefix := strings.TrimSuffix(tableLocation, "/") + "/metadata/"
name := strings.TrimPrefix(location, prefix)
if name == location || name == "" || strings.Contains(name, "/") {
return "", false
}
return name, true
}
const repairedManifestPrefix = "repaired-"
// maxRepairableManifestSize bounds how much manifest data the repair path
// buffers in memory; anything larger fails the repair and the commit
// proceeds with the writer's original files.
const maxRepairableManifestSize = 64 << 20
// hasFieldIDAnnotations is the cheap compliance probe: spec-compliant writers
// annotate every manifest avro schema with field-id, and a writer that
// annotates the manifest list annotates its manifests too.
func hasFieldIDAnnotations(raw []byte) bool {
reader, err := goavro.NewOCFReader(bytes.NewReader(raw))
if err != nil {
return false
}
return bytes.Contains(reader.MetaData()["avro.schema"], []byte(`"field-id"`))
}
// repairManifestList loads a snapshot's manifest list and, when it needs
// repair, rewrites it (and the manifests it references) into spec-compliant
// copies next to the originals. Returns the manifest list location the
// snapshot should reference.
func repairManifestList(ctx context.Context, store manifestStore, tableLocation, listLocation string) (string, bool, error) {
absListLocation := absolutizeLocation(listLocation, tableLocation)
listName, inMetadataDir := metadataFileName(tableLocation, absListLocation)
if !inMetadataDir {
return listLocation, false, nil
}
listBytes, err := store.loadFile(ctx, absListLocation)
if err != nil {
return listLocation, false, fmt.Errorf("load manifest list: %w", err)
}
if absListLocation == listLocation && hasFieldIDAnnotations(listBytes) {
return listLocation, false, nil
}
metadataDir := strings.TrimSuffix(tableLocation, "/") + "/metadata/"
var manifestErr error
repairedList, listChanged, err := transcodeManifestAvro(listBytes, manifestListFieldIDs(), func(record map[string]any) bool {
manifestPath, _ := record["manifest_path"].(string)
if manifestPath == "" || manifestErr != nil {
return false
}
absManifest := absolutizeLocation(manifestPath, tableLocation)
recordChanged := absManifest != manifestPath
if recordChanged {
record["manifest_path"] = absManifest
}
manifestName, ok := metadataFileName(tableLocation, absManifest)
if !ok {
return recordChanged
}
manifestBytes, err := store.loadFile(ctx, absManifest)
if err != nil {
manifestErr = fmt.Errorf("load manifest %s: %w", absManifest, err)
return recordChanged
}
repaired, changed, err := repairManifest(manifestBytes, tableLocation, manifestContentValue(record))
if err != nil {
manifestErr = fmt.Errorf("repair manifest %s: %w", absManifest, err)
return recordChanged
}
if !changed {
return recordChanged
}
repairedLocation := metadataDir + repairedManifestPrefix + manifestName
if err := store.saveFile(ctx, repairedLocation, repaired); err != nil {
manifestErr = fmt.Errorf("save repaired manifest %s: %w", repairedLocation, err)
return recordChanged
}
record["manifest_path"] = repairedLocation
record["manifest_length"] = int64(len(repaired))
return true
}, nil)
if err != nil {
return listLocation, false, err
}
if manifestErr != nil {
return listLocation, false, manifestErr
}
if !listChanged && absListLocation == listLocation {
return listLocation, false, nil
}
repairedListLocation := metadataDir + repairedManifestPrefix + listName
if listChanged {
if err := store.saveFile(ctx, repairedListLocation, repairedList); err != nil {
return listLocation, false, fmt.Errorf("save repaired manifest list: %w", err)
}
return repairedListLocation, true, nil
}
// Only the pointer was relative; the file itself is fine.
return absListLocation, true, nil
}
// manifestContentValue reads the manifest-list entry's content field
// (0 = data, 1 = deletes); v1 lists have no such field and default to data.
// Union-typed values arrive from goavro as a single-branch map.
func manifestContentValue(record map[string]any) int {
value := record["content"]
if union, isUnion := value.(map[string]any); isUnion {
for _, branch := range union {
value = branch
break
}
}
switch v := value.(type) {
case int32:
return int(v)
case int64:
return int(v)
case int:
return v
}
return 0
}
// fixManifestOCFMetadata fixes OCF metadata keys the spec requires of
// manifests but lax writers omit or contradict: "content" must agree with the
// manifest-list entry, which is what readers trust for scan planning, and
// "partition-spec-id" (ClickHouse writes the underscore variant).
func fixManifestOCFMetadata(listContent int) func(map[string][]byte) bool {
return func(meta map[string][]byte) bool {
changed := false
content := "data"
if listContent == 1 {
content = "deletes"
}
if string(meta["content"]) != content {
meta["content"] = []byte(content)
changed = true
}
if _, ok := meta["partition-spec-id"]; !ok {
if v, ok := meta["partition_spec_id"]; ok {
meta["partition-spec-id"] = v
changed = true
}
}
return changed
}
}
// repairManifest annotates a single manifest's schema and absolutizes the
// data file paths inside it. listContent is the content declared by the
// manifest-list entry referencing this manifest.
func repairManifest(raw []byte, tableLocation string, listContent int) ([]byte, bool, error) {
reader, err := goavro.NewOCFReader(bytes.NewReader(raw))
if err != nil {
return nil, false, fmt.Errorf("open manifest avro: %w", err)
}
partitionIDs, err := partitionFieldIDsFromSpec(reader.MetaData()["partition-spec"])
if err != nil {
return nil, false, err
}
return transcodeManifestAvro(raw, manifestEntryFieldIDs(partitionIDs), func(record map[string]any) bool {
dataFile, _ := record["data_file"].(map[string]any)
if dataFile == nil {
return false
}
filePath, _ := dataFile["file_path"].(string)
absPath := absolutizeLocation(filePath, tableLocation)
if absPath == filePath {
return false
}
dataFile["file_path"] = absPath
return true
}, fixManifestOCFMetadata(listContent))
}
// repairAddSnapshotUpdates rewrites the manifest-list references of
// add-snapshot updates whose manifests need repair. Repair is best effort: on
// any error the original update is kept and the commit proceeds as it would
// have without repair.
func repairAddSnapshotUpdates(ctx context.Context, store manifestStore, tableLocation string, rawUpdates []json.RawMessage) ([]json.RawMessage, bool) {
changedAny := false
out := make([]json.RawMessage, len(rawUpdates))
copy(out, rawUpdates)
for i, rawUpdate := range rawUpdates {
var probe struct {
Action string `json:"action"`
}
if err := json.Unmarshal(rawUpdate, &probe); err != nil || probe.Action != "add-snapshot" {
continue
}
var update map[string]json.RawMessage
if err := json.Unmarshal(rawUpdate, &update); err != nil {
continue
}
var snapshot map[string]json.RawMessage
if err := json.Unmarshal(update["snapshot"], &snapshot); err != nil {
continue
}
var listLocation string
if err := json.Unmarshal(snapshot["manifest-list"], &listLocation); err != nil || listLocation == "" {
continue
}
repairedLocation, changed, err := repairManifestList(ctx, store, tableLocation, listLocation)
if err != nil {
glog.Warningf("Iceberg: manifest repair for %s skipped: %v", listLocation, err)
continue
}
if !changed {
continue
}
locationJSON, err := json.Marshal(repairedLocation)
if err != nil {
continue
}
snapshot["manifest-list"] = locationJSON
snapshotJSON, err := json.Marshal(snapshot)
if err != nil {
continue
}
update["snapshot"] = snapshotJSON
updateJSON, err := json.Marshal(update)
if err != nil {
continue
}
out[i] = updateJSON
changedAny = true
glog.V(1).Infof("Iceberg: repaired manifest list %s -> %s", listLocation, repairedLocation)
}
return out, changedAny
}
// serverManifestStore reads and writes table metadata files through the filer.
type serverManifestStore struct {
server *Server
tableLocation string
}
func (st *serverManifestStore) split(location string) (bucket, tablePath, fileName string, err error) {
fileName, ok := metadataFileName(st.tableLocation, location)
if !ok {
return "", "", "", fmt.Errorf("%s is outside the table metadata directory", location)
}
bucket, tablePath, err = parseS3Location(st.tableLocation)
return bucket, tablePath, fileName, err
}
func (st *serverManifestStore) loadFile(ctx context.Context, location string) ([]byte, error) {
bucket, tablePath, fileName, err := st.split(location)
if err != nil {
return nil, err
}
return st.server.loadTableMetadataBlob(ctx, bucket, tablePath, fileName)
}
func (st *serverManifestStore) saveFile(ctx context.Context, location string, data []byte) error {
bucket, tablePath, fileName, err := st.split(location)
if err != nil {
return err
}
return st.server.saveMetadataBlob(ctx, bucket, tablePath, fileName, data, "application/avro", false)
}
// repairAddSnapshotManifests is the server entry point used by the commit
// handlers.
func (s *Server) repairAddSnapshotManifests(ctx context.Context, tableLocation string, rawUpdates []json.RawMessage) ([]json.RawMessage, bool) {
store := &serverManifestStore{server: s, tableLocation: tableLocation}
return repairAddSnapshotUpdates(ctx, store, tableLocation, rawUpdates)
}
// lookupFileIDAdapter lets filer.StreamContent resolve volume locations
// through the iceberg server's filer client.
type lookupFileIDAdapter struct {
fn wdclient.LookupFileIdFunctionType
}
func (a *lookupFileIDAdapter) GetLookupFileIdFunction() wdclient.LookupFileIdFunctionType {
return a.fn
}
// loadTableMetadataBlob reads a file from the table's metadata directory,
// following chunks for files that were written through the S3 gateway rather
// than inline by the catalog itself.
func (s *Server) loadTableMetadataBlob(ctx context.Context, bucketName, tablePath, fileName string) ([]byte, error) {
var entry *filer_pb.Entry
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
resp, err := filer_pb.LookupEntry(ctx, client, &filer_pb.LookupDirectoryEntryRequest{
Directory: metadataDirPath(bucketName, tablePath),
Name: fileName,
})
if err != nil {
return err
}
entry = resp.Entry
return nil
})
if err != nil {
return nil, err
}
if entry == nil {
return nil, fmt.Errorf("no entry for %s", fileName)
}
if len(entry.Content) > 0 || len(entry.GetChunks()) == 0 {
return entry.Content, nil
}
if size := filer.FileSize(entry); size > maxRepairableManifestSize {
return nil, fmt.Errorf("%s is %d bytes, larger than the %d byte repair limit", fileName, size, maxRepairableManifestSize)
}
fullClient, ok := s.filerClient.(filer_pb.FilerClient)
if !ok {
return nil, fmt.Errorf("filer client cannot resolve chunk locations")
}
var buf bytes.Buffer
lookup := &lookupFileIDAdapter{fn: filer.LookupFn(fullClient)}
if err := filer.StreamContent(lookup, &buf, entry.GetChunks(), 0, int64(filer.FileSize(entry))); err != nil {
return nil, fmt.Errorf("read %s: %w", fileName, err)
}
return buf.Bytes(), nil
}

View File

@@ -0,0 +1,281 @@
package iceberg
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
icebergmanifest "github.com/apache/iceberg-go"
"github.com/linkedin/goavro/v2"
)
// The testdata fixtures were written by ClickHouse 25.8's experimental
// Iceberg insert against a SeaweedFS table bucket. clickhouse_rel holds the
// default output (bucket-relative paths, no field-id annotations);
// clickhouse_abs was written with write_full_path_in_iceberg_metadata=1
// (absolute paths, still no field-ids).
type fakeManifestStore struct {
files map[string][]byte
saved map[string][]byte
}
func newFakeManifestStore() *fakeManifestStore {
return &fakeManifestStore{files: map[string][]byte{}, saved: map[string][]byte{}}
}
func (f *fakeManifestStore) loadFile(_ context.Context, location string) ([]byte, error) {
if data, ok := f.files[location]; ok {
return data, nil
}
return nil, fmt.Errorf("no file at %s", location)
}
func (f *fakeManifestStore) saveFile(_ context.Context, location string, data []byte) error {
f.files[location] = data
f.saved[location] = data
return nil
}
func loadFixture(t *testing.T, parts ...string) []byte {
t.Helper()
data, err := os.ReadFile(filepath.Join(append([]string{"testdata"}, parts...)...))
if err != nil {
t.Fatalf("read fixture: %v", err)
}
return data
}
const (
relTableLocation = "s3://iceberg-tables/wprobe/target_rel"
relListName = "snap-449670589-2-48b874af-90f6-4e5e-8002-df91e59c8a0c.avro"
relManifestName = "690958a2-9936-4a04-8117-d8104c977195.avro"
absTableLocation = "s3://iceberg-tables/wprobe/target"
absListName = "snap-331611856-2-51798e6c-427b-4fc2-9149-99e75d96e045.avro"
absManifestName = "b317f0e5-10f3-41d6-86bc-8b42f01f3e86.avro"
)
// verifyRepairedManifests checks the repaired files with iceberg-go's strict
// typed readers: the manifest list parses, references the repaired manifest
// with the right length, and the manifest's data file path is absolute.
func verifyRepairedManifests(t *testing.T, store *fakeManifestStore, listLocation, tableLocation string) {
t.Helper()
listBytes, ok := store.files[listLocation]
if !ok {
t.Fatalf("repaired manifest list %s was not saved", listLocation)
}
if !bytes.Contains(listBytes, []byte(`"field-id"`)) {
t.Fatalf("repaired manifest list schema has no field-id annotations")
}
manifests, err := icebergmanifest.ReadManifestList(bytes.NewReader(listBytes))
if err != nil {
t.Fatalf("iceberg-go cannot read repaired manifest list: %v", err)
}
if len(manifests) != 1 {
t.Fatalf("manifest list has %d entries, want 1", len(manifests))
}
manifestPath := manifests[0].FilePath()
if !strings.HasPrefix(manifestPath, tableLocation+"/metadata/") {
t.Fatalf("manifest path %s is not absolute under the table location", manifestPath)
}
manifestBytes, ok := store.files[manifestPath]
if !ok {
t.Fatalf("manifest %s not present in store", manifestPath)
}
if manifests[0].Length() != int64(len(manifestBytes)) {
t.Fatalf("manifest_length %d != stored size %d", manifests[0].Length(), len(manifestBytes))
}
if !bytes.Contains(manifestBytes, []byte(`"field-id"`)) {
t.Fatalf("repaired manifest schema has no field-id annotations")
}
entries, err := icebergmanifest.ReadManifest(manifests[0], bytes.NewReader(manifestBytes), false)
if err != nil {
t.Fatalf("iceberg-go cannot read repaired manifest: %v", err)
}
if len(entries) != 1 {
t.Fatalf("manifest has %d entries, want 1", len(entries))
}
dataFile := entries[0].DataFile()
if !strings.HasPrefix(dataFile.FilePath(), "s3://iceberg-tables/") {
t.Fatalf("data file path %s is not absolute", dataFile.FilePath())
}
if dataFile.Count() != 2 {
t.Fatalf("record count %d, want 2", dataFile.Count())
}
if len(dataFile.LowerBoundValues()) == 0 {
t.Fatalf("lower bounds were lost in repair")
}
}
func addSnapshotUpdateJSON(t *testing.T, manifestList string) []json.RawMessage {
t.Helper()
update := fmt.Sprintf(`{"action":"add-snapshot","snapshot":{"snapshot-id":449670589,"sequence-number":1,"timestamp-ms":1754625869000,"manifest-list":%q,"summary":{"operation":"append"},"schema-id":0}}`, manifestList)
return []json.RawMessage{json.RawMessage(update)}
}
func TestRepairClickHouseRelativeManifests(t *testing.T) {
store := newFakeManifestStore()
store.files[relTableLocation+"/metadata/"+relListName] = loadFixture(t, "clickhouse_rel", relListName)
store.files[relTableLocation+"/metadata/"+relManifestName] = loadFixture(t, "clickhouse_rel", relManifestName)
rawUpdates := addSnapshotUpdateJSON(t, "wprobe/target_rel/metadata/"+relListName)
repaired, changed := repairAddSnapshotUpdates(context.Background(), store, relTableLocation, rawUpdates)
if !changed {
t.Fatal("expected repair to change the update")
}
var update struct {
Snapshot struct {
ManifestList string `json:"manifest-list"`
SnapshotID int64 `json:"snapshot-id"`
Summary struct {
Operation string `json:"operation"`
} `json:"summary"`
} `json:"snapshot"`
}
if err := json.Unmarshal(repaired[0], &update); err != nil {
t.Fatalf("unmarshal repaired update: %v", err)
}
wantList := relTableLocation + "/metadata/" + repairedManifestPrefix + relListName
if update.Snapshot.ManifestList != wantList {
t.Fatalf("manifest-list = %s, want %s", update.Snapshot.ManifestList, wantList)
}
// The rest of the snapshot survives the rewrite.
if update.Snapshot.SnapshotID != 449670589 || update.Snapshot.Summary.Operation != "append" {
t.Fatalf("snapshot fields were disturbed: %s", repaired[0])
}
verifyRepairedManifests(t, store, wantList, relTableLocation)
}
func TestRepairClickHouseAbsoluteManifests(t *testing.T) {
store := newFakeManifestStore()
listLocation := absTableLocation + "/metadata/" + absListName
store.files[listLocation] = loadFixture(t, "clickhouse_abs", absListName)
store.files[absTableLocation+"/metadata/"+absManifestName] = loadFixture(t, "clickhouse_abs", absManifestName)
repaired, changed, err := repairManifestList(context.Background(), store, absTableLocation, listLocation)
if err != nil {
t.Fatalf("repairManifestList: %v", err)
}
if !changed {
t.Fatal("expected repair for missing field-ids")
}
verifyRepairedManifests(t, store, repaired, absTableLocation)
}
func TestRepairedManifestsAreLeftAlone(t *testing.T) {
store := newFakeManifestStore()
listLocation := absTableLocation + "/metadata/" + absListName
store.files[listLocation] = loadFixture(t, "clickhouse_abs", absListName)
store.files[absTableLocation+"/metadata/"+absManifestName] = loadFixture(t, "clickhouse_abs", absManifestName)
firstPass, changed, err := repairManifestList(context.Background(), store, absTableLocation, listLocation)
if err != nil || !changed {
t.Fatalf("first pass: changed=%v err=%v", changed, err)
}
store.saved = map[string][]byte{}
secondPass, changed, err := repairManifestList(context.Background(), store, absTableLocation, firstPass)
if err != nil {
t.Fatalf("second pass: %v", err)
}
if changed || secondPass != firstPass {
t.Fatalf("second pass rewrote an already-repaired list: changed=%v location=%s", changed, secondPass)
}
if len(store.saved) != 0 {
t.Fatalf("second pass saved files: %v", store.saved)
}
}
func TestRepairKeepsDeleteManifestContent(t *testing.T) {
// The fixture lacks the OCF "content" key; the repaired copy must take the
// content declared by the manifest-list entry, not default to data.
raw := loadFixture(t, "clickhouse_rel", relManifestName)
repaired, changed, err := repairManifest(raw, relTableLocation, 1)
if err != nil || !changed {
t.Fatalf("repairManifest: changed=%v err=%v", changed, err)
}
reader, err := goavro.NewOCFReader(bytes.NewReader(repaired))
if err != nil {
t.Fatalf("read repaired manifest: %v", err)
}
if got := string(reader.MetaData()["content"]); got != "deletes" {
t.Fatalf(`OCF content = %q, want "deletes"`, got)
}
asData, _, err := repairManifest(raw, relTableLocation, 0)
if err != nil {
t.Fatalf("repairManifest: %v", err)
}
reader, err = goavro.NewOCFReader(bytes.NewReader(asData))
if err != nil {
t.Fatalf("read repaired manifest: %v", err)
}
if got := string(reader.MetaData()["content"]); got != "data" {
t.Fatalf(`OCF content = %q, want "data"`, got)
}
// An existing content value that contradicts the manifest-list entry is
// corrected, not preserved.
relabeled, changed, err := repairManifest(asData, relTableLocation, 1)
if err != nil || !changed {
t.Fatalf("repairManifest on mismatched content: changed=%v err=%v", changed, err)
}
reader, err = goavro.NewOCFReader(bytes.NewReader(relabeled))
if err != nil {
t.Fatalf("read relabeled manifest: %v", err)
}
if got := string(reader.MetaData()["content"]); got != "deletes" {
t.Fatalf(`OCF content after relabel = %q, want "deletes"`, got)
}
}
func TestManifestContentValue(t *testing.T) {
cases := []struct {
record map[string]any
want int
}{
{map[string]any{"content": int32(1)}, 1},
{map[string]any{"content": int64(1)}, 1},
{map[string]any{"content": map[string]any{"int": int32(1)}}, 1},
{map[string]any{"content": nil}, 0},
{map[string]any{}, 0},
}
for _, c := range cases {
if got := manifestContentValue(c.record); got != c.want {
t.Errorf("manifestContentValue(%v) = %d, want %d", c.record, got, c.want)
}
}
}
func TestRepairSkipsLocationsOutsideMetadataDir(t *testing.T) {
store := newFakeManifestStore()
rawUpdates := addSnapshotUpdateJSON(t, "s3://other-bucket/elsewhere/list.avro")
_, changed := repairAddSnapshotUpdates(context.Background(), store, relTableLocation, rawUpdates)
if changed {
t.Fatal("repair touched a manifest list outside the table metadata directory")
}
}
func TestRepairLeavesUnreadableListAlone(t *testing.T) {
store := newFakeManifestStore()
store.files[relTableLocation+"/metadata/junk.avro"] = []byte("not avro")
rawUpdates := addSnapshotUpdateJSON(t, relTableLocation+"/metadata/junk.avro")
repaired, changed := repairAddSnapshotUpdates(context.Background(), store, relTableLocation, rawUpdates)
if changed {
t.Fatal("repair claimed to change an unreadable manifest list")
}
if !bytes.Equal(repaired[0], rawUpdates[0]) {
t.Fatal("update was modified despite repair failure")
}
}

View File

@@ -0,0 +1,132 @@
package iceberg
import (
"bytes"
"encoding/json"
)
// specRequiredEmptyOrder is iterated in this fixed order so the byte-level
// append in ensureMetadataSpecCompliance produces deterministic output.
var specRequiredEmptyOrder = []string{
"current-snapshot-id",
"snapshots",
"snapshot-log",
"metadata-log",
"refs",
}
// specRequiredEmptyDefaults holds the sentinel values Iceberg requires when
// the corresponding state is empty.
var specRequiredEmptyDefaults = map[string]json.RawMessage{
"current-snapshot-id": json.RawMessage("-1"),
"snapshots": json.RawMessage("[]"),
"snapshot-log": json.RawMessage("[]"),
"metadata-log": json.RawMessage("[]"),
"refs": json.RawMessage("{}"),
}
// isJSONNull reports whether raw encodes the JSON null literal, ignoring
// insignificant whitespace allowed by RFC 8259.
func isJSONNull(raw json.RawMessage) bool {
return bytes.Equal(bytes.TrimSpace(raw), []byte("null"))
}
// ensureMetadataSpecCompliance backfills Iceberg spec-required fields that
// iceberg-go v0.5.0 drops via `omitempty` on optional pointer/slice fields.
//
// For tables with no snapshots (e.g. freshly created tables), iceberg-go
// represents an empty state with nil/empty Go values, then strips those keys
// entirely from the JSON output. The Iceberg REST spec, however, requires
// these keys to be present even when empty — Java/Spark/Trino clients
// fail to parse responses missing them (most visibly with
// "Cannot parse missing long current-snapshot-id"). This helper rehydrates
// the missing keys without overwriting any present real values.
//
// Missing keys are spliced in at the byte level just before the closing
// brace so iceberg-go's struct-declared key order is preserved (a naive
// map remarshal would alphabetize every key in the document). The slower
// remarshal path is only used when an explicit JSON null needs replacing,
// which iceberg-go itself never emits.
//
// Returns the original bytes unchanged when parsing fails or no key needs
// to be added, to avoid corrupting an otherwise-valid payload.
func ensureMetadataSpecCompliance(raw []byte) []byte {
if len(raw) == 0 {
return raw
}
var obj map[string]json.RawMessage
if err := json.Unmarshal(raw, &obj); err != nil {
return raw
}
// A top-level JSON null leaves obj == nil with no error; writes would
// panic on the slow path. It is not a valid TableMetadata document, so
// pass it through unchanged.
if obj == nil {
return raw
}
// A field is "missing" for spec purposes if it's absent OR encoded as
// JSON null. Some writers emit `"current-snapshot-id": null` for empty
// state instead of omitting; strict clients reject both the same way.
var toAppend []string
hasExplicitNull := false
for _, key := range specRequiredEmptyOrder {
v, present := obj[key]
switch {
case !present:
toAppend = append(toAppend, key)
case isJSONNull(v):
hasExplicitNull = true
}
}
if len(toAppend) == 0 && !hasExplicitNull {
return raw
}
if !hasExplicitNull {
return appendMissingObjectKeys(raw, toAppend, len(obj) > 0)
}
// Slow path: replace explicit nulls. Rare in practice, so the
// alphabetical key reordering inherent to map remarshal is acceptable.
for key, v := range obj {
if _, required := specRequiredEmptyDefaults[key]; required && isJSONNull(v) {
obj[key] = specRequiredEmptyDefaults[key]
}
}
for _, key := range toAppend {
obj[key] = specRequiredEmptyDefaults[key]
}
fixed, err := json.Marshal(obj)
if err != nil {
return raw
}
return fixed
}
// appendMissingObjectKeys splices the given keys into raw just before the
// closing brace of the top-level JSON object. raw is assumed to be a valid
// JSON object (the caller has already unmarshalled it). hasMembers tells us
// whether the object had any existing members, so we know whether to emit a
// leading comma for the first appended key.
func appendMissingObjectKeys(raw []byte, keys []string, hasMembers bool) []byte {
closeIdx := bytes.LastIndexByte(raw, '}')
if closeIdx < 0 {
return raw
}
var buf bytes.Buffer
buf.Grow(len(raw) + len(keys)*48)
buf.Write(raw[:closeIdx])
for _, key := range keys {
if hasMembers {
buf.WriteByte(',')
}
buf.WriteByte('"')
buf.WriteString(key)
buf.WriteString(`":`)
buf.Write(specRequiredEmptyDefaults[key])
hasMembers = true
}
buf.Write(raw[closeIdx:])
return buf.Bytes()
}

View File

@@ -0,0 +1,162 @@
package iceberg
import (
"encoding/json"
"testing"
)
func TestEnsureMetadataSpecCompliance_BackfillsMissingFields(t *testing.T) {
// Mirrors the real iceberg-go v0.5.0 output for a freshly created table:
// current-snapshot-id, snapshots, snapshot-log, metadata-log, refs all absent.
input := []byte(`{
"format-version": 2,
"table-uuid": "82e3eec4-3aee-414f-a444-94c03c641d20",
"location": "s3://s3table/default/t1",
"last-sequence-number": 0,
"last-updated-ms": 1779866785466,
"last-column-id": 2,
"current-schema-id": 0,
"default-spec-id": 0,
"last-partition-id": 999,
"default-sort-order-id": 0
}`)
out := ensureMetadataSpecCompliance(input)
var got map[string]json.RawMessage
if err := json.Unmarshal(out, &got); err != nil {
t.Fatalf("output is not valid JSON: %v", err)
}
if v, ok := got["current-snapshot-id"]; !ok || string(v) != "-1" {
t.Errorf("current-snapshot-id missing or wrong: present=%v value=%s", ok, string(v))
}
for _, key := range []string{"snapshots", "snapshot-log", "metadata-log"} {
v, ok := got[key]
if !ok || string(v) != "[]" {
t.Errorf("%s missing or wrong: present=%v value=%s", key, ok, string(v))
}
}
if v, ok := got["refs"]; !ok || string(v) != "{}" {
t.Errorf("refs missing or wrong: present=%v value=%s", ok, string(v))
}
}
func TestEnsureMetadataSpecCompliance_PreservesExistingFields(t *testing.T) {
// Real snapshot id and refs must not be overwritten by sentinels.
input := []byte(`{
"format-version": 2,
"current-snapshot-id": 9876543210,
"snapshots": [{"snapshot-id": 9876543210}],
"snapshot-log": [{"snapshot-id": 9876543210, "timestamp-ms": 1}],
"metadata-log": [{"metadata-file": "v1.metadata.json", "timestamp-ms": 1}],
"refs": {"main": {"snapshot-id": 9876543210, "type": "branch"}}
}`)
out := ensureMetadataSpecCompliance(input)
var got map[string]json.RawMessage
if err := json.Unmarshal(out, &got); err != nil {
t.Fatalf("output is not valid JSON: %v", err)
}
if string(got["current-snapshot-id"]) != "9876543210" {
t.Errorf("real snapshot id was overwritten: %s", string(got["current-snapshot-id"]))
}
if string(got["snapshots"]) == "[]" {
t.Errorf("non-empty snapshots was overwritten with empty array")
}
}
func TestEnsureMetadataSpecCompliance_ReplacesExplicitNullsWithSentinels(t *testing.T) {
// Some writers emit explicit JSON null for unset values instead of omitting
// the key. Strict Iceberg clients reject these the same way as missing keys.
input := []byte(`{
"format-version": 2,
"current-snapshot-id": null,
"snapshots": null,
"snapshot-log": null,
"metadata-log": null,
"refs": null
}`)
out := ensureMetadataSpecCompliance(input)
var got map[string]json.RawMessage
if err := json.Unmarshal(out, &got); err != nil {
t.Fatalf("output is not valid JSON: %v", err)
}
if string(got["current-snapshot-id"]) != "-1" {
t.Errorf("current-snapshot-id should be replaced with -1, got %s", string(got["current-snapshot-id"]))
}
for _, key := range []string{"snapshots", "snapshot-log", "metadata-log"} {
if string(got[key]) != "[]" {
t.Errorf("%s should be replaced with [], got %s", key, string(got[key]))
}
}
if string(got["refs"]) != "{}" {
t.Errorf("refs should be replaced with {}, got %s", string(got["refs"]))
}
}
func TestEnsureMetadataSpecCompliance_InvalidJSONReturnedUnchanged(t *testing.T) {
input := []byte(`{not valid json`)
out := ensureMetadataSpecCompliance(input)
if string(out) != string(input) {
t.Errorf("invalid JSON should be returned unchanged; got %s", string(out))
}
}
func TestEnsureMetadataSpecCompliance_EmptyInputReturnedUnchanged(t *testing.T) {
if out := ensureMetadataSpecCompliance(nil); out != nil {
t.Errorf("nil input should be returned unchanged, got %v", out)
}
if out := ensureMetadataSpecCompliance([]byte{}); len(out) != 0 {
t.Errorf("empty input should be returned unchanged, got %v", out)
}
// A top-level JSON null literal must not panic on the slow path.
if out := ensureMetadataSpecCompliance([]byte("null")); string(out) != "null" {
t.Errorf("top-level null should be returned unchanged, got %s", string(out))
}
}
// Original iceberg-go key ordering must survive the backfill: appended
// sentinels go at the end without disturbing prior fields. A map-based
// remarshal would have sorted everything alphabetically.
func TestEnsureMetadataSpecCompliance_PreservesOriginalKeyOrder(t *testing.T) {
// Compact JSON, keys in struct-declared order from iceberg-go.
input := []byte(`{"format-version":2,"table-uuid":"82e3eec4-3aee-414f-a444-94c03c641d20","location":"s3://x/t","last-sequence-number":0,"last-updated-ms":1,"last-column-id":2,"current-schema-id":0,"default-spec-id":0,"last-partition-id":999,"default-sort-order-id":0}`)
out := ensureMetadataSpecCompliance(input)
// Prior keys keep their order, sentinels appended at the end.
want := `{"format-version":2,"table-uuid":"82e3eec4-3aee-414f-a444-94c03c641d20","location":"s3://x/t","last-sequence-number":0,"last-updated-ms":1,"last-column-id":2,"current-schema-id":0,"default-spec-id":0,"last-partition-id":999,"default-sort-order-id":0,"current-snapshot-id":-1,"snapshots":[],"snapshot-log":[],"metadata-log":[],"refs":{}}`
if string(out) != want {
t.Errorf("unexpected output\n got: %s\nwant: %s", string(out), want)
}
// Sanity: still valid JSON.
var parsed map[string]json.RawMessage
if err := json.Unmarshal(out, &parsed); err != nil {
t.Fatalf("output is not valid JSON: %v", err)
}
}
// An empty object {} must round-trip to a valid JSON object containing
// only the spec sentinels (no leading comma).
func TestEnsureMetadataSpecCompliance_EmptyObjectBackfilled(t *testing.T) {
out := ensureMetadataSpecCompliance([]byte(`{}`))
want := `{"current-snapshot-id":-1,"snapshots":[],"snapshot-log":[],"metadata-log":[],"refs":{}}`
if string(out) != want {
t.Errorf("unexpected output\n got: %s\nwant: %s", string(out), want)
}
}
// When all fields are already present, the original bytes must be returned
// untouched (no whitespace normalization, no key reordering).
func TestEnsureMetadataSpecCompliance_AllPresentReturnsSameBytes(t *testing.T) {
input := []byte("{\n \"current-snapshot-id\": 1,\n \"snapshots\": [],\n \"snapshot-log\": [],\n \"metadata-log\": [],\n \"refs\": {}\n}")
out := ensureMetadataSpecCompliance(input)
if string(out) != string(input) {
t.Errorf("expected original bytes returned unchanged\n got: %q\nwant: %q", string(out), string(input))
}
}

View File

@@ -0,0 +1,171 @@
package iceberg
import (
"context"
"fmt"
"os"
"path"
"strings"
"time"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// metadataDirPath returns the filer directory holding a table's metadata files.
func metadataDirPath(bucketName, tablePath string) string {
dir := path.Join(s3tables.TablesPath, bucketName)
if tablePath != "" {
dir = path.Join(dir, tablePath)
}
return path.Join(dir, "metadata")
}
// saveMetadataFile saves the Iceberg metadata JSON file to the filer.
// It constructs the filer path from the S3 location components. With exclusive
// set the write fails with filer_pb.ErrEntryAlreadyExists rather than replacing
// a file that is already there: two commits racing off the same base pick the
// same v{N} name, and letting the second overwrite the first would leave the
// winner's catalog pointer aimed at the loser's metadata.
func (s *Server) saveMetadataFile(ctx context.Context, bucketName, tablePath, metadataFileName string, content []byte, exclusive bool) error {
return s.saveMetadataBlob(ctx, bucketName, tablePath, metadataFileName, content, "application/json", exclusive)
}
// saveMetadataBlob saves a file into the table's metadata directory.
func (s *Server) saveMetadataBlob(ctx context.Context, bucketName, tablePath, metadataFileName string, content []byte, mimeType string, exclusive bool) error {
// Create context with timeout for file operations
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
bucketsPath := s3tables.TablesPath
ensureDir := func(parent, name, errorContext string) error {
_, err := filer_pb.LookupEntry(opCtx, client, &filer_pb.LookupDirectoryEntryRequest{
Directory: parent,
Name: name,
})
if err == nil {
return nil
}
if err != filer_pb.ErrNotFound {
return fmt.Errorf("lookup %s failed: %w", errorContext, err)
}
// If lookup fails with ErrNotFound, try to create the directory.
resp, createErr := client.CreateEntry(opCtx, &filer_pb.CreateEntryRequest{
Directory: parent,
Entry: &filer_pb.Entry{
Name: name,
IsDirectory: true,
Attributes: &filer_pb.FuseAttributes{
Mtime: time.Now().Unix(),
Crtime: time.Now().Unix(),
FileMode: uint32(0755 | os.ModeDir),
},
},
})
if createErr != nil {
return fmt.Errorf("failed to create %s: %w", errorContext, createErr)
}
if resp.ErrorCode != filer_pb.FilerError_OK && resp.ErrorCode != filer_pb.FilerError_ENTRY_ALREADY_EXISTS {
return fmt.Errorf("failed to create %s: %s", errorContext, resp.Error)
}
return nil
}
bucketDir := path.Join(bucketsPath, bucketName)
// 1. Ensure bucket directory exists: <bucketsPath>/<bucket>
if err := ensureDir(bucketsPath, bucketName, "bucket directory"); err != nil {
return err
}
// 2. Ensure table path exists under the bucket directory
tableDir := bucketDir
if tablePath != "" {
segments := strings.Split(tablePath, "/")
for _, segment := range segments {
if segment == "" {
continue
}
if err := ensureDir(tableDir, segment, "table directory"); err != nil {
return err
}
tableDir = path.Join(tableDir, segment)
}
}
// 3. Ensure metadata directory exists: <bucketsPath>/<bucket>/<tablePath>/metadata
metadataDir := path.Join(tableDir, "metadata")
if err := ensureDir(tableDir, "metadata", "metadata directory"); err != nil {
return err
}
// 4. Write the file
resp, err := client.CreateEntry(opCtx, &filer_pb.CreateEntryRequest{
Directory: metadataDir,
OExcl: exclusive,
Entry: &filer_pb.Entry{
Name: metadataFileName,
Attributes: &filer_pb.FuseAttributes{
Mtime: time.Now().Unix(),
Crtime: time.Now().Unix(),
FileMode: uint32(0644),
FileSize: uint64(len(content)),
},
Content: content,
Extended: map[string][]byte{
"Mime-Type": []byte(mimeType),
},
},
})
if err != nil {
return fmt.Errorf("failed to write metadata file: %w", err)
}
if resp.ErrorCode != filer_pb.FilerError_OK {
if sentinel := filer_pb.FilerErrorToSentinel(resp.ErrorCode); sentinel != nil {
return fmt.Errorf("failed to write metadata file: %w", sentinel)
}
return fmt.Errorf("failed to write metadata file: code=%v %s", resp.ErrorCode, resp.Error)
}
return nil
})
}
func (s *Server) deleteMetadataFile(ctx context.Context, bucketName, tablePath, metadataFileName string) error {
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
metadataDir := metadataDirPath(bucketName, tablePath)
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
return filer_pb.DoRemove(opCtx, client, metadataDir, metadataFileName, true, false, true, false, nil)
})
}
func (s *Server) loadMetadataFile(ctx context.Context, bucketName, tablePath, metadataFileName string) ([]byte, error) {
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
metadataDir := metadataDirPath(bucketName, tablePath)
var content []byte
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
resp, err := filer_pb.LookupEntry(opCtx, client, &filer_pb.LookupDirectoryEntryRequest{
Directory: metadataDir,
Name: metadataFileName,
})
if err != nil {
return err
}
if resp == nil || resp.Entry == nil {
return fmt.Errorf("lookup returned nil entry for %s/%s", metadataDir, metadataFileName)
}
content = append([]byte(nil), resp.Entry.Content...)
return nil
})
if err != nil {
return nil, err
}
return content, nil
}

View File

@@ -0,0 +1,128 @@
package iceberg
import (
"context"
"errors"
"path"
"strings"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"google.golang.org/grpc"
)
// stubFilerClient answers only the calls the metadata writers make, tracking
// which paths exist so exclusive creates can be exercised.
type stubFilerClient struct {
filer_pb.SeaweedFilerClient
entries map[string][]byte
}
func newStubFilerClient() *stubFilerClient {
return &stubFilerClient{entries: make(map[string][]byte)}
}
func (c *stubFilerClient) WithFilerClient(_ bool, fn func(client filer_pb.SeaweedFilerClient) error) error {
return fn(c)
}
func (c *stubFilerClient) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest, _ ...grpc.CallOption) (*filer_pb.LookupDirectoryEntryResponse, error) {
key := path.Join(req.Directory, req.Name)
if _, ok := c.entries[key]; !ok {
return nil, filer_pb.ErrNotFound
}
return &filer_pb.LookupDirectoryEntryResponse{Entry: &filer_pb.Entry{Name: req.Name}}, nil
}
func (c *stubFilerClient) CreateEntry(_ context.Context, req *filer_pb.CreateEntryRequest, _ ...grpc.CallOption) (*filer_pb.CreateEntryResponse, error) {
key := path.Join(req.Directory, req.Entry.Name)
if _, exists := c.entries[key]; exists && req.OExcl {
return &filer_pb.CreateEntryResponse{
Error: "entry already exists",
ErrorCode: filer_pb.FilerError_ENTRY_ALREADY_EXISTS,
}, nil
}
c.entries[key] = req.Entry.Content
return &filer_pb.CreateEntryResponse{}, nil
}
// Two commits racing off the same base metadata pick the same v{N} file name.
// The loser must be told, not allowed to replace the winner's metadata.
func TestSaveMetadataFileExclusiveRefusesToOverwrite(t *testing.T) {
client := newStubFilerClient()
s := &Server{filerClient: client}
ctx := context.Background()
if err := s.saveMetadataFile(ctx, "bkt", "ns/tbl", "v2.metadata.json", []byte(`{"winner":true}`), true); err != nil {
t.Fatalf("first write failed: %v", err)
}
err := s.saveMetadataFile(ctx, "bkt", "ns/tbl", "v2.metadata.json", []byte(`{"loser":true}`), true)
if !errors.Is(err, filer_pb.ErrEntryAlreadyExists) {
t.Fatalf("second write err = %v, want ErrEntryAlreadyExists", err)
}
stored := string(client.entries[path.Join(metadataDirPath("bkt", "ns/tbl"), "v2.metadata.json")])
if stored != `{"winner":true}` {
t.Errorf("stored metadata = %s, want the first writer's content", stored)
}
}
// A metadata file left behind by an interrupted commit must not wedge the
// table: the next commit stages under a unique name rather than failing or
// overwriting, and the catalog pointer still decides the winner.
func TestStageCommitMetadataFallsBackToAUniqueName(t *testing.T) {
client := newStubFilerClient()
s := &Server{filerClient: client}
ctx := context.Background()
name, location, err := s.stageCommitMetadata(ctx, "bkt", "ns/tbl", "s3://bkt/ns/tbl", "v2.metadata.json", []byte(`{"orphan":true}`))
if err != nil {
t.Fatalf("first stage failed: %v", err)
}
if name != "v2.metadata.json" {
t.Errorf("first stage used %q, want the plain versioned name", name)
}
if location != "s3://bkt/ns/tbl/metadata/v2.metadata.json" {
t.Errorf("location = %q", location)
}
name, location, err = s.stageCommitMetadata(ctx, "bkt", "ns/tbl", "s3://bkt/ns/tbl", "v2.metadata.json", []byte(`{"second":true}`))
if err != nil {
t.Fatalf("second stage failed: %v", err)
}
if !strings.HasPrefix(name, "v2-") || !strings.HasSuffix(name, ".metadata.json") {
t.Errorf("second stage used %q, want a unique v2-* name", name)
}
if location != "s3://bkt/ns/tbl/metadata/"+name {
t.Errorf("location = %q, want it to match the staged name", location)
}
stored := string(client.entries[path.Join(metadataDirPath("bkt", "ns/tbl"), "v2.metadata.json")])
if stored != `{"orphan":true}` {
t.Errorf("the first file was overwritten: %s", stored)
}
if fallback := string(client.entries[path.Join(metadataDirPath("bkt", "ns/tbl"), name)]); fallback != `{"second":true}` {
t.Errorf("fallback file holds %s, want the second writer's metadata", fallback)
}
}
// Paths that legitimately rewrite a file, such as manifest repair, keep the
// overwriting behaviour.
func TestSaveMetadataFileOverwrites(t *testing.T) {
client := newStubFilerClient()
s := &Server{filerClient: client}
ctx := context.Background()
if err := s.saveMetadataFile(ctx, "bkt", "ns/tbl", "v2.metadata.json", []byte(`{"first":true}`), false); err != nil {
t.Fatalf("first write failed: %v", err)
}
if err := s.saveMetadataFile(ctx, "bkt", "ns/tbl", "v2.metadata.json", []byte(`{"second":true}`), false); err != nil {
t.Fatalf("second write failed: %v", err)
}
stored := string(client.entries[path.Join(metadataDirPath("bkt", "ns/tbl"), "v2.metadata.json")])
if stored != `{"second":true}` {
t.Errorf("stored metadata = %s, want the second write", stored)
}
}

View File

@@ -0,0 +1,142 @@
package iceberg
import (
"encoding/json"
"slices"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/table"
)
// nameMappingFromSchema derives the spec's field-id-to-names mapping from a
// table schema, mirroring Java's MappingUtil.create. List elements and map
// keys/values use the canonical "element"/"key"/"value" names.
func nameMappingFromSchema(schema *iceberg.Schema) iceberg.NameMapping {
return mappedFieldsOf(schema.Fields())
}
func mappedFieldsOf(fields []iceberg.NestedField) []iceberg.MappedField {
mapped := make([]iceberg.MappedField, 0, len(fields))
for _, field := range fields {
id := field.ID
mapped = append(mapped, iceberg.MappedField{
Names: []string{field.Name},
FieldID: &id,
Fields: mappedFieldsOfType(field.Type),
})
}
return mapped
}
func mappedFieldsOfType(typ iceberg.Type) []iceberg.MappedField {
switch t := typ.(type) {
case *iceberg.StructType:
return mappedFieldsOf(t.FieldList)
case *iceberg.ListType:
return mappedFieldsOf([]iceberg.NestedField{t.ElementField()})
case *iceberg.MapType:
return mappedFieldsOf([]iceberg.NestedField{t.KeyField(), t.ValueField()})
default:
return nil
}
}
// refreshDefaultNameMapping keeps schema.name-mapping.default in sync with
// the current schema across commits, so schema evolution does not strand
// readers of field-id-less data files on the creation-time mapping. The
// stored mapping is merged rather than replaced: per field-id, names already
// present are kept alongside the current schema name, so files written under
// a renamed column's old name (and user-added aliases) stay resolvable.
// Returns the (possibly patched) serialized metadata.
func refreshDefaultNameMapping(raw []byte, updated table.Metadata) []byte {
if updated == nil {
return raw
}
updatedSchema := updated.CurrentSchema()
if updatedSchema == nil || len(updatedSchema.Fields()) == 0 {
return raw
}
mapping := nameMappingFromSchema(updatedSchema)
existing := updated.Properties()[table.DefaultNameMappingKey]
if existing != "" {
var existingMapping iceberg.NameMapping
if err := json.Unmarshal([]byte(existing), &existingMapping); err != nil {
// Not a mapping we can merge; leave whatever the user stored.
return raw
}
mapping = mergeMappedFields(existingMapping, mapping)
}
wantJSON, err := json.Marshal(mapping)
if err != nil {
return raw
}
want := string(wantJSON)
if existing == want {
return raw
}
var metadata map[string]json.RawMessage
if err := json.Unmarshal(raw, &metadata); err != nil {
return raw
}
properties := map[string]json.RawMessage{}
if rawProperties, ok := metadata["properties"]; ok {
if err := json.Unmarshal(rawProperties, &properties); err != nil {
return raw
}
}
valueJSON, err := json.Marshal(want)
if err != nil {
return raw
}
properties[table.DefaultNameMappingKey] = valueJSON
propertiesJSON, err := json.Marshal(properties)
if err != nil {
return raw
}
metadata["properties"] = propertiesJSON
patched, err := json.Marshal(metadata)
if err != nil {
return raw
}
return patched
}
// mergeMappedFields folds the names of an existing mapping into the mapping
// derived from the current schema. Fields are matched by field-id per nesting
// level; entries whose ids left the schema are dropped, since current readers
// cannot project those fields anyway. A name can belong to only one field per
// level (Java readers reject ambiguous mappings), so a historical name that
// the current schema assigns to a different field-id stays with its new owner
// instead of being duplicated onto the old one — which also keeps mappings
// clean when a commit replaces the schema with reassigned field ids.
func mergeMappedFields(existing, derived []iceberg.MappedField) []iceberg.MappedField {
byID := make(map[int]*iceberg.MappedField, len(existing))
for i := range existing {
byID[existing[i].ID()] = &existing[i]
}
nameOwner := make(map[string]int)
for _, d := range derived {
for _, name := range d.Names {
nameOwner[name] = d.ID()
}
}
for i := range derived {
prior, ok := byID[derived[i].ID()]
if !ok {
continue
}
for _, name := range prior.Names {
if ownerID, taken := nameOwner[name]; taken && ownerID != derived[i].ID() {
continue
}
if !slices.Contains(derived[i].Names, name) {
derived[i].Names = append(derived[i].Names, name)
nameOwner[name] = derived[i].ID()
}
}
derived[i].Fields = mergeMappedFields(prior.Fields, derived[i].Fields)
}
return derived
}

View File

@@ -0,0 +1,213 @@
package iceberg
import (
"encoding/json"
"slices"
"testing"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/table"
"github.com/google/uuid"
)
func TestNameMappingFromSchema(t *testing.T) {
schema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
iceberg.NestedField{ID: 2, Name: "label", Type: iceberg.PrimitiveTypes.String},
iceberg.NestedField{ID: 3, Name: "tags", Type: &iceberg.ListType{
ElementID: 4, Element: iceberg.PrimitiveTypes.String,
}},
iceberg.NestedField{ID: 5, Name: "attrs", Type: &iceberg.MapType{
KeyID: 6, KeyType: iceberg.PrimitiveTypes.String,
ValueID: 7, ValueType: &iceberg.StructType{FieldList: []iceberg.NestedField{
{ID: 8, Name: "inner", Type: iceberg.PrimitiveTypes.Int32},
}},
}},
)
mapping := nameMappingFromSchema(schema)
got, err := json.Marshal(mapping)
if err != nil {
t.Fatalf("marshal mapping: %v", err)
}
want := `[{"names":["id"],"field-id":1},{"names":["label"],"field-id":2},` +
`{"names":["tags"],"field-id":3,"fields":[{"names":["element"],"field-id":4}]},` +
`{"names":["attrs"],"field-id":5,"fields":[{"names":["key"],"field-id":6},` +
`{"names":["value"],"field-id":7,"fields":[{"names":["inner"],"field-id":8}]}]}]`
if string(got) != want {
t.Fatalf("mapping = %s\nwant %s", got, want)
}
}
func TestRefreshDefaultNameMappingDropsReassignedNames(t *testing.T) {
// The stored mapping was derived from a schema whose ids were later
// reassigned (id<->name swapped). The merged mapping must follow the
// current schema exactly: duplicating names across ids makes Java readers
// reject the whole mapping.
schema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
iceberg.NestedField{ID: 2, Name: "name", Type: iceberg.PrimitiveTypes.String},
)
metadata, err := newTableMetadata(uuid.New(), "s3://bucket/ns/tbl", schema, nil, nil,
iceberg.Properties{table.DefaultNameMappingKey: `[{"names":["name"],"field-id":1},{"names":["id"],"field-id":2}]`})
if err != nil {
t.Fatalf("newTableMetadata: %v", err)
}
raw, err := json.Marshal(metadata)
if err != nil {
t.Fatalf("marshal metadata: %v", err)
}
refreshed := refreshDefaultNameMapping(raw, metadata)
final, err := table.ParseMetadataBytes(refreshed)
if err != nil {
t.Fatalf("parse refreshed metadata: %v", err)
}
var mapping iceberg.NameMapping
if err := json.Unmarshal([]byte(final.Properties()[table.DefaultNameMappingKey]), &mapping); err != nil {
t.Fatalf("parse refreshed mapping: %v", err)
}
seen := map[string]int{}
for _, field := range mapping {
for _, name := range field.Names {
if prior, dup := seen[name]; dup {
t.Fatalf("name %q mapped to both field %d and field %d: %s",
name, prior, field.ID(), final.Properties()[table.DefaultNameMappingKey])
}
seen[name] = field.ID()
}
}
if seen["id"] != 1 || seen["name"] != 2 {
t.Fatalf("current schema names must win: %s", final.Properties()[table.DefaultNameMappingKey])
}
}
// evolveMetadataSchema returns serialized metadata whose schema renamed
// "label" to "tag" and gained an "extra" column while the properties still
// carry the creation-time mapping, simulating a schema-evolution commit.
func evolveMetadataSchema(t *testing.T, base table.Metadata) []byte {
t.Helper()
raw, err := json.Marshal(base)
if err != nil {
t.Fatalf("marshal base metadata: %v", err)
}
evolved := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
iceberg.NestedField{ID: 2, Name: "tag", Type: iceberg.PrimitiveTypes.String},
iceberg.NestedField{ID: 3, Name: "extra", Type: iceberg.PrimitiveTypes.String},
)
schemaJSON, err := json.Marshal(evolved)
if err != nil {
t.Fatalf("marshal evolved schema: %v", err)
}
var metadata map[string]json.RawMessage
if err := json.Unmarshal(raw, &metadata); err != nil {
t.Fatalf("unmarshal metadata: %v", err)
}
metadata["schemas"] = json.RawMessage("[" + string(schemaJSON) + "]")
edited, err := json.Marshal(metadata)
if err != nil {
t.Fatalf("marshal edited metadata: %v", err)
}
return edited
}
func TestRefreshDefaultNameMappingOnSchemaEvolution(t *testing.T) {
schema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
iceberg.NestedField{ID: 2, Name: "label", Type: iceberg.PrimitiveTypes.String},
)
base, err := newTableMetadata(uuid.New(), "s3://bucket/ns/tbl", schema, nil, nil, nil)
if err != nil {
t.Fatalf("newTableMetadata: %v", err)
}
edited := evolveMetadataSchema(t, base)
updated, err := table.ParseMetadataBytes(edited)
if err != nil {
t.Fatalf("parse evolved metadata: %v", err)
}
refreshed := refreshDefaultNameMapping(edited, updated)
final, err := table.ParseMetadataBytes(refreshed)
if err != nil {
t.Fatalf("parse refreshed metadata: %v", err)
}
var mapping iceberg.NameMapping
if err := json.Unmarshal([]byte(final.Properties()[table.DefaultNameMappingKey]), &mapping); err != nil {
t.Fatalf("parse refreshed mapping: %v", err)
}
if len(mapping) != 3 || mapping[2].ID() != 3 || mapping[2].Names[0] != "extra" {
t.Fatalf("mapping was not refreshed for the evolved schema: %s", final.Properties()[table.DefaultNameMappingKey])
}
// The renamed column keeps its historical name so files written under the
// old physical name stay resolvable.
if mapping[1].ID() != 2 || !slices.Contains(mapping[1].Names, "tag") || !slices.Contains(mapping[1].Names, "label") {
t.Fatalf("renamed column lost a name: %s", final.Properties()[table.DefaultNameMappingKey])
}
}
func TestRefreshDefaultNameMappingKeepsUserNames(t *testing.T) {
schema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
iceberg.NestedField{ID: 2, Name: "label", Type: iceberg.PrimitiveTypes.String},
)
base, err := newTableMetadata(uuid.New(), "s3://bucket/ns/tbl", schema, nil, nil,
iceberg.Properties{table.DefaultNameMappingKey: `[{"names":["custom"],"field-id":1}]`})
if err != nil {
t.Fatalf("newTableMetadata: %v", err)
}
edited := evolveMetadataSchema(t, base)
updated, err := table.ParseMetadataBytes(edited)
if err != nil {
t.Fatalf("parse evolved metadata: %v", err)
}
refreshed := refreshDefaultNameMapping(edited, updated)
final, err := table.ParseMetadataBytes(refreshed)
if err != nil {
t.Fatalf("parse refreshed metadata: %v", err)
}
var mapping iceberg.NameMapping
if err := json.Unmarshal([]byte(final.Properties()[table.DefaultNameMappingKey]), &mapping); err != nil {
t.Fatalf("parse refreshed mapping: %v", err)
}
// The user's alias survives the merge alongside the schema name.
if mapping[0].ID() != 1 || !slices.Contains(mapping[0].Names, "custom") || !slices.Contains(mapping[0].Names, "id") {
t.Fatalf("user alias was dropped: %s", final.Properties()[table.DefaultNameMappingKey])
}
}
func TestCreateTableSetsDefaultNameMapping(t *testing.T) {
// Request ids are deliberately non-sequential: the metadata constructor
// reassigns fresh ids, and the stamped mapping must follow the final
// schema, not the request.
schema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 7, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
iceberg.NestedField{ID: 3, Name: "label", Type: iceberg.PrimitiveTypes.String},
)
metadata, err := newTableMetadata(uuid.New(), "s3://bucket/ns/tbl", schema, nil, nil, nil)
if err != nil {
t.Fatalf("newTableMetadata: %v", err)
}
mappingJSON, ok := metadata.Properties()[table.DefaultNameMappingKey]
if !ok {
t.Fatalf("newTableMetadata did not set %s: %v", table.DefaultNameMappingKey, metadata.Properties())
}
var mapping iceberg.NameMapping
if err := json.Unmarshal([]byte(mappingJSON), &mapping); err != nil {
t.Fatalf("parse stamped mapping: %v", err)
}
finalSchema := metadata.CurrentSchema()
if len(mapping) != len(finalSchema.Fields()) {
t.Fatalf("mapping has %d entries, schema has %d", len(mapping), len(finalSchema.Fields()))
}
for i, field := range finalSchema.Fields() {
if mapping[i].ID() != field.ID || mapping[i].Names[0] != field.Name {
t.Fatalf("mapping entry %d = %v, want id %d name %s", i, mapping[i], field.ID, field.Name)
}
}
}

View File

@@ -0,0 +1,104 @@
package iceberg
import (
"fmt"
"net/http"
"strings"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
)
// validateRequestPath rejects Iceberg REST requests whose captured
// {prefix}/{namespace}/{table} mux vars would produce a parent-directory
// traversal when joined into a filer path. The iceberg router runs with
// SkipClean(true), so `..` survives routing; downstream path.Join calls
// (stageCreateMarkerDir, location builders, etc.) then collapse it and
// escape the table-bucket directory.
//
// {prefix} maps to a table-bucket name; {table} is a single path segment;
// {namespace} is unit-separator (0x1F) joined parts that get flattened into
// a single dotted name for the on-disk layout — each part is validated
// individually.
func validateRequestPath(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
// Use the comma-ok form so vars only checked when the matched route
// actually captures them; when captured, an empty value is itself a
// rejection because downstream path.Join would collapse it.
if prefix, ok := vars["prefix"]; ok {
if prefix == "" || !s3_constants.IsValidBucketName(prefix) {
writeError(w, http.StatusBadRequest, "BadRequest", "invalid prefix")
return
}
}
if table, ok := vars["table"]; ok {
if table == "" || !isValidNameSegment(table) {
writeError(w, http.StatusBadRequest, "BadRequest", "invalid table name")
return
}
}
if ns, ok := vars["namespace"]; ok {
if ns == "" {
writeError(w, http.StatusBadRequest, "BadRequest", "invalid namespace")
return
}
// Reject leading/trailing/consecutive unit separators so distinct
// inputs cannot collapse to the same parsed namespace via
// parseNamespace's empty-part filter.
for _, part := range strings.Split(ns, "\x1F") {
if part == "" || !isValidNameSegment(part) {
writeError(w, http.StatusBadRequest, "BadRequest", "invalid namespace")
return
}
}
}
next.ServeHTTP(w, r)
})
}
// isValidTablePath reports whether a "/"-separated table path (the part below
// the bucket) is free of segments that path.Join would collapse to escape the
// bucket directory. Empty segments (from leading/duplicate slashes) are
// ignored, but the path must resolve to at least one real segment so it does
// not collapse to the bucket-level metadata directory.
func isValidTablePath(tablePath string) bool {
hasSegment := false
for _, segment := range strings.Split(tablePath, "/") {
if segment == "" {
continue
}
if !isValidNameSegment(segment) {
return false
}
hasSegment = true
}
return hasSegment
}
// isValidNameSegment rejects a single path-segment value (bucket prefix slot,
// table name, or one namespace part) that would be unsafe to embed in a filer
// path: `.`, `..`, embedded slash/backslash, or NUL.
func isValidNameSegment(s string) bool {
if s == "" {
return true
}
if s == "." || s == ".." {
return false
}
return !strings.ContainsAny(s, "/\\\x00")
}
// confineMetadataLocation checks that a parsed s3 location stays within the
// authorized table bucket and rejects traversal segments that path.Join in
// saveMetadataBlob would collapse to escape it. The table path must resolve
// to at least one real segment so its metadata directory is table-specific.
func confineMetadataLocation(metadataBucket, metadataPath, bucketName string) error {
if metadataBucket != bucketName {
return fmt.Errorf("table location must be within bucket %s", bucketName)
}
if !isValidTablePath(metadataPath) {
return fmt.Errorf("invalid table location path")
}
return nil
}

View File

@@ -0,0 +1,139 @@
package iceberg
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/gorilla/mux"
)
func TestIsValidTablePath(t *testing.T) {
tests := []struct {
tablePath string
want bool
}{
{"sales.orders", true},
{"ns/table", true},
{"ns//table", true},
{"", false},
{"/", false},
{"//", false},
{"../other", false},
{"ns/../../etc", false},
{"ns/./x", false},
{`ns\..\x`, false},
}
for _, tt := range tests {
if got := isValidTablePath(tt.tablePath); got != tt.want {
t.Errorf("isValidTablePath(%q) = %v, want %v", tt.tablePath, got, tt.want)
}
}
}
func TestValidateRequestPath_RejectsTraversal(t *testing.T) {
tests := []struct {
name string
rawPath string
wantCode int
}{
{"clean namespace+table passes", "/v1/namespaces/sales/tables/orders", http.StatusOK},
{"clean prefixed passes", "/v1/wh/namespaces/sales/tables/orders", http.StatusOK},
{"clean namespace only passes", "/v1/namespaces/sales", http.StatusOK},
// SkipClean(true) means raw `..` survives routing — these are the
// realistic traversal shapes the middleware must catch.
{"dotdot as prefix var rejected", "/v1/../namespaces/sales", http.StatusBadRequest},
{"dotdot as namespace var rejected", "/v1/namespaces/..", http.StatusBadRequest},
{"dotdot as namespace var prefixed rejected", "/v1/wh/namespaces/..", http.StatusBadRequest},
{"dotdot as table var rejected", "/v1/namespaces/sales/tables/..", http.StatusBadRequest},
{"dot as table var rejected", "/v1/namespaces/sales/tables/.", http.StatusBadRequest},
// Iceberg clients send the 0x1F unit separator percent-encoded; mux
// decodes it before the middleware sees the namespace var.
{"unit-sep namespace with dotdot part rejected", "/v1/namespaces/sales%1F..%1Fevil", http.StatusBadRequest},
{"leading unit-sep namespace rejected", "/v1/namespaces/%1Fsales", http.StatusBadRequest},
{"trailing unit-sep namespace rejected", "/v1/namespaces/sales%1F", http.StatusBadRequest},
{"consecutive unit-sep namespace rejected", "/v1/namespaces/sales%1F%1Fevil", http.StatusBadRequest},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
router := mux.NewRouter().SkipClean(true)
router.Use(validateRequestPath)
handlerCalled := false
pass := func(w http.ResponseWriter, r *http.Request) {
handlerCalled = true
w.WriteHeader(http.StatusOK)
}
router.HandleFunc("/v1/namespaces/{namespace}", pass)
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", pass)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", pass)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", pass)
req := httptest.NewRequest(http.MethodGet, tt.rawPath, nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != tt.wantCode {
t.Fatalf("path %q: got status %d, want %d (body=%q)", tt.rawPath, rr.Code, tt.wantCode, rr.Body.String())
}
if tt.wantCode == http.StatusBadRequest && handlerCalled {
t.Fatalf("path %q: inner handler reached despite rejection", tt.rawPath)
}
})
}
}
// Defense-in-depth: if a future route or middleware ever leaves one of the
// captured vars empty, the middleware must still reject the request. The
// default mux regex won't normally allow this.
func TestValidateRequestPath_RejectsEmptyCapturedVars(t *testing.T) {
tests := []struct {
name string
vars map[string]string
}{
{"empty prefix", map[string]string{"prefix": "", "namespace": "ns"}},
{"empty table", map[string]string{"namespace": "ns", "table": ""}},
{"empty namespace", map[string]string{"namespace": ""}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
handlerCalled := false
h := validateRequestPath(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
handlerCalled = true
}))
req := mux.SetURLVars(httptest.NewRequest(http.MethodGet, "/", nil), tt.vars)
rr := httptest.NewRecorder()
h.ServeHTTP(rr, req)
if handlerCalled {
t.Fatalf("vars %v: inner handler reached despite empty capture", tt.vars)
}
})
}
}
func TestIsValidNameSegment(t *testing.T) {
tests := []struct {
name string
input string
want bool
}{
{"empty ok", "", true},
{"plain", "orders", true},
{"with dot inside", "my.table", true},
{"hidden", ".hidden", true},
{"bare dot", ".", false},
{"bare dotdot", "..", false},
{"contains slash", "foo/bar", false},
{"contains backslash", "foo\\bar", false},
{"contains nul", "foo\x00bar", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := isValidNameSegment(tt.input); got != tt.want {
t.Errorf("isValidNameSegment(%q) = %v, want %v", tt.input, got, tt.want)
}
})
}
}

View File

@@ -0,0 +1,55 @@
package iceberg
import (
"encoding/json"
)
// normalizeRequirements repairs requirement JSON that iceberg-go's parser
// rejects but the Iceberg REST spec permits.
//
// iceberg-go's assert-ref-snapshot-id parser uses a nullableInt64 that tracks
// whether the "snapshot-id" field was present in the JSON. An absent field
// (set == false) is rejected as "missing required field", even though the spec
// makes snapshot-id optional — null means the ref must not already exist.
// iceberg-go v0.6.0 used a plain *int64, so absent was nil and accepted.
//
// Clients like ClickHouse send assert-ref-snapshot-id without snapshot-id when
// asserting a branch does not yet exist. Splicing in an explicit null before
// unmarshaling restores the v0.6.0 behavior across both iceberg-go versions.
func normalizeRequirements(raw json.RawMessage) (json.RawMessage, error) {
if len(raw) == 0 {
return raw, nil
}
var reqs []json.RawMessage
if err := json.Unmarshal(raw, &reqs); err != nil {
return raw, err
}
changed := false
for i, req := range reqs {
var obj map[string]json.RawMessage
if err := json.Unmarshal(req, &obj); err != nil {
continue // let the downstream parser report the error
}
var typ string
if rawType, ok := obj["type"]; ok {
_ = json.Unmarshal(rawType, &typ)
}
if typ != "assert-ref-snapshot-id" {
continue
}
if _, hasSnapshotID := obj["snapshot-id"]; hasSnapshotID {
continue
}
obj["snapshot-id"] = json.RawMessage("null")
fixed, err := json.Marshal(obj)
if err != nil {
continue
}
reqs[i] = fixed
changed = true
}
if !changed {
return raw, nil
}
return json.Marshal(reqs)
}

View File

@@ -0,0 +1,94 @@
package iceberg
import (
"encoding/json"
"testing"
"github.com/apache/iceberg-go/table"
)
func TestNormalizeRequirementsAddsNullSnapshotID(t *testing.T) {
// ClickHouse sends assert-ref-snapshot-id without snapshot-id when
// asserting a branch does not yet exist. iceberg-go's parser rejects
// an absent field; normalizeRequirements splices in an explicit null.
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main"}]`)
// Without normalization, iceberg-go rejects the requirement.
var reqs table.Requirements
if err := json.Unmarshal(raw, &reqs); err == nil {
t.Fatal("iceberg-go accepted an assert-ref-snapshot-id without snapshot-id; expected an error")
}
normalized, err := normalizeRequirements(raw)
if err != nil {
t.Fatalf("normalizeRequirements: %v", err)
}
if err := json.Unmarshal(normalized, &reqs); err != nil {
t.Fatalf("unmarshal normalized requirements: %v", err)
}
if len(reqs) != 1 {
t.Fatalf("got %d requirements, want 1", len(reqs))
}
if reqs[0].GetType() != "assert-ref-snapshot-id" {
t.Fatalf("requirement type = %s, want assert-ref-snapshot-id", reqs[0].GetType())
}
}
func TestNormalizeRequirementsPreservesPresentSnapshotID(t *testing.T) {
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main","snapshot-id":42}]`)
normalized, err := normalizeRequirements(raw)
if err != nil {
t.Fatalf("normalizeRequirements: %v", err)
}
var reqs table.Requirements
if err := json.Unmarshal(normalized, &reqs); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(reqs) != 1 {
t.Fatalf("got %d requirements, want 1", len(reqs))
}
}
func TestNormalizeRequirementsPreservesExplicitNullSnapshotID(t *testing.T) {
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main","snapshot-id":null}]`)
normalized, err := normalizeRequirements(raw)
if err != nil {
t.Fatalf("normalizeRequirements: %v", err)
}
var reqs table.Requirements
if err := json.Unmarshal(normalized, &reqs); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(reqs) != 1 {
t.Fatalf("got %d requirements, want 1", len(reqs))
}
}
func TestNormalizeRequirementsLeavesOtherTypesUnchanged(t *testing.T) {
raw := json.RawMessage(`[{"type":"assert-create"},{"type":"assert-table-uuid","uuid":"00000000-0000-0000-0000-000000000000"}]`)
normalized, err := normalizeRequirements(raw)
if err != nil {
t.Fatalf("normalizeRequirements: %v", err)
}
var reqs table.Requirements
if err := json.Unmarshal(normalized, &reqs); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(reqs) != 2 {
t.Fatalf("got %d requirements, want 2", len(reqs))
}
}
func TestNormalizeRequirementsEmptyInput(t *testing.T) {
normalized, err := normalizeRequirements(nil)
if err != nil {
t.Fatalf("normalizeRequirements(nil): %v", err)
}
if normalized != nil {
t.Fatalf("expected nil output for nil input, got %s", normalized)
}
}

View File

@@ -0,0 +1,287 @@
package iceberg
import (
"context"
"net/http"
"strings"
"time"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// FilerClient provides access to the filer for storage operations.
type FilerClient interface {
WithFilerClient(streamingMode bool, fn func(client filer_pb.SeaweedFilerClient) error) error
}
type S3Authenticator interface {
AuthenticateRequest(r *http.Request) (string, interface{}, s3err.ErrorCode)
DefaultAllow() bool
}
// CredentialValidator validates S3 access key / secret key pairs
// and provides credential lookup for OAuth token verification.
type CredentialValidator interface {
// ValidateS3Credential checks if the access key and secret key are valid.
// Returns the identity name and identity object on success.
ValidateS3Credential(accessKey, secretKey string) (identityName string, identity interface{}, err error)
// GetCredentialByAccessKey looks up a credential by access key.
// Returns the identity name, identity object, and secret key.
// Used for verifying Bearer tokens signed with a specific credential.
GetCredentialByAccessKey(accessKey string) (identityName string, identity interface{}, secretKey string, err error)
}
// VendedCredentials are short-lived S3 credentials scoped to one table.
type VendedCredentials struct {
AccessKeyID string
SecretAccessKey string
SessionToken string
Expiration time.Time
}
// CredentialVendor mints credentials limited to a single table's prefix for a
// caller the catalog has already authenticated and authorized. A nil result
// with no error means the deployment has vending switched off.
type CredentialVendor interface {
VendTableCredentials(ctx context.Context, principal, bucket, prefix string) (*VendedCredentials, error)
}
// Server implements the Iceberg REST Catalog API.
type Server struct {
filerClient FilerClient
tablesManager *s3tables.Manager
prefix string // optional prefix for routes
authenticator S3Authenticator
credentialValidator CredentialValidator
credentialVendor CredentialVendor
s3Endpoint string // http(s):// URL advertised in LoadTable FileIO config
}
// NewServer creates a new Iceberg REST Catalog server.
func NewServer(filerClient FilerClient, authenticator S3Authenticator) *Server {
manager := s3tables.NewManager()
// Mirror the S3 port: fall open by default only when the gateway itself is
// open. With auth configured, an authenticated catalog caller must pass the
// normal permission check instead of being allowed because no policy denied
// it — even if the full identity struct ever fails to reach the handler.
if authenticator != nil {
manager.SetDefaultAllow(authenticator.DefaultAllow())
}
return &Server{
filerClient: filerClient,
tablesManager: manager,
prefix: "",
authenticator: authenticator,
}
}
// SetCredentialVendor enables credential vending for clients that ask for it
// with X-Iceberg-Access-Delegation: vended-credentials.
func (s *Server) SetCredentialVendor(vendor CredentialVendor) {
s.credentialVendor = vendor
}
// SetCredentialValidator sets the credential validator for OAuth token support.
func (s *Server) SetCredentialValidator(cv CredentialValidator) {
s.credentialValidator = cv
}
// SetS3Endpoint configures the S3 endpoint URL to vend to clients as part of
// the LoadTable FileIO config, so they can read table data files directly
// without separately discovering the S3 API address. See issue #9103.
func (s *Server) SetS3Endpoint(endpoint string) {
s.s3Endpoint = endpoint
}
// RegisterRoutes registers Iceberg REST API routes on the provided router.
func (s *Server) RegisterRoutes(router *mux.Router) {
// Add middleware to log all requests/responses
router.Use(loggingMiddleware)
// Reject `..`/`.`/NUL in {prefix}/{namespace}/{table} vars before any
// handler runs. The router uses SkipClean(true), so traversal segments
// would otherwise reach path.Join in stage-marker / location builders.
// Registered after loggingMiddleware so rejected requests still get
// audit-logged.
router.Use(validateRequestPath)
// Configuration endpoint - no auth needed for config
router.HandleFunc("/v1/config", s.handleConfig).Methods(http.MethodGet)
// OAuth2 token endpoint - no auth needed (this IS the auth endpoint)
router.HandleFunc("/v1/oauth/tokens", s.handleOAuthTokens).Methods(http.MethodPost)
// Namespace endpoints - wrapped with Auth middleware
router.HandleFunc("/v1/namespaces", s.Auth(s.handleListNamespaces)).Methods(http.MethodGet)
router.HandleFunc("/v1/namespaces", s.Auth(s.handleCreateNamespace)).Methods(http.MethodPost)
router.HandleFunc("/v1/namespaces/{namespace}", s.Auth(s.handleGetNamespace)).Methods(http.MethodGet)
router.HandleFunc("/v1/namespaces/{namespace}", s.Auth(s.handleNamespaceExists)).Methods(http.MethodHead)
router.HandleFunc("/v1/namespaces/{namespace}", s.Auth(s.handleDropNamespace)).Methods(http.MethodDelete)
router.HandleFunc("/v1/namespaces/{namespace}/properties", s.Auth(s.handleUpdateNamespaceProperties)).Methods(http.MethodPost)
// Table endpoints - wrapped with Auth middleware
router.HandleFunc("/v1/namespaces/{namespace}/tables", s.Auth(s.handleListTables)).Methods(http.MethodGet)
router.HandleFunc("/v1/namespaces/{namespace}/tables", s.Auth(s.handleCreateTable)).Methods(http.MethodPost)
router.HandleFunc("/v1/namespaces/{namespace}/register", s.Auth(s.handleRegisterTable)).Methods(http.MethodPost)
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.Auth(s.handleLoadTable)).Methods(http.MethodGet)
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.Auth(s.handleTableExists)).Methods(http.MethodHead)
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.Auth(s.handleDropTable)).Methods(http.MethodDelete)
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.Auth(s.handleUpdateTable)).Methods(http.MethodPost)
router.HandleFunc("/v1/tables/rename", s.Auth(s.handleRenameTable)).Methods(http.MethodPost)
// View endpoints - wrapped with Auth middleware
router.HandleFunc("/v1/namespaces/{namespace}/views", s.Auth(s.handleListViews)).Methods(http.MethodGet)
router.HandleFunc("/v1/namespaces/{namespace}/views", s.Auth(s.handleCreateView)).Methods(http.MethodPost)
router.HandleFunc("/v1/namespaces/{namespace}/views/{view}", s.Auth(s.handleLoadView)).Methods(http.MethodGet)
router.HandleFunc("/v1/namespaces/{namespace}/views/{view}", s.Auth(s.handleViewExists)).Methods(http.MethodHead)
router.HandleFunc("/v1/namespaces/{namespace}/views/{view}", s.Auth(s.handleDropView)).Methods(http.MethodDelete)
router.HandleFunc("/v1/namespaces/{namespace}/views/{view}", s.Auth(s.handleUpdateView)).Methods(http.MethodPost)
router.HandleFunc("/v1/views/rename", s.Auth(s.handleRenameView)).Methods(http.MethodPost)
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}/metrics", s.Auth(s.handleReportMetrics)).Methods(http.MethodPost)
// Multi-table transaction commit - wrapped with Auth middleware
router.HandleFunc("/v1/transactions/commit", s.Auth(s.handleCommitTransaction)).Methods(http.MethodPost)
// With prefix support - wrapped with Auth middleware
router.HandleFunc("/v1/{prefix}/namespaces", s.Auth(s.handleListNamespaces)).Methods(http.MethodGet)
router.HandleFunc("/v1/{prefix}/namespaces", s.Auth(s.handleCreateNamespace)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.Auth(s.handleGetNamespace)).Methods(http.MethodGet)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.Auth(s.handleNamespaceExists)).Methods(http.MethodHead)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.Auth(s.handleDropNamespace)).Methods(http.MethodDelete)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/properties", s.Auth(s.handleUpdateNamespaceProperties)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables", s.Auth(s.handleListTables)).Methods(http.MethodGet)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables", s.Auth(s.handleCreateTable)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/register", s.Auth(s.handleRegisterTable)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.Auth(s.handleLoadTable)).Methods(http.MethodGet)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.Auth(s.handleTableExists)).Methods(http.MethodHead)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.Auth(s.handleDropTable)).Methods(http.MethodDelete)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.Auth(s.handleUpdateTable)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/tables/rename", s.Auth(s.handleRenameTable)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views", s.Auth(s.handleListViews)).Methods(http.MethodGet)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views", s.Auth(s.handleCreateView)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views/{view}", s.Auth(s.handleLoadView)).Methods(http.MethodGet)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views/{view}", s.Auth(s.handleViewExists)).Methods(http.MethodHead)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views/{view}", s.Auth(s.handleDropView)).Methods(http.MethodDelete)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views/{view}", s.Auth(s.handleUpdateView)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/views/rename", s.Auth(s.handleRenameView)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics", s.Auth(s.handleReportMetrics)).Methods(http.MethodPost)
router.HandleFunc("/v1/{prefix}/transactions/commit", s.Auth(s.handleCommitTransaction)).Methods(http.MethodPost)
// Catch-all for debugging
router.PathPrefix("/").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
glog.V(2).Infof("Catch-all route hit: %s %s", r.Method, r.RequestURI)
writeError(w, http.StatusNotFound, "NotFound", "Path not found")
})
glog.V(2).Infof("Registered Iceberg REST Catalog routes")
}
func loggingMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
glog.V(2).Infof("Iceberg REST request: %s %s from %s", r.Method, r.RequestURI, r.RemoteAddr)
// Log all headers for debugging
glog.V(2).Infof("Iceberg REST headers:")
for name, values := range r.Header {
for _, value := range values {
// Redact sensitive headers
if name == "Authorization" && len(value) > 20 {
glog.V(2).Infof(" %s: %s...%s", name, value[:20], value[len(value)-10:])
} else {
glog.V(2).Infof(" %s: %s", name, value)
}
}
}
// Create a response writer that captures the status code
wrapped := &responseWriter{ResponseWriter: w}
next.ServeHTTP(wrapped, r)
glog.V(2).Infof("Iceberg REST response: %s %s -> %d", r.Method, r.RequestURI, wrapped.statusCode)
})
}
type responseWriter struct {
http.ResponseWriter
statusCode int
}
func (w *responseWriter) WriteHeader(code int) {
w.statusCode = code
w.ResponseWriter.WriteHeader(code)
}
func (s *Server) Auth(handler http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// A request carrying a Bearer token is an Iceberg REST client. When
// the token is invalid or expired, answer 401 immediately so clients
// (Iceberg Java OAuth2Manager, pyiceberg) refresh their token and
// retry. Falling through to the S3 authenticator instead would parse
// the Authorization header as SigV4, fail with NotImplemented (501),
// and clients would retry the dead token forever.
// The auth scheme is case-insensitive (RFC 7235), matching
// authenticateBearer below.
if strings.HasPrefix(strings.ToLower(r.Header.Get("Authorization")), "bearer ") {
if identityName, identity, ok := s.authenticateBearer(r); ok {
ctx := r.Context()
ctx = s3_constants.SetIdentityNameInContext(ctx, identityName)
if identity != nil {
ctx = s3_constants.SetIdentityInContext(ctx, identity)
}
r = r.WithContext(ctx)
handler(w, r)
return
}
// RFC 6750 / Iceberg REST spec: 401 is the refresh signal.
w.Header().Set("WWW-Authenticate", "Bearer")
writeError(w, http.StatusUnauthorized, "NotAuthorizedException", "Bearer token is invalid or expired")
return
}
if s.authenticator == nil {
writeError(w, http.StatusUnauthorized, "NotAuthorizedException", "Authentication required")
return
}
identityName, identity, errCode := s.authenticator.AuthenticateRequest(r)
if errCode != s3err.ErrNone {
// If authentication failed but DefaultAllow is enabled, proceed without identity
if s.authenticator.DefaultAllow() {
glog.V(2).Infof("Iceberg: AuthenticateRequest failed (%v), but DefaultAllow is true, proceeding", errCode)
} else {
apiErr := s3err.GetAPIError(errCode)
errorType := "RESTException"
switch apiErr.HTTPStatusCode {
case http.StatusForbidden:
errorType = "ForbiddenException"
case http.StatusUnauthorized:
errorType = "NotAuthorizedException"
case http.StatusBadRequest:
errorType = "BadRequestException"
case http.StatusInternalServerError:
errorType = "InternalServerError"
}
writeError(w, apiErr.HTTPStatusCode, errorType, apiErr.Description)
return
}
}
if identityName != "" || identity != nil {
ctx := r.Context()
if identityName != "" {
ctx = s3_constants.SetIdentityNameInContext(ctx, identityName)
}
if identity != nil {
ctx = s3_constants.SetIdentityInContext(ctx, identity)
}
r = r.WithContext(ctx)
}
handler(w, r)
}
}

View File

@@ -0,0 +1,231 @@
package iceberg
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
jwt "github.com/golang-jwt/jwt/v5"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
)
type mockS3Authenticator struct {
called bool
errCode s3err.ErrorCode
}
func (m *mockS3Authenticator) AuthenticateRequest(r *http.Request) (string, interface{}, s3err.ErrorCode) {
m.called = true
if m.errCode != s3err.ErrNone {
return "", nil, m.errCode
}
return "s3user", nil, s3err.ErrNone
}
func (m *mockS3Authenticator) DefaultAllow() bool { return false }
func mintTestToken(t *testing.T, accessKey, secret string, issuedAt, expiresAt time.Time) string {
t.Helper()
key := deriveSigningKey(accessKey, secret)
claims := IcebergClaims{
IdentityName: "testuser",
AccessKey: accessKey,
RegisteredClaims: jwt.RegisteredClaims{
IssuedAt: jwt.NewNumericDate(issuedAt),
ExpiresAt: jwt.NewNumericDate(expiresAt),
Issuer: "seaweedfs-iceberg",
},
}
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
signed, err := token.SignedString(key)
if err != nil {
t.Fatalf("sign token: %v", err)
}
return signed
}
func newAuthTestServer() (*Server, *mockS3Authenticator) {
s := newTestServerWithOAuth()
auth := &mockS3Authenticator{errCode: s3err.ErrNotImplemented}
s.authenticator = auth
return s, auth
}
// TestAuthExpiredBearerReturns401: an expired Bearer token must get 401
// (the refresh signal for Iceberg clients), never fall through to the S3
// authenticator and surface as 501 NotImplemented.
func TestAuthExpiredBearerReturns401(t *testing.T) {
s, auth := newAuthTestServer()
now := time.Now()
expired := mintTestToken(t, "AKID123", "secret456", now.Add(-2*time.Hour), now.Add(-1*time.Hour))
var handlerCalled bool
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
handlerCalled = true
w.WriteHeader(http.StatusOK)
})
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
req.Header.Set("Authorization", "Bearer "+expired)
rec := httptest.NewRecorder()
handler(rec, req)
if rec.Code != http.StatusUnauthorized {
t.Fatalf("expired Bearer: status = %d, want 401 (previously fell through to 501)", rec.Code)
}
if auth.called {
t.Fatalf("expired Bearer must not fall through to the S3 authenticator")
}
if handlerCalled {
t.Fatalf("handler must not run for an expired token")
}
if wa := rec.Header().Get("WWW-Authenticate"); wa != "Bearer" {
t.Fatalf("WWW-Authenticate = %q, want Bearer", wa)
}
if !strings.Contains(rec.Body.String(), "NotAuthorizedException") {
t.Fatalf("error type must be NotAuthorizedException, got: %s", rec.Body.String())
}
}
// TestAuthCaseInsensitiveBearerScheme: RFC 7235 auth schemes are
// case-insensitive; a lowercase or mixed-case "bearer" must take the same
// Bearer path (401 refresh signal) instead of falling through to S3 auth.
func TestAuthCaseInsensitiveBearerScheme(t *testing.T) {
s, auth := newAuthTestServer()
now := time.Now()
expired := mintTestToken(t, "AKID123", "secret456", now.Add(-2*time.Hour), now.Add(-1*time.Hour))
for _, scheme := range []string{"bearer", "BEARER", "BeArEr"} {
var handlerCalled bool
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
handlerCalled = true
w.WriteHeader(http.StatusOK)
})
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
req.Header.Set("Authorization", scheme+" "+expired)
rec := httptest.NewRecorder()
handler(rec, req)
if rec.Code != http.StatusUnauthorized {
t.Fatalf("scheme %q: status = %d, want 401", scheme, rec.Code)
}
if auth.called {
t.Fatalf("scheme %q: must not fall through to the S3 authenticator", scheme)
}
if handlerCalled {
t.Fatalf("scheme %q: handler must not run for an expired token", scheme)
}
if wa := rec.Header().Get("WWW-Authenticate"); wa != "Bearer" {
t.Fatalf("scheme %q: WWW-Authenticate = %q, want Bearer", scheme, wa)
}
}
}
// TestAuthGarbageBearerReturns401 pins the same contract for malformed tokens.
func TestAuthGarbageBearerReturns401(t *testing.T) {
s, auth := newAuthTestServer()
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
})
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
req.Header.Set("Authorization", "Bearer not-a-jwt")
rec := httptest.NewRecorder()
handler(rec, req)
if rec.Code != http.StatusUnauthorized {
t.Fatalf("garbage Bearer: status = %d, want 401", rec.Code)
}
if auth.called {
t.Fatalf("garbage Bearer must not fall through to the S3 authenticator")
}
}
// TestAuthFreshBearerRunsHandler verifies the happy path still authenticates
// and hands the identity to the handler.
func TestAuthFreshBearerRunsHandler(t *testing.T) {
s, auth := newAuthTestServer()
now := time.Now()
fresh := mintTestToken(t, "AKID123", "secret456", now, now.Add(time.Hour))
var gotIdentity string
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
gotIdentity = s3_constants.GetIdentityNameFromContext(r)
w.WriteHeader(http.StatusOK)
})
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
req.Header.Set("Authorization", "Bearer "+fresh)
rec := httptest.NewRecorder()
handler(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("fresh Bearer: status = %d, want 200", rec.Code)
}
if auth.called {
t.Fatalf("fresh Bearer should not need the S3 authenticator")
}
if gotIdentity != "testuser" {
t.Fatalf("identity in context = %q, want testuser", gotIdentity)
}
}
// TestAuthNoBearerStillUsesS3Authenticator keeps the non-Bearer path intact:
// SigV4 requests keep flowing through the S3 authenticator as before.
func TestAuthNoBearerStillUsesS3Authenticator(t *testing.T) {
s := newTestServerWithOAuth()
auth := &mockS3Authenticator{errCode: s3err.ErrNone}
s.authenticator = auth
var handlerCalled bool
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
handlerCalled = true
w.WriteHeader(http.StatusOK)
})
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
rec := httptest.NewRecorder()
handler(rec, req)
if !auth.called {
t.Fatalf("request without Bearer header must use the S3 authenticator")
}
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200 after successful S3 auth", rec.Code)
}
if !handlerCalled {
t.Fatalf("handler must run after S3 auth succeeds")
}
}
// TestAuthNonBearerAuthorizationHeaderUsesS3Authenticator pins that other
// Authorization schemes (e.g. SigV4) never take the Bearer fast path.
func TestAuthNonBearerAuthorizationHeaderUsesS3Authenticator(t *testing.T) {
s := newTestServerWithOAuth()
auth := &mockS3Authenticator{errCode: s3err.ErrNone}
s.authenticator = auth
var handlerCalled bool
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
handlerCalled = true
w.WriteHeader(http.StatusOK)
})
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential=AKID123/...")
rec := httptest.NewRecorder()
handler(rec, req)
if !auth.called {
t.Fatalf("SigV4 Authorization header must use the S3 authenticator")
}
if !handlerCalled {
t.Fatalf("handler must run after S3 auth succeeds")
}
}

View File

@@ -0,0 +1,41 @@
package iceberg
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/gorilla/mux"
)
// The catch-all answers anything unrouted, so a path that reaches it is a
// route that does not exist.
func TestRegisteredRoutesReachAHandler(t *testing.T) {
router := mux.NewRouter().SkipClean(true)
(&Server{}).RegisterRoutes(router)
cases := []struct {
method string
target string
}{
{http.MethodPost, "/v1/views/rename"},
{http.MethodPost, "/v1/warehouse/views/rename"},
{http.MethodPost, "/v1/namespaces/ns/tables/t/metrics"},
{http.MethodPost, "/v1/warehouse/namespaces/ns/tables/t/metrics"},
}
for _, tc := range cases {
var match mux.RouteMatch
if !router.Match(httptest.NewRequest(tc.method, tc.target, nil), &match) {
t.Errorf("%s %s matched no route", tc.method, tc.target)
continue
}
if match.MatchErr != nil {
t.Errorf("%s %s: %v", tc.method, tc.target, match.MatchErr)
continue
}
if tmpl, err := match.Route.GetPathTemplate(); err == nil && tmpl == "/" {
t.Errorf("%s %s fell through to the catch-all", tc.method, tc.target)
}
}
}

View File

@@ -0,0 +1,276 @@
package iceberg
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/url"
"os"
"path"
"strings"
"time"
"github.com/google/uuid"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
const (
stageCreateMarkerDirName = ".iceberg_staged"
// Keep staged markers long enough to avoid deleting in-progress create transactions.
stageCreateMarkerTTL = 72 * time.Hour
)
type stageCreateMarker struct {
TableUUID string `json:"table_uuid"`
Location string `json:"location"`
StagedMetadataLocation string `json:"staged_metadata_location,omitempty"`
CreatedAt string `json:"created_at"`
ExpiresAt string `json:"expires_at"`
}
func stageCreateMarkerNamespaceKey(namespace []string) string {
return url.PathEscape(encodeNamespace(namespace))
}
func stageCreateMarkerDir(bucketName string, namespace []string, tableName string) string {
return path.Join(s3tables.TablesPath, bucketName, stageCreateMarkerDirName, stageCreateMarkerNamespaceKey(namespace), tableName)
}
func stageCreateStagedTablePath(namespace []string, tableName string, tableUUID uuid.UUID) string {
return path.Join(stageCreateMarkerDirName, stageCreateMarkerNamespaceKey(namespace), tableName, tableUUID.String())
}
func (s *Server) pruneExpiredStageCreateMarkers(ctx context.Context, bucketName string, namespace []string, tableName string) error {
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
markerDir := stageCreateMarkerDir(bucketName, namespace, tableName)
now := time.Now().UTC()
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
stream, err := client.ListEntries(opCtx, &filer_pb.ListEntriesRequest{
Directory: markerDir,
Limit: 1024,
})
if err != nil {
if errors.Is(err, filer_pb.ErrNotFound) || strings.Contains(strings.ToLower(err.Error()), "not found") {
return nil
}
return err
}
for {
resp, recvErr := stream.Recv()
if recvErr == io.EOF {
return nil
}
if recvErr != nil {
return recvErr
}
if resp.Entry == nil || resp.Entry.IsDirectory || len(resp.Entry.Content) == 0 {
continue
}
var marker stageCreateMarker
if err := json.Unmarshal(resp.Entry.Content, &marker); err != nil {
continue
}
expiresAt, err := time.Parse(time.RFC3339Nano, marker.ExpiresAt)
if err != nil || !expiresAt.Before(now) {
continue
}
if err := filer_pb.DoRemove(opCtx, client, markerDir, resp.Entry.Name, true, false, true, false, nil); err != nil && !errors.Is(err, filer_pb.ErrNotFound) {
return err
}
}
})
}
func (s *Server) loadLatestStageCreateMarker(ctx context.Context, bucketName string, namespace []string, tableName string) (*stageCreateMarker, error) {
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
markerDir := stageCreateMarkerDir(bucketName, namespace, tableName)
now := time.Now().UTC()
var latestMarker *stageCreateMarker
var latestCreatedAt time.Time
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
stream, err := client.ListEntries(opCtx, &filer_pb.ListEntriesRequest{
Directory: markerDir,
Limit: 1024,
})
if err != nil {
if errors.Is(err, filer_pb.ErrNotFound) || strings.Contains(strings.ToLower(err.Error()), "not found") {
return nil
}
return err
}
for {
resp, recvErr := stream.Recv()
if recvErr == io.EOF {
return nil
}
if recvErr != nil {
return recvErr
}
if resp.Entry == nil || resp.Entry.IsDirectory || len(resp.Entry.Content) == 0 {
continue
}
var marker stageCreateMarker
if err := json.Unmarshal(resp.Entry.Content, &marker); err != nil {
continue
}
expiresAt, err := time.Parse(time.RFC3339Nano, marker.ExpiresAt)
if err != nil || !expiresAt.After(now) {
continue
}
createdAt, err := time.Parse(time.RFC3339Nano, marker.CreatedAt)
if err != nil {
createdAt = time.Time{}
}
if latestMarker == nil || createdAt.After(latestCreatedAt) {
candidate := marker
latestMarker = &candidate
latestCreatedAt = createdAt
}
}
})
if err != nil {
return nil, err
}
return latestMarker, nil
}
func (s *Server) writeStageCreateMarker(ctx context.Context, bucketName string, namespace []string, tableName string, tableUUID uuid.UUID, location string, stagedMetadataLocation string) error {
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
marker := stageCreateMarker{
TableUUID: tableUUID.String(),
Location: location,
StagedMetadataLocation: stagedMetadataLocation,
CreatedAt: time.Now().UTC().Format(time.RFC3339Nano),
ExpiresAt: time.Now().UTC().Add(stageCreateMarkerTTL).Format(time.RFC3339Nano),
}
content, err := json.Marshal(marker)
if err != nil {
return err
}
if err := s.pruneExpiredStageCreateMarkers(ctx, bucketName, namespace, tableName); err != nil {
return err
}
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
ensureDir := func(parent, name, errorContext string) error {
_, lookupErr := filer_pb.LookupEntry(opCtx, client, &filer_pb.LookupDirectoryEntryRequest{
Directory: parent,
Name: name,
})
if lookupErr == nil {
return nil
}
if lookupErr != filer_pb.ErrNotFound {
return fmt.Errorf("lookup %s failed: %w", errorContext, lookupErr)
}
resp, createErr := client.CreateEntry(opCtx, &filer_pb.CreateEntryRequest{
Directory: parent,
Entry: &filer_pb.Entry{
Name: name,
IsDirectory: true,
Attributes: &filer_pb.FuseAttributes{
Mtime: time.Now().Unix(),
Crtime: time.Now().Unix(),
FileMode: uint32(0755 | os.ModeDir),
},
},
})
if createErr != nil {
return fmt.Errorf("failed to create %s: %w", errorContext, createErr)
}
if resp.ErrorCode != filer_pb.FilerError_OK && resp.ErrorCode != filer_pb.FilerError_ENTRY_ALREADY_EXISTS {
return fmt.Errorf("failed to create %s: %s", errorContext, resp.Error)
}
return nil
}
segments := []string{bucketName, stageCreateMarkerDirName, stageCreateMarkerNamespaceKey(namespace), tableName}
currentDir := s3tables.TablesPath
for _, segment := range segments {
if segment == "" {
continue
}
if err := ensureDir(currentDir, segment, "stage-create marker directory"); err != nil {
return err
}
currentDir = path.Join(currentDir, segment)
}
entryName := tableUUID.String() + ".json"
resp, createErr := client.CreateEntry(opCtx, &filer_pb.CreateEntryRequest{
Directory: currentDir,
Entry: &filer_pb.Entry{
Name: entryName,
Attributes: &filer_pb.FuseAttributes{
Mtime: time.Now().Unix(),
Crtime: time.Now().Unix(),
FileMode: uint32(0644),
FileSize: uint64(len(content)),
},
Content: content,
Extended: map[string][]byte{
"Mime-Type": []byte("application/json"),
},
},
})
if createErr != nil {
return createErr
}
if resp.ErrorCode != filer_pb.FilerError_OK {
if sentinel := filer_pb.FilerErrorToSentinel(resp.ErrorCode); sentinel != nil {
return fmt.Errorf("create stage marker: %w", sentinel)
}
return fmt.Errorf("create stage marker: code=%v %s", resp.ErrorCode, resp.Error)
}
return nil
})
}
func (s *Server) deleteStageCreateMarkers(ctx context.Context, bucketName string, namespace []string, tableName string) error {
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
parentDir := path.Dir(stageCreateMarkerDir(bucketName, namespace, tableName))
targetName := path.Base(stageCreateMarkerDir(bucketName, namespace, tableName))
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
err := filer_pb.DoRemove(opCtx, client, parentDir, targetName, true, true, true, false, nil)
if errors.Is(err, filer_pb.ErrNotFound) {
return nil
}
return err
})
}
func isStageCreateEnabled() bool {
value := strings.TrimSpace(strings.ToLower(os.Getenv("ICEBERG_ENABLE_STAGE_CREATE")))
if value == "" {
return true
}
switch value {
case "0", "false", "no", "off":
return false
default:
return true
}
}

289
weed/s3api/iceberg/types.go Normal file
View File

@@ -0,0 +1,289 @@
// Package iceberg defines types for the Iceberg REST Catalog API.
// This package uses types from github.com/apache/iceberg-go for spec compliance.
package iceberg
import (
"encoding/json"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/table"
"github.com/apache/iceberg-go/view"
)
// CatalogConfig is returned by GET /v1/config.
type CatalogConfig struct {
Defaults map[string]string `json:"defaults"`
Overrides map[string]string `json:"overrides"`
Endpoints []string `json:"endpoints,omitempty"`
}
// ErrorModel represents an Iceberg error.
type ErrorModel struct {
Message string `json:"message"`
Type string `json:"type"`
Code int `json:"code"`
Stack []string `json:"stack,omitempty"`
}
// ErrorResponse wraps an error in the standard Iceberg format.
type ErrorResponse struct {
Error ErrorModel `json:"error"`
}
// Namespace is a reference to one or more levels of a namespace.
type Namespace []string
// TableIdentifier identifies a table within a namespace.
type TableIdentifier struct {
Namespace Namespace `json:"namespace"`
Name string `json:"name"`
}
// ListNamespacesResponse is returned by GET /v1/namespaces.
type ListNamespacesResponse struct {
NextPageToken string `json:"next-page-token,omitempty"`
Namespaces []Namespace `json:"namespaces"`
}
// CreateNamespaceRequest is sent to POST /v1/namespaces.
type CreateNamespaceRequest struct {
Namespace Namespace `json:"namespace"`
Properties map[string]string `json:"properties,omitempty"`
}
// CreateNamespaceResponse is returned by POST /v1/namespaces.
type CreateNamespaceResponse struct {
Namespace Namespace `json:"namespace"`
Properties map[string]string `json:"properties"`
}
// GetNamespaceResponse is returned by GET /v1/namespaces/{namespace}.
type GetNamespaceResponse struct {
Namespace Namespace `json:"namespace"`
Properties map[string]string `json:"properties"`
}
// UpdateNamespacePropertiesRequest is sent to POST /v1/namespaces/{namespace}/properties.
type UpdateNamespacePropertiesRequest struct {
Removals []string `json:"removals,omitempty"`
Updates map[string]string `json:"updates,omitempty"`
}
// UpdateNamespacePropertiesResponse is returned by POST /v1/namespaces/{namespace}/properties.
type UpdateNamespacePropertiesResponse struct {
Removed []string `json:"removed"`
Updated []string `json:"updated"`
Missing []string `json:"missing"`
}
// ListTablesResponse is returned by GET /v1/namespaces/{namespace}/tables.
type ListTablesResponse struct {
NextPageToken string `json:"next-page-token,omitempty"`
Identifiers []TableIdentifier `json:"identifiers"`
}
// CreateTableRequest is sent to POST /v1/namespaces/{namespace}/tables.
// Uses iceberg-go types for Schema, PartitionSpec, and SortOrder.
type CreateTableRequest struct {
Name string `json:"name"`
Location string `json:"location,omitempty"`
Schema *iceberg.Schema `json:"schema,omitempty"`
PartitionSpec *iceberg.PartitionSpec `json:"partition-spec,omitempty"`
WriteOrder *table.SortOrder `json:"write-order,omitempty"`
StageCreate bool `json:"stage-create,omitempty"`
Properties iceberg.Properties `json:"properties,omitempty"`
}
// RegisterTableRequest is sent to POST /v1/namespaces/{namespace}/register.
type RegisterTableRequest struct {
Name string `json:"name"`
MetadataLocation string `json:"metadata-location"`
}
type LoadTableResult struct {
MetadataLocation string `json:"metadata-location,omitempty"`
Metadata table.Metadata `json:"metadata"`
Config iceberg.Properties `json:"config"`
StorageCredentials []StorageCredential `json:"storage-credentials,omitempty"`
}
// StorageCredential is one set of credentials scoped to a location prefix, as
// returned to clients that asked for credential vending.
type StorageCredential struct {
Prefix string `json:"prefix"`
Config iceberg.Properties `json:"config"`
}
// loadTableResultAlias is used for custom JSON unmarshaling.
type loadTableResultAlias struct {
MetadataLocation string `json:"metadata-location,omitempty"`
RawMetadata json.RawMessage `json:"metadata"`
Config iceberg.Properties `json:"config,omitempty"`
StorageCredentials []StorageCredential `json:"storage-credentials,omitempty"`
}
// MarshalJSON serializes LoadTableResult while backfilling spec-required
// metadata fields that iceberg-go v0.5.0 drops via `omitempty`. Without this,
// strict Iceberg REST clients (Java/Spark/Trino) reject otherwise valid
// responses for empty tables with errors like
// "Cannot parse missing long current-snapshot-id".
func (r LoadTableResult) MarshalJSON() ([]byte, error) {
metaBytes, err := json.Marshal(r.Metadata)
if err != nil {
return nil, err
}
metaBytes = ensureMetadataSpecCompliance(metaBytes)
return json.Marshal(struct {
MetadataLocation string `json:"metadata-location,omitempty"`
Metadata json.RawMessage `json:"metadata"`
Config iceberg.Properties `json:"config"`
StorageCredentials []StorageCredential `json:"storage-credentials,omitempty"`
}{
MetadataLocation: r.MetadataLocation,
Metadata: metaBytes,
Config: r.Config,
StorageCredentials: r.StorageCredentials,
})
}
// UnmarshalJSON implements custom unmarshaling for LoadTableResult
// to properly parse table.Metadata using iceberg-go's parser.
func (r *LoadTableResult) UnmarshalJSON(data []byte) error {
var alias loadTableResultAlias
if err := json.Unmarshal(data, &alias); err != nil {
return err
}
r.MetadataLocation = alias.MetadataLocation
r.Config = alias.Config
r.StorageCredentials = alias.StorageCredentials
if len(alias.RawMetadata) > 0 {
metadata, err := table.ParseMetadataBytes(alias.RawMetadata)
if err != nil {
return err
}
r.Metadata = metadata
}
return nil
}
// RenameTableRequest is sent to POST /v1/tables/rename.
type RenameTableRequest struct {
Source TableIdentifier `json:"source"`
Destination TableIdentifier `json:"destination"`
}
// CommitTableRequest is sent to POST /v1/namespaces/{namespace}/tables/{table}.
type CommitTableRequest struct {
Identifier *TableIdentifier `json:"identifier,omitempty"`
Requirements table.Requirements `json:"requirements"`
Updates table.Updates `json:"updates"`
}
// CommitTableResponse is returned by POST table commit operations.
type CommitTableResponse struct {
MetadataLocation string `json:"metadata-location"`
Metadata table.Metadata `json:"metadata"`
}
// commitTableResponseAlias is used for custom JSON unmarshaling.
type commitTableResponseAlias struct {
MetadataLocation string `json:"metadata-location"`
RawMetadata json.RawMessage `json:"metadata"`
}
// MarshalJSON mirrors LoadTableResult.MarshalJSON: it backfills spec-required
// fields that iceberg-go v0.5.0 omits, so commit responses also parse cleanly
// on strict Iceberg clients.
func (r CommitTableResponse) MarshalJSON() ([]byte, error) {
metaBytes, err := json.Marshal(r.Metadata)
if err != nil {
return nil, err
}
metaBytes = ensureMetadataSpecCompliance(metaBytes)
return json.Marshal(struct {
MetadataLocation string `json:"metadata-location"`
Metadata json.RawMessage `json:"metadata"`
}{
MetadataLocation: r.MetadataLocation,
Metadata: metaBytes,
})
}
// UnmarshalJSON implements custom unmarshaling for CommitTableResponse.
func (r *CommitTableResponse) UnmarshalJSON(data []byte) error {
var alias commitTableResponseAlias
if err := json.Unmarshal(data, &alias); err != nil {
return err
}
r.MetadataLocation = alias.MetadataLocation
if len(alias.RawMetadata) > 0 {
metadata, err := table.ParseMetadataBytes(alias.RawMetadata)
if err != nil {
return err
}
r.Metadata = metadata
}
return nil
}
// ListViewsResponse is returned by GET /v1/namespaces/{namespace}/views.
type ListViewsResponse struct {
NextPageToken string `json:"next-page-token,omitempty"`
Identifiers []TableIdentifier `json:"identifiers"`
}
// CreateViewRequest is sent to POST /v1/namespaces/{namespace}/views.
type CreateViewRequest struct {
Name string `json:"name"`
Schema *iceberg.Schema `json:"schema"`
Location string `json:"location,omitempty"`
Properties iceberg.Properties `json:"properties,omitempty"`
ViewVersion *view.Version `json:"view-version"`
}
// ViewResponse is returned by view create/update and load operations.
type ViewResponse struct {
MetadataLocation string `json:"metadata-location"`
Metadata view.Metadata `json:"metadata"`
Config iceberg.Properties `json:"config"`
}
// viewResponseAlias is used for custom JSON unmarshaling.
type viewResponseAlias struct {
MetadataLocation string `json:"metadata-location"`
RawMetadata json.RawMessage `json:"metadata"`
Config iceberg.Properties `json:"config,omitempty"`
}
// UnmarshalJSON parses view.Metadata using iceberg-go's view parser.
func (r *ViewResponse) UnmarshalJSON(data []byte) error {
var alias viewResponseAlias
if err := json.Unmarshal(data, &alias); err != nil {
return err
}
r.MetadataLocation = alias.MetadataLocation
r.Config = alias.Config
if len(alias.RawMetadata) > 0 {
metadata, err := view.ParseMetadataBytes(alias.RawMetadata)
if err != nil {
return err
}
r.Metadata = metadata
}
return nil
}
// UpdateViewRequest is sent to POST /v1/namespaces/{namespace}/views/{view}.
type UpdateViewRequest struct {
Identifier *TableIdentifier `json:"identifier,omitempty"`
Requirements view.Requirements `json:"requirements"`
Updates view.Updates `json:"updates"`
}

353
weed/s3api/iceberg/utils.go Normal file
View File

@@ -0,0 +1,353 @@
package iceberg
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"strconv"
"strings"
"github.com/apache/iceberg-go"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
)
// parseNamespace parses the namespace from path parameter.
// Iceberg uses unit separator (0x1F) for multi-level namespaces.
// Note: mux already decodes URL-encoded path parameters, so we only split by unit separator.
func parseNamespace(encoded string) []string {
if encoded == "" {
return nil
}
parts := strings.Split(encoded, "\x1F")
// Filter empty parts
result := make([]string, 0, len(parts))
for _, p := range parts {
if p != "" {
result = append(result, p)
}
}
return result
}
// encodeNamespace encodes namespace parts using the Iceberg REST protocol's
// unit separator (0x1F) convention. This is only appropriate for protocol-level
// encoding (e.g. URL path parameters), NOT for filesystem/S3 paths.
func encodeNamespace(parts []string) string {
return strings.Join(parts, "\x1F")
}
// flattenNamespacePath joins namespace parts with "." for use in S3 location
// and filer paths, matching the S3 Tables storage layer convention.
func flattenNamespacePath(parts []string) string {
return strings.Join(parts, ".")
}
func parseS3Location(location string) (bucketName, tablePath string, err error) {
if !strings.HasPrefix(location, "s3://") {
return "", "", fmt.Errorf("unsupported location: %s", location)
}
trimmed := strings.TrimPrefix(location, "s3://")
trimmed = strings.TrimSuffix(trimmed, "/")
if trimmed == "" {
return "", "", fmt.Errorf("invalid location: %s", location)
}
parts := strings.SplitN(trimmed, "/", 2)
bucketName = parts[0]
if bucketName == "" {
return "", "", fmt.Errorf("invalid location bucket: %s", location)
}
if len(parts) == 2 {
tablePath = parts[1]
}
return bucketName, tablePath, nil
}
func tableLocationFromMetadataLocation(metadataLocation string) string {
trimmed := strings.TrimSuffix(metadataLocation, "/")
if idx := strings.LastIndex(trimmed, "/metadata/"); idx != -1 {
return trimmed[:idx]
}
return trimmed
}
// accessDelegationHeader is how a client asks the catalog to hand back storage
// credentials along with the table metadata.
const accessDelegationHeader = "X-Iceberg-Access-Delegation"
// wantsVendedCredentials reports whether the client asked for vended
// credentials. The header carries a comma-separated list of mechanisms.
func wantsVendedCredentials(r *http.Request) bool {
if r == nil {
return false
}
for _, value := range r.Header.Values(accessDelegationHeader) {
for _, mechanism := range strings.Split(value, ",") {
if strings.EqualFold(strings.TrimSpace(mechanism), "vended-credentials") {
return true
}
}
}
return false
}
// writeLoadResult writes a table or view load response. The FileIO config it
// carries depends on the delegation the client asked for, so a cache between
// us and the client must key on that header and not on the URL alone.
func writeLoadResult(w http.ResponseWriter, status int, v interface{}) {
w.Header().Set("Vary", accessDelegationHeader)
writeJSON(w, status, v)
}
// writeJSON writes a JSON response.
func writeJSON(w http.ResponseWriter, status int, v interface{}) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
if v != nil {
data, err := json.Marshal(v)
if err != nil {
glog.Errorf("Iceberg: failed to encode response: %v", err)
return
}
w.Write(data)
}
}
// writeError writes an Iceberg error response.
func writeError(w http.ResponseWriter, status int, errType, message string) {
resp := ErrorResponse{
Error: ErrorModel{
Message: message,
Type: errType,
Code: status,
},
}
writeJSON(w, status, resp)
}
// nameValidationError reports whether err is an S3 Tables namespace or table
// name validation failure. Such names are client input (the S3 Tables charset
// is stricter than the Iceberg REST spec), so the catalog answers 400 rather
// than treating it as a server fault.
func nameValidationError(err error) bool {
if err == nil {
return false
}
// Match the validator's own phrasings rather than a bare "namespace name"/
// "table name" so unrelated faults (e.g. "failed to resolve table name")
// aren't misreported as client errors. Lowercased for resilience to
// capitalization changes.
msg := strings.ToLower(err.Error())
for _, marker := range []string{
"invalid namespace name", "namespace name must", "namespace name cannot",
"invalid table name", "table name must", "table name cannot",
} {
if strings.Contains(msg, marker) {
return true
}
}
return false
}
// writeManagerError maps a residual error to a response: name validation
// failures and rejected schemas come from the request body, so they are client
// errors (400) and must carry the reason -- a variant field without
// format-version 3 otherwise reads as a bare 500 with "variant is not supported
// until v3" only in the log. Anything else is a server fault (500).
func writeManagerError(w http.ResponseWriter, err error) {
switch {
case nameValidationError(err),
errors.Is(err, iceberg.ErrInvalidSchema),
errors.Is(err, iceberg.ErrInvalidPartitionSpec),
errors.Is(err, iceberg.ErrInvalidTypeString),
errors.Is(err, iceberg.ErrInvalidTransform),
errors.Is(err, iceberg.ErrInvalidArgument):
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
return
}
// Storage-layer failures are mostly the client's, not the server's. Reporting
// a missing namespace or a name conflict as a 500 makes the catalog look
// broken and gives the client nothing to act on.
var tableErr *s3tables.S3TablesError
if errors.As(err, &tableErr) {
switch tableErr.Type {
case s3tables.ErrCodeNoSuchBucket:
// The storage-layer message names the resolved bucket, which for a
// client that sent no warehouse at all is the default one it never
// asked for, so say how to select a real table bucket.
writeError(w, http.StatusNotFound, "NoSuchNamespaceException",
fmt.Sprintf("%s: each table bucket is a separate catalog, select one with warehouse=s3://<table-bucket>/ or /v1/<table-bucket>/", tableErr.Message))
case s3tables.ErrCodeNoSuchNamespace:
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", tableErr.Message)
case s3tables.ErrCodeNoSuchTable:
writeError(w, http.StatusNotFound, "NoSuchTableException", tableErr.Message)
case s3tables.ErrCodeNoSuchView:
writeError(w, http.StatusNotFound, "NoSuchViewException", tableErr.Message)
case s3tables.ErrCodeNamespaceAlreadyExists, s3tables.ErrCodeBucketAlreadyExists,
s3tables.ErrCodeTableAlreadyExists, s3tables.ErrCodeViewAlreadyExists:
writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message)
case s3tables.ErrCodeNamespaceNotEmpty, s3tables.ErrCodeBucketNotEmpty:
writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message)
case s3tables.ErrCodeConflict:
writeError(w, http.StatusConflict, "CommitFailedException", tableErr.Message)
case s3tables.ErrCodeAccessDenied:
writeError(w, http.StatusForbidden, "ForbiddenException", tableErr.Message)
case s3tables.ErrCodeInvalidRequest, s3tables.ErrCodeInvalidIcebergLayout:
writeError(w, http.StatusBadRequest, "BadRequestException", tableErr.Message)
default:
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
}
return
}
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
}
// resolveWarehouseBucket maps a client-supplied warehouse value to a table
// bucket name. Clients spell the warehouse three ways: the s3://<bucket>/
// location this catalog advertises, the table bucket ARN that AWS S3 Tables
// uses, and the bare bucket name. Accepting only the first sends every other
// spelling to the default bucket, where the request fails naming a bucket the
// client never asked for. Returns "" when the value names no usable bucket so
// the caller keeps its own default.
func resolveWarehouseBucket(warehouse string) string {
warehouse = strings.TrimSpace(warehouse)
var bucket string
switch {
case warehouse == "":
return ""
case strings.HasPrefix(warehouse, "s3://"):
parsed, _, err := parseS3Location(warehouse)
if err != nil {
return ""
}
bucket = parsed
case strings.HasPrefix(warehouse, "arn:"):
parsed, err := s3tables.ParseBucketNameFromARN(warehouse)
if err != nil {
return ""
}
bucket = parsed
default:
// Bare name, possibly with a sub-path that bucket-scoped routing ignores.
bucket, _, _ = strings.Cut(strings.TrimSuffix(warehouse, "/"), "/")
}
if !s3tables.IsValidBucketName(bucket) {
return ""
}
return bucket
}
// getBucketFromPrefix extracts table bucket name from prefix parameter.
// For now, we use the prefix as the table bucket name.
//
// The Iceberg REST spec lets clients identify a catalog either by embedding
// its prefix in the URL (/v1/{prefix}/...) or by passing ?warehouse= as a
// query parameter. Clients that skip the /v1/config handshake (or ignore its
// overrides) still routinely send the warehouse parameter on every request,
// so honor it as a fallback before the env-var default.
func getBucketFromPrefix(r *http.Request) string {
vars := mux.Vars(r)
if prefix := vars["prefix"]; prefix != "" {
return prefix
}
if bucket := resolveWarehouseBucket(r.URL.Query().Get("warehouse")); bucket != "" {
return bucket
}
if bucket := os.Getenv("S3TABLES_DEFAULT_BUCKET"); bucket != "" {
return bucket
}
// Some writers commit to the unprefixed path even though their reads
// honor the /v1/config prefix; the deployment's first table bucket is
// where those commits belong.
for _, name := range strings.Split(os.Getenv("S3_TABLE_BUCKET"), ",") {
if name = strings.TrimSpace(name); name != "" {
return name
}
}
// Default bucket if no prefix - use "warehouse" for Iceberg
return "warehouse"
}
// buildTableBucketARN builds an ARN for a table bucket.
func buildTableBucketARN(bucketName string) string {
arn, _ := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3_constants.AccountAdminId, bucketName)
return arn
}
const (
defaultListPageSize = 1000
maxListPageSize = 1000
)
func getPaginationQueryParam(r *http.Request, primary, fallback string) string {
if v := strings.TrimSpace(r.URL.Query().Get(primary)); v != "" {
return v
}
return strings.TrimSpace(r.URL.Query().Get(fallback))
}
func parsePagination(r *http.Request) (pageToken string, pageSize int, err error) {
pageToken = getPaginationQueryParam(r, "pageToken", "page-token")
pageSize = defaultListPageSize
pageSizeValue := getPaginationQueryParam(r, "pageSize", "page-size")
if pageSizeValue == "" {
return pageToken, pageSize, nil
}
parsedPageSize, parseErr := strconv.Atoi(pageSizeValue)
if parseErr != nil || parsedPageSize <= 0 {
return pageToken, 0, fmt.Errorf("invalid pageSize %q: must be a positive integer", pageSizeValue)
}
if parsedPageSize > maxListPageSize {
return pageToken, 0, fmt.Errorf("invalid pageSize %q: must be <= %d", pageSizeValue, maxListPageSize)
}
return pageToken, parsedPageSize, nil
}
func normalizeNamespaceProperties(properties map[string]string) map[string]string {
if properties == nil {
return map[string]string{}
}
return properties
}
// namespaceLocationProperty is the well-known Iceberg key used to advertise a
// default base path for tables created in a namespace.
const namespaceLocationProperty = "location"
// defaultNamespaceLocation returns the s3 path under which tables in the
// namespace should be stored when the namespace itself does not carry an
// explicit "location" property. The flattened namespace path mirrors the on-
// disk layout used by handleCreateTable so a deferred client-side commit ends
// up at the same place a server-computed one would.
func defaultNamespaceLocation(bucket string, namespace []string) string {
if bucket == "" || len(namespace) == 0 {
return ""
}
return fmt.Sprintf("s3://%s/%s", bucket, flattenNamespacePath(namespace))
}
// withDefaultNamespaceLocation populates the Iceberg "location" property on a
// namespace response when the storage layer does not carry one. Trino's REST
// catalog falls back to an eager createTable code path when the namespace
// has no location, which races our metadata write against Trino's emptiness
// check and surfaces as a "Cannot create a table on a non-empty location"
// error on the very first CREATE TABLE. Advertising a default location lets
// Trino take the deferred-transaction path and pick a unique per-table path.
// See issue #9074.
func withDefaultNamespaceLocation(properties map[string]string, bucket string, namespace []string) map[string]string {
properties = normalizeNamespaceProperties(properties)
if _, ok := properties[namespaceLocationProperty]; ok {
return properties
}
if def := defaultNamespaceLocation(bucket, namespace); def != "" {
properties[namespaceLocationProperty] = def
}
return properties
}