snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)
Some checks failed
go: build dev binaries / cleanup (push) Successful in 4s
Codespell / Check for spelling errors (push) Failing after 3s
docker: build dev containers / build-rust-binaries (amd64, x86_64-unknown-linux-musl) (push) Failing after 3s
docker: build dev containers / build-dev-containers (push) Has been skipped
docker: build dev containers / build-rust-binaries (arm64, true, aarch64-unknown-linux-musl) (push) Failing after 3s
End to End / FUSE Mount (push) Failing after 2s
EC Integration Tests / EC Integration Tests (push) Failing after 30s
FUSE DLM Integration Tests / FUSE DLM Integration Tests (push) Failing after 3s
FUSE Volume Server Failover Tests / FUSE Volume Server Failover (push) Failing after 3s
FUSE Integration Tests / FUSE Integration Testing (push) Failing after 2s
FUSE P2P Peer Chunk Sharing Integration Tests / FUSE P2P Peer Chunk Sharing (push) Failing after 3s
go: build binary / Go Vet (push) Failing after 2s
go: build binary / Go Vet 32-bit (push) Failing after 3s
go: build binary / Build (push) Failing after 4s
go: build binary / Build cross-platform (freebsd/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (darwin/arm64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/arm64) (push) Failing after 3s
go: build binary / Test (push) Failing after 2s
go: build binary / Test 32-bit (push) Failing after 3s
helm: lint and test charts / lint-test (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (11) (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (17) (push) Failing after 3s
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Failing after 2s
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Failing after 3s
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been skipped
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been skipped
Java Client Unit Tests / Java Unit Tests (11) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (17) (push) Failing after 2s
Java Client Unit Tests / Java Unit Tests (21) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (8) (push) Failing after 3s
Kafka Quick Test (Load Test with Schema Registry) / Kafka Client Load Test (Quick) (push) Failing after 3s
Kafka Gateway Tests / Kafka Unit Tests (unit-tests-1) (push) Failing after 11s
Kafka Gateway Tests / Kafka Integration Tests (Critical) (integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka End-to-End Tests (with SMQ) (e2e-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Consumer Group Tests (Highly Isolated) (consumer-group-1) (push) Failing after 2s
Kafka Gateway Tests / Kafka Client Compatibility (with SMQ) (client-compat-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka SMQ Integration Tests (Full Stack) (smq-integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Protocol Tests (Isolated) (protocol-1) (push) Failing after 3s
KMS Tests / KMS Provider Integration Tests (push) Failing after 2s
KMS Tests / S3 KMS End-to-End Tests (push) Failing after 3s
Master Cold Start Tests / Master Cold Start Tests (push) Failing after 4s
Metadata Subscribe Integration Tests / Metadata Subscribe Integration Tests (push) Failing after 3s
Multi-Master Tests / Multi-Master Failover Tests (push) Failing after 3s
Performance / CPU and Heap Profile (push) Failing after 2s
Performance / S3 Read/Write Benchmark (rust volume) (push) Failing after 3s
pjdfstest POSIX Compliance / pjdfstest (push) Failing after 2s
Plugin Worker Integration Tests / Plugin Worker: erasure_coding (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: vacuum (push) Failing after 3s
Rust Volume Server Tests / Rust Integration Tests (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 1) (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 1) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 3) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Distributed Tests (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Performance Tests (push) Failing after 3s
Performance / Throughput Benchmark (go volume) (push) Failing after 3s
Performance / Throughput Benchmark (rust volume) (push) Failing after 2s
Performance / S3 Read/Write Benchmark (go volume) (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: volume_balance (push) Failing after 3s
PostgreSQL Gateway Tests / PostgreSQL Basic Tests (push) Failing after 3s
Rust Volume Server Tests / Rust Unit Tests (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 3) (push) Failing after 3s
Rust Plugin Worker Tests / Rust Plugin Worker Build and Unit Tests (push) Failing after 3s
rust: build dev volume server binaries / cleanup (push) Successful in 3s
S3 ETag and ACL Tests / S3 ETag + ACL Integration Tests (push) Failing after 3s
S3 IAM Integration Tests / IAM Unit Tests (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (advanced) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (basic) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (group) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (policy-enforcement) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (sts) (push) Failing after 4s
S3 Keycloak Integration Tests / S3 Keycloak Integration Tests (push) Failing after 3s
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.11) (push) Failing after 3s
S3 PyArrow Parquet Tests / Go Unit Tests (Implicit Directory) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.12) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-amd64, x86_64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-arm64, aarch64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-linux (linux-amd64, x86_64-unknown-linux-gnu) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.9) (push) Has been cancelled
S3 Proxy Signature Tests / S3 Proxy Signature Verification Tests (push) Has been cancelled
S3 SDK V2 Route Disambiguation Tests / S3 SDK V2 Routing Tests (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (quick) (push) Has been cancelled
S3 SSE Tests / S3 SSE Compatibility Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Metadata Persistence Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Copy Operations Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Multipart Upload Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Performance Test (push) Has been cancelled
S3 SSE Tests / S3 Volume Encryption Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (comprehensive) (push) Has been cancelled
Spark Integration Tests / Spark Integration Tests (push) Failing after 2s
Telemetry Integration Tests / telemetry-integration-test (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 1) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 1) (push) Failing after 3s
Ceph S3 tests / Basic S3 tests (KV store) (push) Has been cancelled
Ceph S3 tests / S3 Versioning & Object Lock tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Unit Tests (push) Failing after 3s
Ceph S3 tests / S3 CORS tests (push) Has been cancelled
Ceph S3 tests / SeaweedFS Custom S3 Copy tests (push) Has been cancelled
Ceph S3 tests / Basic S3 tests (SQL store) (push) Has been cancelled
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
TLS Rotation Integration Tests / TLS Rotation Integration Tests (push) Has been cancelled
Vacuum Integration Tests / Vacuum Integration Tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Variables Integration Tests (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (advanced-policy) (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (basic-policy) (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (contextual-policy) (push) Failing after 4s
S3 Policy Integration Tests / Trusted Proxy Detection Tests (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Shell Integration Tests (push) Failing after 3s
Samba on FUSE Integration / samba-integration (push) Failing after 2s
SFTP Integration Tests / SFTP Integration Testing (push) Failing after 3s
terraform: validate and test modules / fmt, validate, plan-level tests (push) Failing after 3s
terraform: validate and test modules / local cluster smoke test (real weed) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 3) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 2) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 2) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 3) (push) Failing after 5s
mount: windows conformance / WinFsp conformance (push) Has been cancelled
mount: windows / Mount on Windows (push) Has been cancelled
Some checks failed
go: build dev binaries / cleanup (push) Successful in 4s
Codespell / Check for spelling errors (push) Failing after 3s
docker: build dev containers / build-rust-binaries (amd64, x86_64-unknown-linux-musl) (push) Failing after 3s
docker: build dev containers / build-dev-containers (push) Has been skipped
docker: build dev containers / build-rust-binaries (arm64, true, aarch64-unknown-linux-musl) (push) Failing after 3s
End to End / FUSE Mount (push) Failing after 2s
EC Integration Tests / EC Integration Tests (push) Failing after 30s
FUSE DLM Integration Tests / FUSE DLM Integration Tests (push) Failing after 3s
FUSE Volume Server Failover Tests / FUSE Volume Server Failover (push) Failing after 3s
FUSE Integration Tests / FUSE Integration Testing (push) Failing after 2s
FUSE P2P Peer Chunk Sharing Integration Tests / FUSE P2P Peer Chunk Sharing (push) Failing after 3s
go: build binary / Go Vet (push) Failing after 2s
go: build binary / Go Vet 32-bit (push) Failing after 3s
go: build binary / Build (push) Failing after 4s
go: build binary / Build cross-platform (freebsd/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/amd64) (push) Failing after 3s
go: build binary / Build cross-platform (darwin/arm64) (push) Failing after 3s
go: build binary / Build cross-platform (windows/arm64) (push) Failing after 3s
go: build binary / Test (push) Failing after 2s
go: build binary / Test 32-bit (push) Failing after 3s
helm: lint and test charts / lint-test (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (11) (push) Failing after 3s
Java Client Integration Tests / Java Integration Tests (17) (push) Failing after 3s
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Failing after 2s
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Failing after 3s
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been skipped
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been skipped
Java Client Unit Tests / Java Unit Tests (11) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (17) (push) Failing after 2s
Java Client Unit Tests / Java Unit Tests (21) (push) Failing after 3s
Java Client Unit Tests / Java Unit Tests (8) (push) Failing after 3s
Kafka Quick Test (Load Test with Schema Registry) / Kafka Client Load Test (Quick) (push) Failing after 3s
Kafka Gateway Tests / Kafka Unit Tests (unit-tests-1) (push) Failing after 11s
Kafka Gateway Tests / Kafka Integration Tests (Critical) (integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka End-to-End Tests (with SMQ) (e2e-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Consumer Group Tests (Highly Isolated) (consumer-group-1) (push) Failing after 2s
Kafka Gateway Tests / Kafka Client Compatibility (with SMQ) (client-compat-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka SMQ Integration Tests (Full Stack) (smq-integration-1) (push) Failing after 3s
Kafka Gateway Tests / Kafka Protocol Tests (Isolated) (protocol-1) (push) Failing after 3s
KMS Tests / KMS Provider Integration Tests (push) Failing after 2s
KMS Tests / S3 KMS End-to-End Tests (push) Failing after 3s
Master Cold Start Tests / Master Cold Start Tests (push) Failing after 4s
Metadata Subscribe Integration Tests / Metadata Subscribe Integration Tests (push) Failing after 3s
Multi-Master Tests / Multi-Master Failover Tests (push) Failing after 3s
Performance / CPU and Heap Profile (push) Failing after 2s
Performance / S3 Read/Write Benchmark (rust volume) (push) Failing after 3s
pjdfstest POSIX Compliance / pjdfstest (push) Failing after 2s
Plugin Worker Integration Tests / Plugin Worker: erasure_coding (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: vacuum (push) Failing after 3s
Rust Volume Server Tests / Rust Integration Tests (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 1) (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 1) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 2) (push) Failing after 3s
Rust Volume Server Tests / Go Tests with Rust Volume (grpc - Shard 3) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Distributed Tests (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Performance Tests (push) Failing after 3s
Performance / Throughput Benchmark (go volume) (push) Failing after 3s
Performance / Throughput Benchmark (rust volume) (push) Failing after 2s
Performance / S3 Read/Write Benchmark (go volume) (push) Failing after 3s
Plugin Worker Integration Tests / Plugin Worker: volume_balance (push) Failing after 3s
PostgreSQL Gateway Tests / PostgreSQL Basic Tests (push) Failing after 3s
Rust Volume Server Tests / Rust Unit Tests (push) Failing after 2s
Rust Volume Server Tests / Go Tests with Rust Volume (http - Shard 3) (push) Failing after 3s
Rust Plugin Worker Tests / Rust Plugin Worker Build and Unit Tests (push) Failing after 3s
rust: build dev volume server binaries / cleanup (push) Successful in 3s
S3 ETag and ACL Tests / S3 ETag + ACL Integration Tests (push) Failing after 3s
S3 IAM Integration Tests / IAM Unit Tests (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (advanced) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (basic) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (group) (push) Failing after 3s
S3 IAM Integration Tests / S3 IAM Integration Tests (policy-enforcement) (push) Failing after 2s
S3 IAM Integration Tests / S3 IAM Integration Tests (sts) (push) Failing after 4s
S3 Keycloak Integration Tests / S3 Keycloak Integration Tests (push) Failing after 3s
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.11) (push) Failing after 3s
S3 PyArrow Parquet Tests / Go Unit Tests (Implicit Directory) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.12) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-amd64, x86_64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-darwin (darwin-arm64, aarch64-apple-darwin) (push) Has been cancelled
rust: build dev volume server binaries / build-rust-volume-dev-linux (linux-amd64, x86_64-unknown-linux-gnu) (push) Has been cancelled
S3 PyArrow Parquet Tests / PyArrow Parquet Tests (Python 3.9) (push) Has been cancelled
S3 Proxy Signature Tests / S3 Proxy Signature Verification Tests (push) Has been cancelled
S3 SDK V2 Route Disambiguation Tests / S3 SDK V2 Routing Tests (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (quick) (push) Has been cancelled
S3 SSE Tests / S3 SSE Compatibility Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Metadata Persistence Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Copy Operations Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Multipart Upload Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Performance Test (push) Has been cancelled
S3 SSE Tests / S3 Volume Encryption Test (push) Has been cancelled
S3 SSE Tests / S3 SSE Integration Tests (comprehensive) (push) Has been cancelled
Spark Integration Tests / Spark Integration Tests (push) Failing after 2s
Telemetry Integration Tests / telemetry-integration-test (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 1) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 1) (push) Failing after 3s
Ceph S3 tests / Basic S3 tests (KV store) (push) Has been cancelled
Ceph S3 tests / S3 Versioning & Object Lock tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Unit Tests (push) Failing after 3s
Ceph S3 tests / S3 CORS tests (push) Has been cancelled
Ceph S3 tests / SeaweedFS Custom S3 Copy tests (push) Has been cancelled
Ceph S3 tests / Basic S3 tests (SQL store) (push) Has been cancelled
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
TLS Rotation Integration Tests / TLS Rotation Integration Tests (push) Has been cancelled
Vacuum Integration Tests / Vacuum Integration Tests (push) Has been cancelled
S3 Policy Integration Tests / S3 Policy Variables Integration Tests (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (advanced-policy) (push) Failing after 3s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (basic-policy) (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Enforcement Integration Tests (contextual-policy) (push) Failing after 4s
S3 Policy Integration Tests / Trusted Proxy Detection Tests (push) Failing after 2s
S3 Policy Integration Tests / S3 Policy Shell Integration Tests (push) Failing after 3s
Samba on FUSE Integration / samba-integration (push) Failing after 2s
SFTP Integration Tests / SFTP Integration Testing (push) Failing after 3s
terraform: validate and test modules / fmt, validate, plan-level tests (push) Failing after 3s
terraform: validate and test modules / local cluster smoke test (real weed) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 3) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (grpc - Shard 2) (push) Failing after 3s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 2) (push) Failing after 2s
Volume Server Integration Tests / Volume Server Integration Tests (http - Shard 3) (push) Failing after 5s
mount: windows conformance / WinFsp conformance (push) Has been cancelled
mount: windows / Mount on Windows (push) Has been cancelled
This commit is contained in:
254
weed/s3api/iceberg/commit_helpers.go
Normal file
254
weed/s3api/iceberg/commit_helpers.go
Normal file
@@ -0,0 +1,254 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/rand/v2"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/google/uuid"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
const requirementAssertCreate = "assert-create"
|
||||
|
||||
// sleepBeforeCommitRetry backs off between commit attempts, jittered so that
|
||||
// writers that collided do not line up again on the next try.
|
||||
func sleepBeforeCommitRetry(attempt int) {
|
||||
jitter := time.Duration(rand.Int64N(int64(25 * time.Millisecond)))
|
||||
time.Sleep(time.Duration(50*attempt)*time.Millisecond + jitter)
|
||||
}
|
||||
|
||||
// stageCommitMetadata writes the metadata file a commit will point the catalog
|
||||
// at, returning the name and location it landed on. The exclusive create keeps
|
||||
// one writer from overwriting another's file; when the name is already taken
|
||||
// the metadata goes to a unique one instead of failing, because the file there
|
||||
// may be an orphan left by an interrupted commit, and refusing would wedge the
|
||||
// table until an orphan sweep ran. The catalog pointer, not the file name,
|
||||
// decides which commit won. This mirrors how the maintenance worker stages its
|
||||
// own metadata.
|
||||
func (s *Server) stageCommitMetadata(ctx context.Context, metadataBucket, metadataPath, location, metadataFileName string, metadataBytes []byte) (string, string, error) {
|
||||
err := s.saveMetadataFile(ctx, metadataBucket, metadataPath, metadataFileName, metadataBytes, true)
|
||||
if errors.Is(err, filer_pb.ErrEntryAlreadyExists) {
|
||||
// v{N}-{uuid}, a form both the catalog and the maintenance worker
|
||||
// still read the version from.
|
||||
metadataFileName = fmt.Sprintf("%s-%s.metadata.json", strings.TrimSuffix(metadataFileName, ".metadata.json"), uuid.NewString())
|
||||
glog.V(1).Infof("Iceberg: metadata file already staged for %s, using %s", location, metadataFileName)
|
||||
err = s.saveMetadataFile(ctx, metadataBucket, metadataPath, metadataFileName, metadataBytes, true)
|
||||
}
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
return metadataFileName, fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(location, "/"), metadataFileName), nil
|
||||
}
|
||||
|
||||
type icebergRequestError struct {
|
||||
status int
|
||||
errType string
|
||||
message string
|
||||
}
|
||||
|
||||
type createOnCommitInput struct {
|
||||
bucketARN string
|
||||
markerBucket string
|
||||
namespace []string
|
||||
tableName string
|
||||
identityName string
|
||||
location string
|
||||
tableUUID uuid.UUID
|
||||
baseMetadata table.Metadata
|
||||
baseMetadataLoc string
|
||||
baseMetadataVer int
|
||||
updates table.Updates
|
||||
statisticsUpdates []statisticsUpdate
|
||||
}
|
||||
|
||||
func isS3TablesConflict(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
if errors.Is(err, s3tables.ErrVersionTokenMismatch) {
|
||||
return true
|
||||
}
|
||||
var tableErr *s3tables.S3TablesError
|
||||
return errors.As(err, &tableErr) && tableErr.Type == s3tables.ErrCodeConflict
|
||||
}
|
||||
|
||||
func isS3TablesNotFound(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
if strings.Contains(strings.ToLower(err.Error()), "not found") {
|
||||
return true
|
||||
}
|
||||
var tableErr *s3tables.S3TablesError
|
||||
return errors.As(err, &tableErr) &&
|
||||
(tableErr.Type == s3tables.ErrCodeNoSuchTable || tableErr.Type == s3tables.ErrCodeNoSuchNamespace || strings.Contains(strings.ToLower(tableErr.Message), "not found"))
|
||||
}
|
||||
|
||||
func hasAssertCreateRequirement(requirements table.Requirements) bool {
|
||||
for _, requirement := range requirements {
|
||||
if requirement.GetType() == requirementAssertCreate {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func isS3TablesAlreadyExists(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
if strings.Contains(strings.ToLower(err.Error()), "already exists") {
|
||||
return true
|
||||
}
|
||||
var tableErr *s3tables.S3TablesError
|
||||
return errors.As(err, &tableErr) &&
|
||||
(tableErr.Type == s3tables.ErrCodeTableAlreadyExists || tableErr.Type == s3tables.ErrCodeNamespaceAlreadyExists || strings.Contains(strings.ToLower(tableErr.Message), "already exists"))
|
||||
}
|
||||
|
||||
func (s *Server) finalizeCreateOnCommit(ctx context.Context, input createOnCommitInput) (*CommitTableResponse, *icebergRequestError) {
|
||||
builder, err := table.MetadataBuilderFromBase(input.baseMetadata, input.baseMetadataLoc)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusInternalServerError,
|
||||
errType: "InternalServerError",
|
||||
message: "Failed to create metadata builder: " + err.Error(),
|
||||
}
|
||||
}
|
||||
for _, update := range input.updates {
|
||||
if err := update.Apply(builder); err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusBadRequest,
|
||||
errType: "BadRequestException",
|
||||
message: "Failed to apply update: " + err.Error(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
newMetadata, err := builder.Build()
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusBadRequest,
|
||||
errType: "BadRequestException",
|
||||
message: "Failed to build new metadata: " + err.Error(),
|
||||
}
|
||||
}
|
||||
|
||||
metadataVersion := input.baseMetadataVer + 1
|
||||
if metadataVersion <= 0 {
|
||||
metadataVersion = 1
|
||||
}
|
||||
metadataFileName := fmt.Sprintf("v%d.metadata.json", metadataVersion)
|
||||
newMetadataLocation := fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(input.location, "/"), metadataFileName)
|
||||
|
||||
metadataBytes, err := json.Marshal(newMetadata)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusInternalServerError,
|
||||
errType: "InternalServerError",
|
||||
message: "Failed to serialize metadata: " + err.Error(),
|
||||
}
|
||||
}
|
||||
metadataBytes, err = applyStatisticsUpdates(metadataBytes, input.statisticsUpdates)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusBadRequest,
|
||||
errType: "BadRequestException",
|
||||
message: "Failed to apply statistics updates: " + err.Error(),
|
||||
}
|
||||
}
|
||||
metadataBytes = refreshDefaultNameMapping(metadataBytes, newMetadata)
|
||||
// Same spec-compliance fixup we apply on create-table; ensures
|
||||
// v{N}.metadata.json files written through this create-on-commit path are
|
||||
// also readable by strict Iceberg clients reading directly from S3.
|
||||
metadataBytes = ensureMetadataSpecCompliance(metadataBytes)
|
||||
newMetadata, err = table.ParseMetadataBytes(metadataBytes)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusInternalServerError,
|
||||
errType: "InternalServerError",
|
||||
message: "Failed to parse committed metadata: " + err.Error(),
|
||||
}
|
||||
}
|
||||
|
||||
metadataBucket, metadataPath, err := parseS3Location(input.location)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusInternalServerError,
|
||||
errType: "InternalServerError",
|
||||
message: "Invalid table location: " + err.Error(),
|
||||
}
|
||||
}
|
||||
if err := confineMetadataLocation(metadataBucket, metadataPath, input.markerBucket); err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusBadRequest,
|
||||
errType: "BadRequestException",
|
||||
message: err.Error(),
|
||||
}
|
||||
}
|
||||
if err := s.saveMetadataFile(ctx, metadataBucket, metadataPath, metadataFileName, metadataBytes, false); err != nil {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusInternalServerError,
|
||||
errType: "InternalServerError",
|
||||
message: "Failed to save metadata file: " + err.Error(),
|
||||
}
|
||||
}
|
||||
|
||||
createReq := &s3tables.CreateTableRequest{
|
||||
TableBucketARN: input.bucketARN,
|
||||
Namespace: input.namespace,
|
||||
Name: input.tableName,
|
||||
Format: "ICEBERG",
|
||||
Metadata: &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{
|
||||
TableUUID: input.tableUUID.String(),
|
||||
},
|
||||
FullMetadata: metadataBytes,
|
||||
},
|
||||
MetadataVersion: metadataVersion,
|
||||
MetadataLocation: newMetadataLocation,
|
||||
}
|
||||
createErr := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(ctx, mgrClient, "CreateTable", createReq, nil, input.identityName)
|
||||
})
|
||||
if createErr != nil {
|
||||
if cleanupErr := s.deleteMetadataFile(ctx, metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to cleanup metadata file %s after create-on-commit failure: %v", newMetadataLocation, cleanupErr)
|
||||
}
|
||||
if isS3TablesConflict(createErr) || isS3TablesAlreadyExists(createErr) {
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusConflict,
|
||||
errType: "CommitFailedException",
|
||||
message: "Table was created concurrently",
|
||||
}
|
||||
}
|
||||
glog.Errorf("Iceberg: CommitTable CreateTable error: %v", createErr)
|
||||
return nil, &icebergRequestError{
|
||||
status: http.StatusInternalServerError,
|
||||
errType: "InternalServerError",
|
||||
message: "Failed to commit table creation: " + createErr.Error(),
|
||||
}
|
||||
}
|
||||
|
||||
markerBucket := input.markerBucket
|
||||
if markerBucket == "" {
|
||||
markerBucket = metadataBucket
|
||||
}
|
||||
if markerErr := s.deleteStageCreateMarkers(ctx, markerBucket, input.namespace, input.tableName); markerErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to cleanup stage-create markers for %s.%s after finalize: %v", flattenNamespacePath(input.namespace), input.tableName, markerErr)
|
||||
}
|
||||
|
||||
return &CommitTableResponse{
|
||||
MetadataLocation: newMetadataLocation,
|
||||
Metadata: newMetadata,
|
||||
}, nil
|
||||
}
|
||||
173
weed/s3api/iceberg/commit_updates.go
Normal file
173
weed/s3api/iceberg/commit_updates.go
Normal file
@@ -0,0 +1,173 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
|
||||
"github.com/apache/iceberg-go/table"
|
||||
)
|
||||
|
||||
type statisticsUpdate struct {
|
||||
set *table.StatisticsFile
|
||||
remove *int64
|
||||
}
|
||||
|
||||
var ErrIncompleteSetStatistics = errors.New("set-statistics requires snapshot-id, statistics-path, file-size-in-bytes, and file-footer-size-in-bytes")
|
||||
|
||||
type commitAction struct {
|
||||
Action string `json:"action"`
|
||||
}
|
||||
|
||||
type setStatisticsUpdate struct {
|
||||
Action string `json:"action"`
|
||||
SnapshotID *int64 `json:"snapshot-id,omitempty"`
|
||||
StatisticsPath string `json:"statistics-path,omitempty"`
|
||||
FileSizeInBytes *int64 `json:"file-size-in-bytes,omitempty"`
|
||||
FileFooterSizeInBytes *int64 `json:"file-footer-size-in-bytes,omitempty"`
|
||||
KeyMetadata *string `json:"key-metadata,omitempty"`
|
||||
BlobMetadata []table.BlobMetadata `json:"blob-metadata,omitempty"`
|
||||
Statistics *table.StatisticsFile `json:"statistics,omitempty"`
|
||||
}
|
||||
|
||||
func (u *setStatisticsUpdate) asStatisticsFile() (*table.StatisticsFile, error) {
|
||||
if u.Statistics != nil {
|
||||
if u.Statistics.BlobMetadata == nil {
|
||||
u.Statistics.BlobMetadata = []table.BlobMetadata{}
|
||||
}
|
||||
return u.Statistics, nil
|
||||
}
|
||||
if u.SnapshotID == nil || u.StatisticsPath == "" || u.FileSizeInBytes == nil || u.FileFooterSizeInBytes == nil {
|
||||
return nil, ErrIncompleteSetStatistics
|
||||
}
|
||||
|
||||
stats := &table.StatisticsFile{
|
||||
SnapshotID: *u.SnapshotID,
|
||||
StatisticsPath: u.StatisticsPath,
|
||||
FileSizeInBytes: *u.FileSizeInBytes,
|
||||
FileFooterSizeInBytes: *u.FileFooterSizeInBytes,
|
||||
KeyMetadata: u.KeyMetadata,
|
||||
BlobMetadata: u.BlobMetadata,
|
||||
}
|
||||
if stats.BlobMetadata == nil {
|
||||
stats.BlobMetadata = []table.BlobMetadata{}
|
||||
}
|
||||
return stats, nil
|
||||
}
|
||||
|
||||
type removeStatisticsUpdate struct {
|
||||
Action string `json:"action"`
|
||||
SnapshotID int64 `json:"snapshot-id"`
|
||||
}
|
||||
|
||||
func parseCommitUpdates(rawUpdates []json.RawMessage) (table.Updates, []statisticsUpdate, error) {
|
||||
filtered := make([]json.RawMessage, 0, len(rawUpdates))
|
||||
statisticsUpdates := make([]statisticsUpdate, 0)
|
||||
|
||||
for _, raw := range rawUpdates {
|
||||
var action commitAction
|
||||
if err := json.Unmarshal(raw, &action); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
switch action.Action {
|
||||
case "set-statistics":
|
||||
var setUpdate setStatisticsUpdate
|
||||
if err := json.Unmarshal(raw, &setUpdate); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
stats, err := setUpdate.asStatisticsFile()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
statisticsUpdates = append(statisticsUpdates, statisticsUpdate{set: stats})
|
||||
case "remove-statistics":
|
||||
var removeUpdate removeStatisticsUpdate
|
||||
if err := json.Unmarshal(raw, &removeUpdate); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
snapshotID := removeUpdate.SnapshotID
|
||||
statisticsUpdates = append(statisticsUpdates, statisticsUpdate{remove: &snapshotID})
|
||||
default:
|
||||
filtered = append(filtered, raw)
|
||||
}
|
||||
}
|
||||
|
||||
if len(filtered) == 0 {
|
||||
return nil, statisticsUpdates, nil
|
||||
}
|
||||
|
||||
data, err := json.Marshal(filtered)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
var updates table.Updates
|
||||
if err := json.Unmarshal(data, &updates); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
return updates, statisticsUpdates, nil
|
||||
}
|
||||
|
||||
func applyStatisticsUpdates(metadataBytes []byte, updates []statisticsUpdate) ([]byte, error) {
|
||||
if len(updates) == 0 {
|
||||
return metadataBytes, nil
|
||||
}
|
||||
|
||||
var metadata map[string]json.RawMessage
|
||||
if err := json.Unmarshal(metadataBytes, &metadata); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var statistics []table.StatisticsFile
|
||||
if rawStatistics, ok := metadata["statistics"]; ok && len(rawStatistics) > 0 {
|
||||
if err := json.Unmarshal(rawStatistics, &statistics); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
statisticsBySnapshot := make(map[int64]table.StatisticsFile, len(statistics))
|
||||
orderedSnapshotIDs := make([]int64, 0, len(statistics))
|
||||
inOrder := make(map[int64]bool, len(statistics))
|
||||
for _, stat := range statistics {
|
||||
statisticsBySnapshot[stat.SnapshotID] = stat
|
||||
if !inOrder[stat.SnapshotID] {
|
||||
orderedSnapshotIDs = append(orderedSnapshotIDs, stat.SnapshotID)
|
||||
inOrder[stat.SnapshotID] = true
|
||||
}
|
||||
}
|
||||
|
||||
for _, update := range updates {
|
||||
if update.set != nil {
|
||||
statisticsBySnapshot[update.set.SnapshotID] = *update.set
|
||||
if !inOrder[update.set.SnapshotID] {
|
||||
orderedSnapshotIDs = append(orderedSnapshotIDs, update.set.SnapshotID)
|
||||
inOrder[update.set.SnapshotID] = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
if update.remove != nil {
|
||||
delete(statisticsBySnapshot, *update.remove)
|
||||
}
|
||||
}
|
||||
|
||||
statistics = make([]table.StatisticsFile, 0, len(statisticsBySnapshot))
|
||||
for _, snapshotID := range orderedSnapshotIDs {
|
||||
stat, ok := statisticsBySnapshot[snapshotID]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
statistics = append(statistics, stat)
|
||||
}
|
||||
|
||||
if len(statistics) == 0 {
|
||||
delete(metadata, "statistics")
|
||||
} else {
|
||||
data, err := json.Marshal(statistics)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
metadata["statistics"] = data
|
||||
}
|
||||
|
||||
return json.Marshal(metadata)
|
||||
}
|
||||
404
weed/s3api/iceberg/handlers_commit.go
Normal file
404
weed/s3api/iceberg/handlers_commit.go
Normal file
@@ -0,0 +1,404 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"path"
|
||||
"strings"
|
||||
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/google/uuid"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// handleUpdateTable commits updates to a table.
|
||||
// Implements the Iceberg REST Catalog commit protocol.
|
||||
func (s *Server) handleUpdateTable(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
tableName := vars["table"]
|
||||
|
||||
if len(namespace) == 0 || tableName == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and table name are required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Extract identity from context
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
// Parse commit request and keep statistics updates separate because iceberg-go v0.4.0
|
||||
// does not decode set/remove-statistics update actions yet.
|
||||
var raw struct {
|
||||
Identifier *TableIdentifier `json:"identifier,omitempty"`
|
||||
Requirements json.RawMessage `json:"requirements"`
|
||||
Updates []json.RawMessage `json:"updates"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&raw); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
var req CommitTableRequest
|
||||
req.Identifier = raw.Identifier
|
||||
var statisticsUpdates []statisticsUpdate
|
||||
if len(raw.Requirements) > 0 {
|
||||
normalized, err := normalizeRequirements(raw.Requirements)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid requirements: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := json.Unmarshal(normalized, &req.Requirements); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid requirements: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
if len(raw.Updates) > 0 {
|
||||
var err error
|
||||
req.Updates, statisticsUpdates, err = parseCommitUpdates(raw.Updates)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid updates: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
// Manifest repair runs once, as soon as the table location is known; on
|
||||
// commit retries the updates already reference the repaired files. Repair
|
||||
// is best effort end to end: the originals parsed already, so a repair
|
||||
// that fails to re-parse is discarded rather than failing the commit.
|
||||
manifestsRepaired := false
|
||||
repairManifests := func(location string) {
|
||||
if manifestsRepaired {
|
||||
return
|
||||
}
|
||||
manifestsRepaired = true
|
||||
repaired, changed := s.repairAddSnapshotManifests(r.Context(), location, raw.Updates)
|
||||
if !changed {
|
||||
return
|
||||
}
|
||||
repairedUpdates, repairedStatistics, err := parseCommitUpdates(repaired)
|
||||
if err != nil {
|
||||
glog.Warningf("Iceberg: repaired updates failed to parse, keeping originals: %v", err)
|
||||
return
|
||||
}
|
||||
raw.Updates = repaired
|
||||
req.Updates = repairedUpdates
|
||||
statisticsUpdates = repairedStatistics
|
||||
}
|
||||
|
||||
maxCommitAttempts := 3
|
||||
generatedLegacyUUID := uuid.New()
|
||||
stageCreateEnabled := isStageCreateEnabled()
|
||||
for attempt := 1; attempt <= maxCommitAttempts; attempt++ {
|
||||
getReq := &s3tables.GetTableRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
Name: tableName,
|
||||
}
|
||||
var getResp s3tables.GetTableResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetTable", getReq, &getResp, identityName)
|
||||
})
|
||||
if err != nil {
|
||||
if isS3TablesNotFound(err) {
|
||||
location := fmt.Sprintf("s3://%s/%s", bucketName, path.Join(flattenNamespacePath(namespace), tableName))
|
||||
tableUUID := generatedLegacyUUID
|
||||
baseMetadataVersion := 0
|
||||
baseMetadataLocation := ""
|
||||
var baseMetadata table.Metadata
|
||||
|
||||
var latestMarker *stageCreateMarker
|
||||
if stageCreateEnabled {
|
||||
var markerErr error
|
||||
latestMarker, markerErr = s.loadLatestStageCreateMarker(r.Context(), bucketName, namespace, tableName)
|
||||
if markerErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to load stage-create marker: "+markerErr.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
if latestMarker != nil {
|
||||
if latestMarker.Location != "" {
|
||||
location = strings.TrimSuffix(latestMarker.Location, "/")
|
||||
}
|
||||
if latestMarker.TableUUID != "" {
|
||||
if parsedUUID, parseErr := uuid.Parse(latestMarker.TableUUID); parseErr == nil {
|
||||
tableUUID = parsedUUID
|
||||
}
|
||||
}
|
||||
|
||||
stagedMetadataLocation := latestMarker.StagedMetadataLocation
|
||||
if stagedMetadataLocation == "" {
|
||||
stagedMetadataLocation = fmt.Sprintf("%s/metadata/v1.metadata.json", strings.TrimSuffix(location, "/"))
|
||||
}
|
||||
stagedLocation := tableLocationFromMetadataLocation(stagedMetadataLocation)
|
||||
stagedFileName := path.Base(stagedMetadataLocation)
|
||||
stagedBucket, stagedPath, parseLocationErr := parseS3Location(stagedLocation)
|
||||
if parseLocationErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid staged metadata location: "+parseLocationErr.Error())
|
||||
return
|
||||
}
|
||||
if err := confineMetadataLocation(stagedBucket, stagedPath, bucketName); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
stagedMetadataBytes, loadErr := s.loadMetadataFile(r.Context(), stagedBucket, stagedPath, stagedFileName)
|
||||
if loadErr != nil {
|
||||
if !errors.Is(loadErr, filer_pb.ErrNotFound) {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to load staged metadata: "+loadErr.Error())
|
||||
return
|
||||
}
|
||||
} else if len(stagedMetadataBytes) > 0 {
|
||||
stagedMetadata, parseErr := table.ParseMetadataBytes(stagedMetadataBytes)
|
||||
if parseErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to parse staged metadata: "+parseErr.Error())
|
||||
return
|
||||
}
|
||||
// Staged metadata is only a template for table creation; commit starts from version 1.
|
||||
baseMetadata = stagedMetadata
|
||||
baseMetadataLocation = ""
|
||||
baseMetadataVersion = 0
|
||||
if stagedMetadata.TableUUID() != uuid.Nil {
|
||||
tableUUID = stagedMetadata.TableUUID()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
hasAssertCreate := hasAssertCreateRequirement(req.Requirements)
|
||||
hasStagedTemplate := baseMetadata != nil
|
||||
if !(stageCreateEnabled && (hasAssertCreate || hasStagedTemplate)) {
|
||||
writeError(w, http.StatusNotFound, "NoSuchTableException", fmt.Sprintf("Table does not exist: %s", tableName))
|
||||
return
|
||||
}
|
||||
// From here the commit creates the table, writing its metadata
|
||||
// file before the create that authorizes it.
|
||||
if authErr := s.authorizeCreateTable(r.Context(), bucketARN, namespace, tableName, identityName); authErr != nil {
|
||||
writeManagerError(w, authErr)
|
||||
return
|
||||
}
|
||||
|
||||
for _, requirement := range req.Requirements {
|
||||
validateAgainst := table.Metadata(nil)
|
||||
if hasStagedTemplate && requirement.GetType() != requirementAssertCreate {
|
||||
validateAgainst = baseMetadata
|
||||
}
|
||||
if requirementErr := requirement.Validate(validateAgainst); requirementErr != nil {
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", "Requirement failed: "+requirementErr.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if baseMetadata == nil {
|
||||
var buildErr error
|
||||
if baseMetadata, buildErr = newTableMetadata(tableUUID, location, nil, nil, nil, nil); buildErr != nil {
|
||||
glog.Errorf("Iceberg: CommitTable placeholder metadata for %s: %v", tableName, buildErr)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to build current metadata")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
createBucket, createPath, createLocErr := parseS3Location(location)
|
||||
if createLocErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid table location: "+createLocErr.Error())
|
||||
return
|
||||
}
|
||||
if err := confineMetadataLocation(createBucket, createPath, bucketName); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
repairManifests(location)
|
||||
|
||||
result, reqErr := s.finalizeCreateOnCommit(r.Context(), createOnCommitInput{
|
||||
bucketARN: bucketARN,
|
||||
markerBucket: bucketName,
|
||||
namespace: namespace,
|
||||
tableName: tableName,
|
||||
identityName: identityName,
|
||||
location: location,
|
||||
tableUUID: tableUUID,
|
||||
baseMetadata: baseMetadata,
|
||||
baseMetadataLoc: baseMetadataLocation,
|
||||
baseMetadataVer: baseMetadataVersion,
|
||||
updates: req.Updates,
|
||||
statisticsUpdates: statisticsUpdates,
|
||||
})
|
||||
if reqErr != nil {
|
||||
writeError(w, reqErr.status, reqErr.errType, reqErr.message)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: CommitTable GetTable error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
location := tableLocationFromMetadataLocation(getResp.MetadataLocation)
|
||||
if location == "" {
|
||||
location = fmt.Sprintf("s3://%s/%s", bucketName, path.Join(flattenNamespacePath(namespace), tableName))
|
||||
}
|
||||
locBucket, locPath, locErr := parseS3Location(location)
|
||||
if locErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid table location: "+locErr.Error())
|
||||
return
|
||||
}
|
||||
if err := confineMetadataLocation(locBucket, locPath, bucketName); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
tableUUID := uuid.Nil
|
||||
if getResp.Metadata != nil && getResp.Metadata.Iceberg != nil && getResp.Metadata.Iceberg.TableUUID != "" {
|
||||
if parsed, parseErr := uuid.Parse(getResp.Metadata.Iceberg.TableUUID); parseErr == nil {
|
||||
tableUUID = parsed
|
||||
}
|
||||
}
|
||||
if tableUUID == uuid.Nil {
|
||||
tableUUID = generatedLegacyUUID
|
||||
}
|
||||
|
||||
var currentMetadata table.Metadata
|
||||
if getResp.Metadata != nil && len(getResp.Metadata.FullMetadata) > 0 {
|
||||
currentMetadata, err = table.ParseMetadataBytes(getResp.Metadata.FullMetadata)
|
||||
if err != nil {
|
||||
glog.Errorf("Iceberg: Failed to parse current metadata for %s: %v", tableName, err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to parse current metadata")
|
||||
return
|
||||
}
|
||||
} else {
|
||||
currentMetadata, err = newTableMetadata(tableUUID, location, nil, nil, nil, nil)
|
||||
if err != nil {
|
||||
glog.Errorf("Iceberg: CommitTable placeholder metadata for %s: %v", tableName, err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to build current metadata")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
for _, requirement := range req.Requirements {
|
||||
if err := requirement.Validate(currentMetadata); err != nil {
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", "Requirement failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
repairManifests(location)
|
||||
|
||||
builder, err := table.MetadataBuilderFromBase(currentMetadata, getResp.MetadataLocation)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to create metadata builder: "+err.Error())
|
||||
return
|
||||
}
|
||||
for _, update := range req.Updates {
|
||||
if err := update.Apply(builder); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to apply update: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
newMetadata, err := builder.Build()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to build new metadata: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
metadataVersion := getResp.MetadataVersion + 1
|
||||
metadataFileName := fmt.Sprintf("v%d.metadata.json", metadataVersion)
|
||||
newMetadataLocation := fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(location, "/"), metadataFileName)
|
||||
|
||||
metadataBytes, err := json.Marshal(newMetadata)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to serialize metadata: "+err.Error())
|
||||
return
|
||||
}
|
||||
// iceberg-go does not currently support set/remove-statistics updates in MetadataBuilder.
|
||||
// Patch the encoded metadata JSON and parse it back to keep the response object consistent.
|
||||
metadataBytes, err = applyStatisticsUpdates(metadataBytes, statisticsUpdates)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to apply statistics updates: "+err.Error())
|
||||
return
|
||||
}
|
||||
metadataBytes = refreshDefaultNameMapping(metadataBytes, newMetadata)
|
||||
// Same spec-compliance fixup we apply on create-table; ensures
|
||||
// v{N}.metadata.json files written during commit are also readable by
|
||||
// strict Iceberg clients reading directly from S3, and that the
|
||||
// FullMetadata persisted in S3Tables stays consistent.
|
||||
metadataBytes = ensureMetadataSpecCompliance(metadataBytes)
|
||||
newMetadata, err = table.ParseMetadataBytes(metadataBytes)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to parse committed metadata: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
metadataBucket, metadataPath, err := parseS3Location(location)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid table location: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := confineMetadataLocation(metadataBucket, metadataPath, bucketName); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
metadataFileName, newMetadataLocation, err = s.stageCommitMetadata(r.Context(), metadataBucket, metadataPath, location, metadataFileName, metadataBytes)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to save metadata file: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
updateReq := &s3tables.UpdateTableRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
Name: tableName,
|
||||
VersionToken: getResp.VersionToken,
|
||||
Metadata: &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{
|
||||
TableUUID: tableUUID.String(),
|
||||
},
|
||||
FullMetadata: metadataBytes,
|
||||
},
|
||||
MetadataVersion: metadataVersion,
|
||||
MetadataLocation: newMetadataLocation,
|
||||
}
|
||||
|
||||
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "UpdateTable", updateReq, nil, identityName)
|
||||
})
|
||||
if err == nil {
|
||||
result := CommitTableResponse{
|
||||
MetadataLocation: newMetadataLocation,
|
||||
Metadata: newMetadata,
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
return
|
||||
}
|
||||
|
||||
if isS3TablesConflict(err) {
|
||||
if cleanupErr := s.deleteMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to cleanup metadata file %s on conflict: %v", newMetadataLocation, cleanupErr)
|
||||
}
|
||||
if attempt < maxCommitAttempts {
|
||||
glog.V(1).Infof("Iceberg: CommitTable conflict for %s (attempt %d/%d), retrying", tableName, attempt, maxCommitAttempts)
|
||||
sleepBeforeCommitRetry(attempt)
|
||||
continue
|
||||
}
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", "Version token mismatch")
|
||||
return
|
||||
}
|
||||
|
||||
if cleanupErr := s.deleteMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to cleanup metadata file %s after update failure: %v", newMetadataLocation, cleanupErr)
|
||||
}
|
||||
glog.Errorf("Iceberg: CommitTable UpdateTable error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to commit table update: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
58
weed/s3api/iceberg/handlers_metrics.go
Normal file
58
weed/s3api/iceberg/handlers_metrics.go
Normal file
@@ -0,0 +1,58 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
)
|
||||
|
||||
// Reports carry per-file scan metrics and grow with the table. The content is
|
||||
// discarded either way, so anything past this is accepted without being read
|
||||
// into memory rather than rejected.
|
||||
const maxMetricsReportBytes = 1 << 20
|
||||
|
||||
// handleReportMetrics accepts the scan and commit reports engines send after
|
||||
// planning or committing. The catalog keeps no metrics store, but the endpoint
|
||||
// has to exist: clients that get a 404 here log an error per query, and some
|
||||
// treat repeated failures as a broken catalog.
|
||||
func (s *Server) handleReportMetrics(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
tableName := vars["table"]
|
||||
|
||||
if len(namespace) == 0 || tableName == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and table name are required")
|
||||
return
|
||||
}
|
||||
|
||||
// One byte past the limit distinguishes a report that fits from one that
|
||||
// was cut short, which would otherwise fail to parse and look malformed.
|
||||
body, err := io.ReadAll(io.LimitReader(r.Body, maxMetricsReportBytes+1))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
|
||||
return
|
||||
}
|
||||
if len(body) > maxMetricsReportBytes {
|
||||
glog.V(3).Infof("Iceberg: discarding oversized metrics report for %s.%s", flattenNamespacePath(namespace), tableName)
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
|
||||
var report struct {
|
||||
ReportType string `json:"report-type"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &report); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body: "+err.Error())
|
||||
return
|
||||
}
|
||||
if report.ReportType == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "report-type is required")
|
||||
return
|
||||
}
|
||||
|
||||
glog.V(3).Infof("Iceberg: metrics report %q for %s.%s", report.ReportType, flattenNamespacePath(namespace), tableName)
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
63
weed/s3api/iceberg/handlers_metrics_test.go
Normal file
63
weed/s3api/iceberg/handlers_metrics_test.go
Normal file
@@ -0,0 +1,63 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
)
|
||||
|
||||
func postMetrics(t *testing.T, body string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
r := httptest.NewRequest(http.MethodPost, "/v1/namespaces/ns/tables/t/metrics", strings.NewReader(body))
|
||||
r = mux.SetURLVars(r, map[string]string{"namespace": "ns", "table": "t"})
|
||||
w := httptest.NewRecorder()
|
||||
(&Server{}).handleReportMetrics(w, r)
|
||||
return w
|
||||
}
|
||||
|
||||
func TestHandleReportMetricsAcceptsScanReport(t *testing.T) {
|
||||
w := postMetrics(t, `{"report-type":"scan-report","table-name":"t","snapshot-id":1}`)
|
||||
if w.Code != http.StatusNoContent {
|
||||
t.Errorf("status = %d, want %d: %s", w.Code, http.StatusNoContent, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleReportMetricsRejectsGarbage(t *testing.T) {
|
||||
w := postMetrics(t, `not json`)
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want %d", w.Code, http.StatusBadRequest)
|
||||
}
|
||||
}
|
||||
|
||||
// The spec makes the body and its report-type required.
|
||||
func TestHandleReportMetricsRequiresReportType(t *testing.T) {
|
||||
for _, body := range []string{``, `{}`, `{"report-type":""}`} {
|
||||
if w := postMetrics(t, body); w.Code != http.StatusBadRequest {
|
||||
t.Errorf("body %q: status = %d, want %d", body, w.Code, http.StatusBadRequest)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A report larger than the read limit is still a valid report; truncating it
|
||||
// and then failing to parse would hand the client an error for a query that
|
||||
// worked.
|
||||
func TestHandleReportMetricsAcceptsOversizedReport(t *testing.T) {
|
||||
padding := strings.Repeat("x", maxMetricsReportBytes)
|
||||
w := postMetrics(t, `{"report-type":"scan-report","padding":"`+padding+`"}`)
|
||||
if w.Code != http.StatusNoContent {
|
||||
t.Errorf("status = %d, want %d: %s", w.Code, http.StatusNoContent, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleReportMetricsRequiresTable(t *testing.T) {
|
||||
r := httptest.NewRequest(http.MethodPost, "/v1/namespaces/ns/tables//metrics", strings.NewReader(`{}`))
|
||||
r = mux.SetURLVars(r, map[string]string{"namespace": "ns"})
|
||||
w := httptest.NewRecorder()
|
||||
(&Server{}).handleReportMetrics(w, r)
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want %d", w.Code, http.StatusBadRequest)
|
||||
}
|
||||
}
|
||||
389
weed/s3api/iceberg/handlers_namespace.go
Normal file
389
weed/s3api/iceberg/handlers_namespace.go
Normal file
@@ -0,0 +1,389 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// handleConfig returns catalog configuration.
|
||||
//
|
||||
// When a client passes ?warehouse=, the Iceberg REST spec expects the server
|
||||
// to echo a catalog identifier back as overrides.prefix so subsequent calls
|
||||
// use /v1/{prefix}/... and land on the right table bucket. Without this,
|
||||
// clients like DuckDB's ATTACH flow fall back to an unprefixed path that
|
||||
// resolves to the wrong bucket and report phantom "schema does not exist"
|
||||
// errors.
|
||||
func (s *Server) handleConfig(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
config := CatalogConfig{
|
||||
Defaults: map[string]string{},
|
||||
Overrides: map[string]string{},
|
||||
}
|
||||
if warehouse := strings.TrimSpace(r.URL.Query().Get("warehouse")); warehouse != "" {
|
||||
// Only the bucket portion of the warehouse is meaningful today —
|
||||
// SeaweedFS table-bucket routing is bucket-scoped, so any sub-path
|
||||
// (e.g. s3://bucket/prefix/) is ignored here and clients that try to
|
||||
// scope a catalog under a sub-prefix will still land on the bucket.
|
||||
if bucket := resolveWarehouseBucket(warehouse); bucket != "" {
|
||||
config.Overrides["prefix"] = bucket
|
||||
// Echo the location form regardless of how the client spelled the
|
||||
// warehouse: clients derive default table locations from it.
|
||||
config.Defaults["warehouse"] = "s3://" + bucket
|
||||
}
|
||||
}
|
||||
if err := json.NewEncoder(w).Encode(config); err != nil {
|
||||
glog.Warningf("handleConfig: Failed to encode config: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// handleListNamespaces lists namespaces in a catalog.
|
||||
func (s *Server) handleListNamespaces(w http.ResponseWriter, r *http.Request) {
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Extract identity from context
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
pageToken, pageSize, err := parsePagination(r)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// The Iceberg REST spec allows a "parent" query parameter for hierarchical
|
||||
// namespace listing. Convert it to the dot-separated prefix used by S3 Tables.
|
||||
var prefix string
|
||||
if parent := r.URL.Query().Get("parent"); parent != "" {
|
||||
parentParts := parseNamespace(parent)
|
||||
if len(parentParts) > 0 {
|
||||
prefix = flattenNamespacePath(parentParts) + "."
|
||||
}
|
||||
}
|
||||
|
||||
// Use S3 Tables manager to list namespaces
|
||||
var resp s3tables.ListNamespacesResponse
|
||||
req := &s3tables.ListNamespacesRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Prefix: prefix,
|
||||
ContinuationToken: pageToken,
|
||||
MaxNamespaces: pageSize,
|
||||
}
|
||||
|
||||
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "ListNamespaces", req, &resp, identityName)
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
glog.Infof("Iceberg: ListNamespaces error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
// Convert to Iceberg format
|
||||
namespaces := make([]Namespace, 0, len(resp.Namespaces))
|
||||
for _, ns := range resp.Namespaces {
|
||||
namespaces = append(namespaces, Namespace(ns.Namespace))
|
||||
}
|
||||
|
||||
result := ListNamespacesResponse{
|
||||
NextPageToken: resp.ContinuationToken,
|
||||
Namespaces: namespaces,
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleCreateNamespace creates a new namespace.
|
||||
func (s *Server) handleCreateNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Extract identity from context
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
var req CreateNamespaceRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
if len(req.Namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
// Use S3 Tables manager to create namespace
|
||||
createReq := &s3tables.CreateNamespaceRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: req.Namespace,
|
||||
Properties: normalizeNamespaceProperties(req.Properties),
|
||||
}
|
||||
var createResp s3tables.CreateNamespaceResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
glog.V(2).Infof("Iceberg: handleCreateNamespace calling Execute with identityName=%s", identityName)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "CreateNamespace", createReq, &createResp, identityName)
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "already exists") {
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", err.Error())
|
||||
return
|
||||
}
|
||||
glog.Errorf("Iceberg: CreateNamespace error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
result := CreateNamespaceResponse{
|
||||
Namespace: Namespace(createResp.Namespace),
|
||||
Properties: withDefaultNamespaceLocation(createResp.Properties, bucketName, createResp.Namespace),
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleGetNamespace gets namespace metadata.
|
||||
func (s *Server) handleGetNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Extract identity from context
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
// Use S3 Tables manager to get namespace
|
||||
getReq := &s3tables.GetNamespaceRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
}
|
||||
var getResp s3tables.GetNamespaceResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetNamespace", getReq, &getResp, identityName)
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: GetNamespace error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
result := GetNamespaceResponse{
|
||||
Namespace: Namespace(getResp.Namespace),
|
||||
Properties: withDefaultNamespaceLocation(getResp.Properties, bucketName, getResp.Namespace),
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// applyNamespacePropertyUpdates applies removals and updates to a copy of
|
||||
// current and returns the merged property map alongside the Iceberg REST
|
||||
// summary of which keys were removed, updated, or were missing.
|
||||
func applyNamespacePropertyUpdates(current map[string]string, removals []string, updates map[string]string) (map[string]string, UpdateNamespacePropertiesResponse) {
|
||||
properties := make(map[string]string, len(current))
|
||||
for k, v := range current {
|
||||
properties[k] = v
|
||||
}
|
||||
|
||||
summary := UpdateNamespacePropertiesResponse{Removed: []string{}, Updated: []string{}, Missing: []string{}}
|
||||
seen := make(map[string]struct{}, len(removals))
|
||||
for _, key := range removals {
|
||||
if _, dup := seen[key]; dup {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
if _, ok := properties[key]; ok {
|
||||
delete(properties, key)
|
||||
summary.Removed = append(summary.Removed, key)
|
||||
} else {
|
||||
summary.Missing = append(summary.Missing, key)
|
||||
}
|
||||
}
|
||||
for key, value := range updates {
|
||||
properties[key] = value
|
||||
summary.Updated = append(summary.Updated, key)
|
||||
}
|
||||
return properties, summary
|
||||
}
|
||||
|
||||
// handleUpdateNamespaceProperties applies a set of removals and updates to a
|
||||
// namespace's properties and returns which keys were removed, updated, or
|
||||
// missing, per the Iceberg REST spec.
|
||||
func (s *Server) handleUpdateNamespaceProperties(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
var req UpdateNamespacePropertiesRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
// A key cannot be both removed and updated in the same request.
|
||||
for _, key := range req.Removals {
|
||||
if _, ok := req.Updates[key]; ok {
|
||||
writeError(w, http.StatusUnprocessableEntity, "UnprocessableEntityException",
|
||||
fmt.Sprintf("Cannot remove and update the same key: %s", key))
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
// Load the current properties so we can compute the summary and the merged map.
|
||||
getReq := &s3tables.GetNamespaceRequest{TableBucketARN: bucketARN, Namespace: namespace}
|
||||
var getResp s3tables.GetNamespaceResponse
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetNamespace", getReq, &getResp, identityName)
|
||||
})
|
||||
if err != nil {
|
||||
writeNamespaceManagerError(w, err, namespace)
|
||||
return
|
||||
}
|
||||
|
||||
properties, summary := applyNamespacePropertyUpdates(getResp.Properties, req.Removals, req.Updates)
|
||||
|
||||
updReq := &s3tables.UpdateNamespaceRequest{TableBucketARN: bucketARN, Namespace: namespace, Properties: properties}
|
||||
var updResp s3tables.UpdateNamespaceResponse
|
||||
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "UpdateNamespace", updReq, &updResp, identityName)
|
||||
})
|
||||
if err != nil {
|
||||
writeNamespaceManagerError(w, err, namespace)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, summary)
|
||||
}
|
||||
|
||||
// writeNamespaceManagerError maps an s3tables manager error to the matching
|
||||
// Iceberg REST response. A namespace dropped between read and write surfaces
|
||||
// as 404 and a denied caller as 403; anything else is a 500.
|
||||
func writeNamespaceManagerError(w http.ResponseWriter, err error, namespace Namespace) {
|
||||
var s3Err *s3tables.S3TablesError
|
||||
if errors.As(err, &s3Err) {
|
||||
switch s3Err.Type {
|
||||
case s3tables.ErrCodeNoSuchNamespace:
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
|
||||
return
|
||||
case s3tables.ErrCodeAccessDenied:
|
||||
writeError(w, http.StatusForbidden, "ForbiddenException", "Not authorized to update namespace properties")
|
||||
return
|
||||
}
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: UpdateNamespaceProperties error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
}
|
||||
|
||||
// handleNamespaceExists checks if a namespace exists.
|
||||
func (s *Server) handleNamespaceExists(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Extract identity from context
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
getReq := &s3tables.GetNamespaceRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
}
|
||||
var getResp s3tables.GetNamespaceResponse
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetNamespace", getReq, &getResp, identityName)
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
if nameValidationError(err) {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: NamespaceExists error: %v", err)
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleDropNamespace deletes a namespace.
|
||||
func (s *Server) handleDropNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
|
||||
// Extract identity from context
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
deleteReq := &s3tables.DeleteNamespaceRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
}
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "DeleteNamespace", deleteReq, nil, identityName)
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
|
||||
return
|
||||
}
|
||||
if strings.Contains(err.Error(), "not empty") {
|
||||
writeError(w, http.StatusConflict, "NamespaceNotEmptyException", "Namespace is not empty")
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: DropNamespace error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
397
weed/s3api/iceberg/handlers_oauth.go
Normal file
397
weed/s3api/iceberg/handlers_oauth.go
Normal file
@@ -0,0 +1,397 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
jwt "github.com/golang-jwt/jwt/v5"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
)
|
||||
|
||||
// OAuthTokenResponse is the response for POST /v1/oauth/tokens.
|
||||
type OAuthTokenResponse struct {
|
||||
AccessToken string `json:"access_token"`
|
||||
TokenType string `json:"token_type"`
|
||||
ExpiresIn int `json:"expires_in"`
|
||||
Scope string `json:"scope,omitempty"`
|
||||
// IssuedTokenType is set to the access-token URN on token-exchange
|
||||
// responses, where RFC 8693 requires it.
|
||||
IssuedTokenType string `json:"issued_token_type,omitempty"`
|
||||
}
|
||||
|
||||
// OAuthErrorResponse is the error response for the OAuth endpoint.
|
||||
type OAuthErrorResponse struct {
|
||||
Error string `json:"error"`
|
||||
Description string `json:"error_description,omitempty"`
|
||||
}
|
||||
|
||||
// IcebergClaims are JWT claims for Iceberg catalog OAuth tokens.
|
||||
type IcebergClaims struct {
|
||||
IdentityName string `json:"identity_name"`
|
||||
AccessKey string `json:"access_key"`
|
||||
jwt.RegisteredClaims
|
||||
}
|
||||
|
||||
const defaultOauthTokenExpiry = 3600
|
||||
|
||||
// maxOauthTokenExpiry bounds the configured TTL so seconds-to-Duration
|
||||
// conversions cannot overflow into already-expired tokens.
|
||||
const maxOauthTokenExpiry = 365 * 24 * 3600
|
||||
|
||||
// accessTokenTokenType is the RFC 8693 token-type URN for access tokens,
|
||||
// required in issued_token_type on token-exchange responses.
|
||||
const accessTokenTokenType = "urn:ietf:params:oauth:token-type:access_token"
|
||||
|
||||
// grant types accepted by POST /v1/oauth/tokens. Iceberg Java 1.10.x
|
||||
// refreshes tokens via token-exchange (exchangeEnabled defaults true), so a
|
||||
// server that only accepts client_credentials leaves those clients unable
|
||||
// to refresh.
|
||||
const (
|
||||
grantTypeClientCredentials = "client_credentials"
|
||||
grantTypeTokenExchange = "urn:ietf:params:oauth:grant-type:token-exchange"
|
||||
)
|
||||
|
||||
// oauthExpirySeconds returns the OAuth token TTL. Deployments whose
|
||||
// clients cannot refresh tokens on 401 can raise this to survive client
|
||||
// restart cycles instead of dying every hour.
|
||||
func oauthExpirySeconds() int {
|
||||
if v := os.Getenv("ICEBERG_OAUTH_TOKEN_EXPIRY"); v != "" {
|
||||
// Parse in 64-bit space: on 32-bit platforms Atoi overflows and
|
||||
// errors on oversized values, which would silently fall back to
|
||||
// the default instead of clamping.
|
||||
if n, err := strconv.ParseInt(v, 10, 64); err == nil && n > 0 {
|
||||
if n > maxOauthTokenExpiry {
|
||||
return maxOauthTokenExpiry
|
||||
}
|
||||
return int(n)
|
||||
}
|
||||
}
|
||||
return defaultOauthTokenExpiry
|
||||
}
|
||||
|
||||
// tokenExchangeGrace is how long an expired subject token may still be
|
||||
// exchanged for a fresh access token. Signature verification is the real
|
||||
// gate; the grace exists so a client holding a token that expired while the
|
||||
// exchange grant was unsupported (or during a server outage) recovers
|
||||
// without a process restart.
|
||||
func tokenExchangeGrace() time.Duration {
|
||||
grace := 2 * oauthExpirySeconds()
|
||||
if grace < 3600 {
|
||||
grace = 3600
|
||||
}
|
||||
if grace > 86400 {
|
||||
grace = 86400
|
||||
}
|
||||
return time.Duration(grace) * time.Second
|
||||
}
|
||||
|
||||
// handleOAuthTokens implements the OAuth2 client_credentials flow.
|
||||
// POST /v1/oauth/tokens
|
||||
func (s *Server) handleOAuthTokens(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_request", "Could not parse form body")
|
||||
return
|
||||
}
|
||||
|
||||
// Reject credentials in query string to prevent leaking secrets into logs and caches.
|
||||
if r.URL.Query().Get("client_secret") != "" {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_request", "client_secret must not be sent in the URL")
|
||||
return
|
||||
}
|
||||
|
||||
grantType := r.PostFormValue("grant_type")
|
||||
switch grantType {
|
||||
case grantTypeClientCredentials:
|
||||
// handled below
|
||||
case grantTypeTokenExchange, "token_exchange":
|
||||
s.handleTokenExchange(w, r)
|
||||
return
|
||||
default:
|
||||
writeOAuthError(w, http.StatusBadRequest, "unsupported_grant_type",
|
||||
fmt.Sprintf("Unsupported grant_type: %s", grantType))
|
||||
return
|
||||
}
|
||||
|
||||
clientID := r.PostFormValue("client_id")
|
||||
clientSecret := r.PostFormValue("client_secret")
|
||||
|
||||
// Also support HTTP Basic auth per OAuth2 spec
|
||||
if clientID == "" && clientSecret == "" {
|
||||
var ok bool
|
||||
clientID, clientSecret, ok = r.BasicAuth()
|
||||
if !ok {
|
||||
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "Missing client credentials")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if clientID == "" || clientSecret == "" {
|
||||
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "Missing client_id or client_secret")
|
||||
return
|
||||
}
|
||||
|
||||
if s.credentialValidator == nil {
|
||||
writeOAuthError(w, http.StatusInternalServerError, "server_error", "Credential validation not configured")
|
||||
return
|
||||
}
|
||||
|
||||
identityName, _, err := s.credentialValidator.ValidateS3Credential(clientID, clientSecret)
|
||||
if err != nil {
|
||||
glog.V(2).Infof("Iceberg OAuth: credential validation failed for client_id=%s: %v", clientID, err)
|
||||
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "Invalid client credentials")
|
||||
return
|
||||
}
|
||||
|
||||
// Generate a JWT signed with a key derived from the client secret.
|
||||
// Include the access key in claims so we can look up the exact credential for verification.
|
||||
tokenString, err := mintIcebergToken(identityName, clientID, clientSecret, oauthExpirySeconds())
|
||||
if err != nil {
|
||||
glog.Errorf("Iceberg OAuth: failed to sign token: %v", err)
|
||||
writeOAuthError(w, http.StatusInternalServerError, "server_error", "Failed to generate token")
|
||||
return
|
||||
}
|
||||
|
||||
scope := r.PostFormValue("scope")
|
||||
resp := OAuthTokenResponse{
|
||||
AccessToken: tokenString,
|
||||
TokenType: "bearer",
|
||||
ExpiresIn: oauthExpirySeconds(),
|
||||
Scope: scope,
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
// handleTokenExchange implements RFC 8693 token exchange for Iceberg REST
|
||||
// clients: a previously issued access token (subject_token) is exchanged for
|
||||
// a fresh one. Iceberg Java's OAuth2Manager refreshes via this grant
|
||||
// (exchangeEnabled defaults true), so supporting it lets those clients
|
||||
// self-heal before their token expires — no client restart needed.
|
||||
//
|
||||
// Client authentication (RFC 8693 §2.1) is accepted but not required:
|
||||
// Iceberg Java's refreshExpiredToken sends Basic credentials with the
|
||||
// exchange, while its proactive scheduled refresh sends only the Bearer
|
||||
// session headers. An expired subject_token is only exchangeable with valid
|
||||
// client credentials, and an unauthenticated exchange never extends the
|
||||
// token's lifetime beyond the subject_token's own expiry.
|
||||
func (s *Server) handleTokenExchange(w http.ResponseWriter, r *http.Request) {
|
||||
if s.credentialValidator == nil {
|
||||
writeOAuthError(w, http.StatusInternalServerError, "server_error", "Credential validation not configured")
|
||||
return
|
||||
}
|
||||
|
||||
subjectToken := r.PostFormValue("subject_token")
|
||||
if subjectToken == "" {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_request", "Missing subject_token")
|
||||
return
|
||||
}
|
||||
|
||||
// Verify the subject token by signature (exp checked separately against
|
||||
// the recovery grace).
|
||||
unverified := &IcebergClaims{}
|
||||
parser := jwt.NewParser(jwt.WithoutClaimsValidation())
|
||||
if _, _, err := parser.ParseUnverified(subjectToken, unverified); err != nil {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
|
||||
return
|
||||
}
|
||||
if unverified.AccessKey == "" || unverified.Issuer != "seaweedfs-iceberg" {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
|
||||
return
|
||||
}
|
||||
identityName, _, secretKey, err := s.credentialValidator.GetCredentialByAccessKey(unverified.AccessKey)
|
||||
if err != nil {
|
||||
glog.V(2).Infof("Iceberg OAuth: token exchange failed to get credential for access key: %v", err)
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
|
||||
return
|
||||
}
|
||||
|
||||
// Optional client authentication. When present, the client must
|
||||
// authenticate with the same credential that issued the subject_token.
|
||||
clientID, clientSecret, hasBasic := r.BasicAuth()
|
||||
if !hasBasic {
|
||||
clientID = r.PostFormValue("client_id")
|
||||
clientSecret = r.PostFormValue("client_secret")
|
||||
hasBasic = clientID != "" && clientSecret != ""
|
||||
}
|
||||
clientAuthenticated := false
|
||||
if hasBasic {
|
||||
if _, _, err := s.credentialValidator.ValidateS3Credential(clientID, clientSecret); err != nil {
|
||||
glog.V(2).Infof("Iceberg OAuth: token exchange client authentication failed for client_id=%s: %v", clientID, err)
|
||||
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "Invalid client credentials")
|
||||
return
|
||||
}
|
||||
if clientID != unverified.AccessKey {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "subject_token was issued to a different client")
|
||||
return
|
||||
}
|
||||
clientAuthenticated = true
|
||||
}
|
||||
|
||||
signingKey := deriveSigningKey(unverified.AccessKey, secretKey)
|
||||
claims := &IcebergClaims{}
|
||||
parsed, err := jwt.ParseWithClaims(subjectToken, claims, func(token *jwt.Token) (interface{}, error) {
|
||||
if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
|
||||
return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
|
||||
}
|
||||
return signingKey, nil
|
||||
}, jwt.WithoutClaimsValidation())
|
||||
if err != nil || !parsed.Valid {
|
||||
glog.V(2).Infof("Iceberg OAuth: token exchange signature verification failed: %v", err)
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
|
||||
return
|
||||
}
|
||||
if claims.Issuer != "seaweedfs-iceberg" {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "Invalid subject_token")
|
||||
return
|
||||
}
|
||||
if claims.ExpiresAt == nil {
|
||||
// Tokens minted by this server always carry an expiry; a subject
|
||||
// token without one must not be exchangeable forever.
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "subject_token has no expiry")
|
||||
return
|
||||
}
|
||||
|
||||
ttlSeconds := oauthExpirySeconds()
|
||||
if time.Now().After(claims.ExpiresAt.Time) {
|
||||
// Expired subject tokens are only exchangeable by an authenticated
|
||||
// client, within the recovery grace.
|
||||
if !clientAuthenticated {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant",
|
||||
"subject_token is expired; client authentication is required to exchange it")
|
||||
return
|
||||
}
|
||||
if time.Since(claims.ExpiresAt.Time) > tokenExchangeGrace() {
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "subject_token expired beyond the exchange grace window")
|
||||
return
|
||||
}
|
||||
} else if remaining := int(time.Until(claims.ExpiresAt.Time).Seconds()); !clientAuthenticated && remaining < ttlSeconds {
|
||||
// Unauthenticated exchange (proactive refresh carrying only the
|
||||
// Bearer) must not extend the lifetime past the subject's expiry;
|
||||
// otherwise a leaked token could chain-refresh forever. An
|
||||
// authenticated client gets a fresh full TTL — it could mint one
|
||||
// via client_credentials anyway.
|
||||
ttlSeconds = remaining
|
||||
}
|
||||
if ttlSeconds <= 0 {
|
||||
// A subject token with under a second left would otherwise be
|
||||
// exchanged for a token that is born expired.
|
||||
writeOAuthError(w, http.StatusBadRequest, "invalid_grant", "subject_token has no remaining lifetime")
|
||||
return
|
||||
}
|
||||
|
||||
tokenString, err := mintIcebergToken(identityName, unverified.AccessKey, secretKey, ttlSeconds)
|
||||
if err != nil {
|
||||
glog.Errorf("Iceberg OAuth: failed to sign exchanged token: %v", err)
|
||||
writeOAuthError(w, http.StatusInternalServerError, "server_error", "Failed to generate token")
|
||||
return
|
||||
}
|
||||
|
||||
resp := OAuthTokenResponse{
|
||||
AccessToken: tokenString,
|
||||
TokenType: "bearer",
|
||||
ExpiresIn: ttlSeconds,
|
||||
Scope: r.PostFormValue("scope"),
|
||||
IssuedTokenType: accessTokenTokenType,
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
// mintIcebergToken issues a signed access token for the given identity and
|
||||
// credential, valid for ttlSeconds.
|
||||
func mintIcebergToken(identityName, accessKey, secret string, ttlSeconds int) (string, error) {
|
||||
signingKey := deriveSigningKey(accessKey, secret)
|
||||
now := time.Now()
|
||||
claims := IcebergClaims{
|
||||
IdentityName: identityName,
|
||||
AccessKey: accessKey,
|
||||
RegisteredClaims: jwt.RegisteredClaims{
|
||||
IssuedAt: jwt.NewNumericDate(now),
|
||||
ExpiresAt: jwt.NewNumericDate(now.Add(time.Duration(ttlSeconds) * time.Second)),
|
||||
Issuer: "seaweedfs-iceberg",
|
||||
},
|
||||
}
|
||||
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
|
||||
return token.SignedString(signingKey)
|
||||
}
|
||||
|
||||
// authenticateBearer validates a Bearer token from the Authorization header.
|
||||
// Returns the identity name, identity object, and whether auth succeeded.
|
||||
func (s *Server) authenticateBearer(r *http.Request) (string, interface{}, bool) {
|
||||
auth := r.Header.Get("Authorization")
|
||||
if auth == "" {
|
||||
return "", nil, false
|
||||
}
|
||||
if !strings.HasPrefix(strings.ToLower(auth), "bearer ") {
|
||||
return "", nil, false
|
||||
}
|
||||
tokenString := strings.TrimSpace(auth[7:])
|
||||
if tokenString == "" {
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
if s.credentialValidator == nil {
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
// Parse the token without verification first to get the access key,
|
||||
// then look up the exact credential to verify the signature.
|
||||
parser := jwt.NewParser(jwt.WithoutClaimsValidation())
|
||||
unverified := &IcebergClaims{}
|
||||
_, _, err := parser.ParseUnverified(tokenString, unverified)
|
||||
if err != nil {
|
||||
glog.V(2).Infof("Iceberg OAuth: failed to parse token: %v", err)
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
if unverified.AccessKey == "" {
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
// Look up the credential by access key to get the signing key for verification
|
||||
identityName, identity, secretKey, err := s.credentialValidator.GetCredentialByAccessKey(unverified.AccessKey)
|
||||
if err != nil {
|
||||
glog.V(2).Infof("Iceberg OAuth: failed to get credential for access key: %v", err)
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
signingKey := deriveSigningKey(unverified.AccessKey, secretKey)
|
||||
claims := &IcebergClaims{}
|
||||
verified, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
|
||||
if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
|
||||
return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
|
||||
}
|
||||
return signingKey, nil
|
||||
})
|
||||
if err != nil || !verified.Valid {
|
||||
glog.V(2).Infof("Iceberg OAuth: token verification failed: %v", err)
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
return identityName, identity, true
|
||||
}
|
||||
|
||||
// deriveSigningKey derives a signing key from the access key and secret using HMAC-SHA256.
|
||||
// Including the access key prevents cross-credential token forgery when two
|
||||
// credentials happen to share the same secret.
|
||||
func deriveSigningKey(accessKey, secret string) []byte {
|
||||
h := hmac.New(sha256.New, []byte("seaweedfs-iceberg-oauth"))
|
||||
h.Write([]byte(accessKey))
|
||||
h.Write([]byte{0}) // null separator
|
||||
h.Write([]byte(secret))
|
||||
return h.Sum(nil)
|
||||
}
|
||||
|
||||
func writeOAuthError(w http.ResponseWriter, status int, errCode, description string) {
|
||||
resp := OAuthErrorResponse{
|
||||
Error: errCode,
|
||||
Description: description,
|
||||
}
|
||||
writeJSON(w, status, resp)
|
||||
}
|
||||
410
weed/s3api/iceberg/handlers_oauth_test.go
Normal file
410
weed/s3api/iceberg/handlers_oauth_test.go
Normal file
@@ -0,0 +1,410 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
jwt "github.com/golang-jwt/jwt/v5"
|
||||
)
|
||||
|
||||
type mockCredentialValidator struct {
|
||||
credentials map[string]string // accessKey -> secretKey
|
||||
identities map[string]string // accessKey -> identityName
|
||||
}
|
||||
|
||||
func (m *mockCredentialValidator) ValidateS3Credential(accessKey, secretKey string) (string, interface{}, error) {
|
||||
expected, ok := m.credentials[accessKey]
|
||||
if !ok {
|
||||
return "", nil, fmt.Errorf("access key not found")
|
||||
}
|
||||
if expected != secretKey {
|
||||
return "", nil, fmt.Errorf("invalid secret key")
|
||||
}
|
||||
return m.identities[accessKey], nil, nil
|
||||
}
|
||||
|
||||
func (m *mockCredentialValidator) GetCredentialByAccessKey(accessKey string) (string, interface{}, string, error) {
|
||||
secret, ok := m.credentials[accessKey]
|
||||
if !ok {
|
||||
return "", nil, "", fmt.Errorf("access key not found")
|
||||
}
|
||||
return m.identities[accessKey], nil, secret, nil
|
||||
}
|
||||
|
||||
func newTestServerWithOAuth() *Server {
|
||||
cv := &mockCredentialValidator{
|
||||
credentials: map[string]string{"AKID123": "secret456"},
|
||||
identities: map[string]string{"AKID123": "testuser"},
|
||||
}
|
||||
s := &Server{
|
||||
credentialValidator: cv,
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func TestHandleOAuthTokens_Success(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
|
||||
body := "grant_type=client_credentials&client_id=AKID123&client_secret=secret456"
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
s.handleOAuthTokens(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
|
||||
var resp OAuthTokenResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if resp.TokenType != "bearer" {
|
||||
t.Errorf("expected token_type=bearer, got %s", resp.TokenType)
|
||||
}
|
||||
if resp.AccessToken == "" {
|
||||
t.Error("expected non-empty access_token")
|
||||
}
|
||||
if resp.ExpiresIn != oauthExpirySeconds() {
|
||||
t.Errorf("expected expires_in=%d, got %d", oauthExpirySeconds(), resp.ExpiresIn)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleOAuthTokens_InvalidCredentials(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
|
||||
body := "grant_type=client_credentials&client_id=AKID123&client_secret=wrongsecret"
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
s.handleOAuthTokens(w, req)
|
||||
|
||||
if w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("expected 401, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleOAuthTokens_UnsupportedGrantType(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
|
||||
body := "grant_type=authorization_code&client_id=AKID123&client_secret=secret456"
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
s.handleOAuthTokens(w, req)
|
||||
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestBearerTokenRoundTrip(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
|
||||
// Get a token
|
||||
body := "grant_type=client_credentials&client_id=AKID123&client_secret=secret456"
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
w := httptest.NewRecorder()
|
||||
s.handleOAuthTokens(w, req)
|
||||
|
||||
var resp OAuthTokenResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Use the token for Bearer auth
|
||||
authReq := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
authReq.Header.Set("Authorization", "Bearer "+resp.AccessToken)
|
||||
|
||||
identityName, _, ok := s.authenticateBearer(authReq)
|
||||
if !ok {
|
||||
t.Fatal("expected Bearer auth to succeed")
|
||||
}
|
||||
if identityName != "testuser" {
|
||||
t.Errorf("expected identity 'testuser', got '%s'", identityName)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBearerTokenInvalid(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
req.Header.Set("Authorization", "Bearer invalid-token")
|
||||
|
||||
_, _, ok := s.authenticateBearer(req)
|
||||
if ok {
|
||||
t.Error("expected Bearer auth to fail with invalid token")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBearerTokenNone(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
|
||||
_, _, ok := s.authenticateBearer(req)
|
||||
if ok {
|
||||
t.Error("expected Bearer auth to fail with no token")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOauthExpirySecondsEnvOverride pins the TTL knob: the token TTL must
|
||||
// be configurable so clients that cannot refresh on 401 can be given
|
||||
// longer-lived tokens instead of dying every hour.
|
||||
func TestOauthExpirySecondsEnvOverride(t *testing.T) {
|
||||
if got := oauthExpirySeconds(); got != defaultOauthTokenExpiry {
|
||||
t.Fatalf("default TTL = %d, want %d", got, defaultOauthTokenExpiry)
|
||||
}
|
||||
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "86400")
|
||||
if got := oauthExpirySeconds(); got != 86400 {
|
||||
t.Fatalf("env TTL = %d, want 86400", got)
|
||||
}
|
||||
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "-5")
|
||||
if got := oauthExpirySeconds(); got != defaultOauthTokenExpiry {
|
||||
t.Fatalf("negative TTL must fall back to default, got %d", got)
|
||||
}
|
||||
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "garbage")
|
||||
if got := oauthExpirySeconds(); got != defaultOauthTokenExpiry {
|
||||
t.Fatalf("invalid TTL must fall back to default, got %d", got)
|
||||
}
|
||||
// oversized values are clamped so Duration math cannot overflow
|
||||
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "99999999999999999")
|
||||
if got := oauthExpirySeconds(); got != maxOauthTokenExpiry {
|
||||
t.Fatalf("oversized TTL must clamp to %d, got %d", maxOauthTokenExpiry, got)
|
||||
}
|
||||
|
||||
// a token minted under an override carries the override's expiry
|
||||
t.Setenv("ICEBERG_OAUTH_TOKEN_EXPIRY", "7200")
|
||||
s := newTestServerWithOAuth()
|
||||
body := "grant_type=client_credentials&client_id=AKID123&client_secret=secret456"
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
w := httptest.NewRecorder()
|
||||
s.handleOAuthTokens(w, req)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d", w.Code)
|
||||
}
|
||||
var resp OAuthTokenResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if resp.ExpiresIn != 7200 {
|
||||
t.Fatalf("minted token expires_in = %d, want 7200", resp.ExpiresIn)
|
||||
}
|
||||
}
|
||||
|
||||
func exchangeRequest(t *testing.T, subjectToken string) *httptest.ResponseRecorder {
|
||||
return exchangeRequestWithBasic(t, subjectToken, "", "")
|
||||
}
|
||||
|
||||
// exchangeRequestWithBasic mirrors Iceberg Java: refreshExpiredToken sends
|
||||
// Basic client credentials with the token-exchange grant, the proactive
|
||||
// scheduled refresh sends only the form body.
|
||||
func exchangeRequestWithBasic(t *testing.T, subjectToken, basicUser, basicPass string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
s := newTestServerWithOAuth()
|
||||
body := "grant_type=urn:ietf:params:oauth:grant-type:token-exchange&subject_token=" + url.QueryEscape(subjectToken)
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
if basicUser != "" || basicPass != "" {
|
||||
req.SetBasicAuth(basicUser, basicPass)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
s.handleOAuthTokens(w, req)
|
||||
return w
|
||||
}
|
||||
|
||||
// TestTokenExchangeLiveSubject: the proactive refresh path Iceberg Java
|
||||
// 1.10.x uses (exchangeEnabled defaults true, Bearer-only headers) must
|
||||
// mint a fresh working token.
|
||||
func TestTokenExchangeLiveSubject(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
now := time.Now()
|
||||
live := mintTestToken(t, "AKID123", "secret456", now, now.Add(30*time.Minute))
|
||||
|
||||
w := exchangeRequest(t, live)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
var resp OAuthTokenResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if resp.TokenType != "bearer" || resp.AccessToken == "" {
|
||||
t.Fatalf("bad token response: %+v", resp)
|
||||
}
|
||||
if resp.IssuedTokenType != accessTokenTokenType {
|
||||
t.Fatalf("issued_token_type = %q, want %s", resp.IssuedTokenType, accessTokenTokenType)
|
||||
}
|
||||
// Unauthenticated exchange must not outlive the subject token.
|
||||
if resp.ExpiresIn > 30*60 {
|
||||
t.Fatalf("expires_in = %d, must be bounded by the subject's remaining lifetime (<= 1800)", resp.ExpiresIn)
|
||||
}
|
||||
// the exchanged token must authenticate like a normal Bearer
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+resp.AccessToken)
|
||||
if _, _, ok := s.authenticateBearer(req); !ok {
|
||||
t.Fatalf("exchanged token must pass authenticateBearer")
|
||||
}
|
||||
}
|
||||
|
||||
// TestTokenExchangeExpiredWithinGrace: a client whose token expired while
|
||||
// exchange was unsupported must recover without a restart. This is the
|
||||
// refreshExpiredToken path, which authenticates with Basic credentials.
|
||||
func TestTokenExchangeExpiredWithinGrace(t *testing.T) {
|
||||
now := time.Now()
|
||||
expiredRecently := mintTestToken(t, "AKID123", "secret456", now.Add(-20*time.Minute), now.Add(-10*time.Minute))
|
||||
|
||||
w := exchangeRequestWithBasic(t, expiredRecently, "AKID123", "secret456")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200 within grace, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
var resp OAuthTokenResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Authenticated refresh gets the full configured TTL.
|
||||
if resp.ExpiresIn != oauthExpirySeconds() {
|
||||
t.Fatalf("expires_in = %d, want %d", resp.ExpiresIn, oauthExpirySeconds())
|
||||
}
|
||||
}
|
||||
|
||||
// TestTokenExchangeAuthenticatedLiveSubjectFullTTL: an authenticated client
|
||||
// exchanging a live token renews the session — the TTL cap only applies to
|
||||
// unauthenticated (Bearer-only) exchanges.
|
||||
func TestTokenExchangeAuthenticatedLiveSubjectFullTTL(t *testing.T) {
|
||||
now := time.Now()
|
||||
live := mintTestToken(t, "AKID123", "secret456", now, now.Add(30*time.Minute))
|
||||
|
||||
w := exchangeRequestWithBasic(t, live, "AKID123", "secret456")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
var resp OAuthTokenResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if resp.ExpiresIn != oauthExpirySeconds() {
|
||||
t.Fatalf("authenticated live exchange expires_in = %d, want full TTL %d", resp.ExpiresIn, oauthExpirySeconds())
|
||||
}
|
||||
}
|
||||
|
||||
// TestTokenExchangeSubjectNearlyExpired: a Bearer-only exchange when the
|
||||
// subject token has under a second left must be rejected, not minted into
|
||||
// an already-expired token (expires_in: 0).
|
||||
func TestTokenExchangeSubjectNearlyExpired(t *testing.T) {
|
||||
// jwt/v5 serializes exp at one-second precision, so pin the expiry to
|
||||
// the next whole-second boundary: the token is always live when minted
|
||||
// and exchanged, yet has under a second of remaining lifetime.
|
||||
exp := time.Now().Truncate(time.Second).Add(time.Second)
|
||||
nearlyDead := mintTestToken(t, "AKID123", "secret456", exp.Add(-time.Hour), exp)
|
||||
|
||||
w := exchangeRequest(t, nearlyDead)
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
if !strings.Contains(w.Body.String(), "invalid_grant") {
|
||||
t.Fatalf("expected invalid_grant, got: %s", w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestTokenExchangeExpiredWithoutClientAuth: a leaked expired token must
|
||||
// not be exchangeable without client credentials.
|
||||
func TestTokenExchangeExpiredWithoutClientAuth(t *testing.T) {
|
||||
now := time.Now()
|
||||
expiredRecently := mintTestToken(t, "AKID123", "secret456", now.Add(-20*time.Minute), now.Add(-10*time.Minute))
|
||||
|
||||
w := exchangeRequest(t, expiredRecently)
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("expected 400, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
if !strings.Contains(w.Body.String(), "invalid_grant") {
|
||||
t.Fatalf("expected invalid_grant, got: %s", w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestTokenExchangeExpiredBeyondGrace: stale tokens far past expiry must not
|
||||
// act as eternal credentials, even with client auth.
|
||||
func TestTokenExchangeExpiredBeyondGrace(t *testing.T) {
|
||||
now := time.Now()
|
||||
longDead := mintTestToken(t, "AKID123", "secret456", now.Add(-48*time.Hour), now.Add(-47*time.Hour))
|
||||
|
||||
w := exchangeRequestWithBasic(t, longDead, "AKID123", "secret456")
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("expected 400 beyond grace, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
if !strings.Contains(w.Body.String(), "invalid_grant") {
|
||||
t.Fatalf("expected invalid_grant, got: %s", w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestTokenExchangeGarbageSubject: malformed subject tokens are rejected
|
||||
// with 400 invalid_grant per RFC 6749 §5.2.
|
||||
func TestTokenExchangeGarbageSubject(t *testing.T) {
|
||||
w := exchangeRequest(t, "not-a-jwt")
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("expected 400, got %d", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTokenExchangeWrongClientCredentials: valid subject token, but Basic
|
||||
// auth with wrong credentials or a different client is rejected.
|
||||
func TestTokenExchangeWrongClientCredentials(t *testing.T) {
|
||||
now := time.Now()
|
||||
live := mintTestToken(t, "AKID123", "secret456", now, now.Add(30*time.Minute))
|
||||
|
||||
// wrong secret → invalid_client 401
|
||||
w := exchangeRequestWithBasic(t, live, "AKID123", "wrongsecret")
|
||||
if w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("wrong secret: expected 401, got %d", w.Code)
|
||||
}
|
||||
|
||||
// a different valid client must not exchange someone else's token
|
||||
s := &Server{credentialValidator: &mockCredentialValidator{
|
||||
credentials: map[string]string{"AKID123": "secret456", "AKID999": "other-secret"},
|
||||
identities: map[string]string{"AKID123": "testuser", "AKID999": "otheruser"},
|
||||
}}
|
||||
body := "grant_type=urn:ietf:params:oauth:grant-type:token-exchange&subject_token=" + url.QueryEscape(live)
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/oauth/tokens", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
req.SetBasicAuth("AKID999", "other-secret")
|
||||
rec := httptest.NewRecorder()
|
||||
s.handleOAuthTokens(rec, req)
|
||||
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "invalid_grant") {
|
||||
t.Fatalf("cross-client exchange: expected 400 invalid_grant, got %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestTokenExchangeSubjectWithoutExpiry: a correctly signed subject token
|
||||
// without an exp claim must be rejected — it would otherwise refresh forever.
|
||||
func TestTokenExchangeSubjectWithoutExpiry(t *testing.T) {
|
||||
claims := IcebergClaims{
|
||||
IdentityName: "testuser",
|
||||
AccessKey: "AKID123",
|
||||
RegisteredClaims: jwt.RegisteredClaims{
|
||||
IssuedAt: jwt.NewNumericDate(time.Now()),
|
||||
Issuer: "seaweedfs-iceberg",
|
||||
// no ExpiresAt
|
||||
},
|
||||
}
|
||||
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
|
||||
signed, err := token.SignedString(deriveSigningKey("AKID123", "secret456"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
w := exchangeRequestWithBasic(t, signed, "AKID123", "secret456")
|
||||
if w.Code != http.StatusBadRequest || !strings.Contains(w.Body.String(), "invalid_grant") {
|
||||
t.Fatalf("expected 400 invalid_grant for nil-exp subject, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
1000
weed/s3api/iceberg/handlers_table.go
Normal file
1000
weed/s3api/iceberg/handlers_table.go
Normal file
File diff suppressed because it is too large
Load Diff
364
weed/s3api/iceberg/handlers_transaction.go
Normal file
364
weed/s3api/iceberg/handlers_transaction.go
Normal file
@@ -0,0 +1,364 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"path"
|
||||
"strings"
|
||||
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/google/uuid"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// CommitTransactionRequest is sent to POST /v1/transactions/commit.
|
||||
type CommitTransactionRequest struct {
|
||||
TableChanges []tableChangeRequest `json:"table-changes"`
|
||||
}
|
||||
|
||||
type tableChangeRequest struct {
|
||||
Identifier *TableIdentifier `json:"identifier"`
|
||||
Requirements json.RawMessage `json:"requirements"`
|
||||
Updates []json.RawMessage `json:"updates"`
|
||||
}
|
||||
|
||||
// preparedTableCommit holds the per-table work resolved during validation so
|
||||
// pointer flips (and their rollback) can run after every table is validated.
|
||||
type preparedTableCommit struct {
|
||||
namespace []string
|
||||
tableName string
|
||||
tableUUID uuid.UUID
|
||||
versionToken string
|
||||
metadataBucket string
|
||||
metadataPath string
|
||||
location string
|
||||
metadataFileName string
|
||||
metadataBytes []byte
|
||||
metadataVersion int
|
||||
newMetadataLoc string
|
||||
|
||||
// prior table state captured before the flip, so a rollback can revert
|
||||
// every field handleUpdateTable would have mutated (not just the location).
|
||||
prevMetadataLoc string
|
||||
prevMetadataVersion int
|
||||
prevMetadata *s3tables.TableMetadata
|
||||
}
|
||||
|
||||
// handleCommitTransaction commits changes to multiple tables in one request.
|
||||
// Validation is atomic (all requirements evaluated before any write); pointer
|
||||
// flips are best-effort with rollback, so this is not crash-atomic.
|
||||
func (s *Server) handleCommitTransaction(w http.ResponseWriter, r *http.Request) {
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
var req CommitTransactionRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body: "+err.Error())
|
||||
return
|
||||
}
|
||||
if len(req.TableChanges) == 0 {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
|
||||
// Phase 1: resolve, load, and validate every table; build new metadata bytes.
|
||||
prepared := make([]preparedTableCommit, 0, len(req.TableChanges))
|
||||
for _, change := range req.TableChanges {
|
||||
if change.Identifier == nil || len(change.Identifier.Namespace) == 0 || change.Identifier.Name == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Each table change requires identifier namespace and name")
|
||||
return
|
||||
}
|
||||
pc, reqErr := s.prepareTableCommit(r.Context(), bucketName, bucketARN, identityName, change)
|
||||
if reqErr != nil {
|
||||
writeError(w, reqErr.status, reqErr.errType, reqErr.message)
|
||||
return
|
||||
}
|
||||
prepared = append(prepared, *pc)
|
||||
}
|
||||
|
||||
// Phase 2: write each new metadata.json object. Staging is exclusive for the
|
||||
// same reason a single-table commit stages exclusively: a transaction racing
|
||||
// another writer on one of its tables would otherwise overwrite that
|
||||
// writer's metadata, and the pointer flip below decides who won.
|
||||
for i := range prepared {
|
||||
pc := &prepared[i]
|
||||
fileName, location, err := s.stageCommitMetadata(r.Context(), pc.metadataBucket, pc.metadataPath, pc.location, pc.metadataFileName, pc.metadataBytes)
|
||||
if err != nil {
|
||||
// No pointer flipped yet, so every written file is safe to delete.
|
||||
s.cleanupPreparedMetadata(r.Context(), prepared[:i+1], nil)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to save metadata file: "+err.Error())
|
||||
return
|
||||
}
|
||||
pc.metadataFileName = fileName
|
||||
pc.newMetadataLoc = location
|
||||
}
|
||||
|
||||
// Phase 3: flip each table's pointer xattr; on failure roll back prior flips.
|
||||
for i := range prepared {
|
||||
pc := &prepared[i]
|
||||
if err := s.flipTablePointer(r.Context(), bucketARN, identityName, pc); err != nil {
|
||||
rolledBack := s.rollbackTablePointers(r.Context(), bucketARN, identityName, prepared[:i])
|
||||
s.cleanupPreparedMetadata(r.Context(), prepared, cleanupSafeMetadata(prepared, i, rolledBack))
|
||||
if isS3TablesConflict(err) {
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", "Version token mismatch")
|
||||
return
|
||||
}
|
||||
glog.Errorf("Iceberg: CommitTransaction UpdateTable error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to commit table update: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (s *Server) prepareTableCommit(ctx context.Context, bucketName, bucketARN, identityName string, change tableChangeRequest) (*preparedTableCommit, *icebergRequestError) {
|
||||
namespace := []string(change.Identifier.Namespace)
|
||||
tableName := change.Identifier.Name
|
||||
|
||||
requirements, updates, statisticsUpdates, reqErr := parseTableChange(change)
|
||||
if reqErr != nil {
|
||||
return nil, reqErr
|
||||
}
|
||||
|
||||
getReq := &s3tables.GetTableRequest{TableBucketARN: bucketARN, Namespace: namespace, Name: tableName}
|
||||
var getResp s3tables.GetTableResponse
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(ctx, mgrClient, "GetTable", getReq, &getResp, identityName)
|
||||
})
|
||||
if err != nil {
|
||||
if isS3TablesNotFound(err) {
|
||||
return nil, &icebergRequestError{http.StatusNotFound, "NoSuchTableException", fmt.Sprintf("Table does not exist: %s", tableName)}
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: CommitTransaction GetTable error: %v", err)
|
||||
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", err.Error()}
|
||||
}
|
||||
|
||||
location := tableLocationFromMetadataLocation(getResp.MetadataLocation)
|
||||
if location == "" {
|
||||
location = fmt.Sprintf("s3://%s/%s", bucketName, path.Join(flattenNamespacePath(namespace), tableName))
|
||||
}
|
||||
tableUUID := uuid.Nil
|
||||
if getResp.Metadata != nil && getResp.Metadata.Iceberg != nil && getResp.Metadata.Iceberg.TableUUID != "" {
|
||||
if parsed, parseErr := uuid.Parse(getResp.Metadata.Iceberg.TableUUID); parseErr == nil {
|
||||
tableUUID = parsed
|
||||
}
|
||||
}
|
||||
if tableUUID == uuid.Nil {
|
||||
tableUUID = uuid.New()
|
||||
}
|
||||
|
||||
var currentMetadata table.Metadata
|
||||
if getResp.Metadata != nil && len(getResp.Metadata.FullMetadata) > 0 {
|
||||
currentMetadata, err = table.ParseMetadataBytes(getResp.Metadata.FullMetadata)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Failed to parse current metadata"}
|
||||
}
|
||||
} else {
|
||||
currentMetadata, err = newTableMetadata(tableUUID, location, nil, nil, nil, nil)
|
||||
if err != nil {
|
||||
glog.Errorf("Iceberg: CommitTransaction placeholder metadata for %s: %v", tableName, err)
|
||||
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Failed to build current metadata"}
|
||||
}
|
||||
}
|
||||
|
||||
for _, requirement := range requirements {
|
||||
if err := requirement.Validate(currentMetadata); err != nil {
|
||||
return nil, &icebergRequestError{http.StatusConflict, "CommitFailedException", "Requirement failed: " + err.Error()}
|
||||
}
|
||||
}
|
||||
|
||||
builder, err := table.MetadataBuilderFromBase(currentMetadata, getResp.MetadataLocation)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Failed to create metadata builder: " + err.Error()}
|
||||
}
|
||||
for _, update := range updates {
|
||||
if err := update.Apply(builder); err != nil {
|
||||
return nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Failed to apply update: " + err.Error()}
|
||||
}
|
||||
}
|
||||
newMetadata, err := builder.Build()
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Failed to build new metadata: " + err.Error()}
|
||||
}
|
||||
|
||||
metadataVersion := getResp.MetadataVersion + 1
|
||||
metadataFileName := fmt.Sprintf("v%d.metadata.json", metadataVersion)
|
||||
newMetadataLocation := fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(location, "/"), metadataFileName)
|
||||
|
||||
metadataBytes, err := json.Marshal(newMetadata)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Failed to serialize metadata: " + err.Error()}
|
||||
}
|
||||
metadataBytes, err = applyStatisticsUpdates(metadataBytes, statisticsUpdates)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Failed to apply statistics updates: " + err.Error()}
|
||||
}
|
||||
metadataBytes = ensureMetadataSpecCompliance(metadataBytes)
|
||||
|
||||
metadataBucket, metadataPath, err := parseS3Location(location)
|
||||
if err != nil {
|
||||
return nil, &icebergRequestError{http.StatusInternalServerError, "InternalServerError", "Invalid table location: " + err.Error()}
|
||||
}
|
||||
if err := confineMetadataLocation(metadataBucket, metadataPath, bucketName); err != nil {
|
||||
return nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", err.Error()}
|
||||
}
|
||||
|
||||
return &preparedTableCommit{
|
||||
namespace: namespace,
|
||||
tableName: tableName,
|
||||
tableUUID: tableUUID,
|
||||
versionToken: getResp.VersionToken,
|
||||
metadataBucket: metadataBucket,
|
||||
metadataPath: metadataPath,
|
||||
location: location,
|
||||
metadataFileName: metadataFileName,
|
||||
metadataBytes: metadataBytes,
|
||||
metadataVersion: metadataVersion,
|
||||
newMetadataLoc: newMetadataLocation,
|
||||
prevMetadataLoc: getResp.MetadataLocation,
|
||||
prevMetadataVersion: getResp.MetadataVersion,
|
||||
prevMetadata: cloneTableMetadata(getResp.Metadata),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// cloneTableMetadata deep-copies the prior table metadata so a later rollback
|
||||
// restores the exact pre-transaction bytes without aliasing the get response.
|
||||
func cloneTableMetadata(m *s3tables.TableMetadata) *s3tables.TableMetadata {
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
clone := &s3tables.TableMetadata{}
|
||||
if m.Iceberg != nil {
|
||||
iceberg := *m.Iceberg
|
||||
clone.Iceberg = &iceberg
|
||||
}
|
||||
if len(m.FullMetadata) > 0 {
|
||||
clone.FullMetadata = append(json.RawMessage(nil), m.FullMetadata...)
|
||||
}
|
||||
return clone
|
||||
}
|
||||
|
||||
func parseTableChange(change tableChangeRequest) (table.Requirements, table.Updates, []statisticsUpdate, *icebergRequestError) {
|
||||
var requirements table.Requirements
|
||||
if len(change.Requirements) > 0 {
|
||||
normalized, err := normalizeRequirements(change.Requirements)
|
||||
if err != nil {
|
||||
return nil, nil, nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Invalid requirements: " + err.Error()}
|
||||
}
|
||||
if err := json.Unmarshal(normalized, &requirements); err != nil {
|
||||
return nil, nil, nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Invalid requirements: " + err.Error()}
|
||||
}
|
||||
}
|
||||
var updates table.Updates
|
||||
var statisticsUpdates []statisticsUpdate
|
||||
if len(change.Updates) > 0 {
|
||||
var err error
|
||||
updates, statisticsUpdates, err = parseCommitUpdates(change.Updates)
|
||||
if err != nil {
|
||||
return nil, nil, nil, &icebergRequestError{http.StatusBadRequest, "BadRequestException", "Invalid updates: " + err.Error()}
|
||||
}
|
||||
}
|
||||
return requirements, updates, statisticsUpdates, nil
|
||||
}
|
||||
|
||||
func (s *Server) flipTablePointer(ctx context.Context, bucketARN, identityName string, pc *preparedTableCommit) error {
|
||||
updateReq := &s3tables.UpdateTableRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: pc.namespace,
|
||||
Name: pc.tableName,
|
||||
VersionToken: pc.versionToken,
|
||||
Metadata: &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{TableUUID: pc.tableUUID.String()},
|
||||
FullMetadata: pc.metadataBytes,
|
||||
},
|
||||
MetadataVersion: pc.metadataVersion,
|
||||
MetadataLocation: pc.newMetadataLoc,
|
||||
}
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(ctx, mgrClient, "UpdateTable", updateReq, nil, identityName)
|
||||
})
|
||||
}
|
||||
|
||||
// rollbackTablePointers restores already-flipped tables to their full prior
|
||||
// state. handleUpdateTable applies partial field updates, so the restore must
|
||||
// re-send every field the flip changed (location, version, and full metadata),
|
||||
// not just the location. Best-effort: a failed rollback is logged, not surfaced.
|
||||
// The returned slice flags, per flipped table, whether the restore succeeded so
|
||||
// the caller can avoid deleting metadata a still-flipped pointer references.
|
||||
func (s *Server) rollbackTablePointers(ctx context.Context, bucketARN, identityName string, flipped []preparedTableCommit) []bool {
|
||||
restored := make([]bool, len(flipped))
|
||||
for i := range flipped {
|
||||
pc := &flipped[i]
|
||||
updateReq := buildTableRestoreRequest(bucketARN, pc)
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(ctx, mgrClient, "UpdateTable", updateReq, nil, identityName)
|
||||
})
|
||||
if err != nil {
|
||||
glog.Errorf("Iceberg: CommitTransaction rollback of %s failed: %v", pc.tableName, err)
|
||||
continue
|
||||
}
|
||||
restored[i] = true
|
||||
}
|
||||
return restored
|
||||
}
|
||||
|
||||
// buildTableRestoreRequest reconstructs the UpdateTableRequest that reverts a
|
||||
// flipped table to its captured prior state. Every field handleUpdateTable
|
||||
// would have mutated on the flip is re-supplied here so the partial update
|
||||
// fully restores it (ModifiedAt and VersionToken are always regenerated by the
|
||||
// handler and cannot be pinned back).
|
||||
func buildTableRestoreRequest(bucketARN string, pc *preparedTableCommit) *s3tables.UpdateTableRequest {
|
||||
return &s3tables.UpdateTableRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: pc.namespace,
|
||||
Name: pc.tableName,
|
||||
Metadata: cloneTableMetadata(pc.prevMetadata),
|
||||
MetadataVersion: pc.prevMetadataVersion,
|
||||
MetadataLocation: pc.prevMetadataLoc,
|
||||
}
|
||||
}
|
||||
|
||||
// cleanupSafeMetadata reports, per prepared table, whether deleting its newly
|
||||
// written metadata file is safe after a flip failed at failedIndex. A file is
|
||||
// safe to delete only when its table's pointer no longer references it: tables
|
||||
// at or past failedIndex were never flipped, and earlier tables are safe only
|
||||
// if their rollback succeeded. A table whose rollback failed still points at the
|
||||
// new metadata, so deleting it would strand the pointer on a missing file.
|
||||
func cleanupSafeMetadata(prepared []preparedTableCommit, failedIndex int, rolledBack []bool) []bool {
|
||||
safe := make([]bool, len(prepared))
|
||||
for i := range prepared {
|
||||
switch {
|
||||
case i >= failedIndex:
|
||||
safe[i] = true
|
||||
case i < len(rolledBack):
|
||||
safe[i] = rolledBack[i]
|
||||
}
|
||||
}
|
||||
return safe
|
||||
}
|
||||
|
||||
// cleanupPreparedMetadata deletes the newly written metadata file for each
|
||||
// prepared table flagged safe; unflagged tables are left in place because a
|
||||
// still-flipped pointer references them.
|
||||
func (s *Server) cleanupPreparedMetadata(ctx context.Context, prepared []preparedTableCommit, safe []bool) {
|
||||
for i := range prepared {
|
||||
pc := &prepared[i]
|
||||
if i < len(safe) && !safe[i] {
|
||||
glog.Errorf("Iceberg: CommitTransaction keeping metadata %s; %s rollback failed and still references it", pc.newMetadataLoc, pc.tableName)
|
||||
continue
|
||||
}
|
||||
if cleanupErr := s.deleteMetadataFile(ctx, pc.metadataBucket, pc.metadataPath, pc.metadataFileName); cleanupErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to cleanup metadata file %s: %v", pc.newMetadataLoc, cleanupErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
449
weed/s3api/iceberg/handlers_view.go
Normal file
449
weed/s3api/iceberg/handlers_view.go
Normal file
@@ -0,0 +1,449 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path"
|
||||
"strings"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/apache/iceberg-go/view"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// handleListViews lists views in a namespace.
|
||||
func (s *Server) handleListViews(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
pageToken, pageSize, err := parsePagination(r)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
listReq := &s3tables.ListViewsRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
ContinuationToken: pageToken,
|
||||
MaxViews: pageSize,
|
||||
}
|
||||
var listResp s3tables.ListViewsResponse
|
||||
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "ListViews", listReq, &listResp, identityName)
|
||||
})
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: ListViews error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
identifiers := make([]TableIdentifier, 0, len(listResp.Views))
|
||||
for _, v := range listResp.Views {
|
||||
identifiers = append(identifiers, TableIdentifier{Namespace: namespace, Name: v.Name})
|
||||
}
|
||||
writeJSON(w, http.StatusOK, ListViewsResponse{
|
||||
NextPageToken: listResp.ContinuationToken,
|
||||
Identifiers: identifiers,
|
||||
})
|
||||
}
|
||||
|
||||
// handleCreateView creates a new view and writes its v1 metadata.json.
|
||||
func (s *Server) handleCreateView(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
if len(namespace) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace is required")
|
||||
return
|
||||
}
|
||||
|
||||
var req CreateViewRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
|
||||
return
|
||||
}
|
||||
if req.Name == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "view name is required")
|
||||
return
|
||||
}
|
||||
if req.Schema == nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "view schema is required")
|
||||
return
|
||||
}
|
||||
if req.ViewVersion == nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "view-version is required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
viewPath := path.Join(flattenNamespacePath(namespace), req.Name)
|
||||
location := strings.TrimSuffix(req.Location, "/")
|
||||
if location == "" {
|
||||
if req.Properties != nil {
|
||||
if warehouse := strings.TrimSuffix(req.Properties["warehouse"], "/"); warehouse != "" {
|
||||
location = fmt.Sprintf("%s/%s", warehouse, viewPath)
|
||||
}
|
||||
}
|
||||
if location == "" {
|
||||
if warehouse := strings.TrimSuffix(os.Getenv("ICEBERG_WAREHOUSE"), "/"); warehouse != "" {
|
||||
location = fmt.Sprintf("%s/%s", warehouse, viewPath)
|
||||
}
|
||||
}
|
||||
if location == "" {
|
||||
location = fmt.Sprintf("s3://%s/%s", bucketName, viewPath)
|
||||
}
|
||||
} else {
|
||||
parsedBucket, parsedPath, err := parseS3Location(location)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid view location: "+err.Error())
|
||||
return
|
||||
}
|
||||
if parsedPath == "" {
|
||||
location = fmt.Sprintf("s3://%s/%s", parsedBucket, viewPath)
|
||||
}
|
||||
}
|
||||
|
||||
metadataBucket, metadataPath, err := parseS3Location(location)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid view location: "+err.Error())
|
||||
return
|
||||
}
|
||||
if metadataBucket != bucketName {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "view location must be within bucket "+bucketName)
|
||||
return
|
||||
}
|
||||
if !isValidTablePath(metadataPath) {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "invalid view location path")
|
||||
return
|
||||
}
|
||||
|
||||
metadata, err := view.NewMetadata(req.ViewVersion, req.Schema, location, viewProperties(req.Properties))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to build view metadata: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
metadataFileName := "v1.metadata.json"
|
||||
metadataLocation := fmt.Sprintf("%s/metadata/%s", location, metadataFileName)
|
||||
metadataBytes, err := json.Marshal(metadata)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to serialize view metadata: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Authoritative existence check before touching storage: a registered view
|
||||
// short-circuits with its stored definition (idempotent CreateView) so we
|
||||
// never overwrite the persisted metadata of an existing view.
|
||||
if existsResp, existsErr := s.getView(r, namespace, req.Name); existsErr == nil {
|
||||
result, buildErr := s.buildViewResponse(r, existsResp, bucketName, namespace, req.Name)
|
||||
if buildErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", buildErr.Error())
|
||||
return
|
||||
}
|
||||
writeLoadResult(w, http.StatusOK, result)
|
||||
return
|
||||
} else if !isViewNotFound(existsErr) {
|
||||
glog.V(1).Infof("Iceberg: CreateView existence check failed for %s.%s: %v", flattenNamespacePath(namespace), req.Name, existsErr)
|
||||
writeManagerError(w, existsErr)
|
||||
return
|
||||
}
|
||||
|
||||
createReq := &s3tables.CreateViewRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
Name: req.Name,
|
||||
Metadata: &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{TableUUID: metadata.ViewUUID().String()},
|
||||
FullMetadata: metadataBytes,
|
||||
},
|
||||
MetadataLocation: metadataLocation,
|
||||
MetadataVersion: 1,
|
||||
}
|
||||
var createResp s3tables.CreateViewResponse
|
||||
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "CreateView", createReq, &createResp, identityName)
|
||||
})
|
||||
if err != nil {
|
||||
if isViewAlreadyExists(err) {
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", err.Error())
|
||||
return
|
||||
}
|
||||
var viewErr *s3tables.S3TablesError
|
||||
if errors.As(err, &viewErr) && viewErr.Type == s3tables.ErrCodeNoSuchNamespace {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", namespace))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: CreateView error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
// Persist the metadata file only after the catalog registers the view, so a
|
||||
// missing namespace or name collision fails before any bytes hit storage.
|
||||
if err := s.saveMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName, metadataBytes, false); err != nil {
|
||||
// Roll back the registered view so it doesn't linger pointing at metadata
|
||||
// that was never written.
|
||||
if dropErr := s.dropView(r, namespace, req.Name); dropErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to roll back view %s.%s after metadata write error: %v", flattenNamespacePath(namespace), req.Name, dropErr)
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to save view metadata file: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
finalLocation := createResp.MetadataLocation
|
||||
if finalLocation == "" {
|
||||
finalLocation = metadataLocation
|
||||
}
|
||||
config, _ := s.buildFileIOConfig(r, location)
|
||||
writeLoadResult(w, http.StatusOK, ViewResponse{
|
||||
MetadataLocation: finalLocation,
|
||||
Metadata: metadata,
|
||||
Config: config,
|
||||
})
|
||||
}
|
||||
|
||||
// handleLoadView loads view metadata.
|
||||
func (s *Server) handleLoadView(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
viewName := vars["view"]
|
||||
if len(namespace) == 0 || viewName == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and view name are required")
|
||||
return
|
||||
}
|
||||
|
||||
getResp, err := s.getView(r, namespace, viewName)
|
||||
if err != nil {
|
||||
if isViewNotFound(err) {
|
||||
writeError(w, http.StatusNotFound, "NoSuchViewException", fmt.Sprintf("View does not exist: %s", viewName))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: LoadView error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
result, err := s.buildViewResponse(r, getResp, getBucketFromPrefix(r), namespace, viewName)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
return
|
||||
}
|
||||
writeLoadResult(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleViewExists checks if a view exists.
|
||||
func (s *Server) handleViewExists(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
viewName := vars["view"]
|
||||
if len(namespace) == 0 || viewName == "" {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if _, err := s.getView(r, namespace, viewName); err != nil {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleDropView deletes a view.
|
||||
func (s *Server) handleDropView(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
viewName := vars["view"]
|
||||
if len(namespace) == 0 || viewName == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and view name are required")
|
||||
return
|
||||
}
|
||||
|
||||
var storedMetadataLocation string
|
||||
if getResp, err := s.getView(r, namespace, viewName); err == nil {
|
||||
storedMetadataLocation = getResp.MetadataLocation
|
||||
}
|
||||
|
||||
err := s.dropView(r, namespace, viewName)
|
||||
if err != nil {
|
||||
if isViewNotFound(err) {
|
||||
writeError(w, http.StatusNotFound, "NoSuchViewException", fmt.Sprintf("View does not exist: %s", viewName))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: DropView error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
if storedMetadataLocation != "" {
|
||||
if viewLoc := tableLocationFromMetadataLocation(storedMetadataLocation); viewLoc != "" {
|
||||
if dataBucket, dataPath, parseErr := parseS3Location(viewLoc); parseErr == nil {
|
||||
if cleanupErr := s.cleanupStaleTableLocation(r.Context(), dataBucket, dataPath); cleanupErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to purge dropped view location s3://%s/%s: %v", dataBucket, dataPath, cleanupErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// dropView removes a view registration from the catalog.
|
||||
func (s *Server) dropView(r *http.Request, namespace []string, viewName string) error {
|
||||
deleteReq := &s3tables.DeleteViewRequest{
|
||||
TableBucketARN: buildTableBucketARN(getBucketFromPrefix(r)),
|
||||
Namespace: namespace,
|
||||
Name: viewName,
|
||||
}
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "DeleteView", deleteReq, nil, identityName)
|
||||
})
|
||||
}
|
||||
|
||||
// getView fetches a view's stored metadata pointer from the catalog.
|
||||
func (s *Server) getView(r *http.Request, namespace []string, viewName string) (s3tables.GetViewResponse, error) {
|
||||
getReq := &s3tables.GetViewRequest{
|
||||
TableBucketARN: buildTableBucketARN(getBucketFromPrefix(r)),
|
||||
Namespace: namespace,
|
||||
Name: viewName,
|
||||
}
|
||||
var getResp s3tables.GetViewResponse
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "GetView", getReq, &getResp, identityName)
|
||||
})
|
||||
return getResp, err
|
||||
}
|
||||
|
||||
// buildViewResponse parses the stored view metadata into a ViewResponse.
|
||||
func (s *Server) buildViewResponse(r *http.Request, getResp s3tables.GetViewResponse, bucketName string, namespace []string, viewName string) (ViewResponse, error) {
|
||||
if getResp.Metadata == nil || len(getResp.Metadata.FullMetadata) == 0 {
|
||||
return ViewResponse{}, fmt.Errorf("view %s has no metadata", viewName)
|
||||
}
|
||||
metadata, err := view.ParseMetadataBytes(getResp.Metadata.FullMetadata)
|
||||
if err != nil {
|
||||
return ViewResponse{}, fmt.Errorf("failed to parse view metadata: %w", err)
|
||||
}
|
||||
config, _ := s.buildFileIOConfig(r, tableLocationFromMetadataLocation(getResp.MetadataLocation))
|
||||
return ViewResponse{
|
||||
MetadataLocation: getResp.MetadataLocation,
|
||||
Metadata: metadata,
|
||||
Config: config,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// viewProperties returns a non-nil copy of props for view metadata construction.
|
||||
func viewProperties(props iceberg.Properties) iceberg.Properties {
|
||||
out := make(iceberg.Properties, len(props))
|
||||
for k, v := range props {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func isViewNotFound(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
var viewErr *s3tables.S3TablesError
|
||||
if errors.As(err, &viewErr) {
|
||||
if viewErr.Type == s3tables.ErrCodeNoSuchView || viewErr.Type == s3tables.ErrCodeNoSuchNamespace {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return strings.Contains(strings.ToLower(err.Error()), "not found")
|
||||
}
|
||||
|
||||
func isViewAlreadyExists(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
var viewErr *s3tables.S3TablesError
|
||||
if errors.As(err, &viewErr) && viewErr.Type == s3tables.ErrCodeViewAlreadyExists {
|
||||
return true
|
||||
}
|
||||
return strings.Contains(strings.ToLower(err.Error()), "already exists")
|
||||
}
|
||||
|
||||
// handleRenameView moves a view's catalog pointer to a new namespace/name,
|
||||
// the view counterpart of POST /v1/{prefix}/tables/rename.
|
||||
func (s *Server) handleRenameView(w http.ResponseWriter, r *http.Request) {
|
||||
var req RenameTableRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
source := parseNamespace(encodeNamespace(req.Source.Namespace))
|
||||
dest := parseNamespace(encodeNamespace(req.Destination.Namespace))
|
||||
if len(source) == 0 || req.Source.Name == "" || len(dest) == 0 || req.Destination.Name == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "source and destination namespace and name are required")
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
renameReq := &s3tables.RenameTableRequest{
|
||||
TableBucketARN: buildTableBucketARN(bucketName),
|
||||
SourceNamespace: source,
|
||||
SourceName: req.Source.Name,
|
||||
DestNamespace: dest,
|
||||
DestName: req.Destination.Name,
|
||||
}
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "RenameView", renameReq, nil, identityName)
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
var viewErr *s3tables.S3TablesError
|
||||
if errors.As(err, &viewErr) {
|
||||
switch viewErr.Type {
|
||||
case s3tables.ErrCodeNoSuchView:
|
||||
writeError(w, http.StatusNotFound, "NoSuchViewException", fmt.Sprintf("View does not exist: %s", req.Source.Name))
|
||||
return
|
||||
case s3tables.ErrCodeNoSuchNamespace:
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", fmt.Sprintf("Namespace does not exist: %v", dest))
|
||||
return
|
||||
case s3tables.ErrCodeViewAlreadyExists:
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", fmt.Sprintf("View already exists: %s", req.Destination.Name))
|
||||
return
|
||||
case s3tables.ErrCodeInvalidRequest:
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", viewErr.Message)
|
||||
return
|
||||
}
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: RenameView error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
165
weed/s3api/iceberg/handlers_view_update.go
Normal file
165
weed/s3api/iceberg/handlers_view_update.go
Normal file
@@ -0,0 +1,165 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/apache/iceberg-go/view"
|
||||
"github.com/google/uuid"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// handleUpdateView applies requirements and updates to a view, writes a new
|
||||
// metadata.json, and flips the stored pointer. Mirrors the table commit flow.
|
||||
func (s *Server) handleUpdateView(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
namespace := parseNamespace(vars["namespace"])
|
||||
viewName := vars["view"]
|
||||
if len(namespace) == 0 || viewName == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Namespace and view name are required")
|
||||
return
|
||||
}
|
||||
|
||||
var req UpdateViewRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Invalid request body: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
bucketName := getBucketFromPrefix(r)
|
||||
bucketARN := buildTableBucketARN(bucketName)
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
|
||||
const maxCommitAttempts = 3
|
||||
for attempt := 1; attempt <= maxCommitAttempts; attempt++ {
|
||||
getResp, err := s.getView(r, namespace, viewName)
|
||||
if err != nil {
|
||||
if isViewNotFound(err) {
|
||||
writeError(w, http.StatusNotFound, "NoSuchViewException", fmt.Sprintf("View does not exist: %s", viewName))
|
||||
return
|
||||
}
|
||||
glog.V(1).Infof("Iceberg: UpdateView GetView error: %v", err)
|
||||
writeManagerError(w, err)
|
||||
return
|
||||
}
|
||||
if getResp.Metadata == nil || len(getResp.Metadata.FullMetadata) == 0 {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "view has no metadata")
|
||||
return
|
||||
}
|
||||
|
||||
currentMetadata, err := view.ParseMetadataBytes(getResp.Metadata.FullMetadata)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to parse current view metadata: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
for _, requirement := range req.Requirements {
|
||||
if err := requirement.Validate(currentMetadata); err != nil {
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", "Requirement failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
builder, err := view.MetadataBuilderFromBase(currentMetadata)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to create view metadata builder: "+err.Error())
|
||||
return
|
||||
}
|
||||
for _, update := range req.Updates {
|
||||
if err := update.Apply(builder); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to apply update: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
newMetadata, err := builder.Build()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", "Failed to build new view metadata: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
location := tableLocationFromMetadataLocation(getResp.MetadataLocation)
|
||||
if location == "" {
|
||||
location = newMetadata.Location()
|
||||
}
|
||||
metadataVersion := getResp.MetadataVersion + 1
|
||||
metadataFileName := fmt.Sprintf("v%d.metadata.json", metadataVersion)
|
||||
newMetadataLocation := fmt.Sprintf("%s/metadata/%s", strings.TrimSuffix(location, "/"), metadataFileName)
|
||||
|
||||
metadataBytes, err := json.Marshal(newMetadata)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to serialize view metadata: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
metadataBucket, metadataPath, err := parseS3Location(location)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Invalid view location: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := confineMetadataLocation(metadataBucket, metadataPath, bucketName); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
metadataFileName, newMetadataLocation, err = s.stageCommitMetadata(r.Context(), metadataBucket, metadataPath, location, metadataFileName, metadataBytes)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to save view metadata file: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
tableUUID := newMetadata.ViewUUID()
|
||||
if tableUUID == uuid.Nil {
|
||||
tableUUID = currentMetadata.ViewUUID()
|
||||
}
|
||||
updateReq := &s3tables.UpdateViewRequest{
|
||||
TableBucketARN: bucketARN,
|
||||
Namespace: namespace,
|
||||
Name: viewName,
|
||||
VersionToken: getResp.VersionToken,
|
||||
Metadata: &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{TableUUID: tableUUID.String()},
|
||||
FullMetadata: metadataBytes,
|
||||
},
|
||||
MetadataVersion: metadataVersion,
|
||||
MetadataLocation: newMetadataLocation,
|
||||
}
|
||||
err = s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
return s.tablesManager.Execute(r.Context(), mgrClient, "UpdateView", updateReq, nil, identityName)
|
||||
})
|
||||
if err == nil {
|
||||
config, _ := s.buildFileIOConfig(r, location)
|
||||
writeLoadResult(w, http.StatusOK, ViewResponse{
|
||||
MetadataLocation: newMetadataLocation,
|
||||
Metadata: newMetadata,
|
||||
Config: config,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
if isS3TablesConflict(err) {
|
||||
if cleanupErr := s.deleteMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to cleanup view metadata file %s on conflict: %v", newMetadataLocation, cleanupErr)
|
||||
}
|
||||
if attempt < maxCommitAttempts {
|
||||
glog.V(1).Infof("Iceberg: UpdateView conflict for %s (attempt %d/%d), retrying", viewName, attempt, maxCommitAttempts)
|
||||
sleepBeforeCommitRetry(attempt)
|
||||
continue
|
||||
}
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", "Version token mismatch")
|
||||
return
|
||||
}
|
||||
|
||||
if cleanupErr := s.deleteMetadataFile(r.Context(), metadataBucket, metadataPath, metadataFileName); cleanupErr != nil {
|
||||
glog.V(1).Infof("Iceberg: failed to cleanup view metadata file %s after update failure: %v", newMetadataLocation, cleanupErr)
|
||||
}
|
||||
glog.Errorf("Iceberg: UpdateView error: %v", err)
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", "Failed to commit view update: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
4
weed/s3api/iceberg/iceberg.go
Normal file
4
weed/s3api/iceberg/iceberg.go
Normal file
@@ -0,0 +1,4 @@
|
||||
// Package iceberg provides Iceberg REST Catalog API support.
|
||||
// It implements the Apache Iceberg REST Catalog specification
|
||||
// backed by S3 Tables metadata storage.
|
||||
package iceberg
|
||||
79
weed/s3api/iceberg/iceberg_access_delegation_test.go
Normal file
79
weed/s3api/iceberg/iceberg_access_delegation_test.go
Normal file
@@ -0,0 +1,79 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
func TestWantsVendedCredentials(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
values []string
|
||||
want bool
|
||||
}{
|
||||
{name: "no header", want: false},
|
||||
{name: "vended credentials", values: []string{"vended-credentials"}, want: true},
|
||||
{name: "mechanism list", values: []string{"remote-signing,vended-credentials"}, want: true},
|
||||
{name: "spaced and mixed case", values: []string{"Remote-Signing, Vended-Credentials"}, want: true},
|
||||
{name: "repeated header", values: []string{"remote-signing", "vended-credentials"}, want: true},
|
||||
{name: "remote signing only", values: []string{"remote-signing"}, want: false},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
r := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
|
||||
for _, v := range tc.values {
|
||||
r.Header.Add(accessDelegationHeader, v)
|
||||
}
|
||||
if got := wantsVendedCredentials(r); got != tc.want {
|
||||
t.Fatalf("wantsVendedCredentials(%v) = %v, want %v", tc.values, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A client that asked for vended credentials replaces its own storage
|
||||
// credentials with whatever the catalog returns. With no vending configured
|
||||
// there are none, so an endpoint by itself would leave the client signing
|
||||
// nothing and every data file read and write would come back 403.
|
||||
func TestBuildFileIOConfigWithholdsEndpointFromCredentialVendingClients(t *testing.T) {
|
||||
s := &Server{s3Endpoint: "http://seaweed.example:8333"}
|
||||
|
||||
r := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
|
||||
r.Header.Set(accessDelegationHeader, "vended-credentials")
|
||||
if got, _ := s.buildFileIOConfig(r, "s3://warehouse/ns/t"); len(got) != 0 {
|
||||
t.Fatalf("buildFileIOConfig() = %v, want empty for a vended-credentials request", got)
|
||||
}
|
||||
|
||||
plain := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
|
||||
if got, _ := s.buildFileIOConfig(plain, "s3://warehouse/ns/t"); got["s3.endpoint"] != s.s3Endpoint {
|
||||
t.Fatalf("s3.endpoint = %q, want %q", got["s3.endpoint"], s.s3Endpoint)
|
||||
}
|
||||
}
|
||||
|
||||
// The body of a load response depends on the delegation header, so a cache in
|
||||
// front of the catalog must not key on the URL alone.
|
||||
func TestWriteLoadResultVariesOnTheDelegationHeader(t *testing.T) {
|
||||
rec := httptest.NewRecorder()
|
||||
writeLoadResult(rec, http.StatusOK, LoadTableResult{})
|
||||
if got := rec.Header().Get("Vary"); got != accessDelegationHeader {
|
||||
t.Fatalf("Vary = %q, want %q", got, accessDelegationHeader)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadTableResultOmitsConfigForCredentialVendingClients(t *testing.T) {
|
||||
s := &Server{s3Endpoint: "http://seaweed.example:8333"}
|
||||
r := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
|
||||
r.Header.Set(accessDelegationHeader, "vended-credentials")
|
||||
|
||||
getResp := s3tables.GetTableResponse{MetadataLocation: "s3://bkt/ns/t/metadata/v1.metadata.json"}
|
||||
result, err := s.buildLoadTableResult(r, getResp, "bkt", []string{"ns"}, "t")
|
||||
if err != nil {
|
||||
t.Fatalf("buildLoadTableResult() error = %v", err)
|
||||
}
|
||||
if len(result.Config) != 0 {
|
||||
t.Fatalf("LoadTableResult.Config = %v, want empty", result.Config)
|
||||
}
|
||||
}
|
||||
97
weed/s3api/iceberg/iceberg_commit_location_test.go
Normal file
97
weed/s3api/iceberg/iceberg_commit_location_test.go
Normal file
@@ -0,0 +1,97 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// seedPoisonedTable registers a table entry whose stored metadataLocation
|
||||
// points outside its own bucket via a ".." segment, simulating a value
|
||||
// persisted verbatim by the raw S3Tables UpdateTable API.
|
||||
func seedPoisonedTable(t *testing.T, fc *memFiler, bucket, namespace, tableName, metadataLocation string) {
|
||||
t.Helper()
|
||||
meta := s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{TableUUID: "00000000-0000-0000-0000-000000000001"},
|
||||
}
|
||||
internal := map[string]any{
|
||||
"name": tableName,
|
||||
"namespace": namespace,
|
||||
"format": "ICEBERG",
|
||||
"ownerAccountId": s3_constants.AccountAdminId,
|
||||
"versionToken": "v1",
|
||||
"metadataVersion": 1,
|
||||
"metadataLocation": metadataLocation,
|
||||
"metadata": meta,
|
||||
}
|
||||
metaBytes, _ := json.Marshal(internal)
|
||||
fc.seed(s3tables.GetTablePath(bucket, namespace, tableName), &filer_pb.Entry{
|
||||
Name: tableName,
|
||||
IsDirectory: true,
|
||||
Extended: map[string][]byte{s3tables.ExtendedKeyMetadata: metaBytes},
|
||||
})
|
||||
}
|
||||
|
||||
func TestCommitTableRejectsStoredTraversalLocation(t *testing.T) {
|
||||
const attacker = "attacker"
|
||||
const victim = "victim"
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, attacker, "ns", s3_constants.AccountAdminId)
|
||||
seedPoisonedTable(t, fc, attacker, "ns", "t",
|
||||
"s3://attacker/../victim/planted/metadata/v1.metadata.json")
|
||||
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
r := httptest.NewRequest(http.MethodPost, "/v1/"+attacker+"/namespaces/ns/tables/t",
|
||||
strings.NewReader(`{"requirements":[],"updates":[]}`))
|
||||
r = mux.SetURLVars(r, map[string]string{"prefix": attacker, "namespace": "ns", "table": "t"})
|
||||
r = r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), s3_constants.AccountAdminId))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleUpdateTable(w, r)
|
||||
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d (body=%s)", w.Code, http.StatusBadRequest, w.Body.String())
|
||||
}
|
||||
for p := range fc.entries {
|
||||
if strings.Contains(p, "/"+victim+"/") && !strings.Contains(p, "/"+attacker+"/") {
|
||||
t.Fatalf("cross-tenant write escaped into victim tree at %s (status=%d body=%s)", p, w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCommitTransactionRejectsStoredTraversalLocation(t *testing.T) {
|
||||
const attacker = "attacker"
|
||||
const victim = "victim"
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, attacker, "ns", s3_constants.AccountAdminId)
|
||||
seedPoisonedTable(t, fc, attacker, "ns", "t",
|
||||
"s3://attacker/../victim/planted/metadata/v1.metadata.json")
|
||||
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
body := `{"table-changes":[{"identifier":{"namespace":["ns"],"name":"t"},"requirements":[],"updates":[]}]}`
|
||||
r := httptest.NewRequest(http.MethodPost, "/v1/"+attacker+"/transactions/commit",
|
||||
strings.NewReader(body))
|
||||
r = mux.SetURLVars(r, map[string]string{"prefix": attacker})
|
||||
r = r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), s3_constants.AccountAdminId))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleCommitTransaction(w, r)
|
||||
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d (body=%s)", w.Code, http.StatusBadRequest, w.Body.String())
|
||||
}
|
||||
for p := range fc.entries {
|
||||
if strings.Contains(p, "/"+victim+"/") && !strings.Contains(p, "/"+attacker+"/") {
|
||||
t.Fatalf("cross-tenant transaction write escaped into victim tree at %s (status=%d body=%s)", p, w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
134
weed/s3api/iceberg/iceberg_commit_updates_test.go
Normal file
134
weed/s3api/iceberg/iceberg_commit_updates_test.go
Normal file
@@ -0,0 +1,134 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
func TestParseCommitUpdatesSeparatesStatistics(t *testing.T) {
|
||||
raw := []json.RawMessage{
|
||||
json.RawMessage(`{"action":"set-statistics","snapshot-id":10,"statistics-path":"s3://bucket/table/metadata/stats.puffin","file-size-in-bytes":100,"file-footer-size-in-bytes":20,"blob-metadata":[]}`),
|
||||
json.RawMessage(`{"action":"set-properties","updates":{"k":"v"}}`),
|
||||
}
|
||||
|
||||
updates, stats, err := parseCommitUpdates(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("parseCommitUpdates() error = %v", err)
|
||||
}
|
||||
if len(stats) != 1 {
|
||||
t.Fatalf("statistics updates = %d, want 1", len(stats))
|
||||
}
|
||||
if stats[0].set == nil || stats[0].set.SnapshotID != 10 {
|
||||
t.Fatalf("unexpected statistics update: %#v", stats[0])
|
||||
}
|
||||
if len(updates) != 1 {
|
||||
t.Fatalf("decoded updates = %d, want 1", len(updates))
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCommitUpdatesRejectsIncompleteSetStatistics(t *testing.T) {
|
||||
raw := []json.RawMessage{
|
||||
json.RawMessage(`{"action":"set-statistics","snapshot-id":10}`),
|
||||
}
|
||||
|
||||
_, _, err := parseCommitUpdates(raw)
|
||||
if err == nil {
|
||||
t.Fatalf("parseCommitUpdates() expected error")
|
||||
}
|
||||
if !errors.Is(err, ErrIncompleteSetStatistics) {
|
||||
t.Fatalf("parseCommitUpdates() error = %v, want ErrIncompleteSetStatistics", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyStatisticsUpdatesUpsertAndRemove(t *testing.T) {
|
||||
metadata := []byte(`{"format-version":2,"statistics":[{"snapshot-id":1,"statistics-path":"s3://bucket/stats-1.puffin","file-size-in-bytes":10,"file-footer-size-in-bytes":1,"blob-metadata":[]},{"snapshot-id":2,"statistics-path":"s3://bucket/stats-2.puffin","file-size-in-bytes":20,"file-footer-size-in-bytes":2,"blob-metadata":[]}]} `)
|
||||
|
||||
snapshotID := int64(2)
|
||||
setUpdates := []statisticsUpdate{
|
||||
{
|
||||
set: &statisticsFileForTest,
|
||||
},
|
||||
{
|
||||
remove: &snapshotID,
|
||||
},
|
||||
}
|
||||
|
||||
updated, err := applyStatisticsUpdates(metadata, setUpdates)
|
||||
if err != nil {
|
||||
t.Fatalf("applyStatisticsUpdates() error = %v", err)
|
||||
}
|
||||
|
||||
var decoded map[string]json.RawMessage
|
||||
if err := json.Unmarshal(updated, &decoded); err != nil {
|
||||
t.Fatalf("json.Unmarshal(updated) error = %v", err)
|
||||
}
|
||||
|
||||
var stats []map[string]any
|
||||
if err := json.Unmarshal(decoded["statistics"], &stats); err != nil {
|
||||
t.Fatalf("json.Unmarshal(statistics) error = %v", err)
|
||||
}
|
||||
if len(stats) != 1 {
|
||||
t.Fatalf("statistics length = %d, want 1", len(stats))
|
||||
}
|
||||
if got := int64(stats[0]["snapshot-id"].(float64)); got != 1 {
|
||||
t.Fatalf("remaining snapshot-id = %d, want 1", got)
|
||||
}
|
||||
if got := int64(stats[0]["file-size-in-bytes"].(float64)); got != 11 {
|
||||
t.Fatalf("remaining file-size-in-bytes = %d, want 11", got)
|
||||
}
|
||||
}
|
||||
|
||||
var statisticsFileForTest = table.StatisticsFile{
|
||||
SnapshotID: 1,
|
||||
StatisticsPath: "s3://bucket/stats-1.puffin",
|
||||
FileSizeInBytes: 11,
|
||||
FileFooterSizeInBytes: 2,
|
||||
BlobMetadata: []table.BlobMetadata{},
|
||||
}
|
||||
|
||||
func TestIsS3TablesConflict(t *testing.T) {
|
||||
if !isS3TablesConflict(s3tables.ErrVersionTokenMismatch) {
|
||||
t.Fatalf("expected ErrVersionTokenMismatch to be conflict")
|
||||
}
|
||||
if !isS3TablesConflict(&s3tables.S3TablesError{Type: s3tables.ErrCodeConflict, Message: "Version token mismatch"}) {
|
||||
t.Fatalf("expected S3Tables conflict error to be conflict")
|
||||
}
|
||||
if isS3TablesConflict(errors.New("other")) {
|
||||
t.Fatalf("unexpected conflict for non-conflict error")
|
||||
}
|
||||
}
|
||||
|
||||
// A create-on-commit is a create, and it writes the new table's metadata file
|
||||
// before the registration that authorizes it.
|
||||
func TestCreateOnCommitDeniedBeforeAnyWrite(t *testing.T) {
|
||||
const bucket = "warehouse"
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, bucket, "finance", "alice")
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
r := httptest.NewRequest(http.MethodPost, "/v1/"+bucket+"/namespaces/finance/tables/quarterly_reports",
|
||||
strings.NewReader(`{"requirements":[{"type":"assert-create"}],"updates":[]}`))
|
||||
r = mux.SetURLVars(r, map[string]string{"prefix": bucket, "namespace": "finance", "table": "quarterly_reports"})
|
||||
r = r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), "mallory"))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleUpdateTable(w, r)
|
||||
|
||||
if w.Code != http.StatusForbidden {
|
||||
t.Errorf("status = %d, want %d (body: %s)", w.Code, http.StatusForbidden, w.Body.String())
|
||||
}
|
||||
for p, entry := range fc.entries {
|
||||
if !entry.IsDirectory && strings.Contains(p, "quarterly_reports") {
|
||||
t.Errorf("unauthorized caller wrote %s", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
200
weed/s3api/iceberg/iceberg_create_table_test.go
Normal file
200
weed/s3api/iceberg/iceberg_create_table_test.go
Normal file
@@ -0,0 +1,200 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/google/uuid"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
func TestValidateCreateTableRequestRequiresName(t *testing.T) {
|
||||
err := validateCreateTableRequest(CreateTableRequest{})
|
||||
if !errors.Is(err, errTableNameRequired) {
|
||||
t.Fatalf("validateCreateTableRequest() error = %v, want errTableNameRequired", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateCreateTableRequestAcceptsWithName(t *testing.T) {
|
||||
err := validateCreateTableRequest(CreateTableRequest{Name: "orders"})
|
||||
if err != nil {
|
||||
t.Fatalf("validateCreateTableRequest() error = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsStageCreateEnabledDefaultsToTrue(t *testing.T) {
|
||||
t.Setenv("ICEBERG_ENABLE_STAGE_CREATE", "")
|
||||
if !isStageCreateEnabled() {
|
||||
t.Fatalf("isStageCreateEnabled() = false, want true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsStageCreateEnabledFalseValues(t *testing.T) {
|
||||
falseValues := []string{"0", "false", "FALSE", "no", "off"}
|
||||
for _, value := range falseValues {
|
||||
t.Setenv("ICEBERG_ENABLE_STAGE_CREATE", value)
|
||||
if isStageCreateEnabled() {
|
||||
t.Fatalf("isStageCreateEnabled() = true for value %q, want false", value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func mustParseSchema(t *testing.T, raw string) *iceberg.Schema {
|
||||
t.Helper()
|
||||
var schema iceberg.Schema
|
||||
if err := json.Unmarshal([]byte(raw), &schema); err != nil {
|
||||
t.Fatalf("parse schema: %v", err)
|
||||
}
|
||||
return &schema
|
||||
}
|
||||
|
||||
const variantSchema = `{"type":"struct","schema-id":0,"fields":[
|
||||
{"id":1,"name":"id","required":true,"type":"long"},
|
||||
{"id":2,"name":"payload","required":false,"type":"variant"}]}`
|
||||
|
||||
// A v3-only column type without format-version 3 is the client's mistake, and
|
||||
// the reason has to travel back to them rather than only into the server log.
|
||||
func TestNewTableMetadataRejectsV3TypeBelowV3(t *testing.T) {
|
||||
_, err := newTableMetadata(uuid.New(), "s3://bkt/ns/t", mustParseSchema(t, variantSchema), nil, nil, nil)
|
||||
if err == nil {
|
||||
t.Fatal("newTableMetadata() error = nil, want invalid schema")
|
||||
}
|
||||
if !errors.Is(err, iceberg.ErrInvalidSchema) {
|
||||
t.Fatalf("newTableMetadata() error = %v, want ErrInvalidSchema", err)
|
||||
}
|
||||
|
||||
// writeManagerError turns this into a 400; see TestWriteManagerError.
|
||||
if !strings.Contains(err.Error(), "variant is not supported until v3") {
|
||||
t.Errorf("error = %q, want the underlying reason", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewTableMetadataAcceptsV3TypeAtV3(t *testing.T) {
|
||||
metadata, err := newTableMetadata(uuid.New(), "s3://bkt/ns/t", mustParseSchema(t, variantSchema), nil, nil,
|
||||
iceberg.Properties{"format-version": "3"})
|
||||
if err != nil {
|
||||
t.Fatalf("newTableMetadata() error = %v, want nil", err)
|
||||
}
|
||||
if got := metadata.Version(); got != 3 {
|
||||
t.Fatalf("metadata.Version() = %d, want 3", got)
|
||||
}
|
||||
if _, found := metadata.CurrentSchema().FindFieldByName("payload"); !found {
|
||||
t.Error("variant field missing from stored schema")
|
||||
}
|
||||
}
|
||||
|
||||
// A LoadTable response must never carry nil metadata: it serializes as
|
||||
// "metadata":null under HTTP 200, which no Iceberg client can parse.
|
||||
func TestBuildLoadTableResultNeverReturnsNilMetadata(t *testing.T) {
|
||||
cases := map[string]s3tables.GetTableResponse{
|
||||
"no stored metadata": {MetadataLocation: "s3://bkt/ns/t/metadata/v1.metadata.json"},
|
||||
"empty full metadata": {Metadata: &s3tables.TableMetadata{}},
|
||||
"unparseable metadata": {Metadata: &s3tables.TableMetadata{FullMetadata: json.RawMessage(`{"nope":`)}},
|
||||
}
|
||||
for name, getResp := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
r := httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
|
||||
result, err := (&Server{}).buildLoadTableResult(r, getResp, "bkt", []string{"ns"}, "t")
|
||||
if err != nil {
|
||||
t.Fatalf("buildLoadTableResult() error = %v, want nil", err)
|
||||
}
|
||||
if result.Metadata == nil {
|
||||
t.Fatal("buildLoadTableResult() returned nil metadata with no error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func newCreateTableRequest(t *testing.T, bucket, namespace, body, identity string) *http.Request {
|
||||
t.Helper()
|
||||
r := httptest.NewRequest(http.MethodPost, "/v1/"+bucket+"/namespaces/"+namespace+"/tables", strings.NewReader(body))
|
||||
r = mux.SetURLVars(r, map[string]string{"prefix": bucket, "namespace": namespace})
|
||||
return r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), identity))
|
||||
}
|
||||
|
||||
// A caller who may not create the table must be refused before anything is
|
||||
// written: both branches of CreateTable write to the bucket, and stage-create
|
||||
// never reaches the registration that carries the authorization.
|
||||
func TestCreateTableDeniedBeforeAnyWrite(t *testing.T) {
|
||||
const bucket = "warehouse"
|
||||
for name, body := range map[string]string{
|
||||
"stage-create": `{"name":"quarterly_reports","stage-create":true}`,
|
||||
"create": `{"name":"quarterly_reports"}`,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, bucket, "finance", "alice")
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleCreateTable(w, newCreateTableRequest(t, bucket, "finance", body, "mallory"))
|
||||
|
||||
if w.Code != http.StatusForbidden {
|
||||
t.Errorf("status = %d, want %d (body: %s)", w.Code, http.StatusForbidden, w.Body.String())
|
||||
}
|
||||
for p, entry := range fc.entries {
|
||||
if !entry.IsDirectory && strings.Contains(p, "quarterly_reports") {
|
||||
t.Errorf("unauthorized caller wrote %s", p)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The namespace owner still gets the staged metadata and marker stage-create
|
||||
// exists to leave behind.
|
||||
func TestStageCreateWritesStagedFilesForOwner(t *testing.T) {
|
||||
const bucket = "warehouse"
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, bucket, "finance", "alice")
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleCreateTable(w, newCreateTableRequest(t, bucket, "finance", `{"name":"quarterly_reports","stage-create":true}`, "alice"))
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d (body: %s)", w.Code, http.StatusOK, w.Body.String())
|
||||
}
|
||||
|
||||
staged := 0
|
||||
for p, entry := range fc.entries {
|
||||
if !entry.IsDirectory && strings.Contains(p, stageCreateMarkerDirName) {
|
||||
staged++
|
||||
}
|
||||
}
|
||||
if staged != 2 {
|
||||
t.Fatalf("staged files = %d, want the metadata file and its marker", staged)
|
||||
}
|
||||
if _, ok := fc.entries[s3tables.GetTablePath(bucket, "finance", "quarterly_reports")]; ok {
|
||||
t.Error("stage-create registered the table")
|
||||
}
|
||||
}
|
||||
|
||||
// A caller allowed by the actions on its own identity, rather than by ownership
|
||||
// or a resource policy, has to survive the hop into the s3tables manager: the
|
||||
// gate must not refuse what the create behind it would allow.
|
||||
func TestCreateTableAllowedByIdentityActions(t *testing.T) {
|
||||
const bucket = "warehouse"
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, bucket, "finance", "alice")
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
r := newCreateTableRequest(t, bucket, "finance", `{"name":"quarterly_reports","stage-create":true}`, "mallory")
|
||||
r = r.WithContext(s3_constants.SetIdentityInContext(r.Context(), &struct {
|
||||
Name string
|
||||
Actions []string
|
||||
}{Name: "mallory", Actions: []string{"s3tables:CreateTable"}}))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleCreateTable(w, r)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d (body: %s)", w.Code, http.StatusOK, w.Body.String())
|
||||
}
|
||||
}
|
||||
233
weed/s3api/iceberg/iceberg_create_view_test.go
Normal file
233
weed/s3api/iceberg/iceberg_create_view_test.go
Normal file
@@ -0,0 +1,233 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/apache/iceberg-go/view"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
"google.golang.org/grpc"
|
||||
)
|
||||
|
||||
// memFiler is a minimal in-memory SeaweedFilerClient backing the view handlers
|
||||
// end-to-end through the s3tables manager. Only the entry operations the create
|
||||
// path touches are implemented; anything else panics so a missing dependency is
|
||||
// loud rather than silently passing.
|
||||
type memFiler struct {
|
||||
filer_pb.SeaweedFilerClient
|
||||
entries map[string]*filer_pb.Entry
|
||||
// failFileCreate, when set, makes CreateEntry fail for non-directory entries,
|
||||
// simulating a metadata-file write error.
|
||||
failFileCreate error
|
||||
}
|
||||
|
||||
func newMemFiler() *memFiler {
|
||||
return &memFiler{entries: map[string]*filer_pb.Entry{}}
|
||||
}
|
||||
|
||||
func (m *memFiler) WithFilerClient(_ bool, fn func(client filer_pb.SeaweedFilerClient) error) error {
|
||||
return fn(m)
|
||||
}
|
||||
|
||||
func (m *memFiler) seed(p string, entry *filer_pb.Entry) {
|
||||
m.entries[p] = entry
|
||||
}
|
||||
|
||||
func (m *memFiler) LookupDirectoryEntry(_ context.Context, in *filer_pb.LookupDirectoryEntryRequest, _ ...grpc.CallOption) (*filer_pb.LookupDirectoryEntryResponse, error) {
|
||||
entry, ok := m.entries[path.Join(in.Directory, in.Name)]
|
||||
if !ok {
|
||||
return nil, filer_pb.ErrNotFound
|
||||
}
|
||||
return &filer_pb.LookupDirectoryEntryResponse{Entry: entry}, nil
|
||||
}
|
||||
|
||||
func (m *memFiler) CreateEntry(_ context.Context, in *filer_pb.CreateEntryRequest, _ ...grpc.CallOption) (*filer_pb.CreateEntryResponse, error) {
|
||||
if m.failFileCreate != nil && !in.Entry.IsDirectory {
|
||||
return nil, m.failFileCreate
|
||||
}
|
||||
m.entries[path.Join(in.Directory, in.Entry.Name)] = in.Entry
|
||||
return &filer_pb.CreateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (m *memFiler) DeleteEntry(_ context.Context, in *filer_pb.DeleteEntryRequest, _ ...grpc.CallOption) (*filer_pb.DeleteEntryResponse, error) {
|
||||
prefix := path.Join(in.Directory, in.Name)
|
||||
for p := range m.entries {
|
||||
if p == prefix || strings.HasPrefix(p, prefix+"/") {
|
||||
delete(m.entries, p)
|
||||
}
|
||||
}
|
||||
return &filer_pb.DeleteEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (m *memFiler) UpdateEntry(_ context.Context, in *filer_pb.UpdateEntryRequest, _ ...grpc.CallOption) (*filer_pb.UpdateEntryResponse, error) {
|
||||
m.entries[path.Join(in.Directory, in.Entry.Name)] = in.Entry
|
||||
return &filer_pb.UpdateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (m *memFiler) ListEntries(_ context.Context, in *filer_pb.ListEntriesRequest, _ ...grpc.CallOption) (filer_pb.SeaweedFiler_ListEntriesClient, error) {
|
||||
var names []string
|
||||
for p := range m.entries {
|
||||
if path.Dir(p) == in.Directory {
|
||||
names = append(names, p)
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
stream := &memListEntries{}
|
||||
for _, name := range names {
|
||||
stream.entries = append(stream.entries, m.entries[name])
|
||||
}
|
||||
return stream, nil
|
||||
}
|
||||
|
||||
type memListEntries struct {
|
||||
grpc.ClientStream
|
||||
entries []*filer_pb.Entry
|
||||
}
|
||||
|
||||
func (m *memListEntries) Recv() (*filer_pb.ListEntriesResponse, error) {
|
||||
if len(m.entries) == 0 {
|
||||
return nil, io.EOF
|
||||
}
|
||||
entry := m.entries[0]
|
||||
m.entries = m.entries[1:]
|
||||
return &filer_pb.ListEntriesResponse{Entry: entry}, nil
|
||||
}
|
||||
|
||||
func newCreateViewRequest(t *testing.T, namespace, name, sql string) *http.Request {
|
||||
t.Helper()
|
||||
schema := iceberg.NewSchemaWithIdentifiers(0, nil,
|
||||
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int32, Required: true},
|
||||
)
|
||||
ver, err := view.NewVersionFromSQL(1, schema.ID, sql, table.Identifier{namespace})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body, err := json.Marshal(CreateViewRequest{Name: name, Schema: schema, ViewVersion: ver})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := httptest.NewRequest(http.MethodPost, "/v1/namespaces/"+namespace+"/views", bytes.NewReader(body))
|
||||
r = mux.SetURLVars(r, map[string]string{"namespace": namespace})
|
||||
r = r.WithContext(s3_constants.SetIdentityNameInContext(r.Context(), s3_constants.AccountAdminId))
|
||||
return r
|
||||
}
|
||||
|
||||
// seedNamespace registers a bucket and namespace owned by owner so the s3tables
|
||||
// existence and auth-context lookups pass.
|
||||
func seedNamespace(fc *memFiler, bucket, namespace, owner string) {
|
||||
fc.seed(s3tables.GetTableBucketPath(bucket), &filer_pb.Entry{Name: bucket, IsDirectory: true})
|
||||
meta, _ := json.Marshal(map[string]any{"namespace": []string{namespace}, "ownerAccountId": owner})
|
||||
fc.seed(s3tables.GetNamespacePath(bucket, namespace), &filer_pb.Entry{
|
||||
Name: namespace,
|
||||
IsDirectory: true,
|
||||
Extended: map[string][]byte{s3tables.ExtendedKeyMetadata: meta},
|
||||
})
|
||||
}
|
||||
|
||||
func TestCreateViewMissingNamespaceReturns404(t *testing.T) {
|
||||
fc := newMemFiler()
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 1"))
|
||||
|
||||
if w.Code != http.StatusNotFound {
|
||||
t.Fatalf("status = %d, want %d (body: %s)", w.Code, http.StatusNotFound, w.Body.String())
|
||||
}
|
||||
var errResp ErrorResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &errResp); err != nil {
|
||||
t.Fatalf("unmarshal error body: %v", err)
|
||||
}
|
||||
if errResp.Error.Type != "NoSuchNamespaceException" {
|
||||
t.Fatalf("error type = %q, want NoSuchNamespaceException", errResp.Error.Type)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateViewTagsEntryAsView(t *testing.T) {
|
||||
const bucket = "warehouse"
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, bucket, "ns", s3_constants.AccountAdminId)
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 1"))
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("create status = %d, want 200 (body: %s)", w.Code, w.Body.String())
|
||||
}
|
||||
|
||||
entry, ok := fc.entries[s3tables.GetTablePath(bucket, "ns", "v")]
|
||||
if !ok {
|
||||
t.Fatalf("view entry not created")
|
||||
}
|
||||
if got := string(entry.Extended[s3tables.ExtendedKeyEntryType]); got != s3tables.EntryTypeView {
|
||||
t.Fatalf("entryType = %q, want %q", got, s3tables.EntryTypeView)
|
||||
}
|
||||
if _, ok := entry.Extended[s3tables.ExtendedKeyMetadata]; !ok {
|
||||
t.Fatalf("view entry missing metadata attribute")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateViewDuplicateDoesNotClobberMetadata(t *testing.T) {
|
||||
const bucket = "warehouse"
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, bucket, "ns", s3_constants.AccountAdminId)
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 1"))
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("first create status = %d, want 200 (body: %s)", w.Code, w.Body.String())
|
||||
}
|
||||
|
||||
metadataKey := path.Join(s3tables.GetTablePath(bucket, "ns", "v"), "metadata", "v1.metadata.json")
|
||||
first, ok := fc.entries[metadataKey]
|
||||
if !ok {
|
||||
t.Fatalf("metadata file not written at %s", metadataKey)
|
||||
}
|
||||
original := append([]byte(nil), first.Content...)
|
||||
|
||||
// Re-create the same view with different SQL: the existence pre-check must
|
||||
// short-circuit so the persisted v1.metadata.json stays byte-for-byte intact.
|
||||
w = httptest.NewRecorder()
|
||||
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 2"))
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("duplicate create status = %d, want 200 (body: %s)", w.Code, w.Body.String())
|
||||
}
|
||||
if got := fc.entries[metadataKey].Content; !bytes.Equal(got, original) {
|
||||
t.Fatalf("duplicate create clobbered stored metadata:\n got %s\nwant %s", got, original)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateViewRollsBackEntryWhenMetadataWriteFails(t *testing.T) {
|
||||
const bucket = "warehouse"
|
||||
fc := newMemFiler()
|
||||
seedNamespace(fc, bucket, "ns", s3_constants.AccountAdminId)
|
||||
fc.failFileCreate = errors.New("disk full")
|
||||
s := NewServer(fc, nil)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
s.handleCreateView(w, newCreateViewRequest(t, "ns", "v", "SELECT 1"))
|
||||
if w.Code != http.StatusInternalServerError {
|
||||
t.Fatalf("status = %d, want %d (body: %s)", w.Code, http.StatusInternalServerError, w.Body.String())
|
||||
}
|
||||
|
||||
// The registered view must be rolled back so it doesn't linger pointing at
|
||||
// metadata that was never written.
|
||||
if _, ok := fc.entries[s3tables.GetTablePath(bucket, "ns", "v")]; ok {
|
||||
t.Fatalf("view entry left behind after metadata write failure")
|
||||
}
|
||||
}
|
||||
111
weed/s3api/iceberg/iceberg_credential_vending_test.go
Normal file
111
weed/s3api/iceberg/iceberg_credential_vending_test.go
Normal file
@@ -0,0 +1,111 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type stubVendor struct {
|
||||
credentials *VendedCredentials
|
||||
err error
|
||||
gotBucket string
|
||||
gotPrefix string
|
||||
gotcaller string
|
||||
}
|
||||
|
||||
func (v *stubVendor) VendTableCredentials(_ context.Context, principal, bucket, prefix string) (*VendedCredentials, error) {
|
||||
v.gotcaller, v.gotBucket, v.gotPrefix = principal, bucket, prefix
|
||||
return v.credentials, v.err
|
||||
}
|
||||
|
||||
func vendingServer(vendor CredentialVendor) *Server {
|
||||
s := &Server{s3Endpoint: "http://s3.example:8333"}
|
||||
s.SetCredentialVendor(vendor)
|
||||
return s
|
||||
}
|
||||
|
||||
func TestBuildFileIOConfigVendingVendsScopedCredentials(t *testing.T) {
|
||||
expiry := time.Now().Add(time.Hour).Truncate(time.Millisecond)
|
||||
vendor := &stubVendor{credentials: &VendedCredentials{
|
||||
AccessKeyID: "ASIAEXAMPLE",
|
||||
SecretAccessKey: "secret",
|
||||
SessionToken: "token",
|
||||
Expiration: expiry,
|
||||
}}
|
||||
s := vendingServer(vendor)
|
||||
|
||||
r := httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t", nil)
|
||||
r.Header.Set(accessDelegationHeader, "vended-credentials")
|
||||
|
||||
config, storageCredentials := s.buildFileIOConfig(r, "s3://warehouse/ns/t")
|
||||
|
||||
if vendor.gotBucket != "warehouse" || vendor.gotPrefix != "ns/t" {
|
||||
t.Errorf("vendor called with bucket=%q prefix=%q, want warehouse and ns/t", vendor.gotBucket, vendor.gotPrefix)
|
||||
}
|
||||
if config["s3.access-key-id"] != "ASIAEXAMPLE" || config["s3.secret-access-key"] != "secret" || config["s3.session-token"] != "token" {
|
||||
t.Errorf("config missing vended credentials: %v", config)
|
||||
}
|
||||
if config["s3.endpoint"] != "http://s3.example:8333" {
|
||||
t.Errorf("config missing the endpoint the credentials are for: %v", config)
|
||||
}
|
||||
if len(storageCredentials) != 1 {
|
||||
t.Fatalf("storage-credentials = %d entries, want 1", len(storageCredentials))
|
||||
}
|
||||
if storageCredentials[0].Prefix != "s3://warehouse/ns/t" {
|
||||
t.Errorf("storage credential prefix = %q", storageCredentials[0].Prefix)
|
||||
}
|
||||
if storageCredentials[0].Config["s3.session-token"] != "token" {
|
||||
t.Errorf("storage credential is missing the session token: %v", storageCredentials[0].Config)
|
||||
}
|
||||
}
|
||||
|
||||
// Without a vendor the response must stay silent about the endpoint too: a
|
||||
// client that asked for vended credentials would otherwise drop its own and
|
||||
// start sending unsigned requests.
|
||||
func TestBuildFileIOConfigVendingWithoutVendorSaysNothing(t *testing.T) {
|
||||
s := &Server{s3Endpoint: "http://s3.example:8333"}
|
||||
r := httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t", nil)
|
||||
r.Header.Set(accessDelegationHeader, "vended-credentials")
|
||||
|
||||
config, storageCredentials := s.buildFileIOConfig(r, "s3://warehouse/ns/t")
|
||||
if len(config) != 0 || storageCredentials != nil {
|
||||
t.Errorf("config = %v, storage-credentials = %v, want both empty", config, storageCredentials)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildFileIOConfigVendingFallsBackWhenVendingFails(t *testing.T) {
|
||||
s := vendingServer(&stubVendor{err: errors.New("no role configured")})
|
||||
r := httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t", nil)
|
||||
r.Header.Set(accessDelegationHeader, "vended-credentials")
|
||||
|
||||
config, storageCredentials := s.buildFileIOConfig(r, "s3://warehouse/ns/t")
|
||||
if len(config) != 0 || storageCredentials != nil {
|
||||
t.Errorf("config = %v, storage-credentials = %v, want both empty", config, storageCredentials)
|
||||
}
|
||||
}
|
||||
|
||||
// A client that did not ask for delegation keeps the plain endpoint config and
|
||||
// must never be handed credentials.
|
||||
func TestBuildFileIOConfigVendingWithoutDelegationHeader(t *testing.T) {
|
||||
vendor := &stubVendor{credentials: &VendedCredentials{AccessKeyID: "ASIAEXAMPLE"}}
|
||||
s := vendingServer(vendor)
|
||||
|
||||
r := httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t", nil)
|
||||
config, storageCredentials := s.buildFileIOConfig(r, "s3://warehouse/ns/t")
|
||||
|
||||
if _, vended := config["s3.access-key-id"]; vended {
|
||||
t.Errorf("credentials vended without the delegation header: %v", config)
|
||||
}
|
||||
if config["s3.endpoint"] != "http://s3.example:8333" {
|
||||
t.Errorf("config = %v, want the endpoint", config)
|
||||
}
|
||||
if storageCredentials != nil {
|
||||
t.Errorf("storage-credentials = %v, want none", storageCredentials)
|
||||
}
|
||||
if vendor.gotBucket != "" {
|
||||
t.Errorf("vendor was called for a request that did not ask for delegation")
|
||||
}
|
||||
}
|
||||
87
weed/s3api/iceberg/iceberg_invalid_name_test.go
Normal file
87
weed/s3api/iceberg/iceberg_invalid_name_test.go
Normal file
@@ -0,0 +1,87 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
func TestNameValidationError(t *testing.T) {
|
||||
cases := []struct {
|
||||
err error
|
||||
want bool
|
||||
}{
|
||||
{nil, false},
|
||||
// Wrapped exactly as the s3tables manager surfaces it.
|
||||
{fmt.Errorf("all filers failed, last error: invalid namespace name: only 'a-z', '0-9', and '_' are allowed"), true},
|
||||
{fmt.Errorf("invalid table name: only 'a-z', '0-9', and '_' are allowed"), true},
|
||||
{fmt.Errorf("namespace name must start with a letter or digit"), true},
|
||||
{fmt.Errorf("namespace name cannot start with reserved prefix 'aws'"), true},
|
||||
{fmt.Errorf("table name must be between 1 and 255 characters"), true},
|
||||
{fmt.Errorf("namespace not found"), false},
|
||||
{fmt.Errorf("all filers failed, last error: rpc timeout"), false},
|
||||
// Unrelated faults that merely mention a name must stay 500.
|
||||
{fmt.Errorf("failed to resolve table name from index"), false},
|
||||
{fmt.Errorf("error fetching namespace name mapping"), false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := nameValidationError(c.err); got != c.want {
|
||||
t.Errorf("nameValidationError(%v) = %v, want %v", c.err, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteManagerError(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
err error
|
||||
wantCode int
|
||||
wantType string
|
||||
wantMessage []string
|
||||
}{
|
||||
{name: "invalid name is a client error", err: fmt.Errorf("invalid namespace name: only 'a-z', '0-9', and '_' are allowed"), wantCode: http.StatusBadRequest, wantType: "BadRequestException"},
|
||||
{name: "rejected schema is a client error", err: fmt.Errorf("%w: for v2: variant is not supported until v3", iceberg.ErrInvalidSchema), wantCode: http.StatusBadRequest, wantType: "BadRequestException"},
|
||||
{name: "bad format version is a client error", err: fmt.Errorf("%w: 4", iceberg.ErrInvalidFormatVersion), wantCode: http.StatusBadRequest, wantType: "BadRequestException"},
|
||||
{
|
||||
name: "missing table bucket is a client error",
|
||||
err: fmt.Errorf("all filers failed, last error: %w", &s3tables.S3TablesError{Type: s3tables.ErrCodeNoSuchBucket, Message: "table bucket warehouse not found"}),
|
||||
wantCode: http.StatusNotFound,
|
||||
wantType: "NoSuchNamespaceException",
|
||||
// The guidance is the point of the mapping: the bucket the request
|
||||
// resolved to is one the client never named, so the response has to
|
||||
// say how to name a real one.
|
||||
wantMessage: []string{"table bucket warehouse not found", "warehouse=s3://<table-bucket>/", "/v1/<table-bucket>/"},
|
||||
},
|
||||
{name: "everything else is a server fault", err: fmt.Errorf("all filers failed, last error: connection refused"), wantCode: http.StatusInternalServerError, wantType: "InternalServerError"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
rec := httptest.NewRecorder()
|
||||
writeManagerError(rec, c.err)
|
||||
if rec.Code != c.wantCode {
|
||||
t.Fatalf("status = %d, want %d", rec.Code, c.wantCode)
|
||||
}
|
||||
var resp ErrorResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if resp.Error.Type != c.wantType {
|
||||
t.Fatalf("error type = %q, want %q", resp.Error.Type, c.wantType)
|
||||
}
|
||||
if resp.Error.Code != c.wantCode {
|
||||
t.Fatalf("error code = %d, want %d", resp.Error.Code, c.wantCode)
|
||||
}
|
||||
for _, want := range c.wantMessage {
|
||||
if !strings.Contains(resp.Error.Message, want) {
|
||||
t.Errorf("message %q does not contain %q", resp.Error.Message, want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
105
weed/s3api/iceberg/iceberg_issue_9103_test.go
Normal file
105
weed/s3api/iceberg/iceberg_issue_9103_test.go
Normal file
@@ -0,0 +1,105 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGetBucketFromPrefix_WarehouseQueryFallback(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
url string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "warehouse query routes to its bucket when no prefix in path",
|
||||
url: "/v1/namespaces?warehouse=s3%3A%2F%2Fmyblkt%2F",
|
||||
want: "myblkt",
|
||||
},
|
||||
{
|
||||
name: "warehouse query with sub-path still picks the bucket",
|
||||
url: "/v1/namespaces?warehouse=s3%3A%2F%2Fanother%2Fextra",
|
||||
want: "another",
|
||||
},
|
||||
{
|
||||
name: "bare bucket name is taken as the table bucket",
|
||||
url: "/v1/namespaces?warehouse=not-a-url",
|
||||
want: "not-a-url",
|
||||
},
|
||||
{
|
||||
name: "table bucket ARN routes to its bucket",
|
||||
url: "/v1/namespaces?warehouse=arn%3Aaws%3As3tables%3Aus-east-1%3Aadmin%3Abucket%2Fseaweed-iceberg",
|
||||
want: "seaweed-iceberg",
|
||||
},
|
||||
{
|
||||
name: "unusable warehouse value falls through to default",
|
||||
url: "/v1/namespaces?warehouse=file%3A%2F%2F%2Ftmp%2Fwh",
|
||||
want: "warehouse",
|
||||
},
|
||||
{
|
||||
name: "no warehouse and no prefix returns default",
|
||||
url: "/v1/namespaces",
|
||||
want: "warehouse",
|
||||
},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
r := httptest.NewRequest("GET", tc.url, nil)
|
||||
got := getBucketFromPrefix(r)
|
||||
if got != tc.want {
|
||||
t.Fatalf("getBucketFromPrefix(%q) = %q, want %q", tc.url, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildFileIOConfig(t *testing.T) {
|
||||
loadTable := func() *http.Request {
|
||||
return httptest.NewRequest(http.MethodGet, "/v1/namespaces/ns/tables/t", nil)
|
||||
}
|
||||
|
||||
t.Run("no endpoint configured yields empty config", func(t *testing.T) {
|
||||
s := &Server{}
|
||||
got, _ := s.buildFileIOConfig(loadTable(), "s3://warehouse/ns/t")
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("buildFileIOConfig() = %v, want empty", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("endpoint is advertised with path-style-access and region", func(t *testing.T) {
|
||||
s := &Server{s3Endpoint: "http://seaweed.example:8333"}
|
||||
got, _ := s.buildFileIOConfig(loadTable(), "s3://warehouse/ns/t")
|
||||
if got["s3.endpoint"] != "http://seaweed.example:8333" {
|
||||
t.Fatalf("s3.endpoint = %q, want %q", got["s3.endpoint"], "http://seaweed.example:8333")
|
||||
}
|
||||
if got["s3.path-style-access"] != "true" {
|
||||
t.Fatalf("s3.path-style-access = %q, want %q", got["s3.path-style-access"], "true")
|
||||
}
|
||||
if got["s3.region"] == "" {
|
||||
t.Fatalf("s3.region was empty, want a non-empty default so clients like DuckDB do not require AWS_REGION")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestGetBucketFromPrefix_TableBucketEnvFallback(t *testing.T) {
|
||||
r := httptest.NewRequest("GET", "/v1/namespaces", nil)
|
||||
|
||||
t.Setenv("S3_TABLE_BUCKET", " ,analytics, other")
|
||||
if got := getBucketFromPrefix(r); got != "analytics" {
|
||||
t.Fatalf("first S3_TABLE_BUCKET entry: got %q, want analytics", got)
|
||||
}
|
||||
|
||||
// The explicit default still wins over the table bucket list.
|
||||
t.Setenv("S3TABLES_DEFAULT_BUCKET", "explicit")
|
||||
if got := getBucketFromPrefix(r); got != "explicit" {
|
||||
t.Fatalf("S3TABLES_DEFAULT_BUCKET override: got %q, want explicit", got)
|
||||
}
|
||||
|
||||
// A prefixless value with neither env set falls through to the default.
|
||||
t.Setenv("S3TABLES_DEFAULT_BUCKET", "")
|
||||
t.Setenv("S3_TABLE_BUCKET", " , ")
|
||||
if got := getBucketFromPrefix(r); got != "warehouse" {
|
||||
t.Fatalf("empty specs: got %q, want warehouse", got)
|
||||
}
|
||||
}
|
||||
29
weed/s3api/iceberg/iceberg_namespace_properties_test.go
Normal file
29
weed/s3api/iceberg/iceberg_namespace_properties_test.go
Normal file
@@ -0,0 +1,29 @@
|
||||
package iceberg
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestNormalizeNamespacePropertiesNil(t *testing.T) {
|
||||
properties := normalizeNamespaceProperties(nil)
|
||||
if properties == nil {
|
||||
t.Fatalf("normalizeNamespaceProperties(nil) returned nil map")
|
||||
}
|
||||
if len(properties) != 0 {
|
||||
t.Fatalf("normalizeNamespaceProperties(nil) length = %d, want 0", len(properties))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeNamespacePropertiesReturnsInputWhenSet(t *testing.T) {
|
||||
input := map[string]string{
|
||||
"owner": "analytics",
|
||||
}
|
||||
|
||||
properties := normalizeNamespaceProperties(input)
|
||||
if properties["owner"] != "analytics" {
|
||||
t.Fatalf("normalized properties value = %q, want %q", properties["owner"], "analytics")
|
||||
}
|
||||
|
||||
input["owner"] = "updated"
|
||||
if properties["owner"] != "updated" {
|
||||
t.Fatalf("normalizeNamespaceProperties should reuse the input map when non-nil")
|
||||
}
|
||||
}
|
||||
69
weed/s3api/iceberg/iceberg_pagination_test.go
Normal file
69
weed/s3api/iceberg/iceberg_pagination_test.go
Normal file
@@ -0,0 +1,69 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParsePaginationDefaultValues(t *testing.T) {
|
||||
req := httptest.NewRequest("GET", "/v1/namespaces", nil)
|
||||
|
||||
pageToken, pageSize, err := parsePagination(req)
|
||||
if err != nil {
|
||||
t.Fatalf("parsePagination() error = %v", err)
|
||||
}
|
||||
if pageToken != "" {
|
||||
t.Fatalf("pageToken = %q, want empty", pageToken)
|
||||
}
|
||||
if pageSize != defaultListPageSize {
|
||||
t.Fatalf("pageSize = %d, want %d", pageSize, defaultListPageSize)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePaginationUsesCamelCaseParameters(t *testing.T) {
|
||||
req := httptest.NewRequest("GET", "/v1/namespaces?pageToken=abc&pageSize=25", nil)
|
||||
|
||||
pageToken, pageSize, err := parsePagination(req)
|
||||
if err != nil {
|
||||
t.Fatalf("parsePagination() error = %v", err)
|
||||
}
|
||||
if pageToken != "abc" {
|
||||
t.Fatalf("pageToken = %q, want %q", pageToken, "abc")
|
||||
}
|
||||
if pageSize != 25 {
|
||||
t.Fatalf("pageSize = %d, want %d", pageSize, 25)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePaginationSupportsHyphenatedFallback(t *testing.T) {
|
||||
req := httptest.NewRequest("GET", "/v1/namespaces?page-token=abc&page-size=17", nil)
|
||||
|
||||
pageToken, pageSize, err := parsePagination(req)
|
||||
if err != nil {
|
||||
t.Fatalf("parsePagination() error = %v", err)
|
||||
}
|
||||
if pageToken != "abc" {
|
||||
t.Fatalf("pageToken = %q, want %q", pageToken, "abc")
|
||||
}
|
||||
if pageSize != 17 {
|
||||
t.Fatalf("pageSize = %d, want %d", pageSize, 17)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePaginationRejectsInvalidPageSize(t *testing.T) {
|
||||
testCases := []string{
|
||||
"/v1/namespaces?pageSize=0",
|
||||
"/v1/namespaces?pageSize=-1",
|
||||
"/v1/namespaces?pageSize=foo",
|
||||
"/v1/namespaces?pageSize=1001",
|
||||
}
|
||||
|
||||
for _, rawURL := range testCases {
|
||||
t.Run(rawURL, func(t *testing.T) {
|
||||
req := httptest.NewRequest("GET", rawURL, nil)
|
||||
if _, _, err := parsePagination(req); err == nil {
|
||||
t.Fatalf("parsePagination() expected error for url %q", rawURL)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
61
weed/s3api/iceberg/iceberg_register_table_test.go
Normal file
61
weed/s3api/iceberg/iceberg_register_table_test.go
Normal file
@@ -0,0 +1,61 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
)
|
||||
|
||||
// TestRegisterTableRequestWireShape pins the field names to the iceberg-go REST
|
||||
// client payload: {"name", "metadata-location"}.
|
||||
func TestRegisterTableRequestWireShape(t *testing.T) {
|
||||
body := `{"name":"orders","metadata-location":"s3://bucket/ns/orders/metadata/v1.metadata.json"}`
|
||||
var req RegisterTableRequest
|
||||
if err := json.Unmarshal([]byte(body), &req); err != nil {
|
||||
t.Fatalf("unmarshal: %v", err)
|
||||
}
|
||||
if req.Name != "orders" {
|
||||
t.Errorf("Name = %q, want orders", req.Name)
|
||||
}
|
||||
if req.MetadataLocation != "s3://bucket/ns/orders/metadata/v1.metadata.json" {
|
||||
t.Errorf("MetadataLocation = %q", req.MetadataLocation)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRegisterTableRejectsOutOfBoundsLocation confirms a metadata-location that
|
||||
// points outside the request's table bucket, or contains a traversal segment, is
|
||||
// rejected with 400 before any metadata read.
|
||||
func TestRegisterTableRejectsOutOfBoundsLocation(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
metadataLocation string
|
||||
}{
|
||||
{"cross bucket", "s3://other/ns/orders/metadata/v1.metadata.json"},
|
||||
{"traversal", "s3://bucket/ns/../../etc/metadata/v1.metadata.json"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
body := `{"name":"orders","metadata-location":"` + c.metadataLocation + `"}`
|
||||
r := httptest.NewRequest(http.MethodPost, "/v1/bucket/namespaces/ns/register", strings.NewReader(body))
|
||||
r = mux.SetURLVars(r, map[string]string{"prefix": "bucket", "namespace": "ns"})
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
(&Server{}).handleRegisterTable(rec, r)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d", rec.Code, http.StatusBadRequest)
|
||||
}
|
||||
var resp ErrorResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if resp.Error.Type != "BadRequestException" {
|
||||
t.Fatalf("error type = %q, want BadRequestException", resp.Error.Type)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
32
weed/s3api/iceberg/iceberg_rename_table_test.go
Normal file
32
weed/s3api/iceberg/iceberg_rename_table_test.go
Normal file
@@ -0,0 +1,32 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestHandleRenameTableRejectsIncompleteIdentifiers(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
body string
|
||||
}{
|
||||
{"missing source name", `{"source":{"namespace":["ns"]},"destination":{"namespace":["ns"],"name":"t2"}}`},
|
||||
{"missing source namespace", `{"source":{"name":"t"},"destination":{"namespace":["ns"],"name":"t2"}}`},
|
||||
{"missing destination name", `{"source":{"namespace":["ns"],"name":"t"},"destination":{"namespace":["ns"]}}`},
|
||||
{"missing destination namespace", `{"source":{"namespace":["ns"],"name":"t"},"destination":{"name":"t2"}}`},
|
||||
{"empty body", `{}`},
|
||||
{"malformed json", `{`},
|
||||
}
|
||||
s := &Server{}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
r := httptest.NewRequest("POST", "/v1/tables/rename", strings.NewReader(tc.body))
|
||||
w := httptest.NewRecorder()
|
||||
s.handleRenameTable(w, r)
|
||||
if w.Code != 400 {
|
||||
t.Fatalf("handleRenameTable() status = %d, want 400", w.Code)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
93
weed/s3api/iceberg/iceberg_snapshots_param_test.go
Normal file
93
weed/s3api/iceberg/iceberg_snapshots_param_test.go
Normal file
@@ -0,0 +1,93 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// metadataWithSnapshots builds metadata whose main branch is on the last
|
||||
// snapshot, optionally tagging one of the earlier ones.
|
||||
func metadataWithSnapshots(t *testing.T, ids []int64, tagged int64) table.Metadata {
|
||||
t.Helper()
|
||||
|
||||
base, err := newTableMetadata(uuid.New(), "s3://bkt/ns/t", nil, nil, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("newTableMetadata() error = %v", err)
|
||||
}
|
||||
builder, err := table.MetadataBuilderFromBase(base, "")
|
||||
if err != nil {
|
||||
t.Fatalf("MetadataBuilderFromBase() error = %v", err)
|
||||
}
|
||||
|
||||
now := time.Now().UnixMilli()
|
||||
for i, id := range ids {
|
||||
snapshot := table.Snapshot{
|
||||
SnapshotID: id,
|
||||
TimestampMs: now + int64(i),
|
||||
ManifestList: "metadata/snap-" + string(rune('0'+i)) + ".avro",
|
||||
}
|
||||
if err := builder.AddSnapshot(&snapshot); err != nil {
|
||||
t.Fatalf("AddSnapshot(%d) error = %v", id, err)
|
||||
}
|
||||
}
|
||||
if err := builder.SetSnapshotRef(table.MainBranch, ids[len(ids)-1], table.BranchRef); err != nil {
|
||||
t.Fatalf("SetSnapshotRef(main) error = %v", err)
|
||||
}
|
||||
if tagged != 0 {
|
||||
if err := builder.SetSnapshotRef("release", tagged, table.TagRef); err != nil {
|
||||
t.Fatalf("SetSnapshotRef(release) error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
metadata, err := builder.Build()
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
func snapshotIDs(metadata table.Metadata) map[int64]bool {
|
||||
ids := map[int64]bool{}
|
||||
for _, snapshot := range metadata.Snapshots() {
|
||||
ids[snapshot.SnapshotID] = true
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
func TestApplySnapshotsParamRefsKeepsOnlyReferenced(t *testing.T) {
|
||||
metadata := metadataWithSnapshots(t, []int64{1, 2, 3}, 1)
|
||||
|
||||
got, err := applySnapshotsParam(httptest.NewRequest("GET", "/v1/namespaces/ns/tables/t?snapshots=refs", nil), metadata)
|
||||
if err != nil {
|
||||
t.Fatalf("applySnapshotsParam() error = %v", err)
|
||||
}
|
||||
|
||||
ids := snapshotIDs(got)
|
||||
if !ids[1] {
|
||||
t.Error("tagged snapshot 1 was dropped")
|
||||
}
|
||||
if !ids[3] {
|
||||
t.Error("current snapshot 3 was dropped")
|
||||
}
|
||||
if ids[2] {
|
||||
t.Error("unreferenced snapshot 2 was returned")
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplySnapshotsParamDefaultsToAll(t *testing.T) {
|
||||
metadata := metadataWithSnapshots(t, []int64{1, 2, 3}, 1)
|
||||
|
||||
for _, target := range []string{"/v1/namespaces/ns/tables/t", "/v1/namespaces/ns/tables/t?snapshots=all"} {
|
||||
got, err := applySnapshotsParam(httptest.NewRequest("GET", target, nil), metadata)
|
||||
if err != nil {
|
||||
t.Fatalf("applySnapshotsParam(%s) error = %v", target, err)
|
||||
}
|
||||
if len(got.Snapshots()) != 3 {
|
||||
t.Errorf("applySnapshotsParam(%s) kept %d snapshots, want 3", target, len(got.Snapshots()))
|
||||
}
|
||||
}
|
||||
}
|
||||
254
weed/s3api/iceberg/iceberg_transaction_test.go
Normal file
254
weed/s3api/iceberg/iceberg_transaction_test.go
Normal file
@@ -0,0 +1,254 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
func TestDecodeCommitTransactionRequest(t *testing.T) {
|
||||
body := `{"table-changes":[
|
||||
{"identifier":{"namespace":["ns"],"name":"t1"},"requirements":[{"type":"assert-table-uuid","uuid":"00000000-0000-0000-0000-000000000001"}],"updates":[{"action":"set-properties","updates":{"k":"v"}}]},
|
||||
{"identifier":{"namespace":["ns","sub"],"name":"t2"},"requirements":[],"updates":[]}
|
||||
]}`
|
||||
|
||||
var req CommitTransactionRequest
|
||||
if err := json.Unmarshal([]byte(body), &req); err != nil {
|
||||
t.Fatalf("Unmarshal() error = %v", err)
|
||||
}
|
||||
if len(req.TableChanges) != 2 {
|
||||
t.Fatalf("table-changes = %d, want 2", len(req.TableChanges))
|
||||
}
|
||||
if req.TableChanges[0].Identifier.Name != "t1" {
|
||||
t.Fatalf("change[0] name = %q, want t1", req.TableChanges[0].Identifier.Name)
|
||||
}
|
||||
if got := []string(req.TableChanges[1].Identifier.Namespace); len(got) != 2 || got[1] != "sub" {
|
||||
t.Fatalf("change[1] namespace = %v, want [ns sub]", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseTableChangeSeparatesStatistics(t *testing.T) {
|
||||
change := tableChangeRequest{
|
||||
Requirements: json.RawMessage(`[{"type":"assert-table-uuid","uuid":"00000000-0000-0000-0000-000000000001"}]`),
|
||||
Updates: []json.RawMessage{
|
||||
json.RawMessage(`{"action":"set-properties","updates":{"k":"v"}}`),
|
||||
json.RawMessage(`{"action":"set-statistics","snapshot-id":10,"statistics-path":"s3://bucket/table/metadata/stats.puffin","file-size-in-bytes":100,"file-footer-size-in-bytes":20,"blob-metadata":[]}`),
|
||||
},
|
||||
}
|
||||
|
||||
requirements, updates, statistics, reqErr := parseTableChange(change)
|
||||
if reqErr != nil {
|
||||
t.Fatalf("parseTableChange() error = %+v", reqErr)
|
||||
}
|
||||
if len(requirements) != 1 {
|
||||
t.Fatalf("requirements = %d, want 1", len(requirements))
|
||||
}
|
||||
if len(updates) != 1 {
|
||||
t.Fatalf("updates = %d, want 1", len(updates))
|
||||
}
|
||||
if len(statistics) != 1 || statistics[0].set == nil || statistics[0].set.SnapshotID != 10 {
|
||||
t.Fatalf("unexpected statistics updates: %#v", statistics)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseTableChangeRejectsInvalidRequirements(t *testing.T) {
|
||||
change := tableChangeRequest{Requirements: json.RawMessage(`{not json}`)}
|
||||
_, _, _, reqErr := parseTableChange(change)
|
||||
if reqErr == nil {
|
||||
t.Fatalf("parseTableChange() expected error for invalid requirements")
|
||||
}
|
||||
if reqErr.status != 400 {
|
||||
t.Fatalf("status = %d, want 400", reqErr.status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloneTableMetadataIsIndependent(t *testing.T) {
|
||||
orig := &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{TableUUID: "uuid-1"},
|
||||
FullMetadata: json.RawMessage(`{"v":1}`),
|
||||
}
|
||||
clone := cloneTableMetadata(orig)
|
||||
if clone == orig || clone.Iceberg == orig.Iceberg {
|
||||
t.Fatalf("cloneTableMetadata returned aliased pointers")
|
||||
}
|
||||
|
||||
orig.Iceberg.TableUUID = "mutated"
|
||||
orig.FullMetadata[0] = 'X'
|
||||
if clone.Iceberg.TableUUID != "uuid-1" {
|
||||
t.Fatalf("clone Iceberg.TableUUID = %q, want uuid-1", clone.Iceberg.TableUUID)
|
||||
}
|
||||
if string(clone.FullMetadata) != `{"v":1}` {
|
||||
t.Fatalf("clone FullMetadata = %q, mutated with source", clone.FullMetadata)
|
||||
}
|
||||
|
||||
if cloneTableMetadata(nil) != nil {
|
||||
t.Fatalf("cloneTableMetadata(nil) = non-nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildTableRestoreRequestCarriesPriorState(t *testing.T) {
|
||||
pc := &preparedTableCommit{
|
||||
namespace: []string{"ns", "sub"},
|
||||
tableName: "t1",
|
||||
prevMetadataLoc: "s3://bucket/ns/t1/metadata/v3.metadata.json",
|
||||
prevMetadataVersion: 3,
|
||||
prevMetadata: &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{TableUUID: uuid.NewString()},
|
||||
FullMetadata: json.RawMessage(`{"format-version":2,"last-updated-ms":3}`),
|
||||
},
|
||||
}
|
||||
|
||||
req := buildTableRestoreRequest("arn:bucket", pc)
|
||||
|
||||
if req.TableBucketARN != "arn:bucket" || req.Name != "t1" {
|
||||
t.Fatalf("restore target = %s/%s, want arn:bucket/t1", req.TableBucketARN, req.Name)
|
||||
}
|
||||
if req.MetadataLocation != pc.prevMetadataLoc {
|
||||
t.Fatalf("restore MetadataLocation = %q, want %q", req.MetadataLocation, pc.prevMetadataLoc)
|
||||
}
|
||||
if req.MetadataVersion != pc.prevMetadataVersion {
|
||||
t.Fatalf("restore MetadataVersion = %d, want %d", req.MetadataVersion, pc.prevMetadataVersion)
|
||||
}
|
||||
if req.Metadata == nil || !bytes.Equal(req.Metadata.FullMetadata, pc.prevMetadata.FullMetadata) {
|
||||
t.Fatalf("restore FullMetadata = %q, want %q", req.Metadata.FullMetadata, pc.prevMetadata.FullMetadata)
|
||||
}
|
||||
if req.Metadata.Iceberg == nil || req.Metadata.Iceberg.TableUUID != pc.prevMetadata.Iceberg.TableUUID {
|
||||
t.Fatalf("restore TableUUID mismatch")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRollbackRevertsEveryFlippedField mirrors handleUpdateTable's partial-field
|
||||
// update semantics to prove that after a flip + rollback the table is back to its
|
||||
// pre-transaction state. With the old location-only rollback the table kept the
|
||||
// new full metadata and version; the restore request must revert all of them.
|
||||
func TestRollbackRevertsEveryFlippedField(t *testing.T) {
|
||||
// prior on-disk state of an already-committed table.
|
||||
prior := tableEntry{
|
||||
metadataLocation: "s3://bucket/ns/t1/metadata/v3.metadata.json",
|
||||
metadataVersion: 3,
|
||||
metadata: &s3tables.TableMetadata{
|
||||
Iceberg: &s3tables.IcebergMetadata{TableUUID: "uuid-1"},
|
||||
FullMetadata: json.RawMessage(`{"format-version":2,"v":3}`),
|
||||
},
|
||||
}
|
||||
|
||||
pc := &preparedTableCommit{
|
||||
namespace: []string{"ns"},
|
||||
tableName: "t1",
|
||||
tableUUID: uuid.MustParse("00000000-0000-0000-0000-000000000001"),
|
||||
metadataBytes: json.RawMessage(`{"format-version":2,"v":4}`),
|
||||
metadataVersion: 4,
|
||||
newMetadataLoc: "s3://bucket/ns/t1/metadata/v4.metadata.json",
|
||||
prevMetadataLoc: prior.metadataLocation,
|
||||
prevMetadataVersion: prior.metadataVersion,
|
||||
prevMetadata: cloneTableMetadata(prior.metadata),
|
||||
}
|
||||
|
||||
// flip: apply the new commit, then roll it back via the restore request.
|
||||
state := prior.clone()
|
||||
state.applyUpdate(&s3tables.UpdateTableRequest{
|
||||
Metadata: &s3tables.TableMetadata{Iceberg: &s3tables.IcebergMetadata{TableUUID: pc.tableUUID.String()}, FullMetadata: pc.metadataBytes},
|
||||
MetadataVersion: pc.metadataVersion,
|
||||
MetadataLocation: pc.newMetadataLoc,
|
||||
})
|
||||
state.applyUpdate(buildTableRestoreRequest("arn:bucket", pc))
|
||||
|
||||
if state.metadataLocation != prior.metadataLocation {
|
||||
t.Fatalf("after rollback location = %q, want %q", state.metadataLocation, prior.metadataLocation)
|
||||
}
|
||||
if state.metadataVersion != prior.metadataVersion {
|
||||
t.Fatalf("after rollback version = %d, want %d", state.metadataVersion, prior.metadataVersion)
|
||||
}
|
||||
if !bytes.Equal(state.metadata.FullMetadata, prior.metadata.FullMetadata) {
|
||||
t.Fatalf("after rollback FullMetadata = %q, want %q", state.metadata.FullMetadata, prior.metadata.FullMetadata)
|
||||
}
|
||||
if state.metadata.Iceberg.TableUUID != prior.metadata.Iceberg.TableUUID {
|
||||
t.Fatalf("after rollback TableUUID = %q, want %q", state.metadata.Iceberg.TableUUID, prior.metadata.Iceberg.TableUUID)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCleanupSafeMetadataKeepsFailedRollback proves a flip failure at index 2
|
||||
// where table 0's rollback also fails leaves table 0's new metadata in place
|
||||
// (its pointer still references it), while the rolled-back table 1 and the
|
||||
// never-flipped tables 2 and 3 are safe to delete.
|
||||
func TestCleanupSafeMetadataKeepsFailedRollback(t *testing.T) {
|
||||
prepared := make([]preparedTableCommit, 4)
|
||||
rolledBack := []bool{false, true} // tables 0 and 1 were flipped; 0's rollback failed
|
||||
|
||||
safe := cleanupSafeMetadata(prepared, 2, rolledBack)
|
||||
|
||||
want := []bool{false, true, true, true}
|
||||
for i := range want {
|
||||
if safe[i] != want[i] {
|
||||
t.Fatalf("safe[%d] = %v, want %v (full %v)", i, safe[i], want[i], safe)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCleanupSafeMetadataFirstFlipFails covers a flip failure at the first table:
|
||||
// nothing was flipped, so every written file is safe to delete.
|
||||
func TestCleanupSafeMetadataFirstFlipFails(t *testing.T) {
|
||||
prepared := make([]preparedTableCommit, 3)
|
||||
safe := cleanupSafeMetadata(prepared, 0, nil)
|
||||
for i, ok := range safe {
|
||||
if !ok {
|
||||
t.Fatalf("safe[%d] = false, want true (full %v)", i, safe)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCleanupSafeMetadataAllRolledBack covers the common case where every prior
|
||||
// flip rolls back cleanly, so all metadata files are safe to delete.
|
||||
func TestCleanupSafeMetadataAllRolledBack(t *testing.T) {
|
||||
prepared := make([]preparedTableCommit, 3)
|
||||
safe := cleanupSafeMetadata(prepared, 2, []bool{true, true})
|
||||
for i, ok := range safe {
|
||||
if !ok {
|
||||
t.Fatalf("safe[%d] = false, want true (full %v)", i, safe)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// tableEntry models the persisted table fields and the partial-update rules
|
||||
// handleUpdateTable applies, so rollback can be exercised without a live filer.
|
||||
type tableEntry struct {
|
||||
metadataLocation string
|
||||
metadataVersion int
|
||||
metadata *s3tables.TableMetadata
|
||||
}
|
||||
|
||||
func (e tableEntry) clone() *tableEntry {
|
||||
return &tableEntry{
|
||||
metadataLocation: e.metadataLocation,
|
||||
metadataVersion: e.metadataVersion,
|
||||
metadata: cloneTableMetadata(e.metadata),
|
||||
}
|
||||
}
|
||||
|
||||
func (e *tableEntry) applyUpdate(req *s3tables.UpdateTableRequest) {
|
||||
if req.Metadata != nil {
|
||||
if e.metadata == nil {
|
||||
e.metadata = &s3tables.TableMetadata{}
|
||||
}
|
||||
if req.Metadata.Iceberg != nil {
|
||||
if e.metadata.Iceberg == nil {
|
||||
e.metadata.Iceberg = &s3tables.IcebergMetadata{}
|
||||
}
|
||||
if req.Metadata.Iceberg.TableUUID != "" {
|
||||
e.metadata.Iceberg.TableUUID = req.Metadata.Iceberg.TableUUID
|
||||
}
|
||||
}
|
||||
if len(req.Metadata.FullMetadata) > 0 {
|
||||
e.metadata.FullMetadata = req.Metadata.FullMetadata
|
||||
}
|
||||
}
|
||||
if req.MetadataLocation != "" {
|
||||
e.metadataLocation = req.MetadataLocation
|
||||
}
|
||||
if req.MetadataVersion > 0 {
|
||||
e.metadataVersion = req.MetadataVersion
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
)
|
||||
|
||||
func TestApplyNamespacePropertyUpdates(t *testing.T) {
|
||||
current := map[string]string{"foo": "bar", "prop": "yes"}
|
||||
|
||||
props, summary := applyNamespacePropertyUpdates(current, []string{"abc"}, map[string]string{"prop": "no"})
|
||||
|
||||
// Removal of a missing key reports missing, not removed.
|
||||
if !reflect.DeepEqual(summary.Removed, []string{}) {
|
||||
t.Errorf("Removed = %v, want []", summary.Removed)
|
||||
}
|
||||
if !reflect.DeepEqual(summary.Updated, []string{"prop"}) {
|
||||
t.Errorf("Updated = %v, want [prop]", summary.Updated)
|
||||
}
|
||||
if !reflect.DeepEqual(summary.Missing, []string{"abc"}) {
|
||||
t.Errorf("Missing = %v, want [abc]", summary.Missing)
|
||||
}
|
||||
if props["prop"] != "no" || props["foo"] != "bar" {
|
||||
t.Errorf("merged properties = %v, want foo=bar prop=no", props)
|
||||
}
|
||||
|
||||
// Input map must not be mutated.
|
||||
if current["prop"] != "yes" {
|
||||
t.Errorf("source map mutated: prop = %q, want yes", current["prop"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyNamespacePropertyUpdatesRemoveExisting(t *testing.T) {
|
||||
props, summary := applyNamespacePropertyUpdates(
|
||||
map[string]string{"a": "1", "b": "2"}, []string{"a"}, nil)
|
||||
|
||||
if !reflect.DeepEqual(summary.Removed, []string{"a"}) {
|
||||
t.Errorf("Removed = %v, want [a]", summary.Removed)
|
||||
}
|
||||
if _, ok := props["a"]; ok {
|
||||
t.Errorf("key a should have been removed: %v", props)
|
||||
}
|
||||
if len(summary.Missing) != 0 || len(summary.Updated) != 0 {
|
||||
t.Errorf("unexpected summary: %+v", summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyNamespacePropertyUpdatesDuplicateRemovals(t *testing.T) {
|
||||
// A repeated removal key must not be reported as both removed and missing.
|
||||
_, summary := applyNamespacePropertyUpdates(
|
||||
map[string]string{"a": "1"}, []string{"a", "a"}, nil)
|
||||
|
||||
if !reflect.DeepEqual(summary.Removed, []string{"a"}) {
|
||||
t.Errorf("Removed = %v, want [a]", summary.Removed)
|
||||
}
|
||||
if len(summary.Missing) != 0 {
|
||||
t.Errorf("Missing = %v, want []", summary.Missing)
|
||||
}
|
||||
}
|
||||
|
||||
// A key in both removals and updates is rejected before any backend call, so
|
||||
// the conflict path is reachable without a filer.
|
||||
func TestHandleUpdateNamespacePropertiesConflict(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/namespaces/ns/properties",
|
||||
strings.NewReader(`{"removals":["x"],"updates":{"x":"1"}}`))
|
||||
req = mux.SetURLVars(req, map[string]string{"namespace": "ns"})
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
(&Server{}).handleUpdateNamespaceProperties(rec, req)
|
||||
|
||||
if rec.Code != http.StatusUnprocessableEntity {
|
||||
t.Fatalf("status = %d, want 422", rec.Code)
|
||||
}
|
||||
var resp ErrorResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if resp.Error.Type != "UnprocessableEntityException" {
|
||||
t.Fatalf("error type = %q, want UnprocessableEntityException", resp.Error.Type)
|
||||
}
|
||||
}
|
||||
115
weed/s3api/iceberg/iceberg_view_test.go
Normal file
115
weed/s3api/iceberg/iceberg_view_test.go
Normal file
@@ -0,0 +1,115 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/apache/iceberg-go/view"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
func TestIsViewNotFound(t *testing.T) {
|
||||
cases := []struct {
|
||||
err error
|
||||
want bool
|
||||
}{
|
||||
{nil, false},
|
||||
{&s3tables.S3TablesError{Type: s3tables.ErrCodeNoSuchView, Message: "view x not found"}, true},
|
||||
{&s3tables.S3TablesError{Type: s3tables.ErrCodeNoSuchNamespace, Message: "namespace x not found"}, true},
|
||||
{errors.New("view x not found"), true},
|
||||
{&s3tables.S3TablesError{Type: s3tables.ErrCodeViewAlreadyExists, Message: "already exists"}, false},
|
||||
{errors.New("connection refused"), false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := isViewNotFound(c.err); got != c.want {
|
||||
t.Errorf("isViewNotFound(%v) = %v, want %v", c.err, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsViewAlreadyExists(t *testing.T) {
|
||||
cases := []struct {
|
||||
err error
|
||||
want bool
|
||||
}{
|
||||
{nil, false},
|
||||
{&s3tables.S3TablesError{Type: s3tables.ErrCodeViewAlreadyExists, Message: "already exists"}, true},
|
||||
{errors.New("view already exists"), true},
|
||||
{&s3tables.S3TablesError{Type: s3tables.ErrCodeNoSuchView, Message: "not found"}, false},
|
||||
{errors.New("connection refused"), false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := isViewAlreadyExists(c.err); got != c.want {
|
||||
t.Errorf("isViewAlreadyExists(%v) = %v, want %v", c.err, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestViewPropertiesCopiesAndIsNonNil(t *testing.T) {
|
||||
if got := viewProperties(nil); got == nil {
|
||||
t.Fatal("viewProperties(nil) = nil, want non-nil")
|
||||
}
|
||||
src := iceberg.Properties{"k": "v"}
|
||||
out := viewProperties(src)
|
||||
out["k2"] = "v2"
|
||||
if _, ok := src["k2"]; ok {
|
||||
t.Fatal("viewProperties did not copy: mutation leaked into source")
|
||||
}
|
||||
if out["k"] != "v" {
|
||||
t.Fatalf("viewProperties dropped key: got %q", out["k"])
|
||||
}
|
||||
}
|
||||
|
||||
// TestViewResponseRoundTrip verifies a built view metadata marshals into the
|
||||
// ViewResponse wire shape and parses back via the iceberg-go view parser.
|
||||
func TestViewResponseRoundTrip(t *testing.T) {
|
||||
schema := iceberg.NewSchemaWithIdentifiers(0, nil,
|
||||
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int32, Required: true},
|
||||
)
|
||||
ver, err := view.NewVersionFromSQL(1, schema.ID, "SELECT 1", table.Identifier{"ns"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
md, err := view.NewMetadata(ver, schema, "s3://bucket/ns/v", iceberg.Properties{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
resp := ViewResponse{
|
||||
MetadataLocation: "s3://bucket/ns/v/metadata/v1.metadata.json",
|
||||
Metadata: md,
|
||||
Config: iceberg.Properties{},
|
||||
}
|
||||
data, err := json.Marshal(resp)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
|
||||
var raw map[string]json.RawMessage
|
||||
if err := json.Unmarshal(data, &raw); err != nil {
|
||||
t.Fatalf("unmarshal envelope: %v", err)
|
||||
}
|
||||
if _, ok := raw["metadata-location"]; !ok {
|
||||
t.Fatal("response missing metadata-location")
|
||||
}
|
||||
if _, ok := raw["metadata"]; !ok {
|
||||
t.Fatal("response missing metadata")
|
||||
}
|
||||
|
||||
var back ViewResponse
|
||||
if err := json.Unmarshal(data, &back); err != nil {
|
||||
t.Fatalf("unmarshal back: %v", err)
|
||||
}
|
||||
if back.Metadata == nil {
|
||||
t.Fatal("metadata nil after round-trip")
|
||||
}
|
||||
if back.Metadata.CurrentVersion().VersionID != 1 {
|
||||
t.Fatalf("current version = %d, want 1", back.Metadata.CurrentVersion().VersionID)
|
||||
}
|
||||
if back.MetadataLocation != resp.MetadataLocation {
|
||||
t.Fatalf("metadata-location = %q, want %q", back.MetadataLocation, resp.MetadataLocation)
|
||||
}
|
||||
}
|
||||
71
weed/s3api/iceberg/iceberg_warehouse_test.go
Normal file
71
weed/s3api/iceberg/iceberg_warehouse_test.go
Normal file
@@ -0,0 +1,71 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The admin console tells users that every table bucket is its own Iceberg
|
||||
// catalog, so the /v1/config handshake must turn whichever spelling of that
|
||||
// bucket the client sends into overrides.prefix. PyIceberg only sends the
|
||||
// warehouse on this one call, so a dropped value leaves it talking to the
|
||||
// default bucket for the rest of the session.
|
||||
func TestHandleConfigWarehouseSpellings(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
warehouse string
|
||||
wantPrefix string
|
||||
wantWarehouse string
|
||||
}{
|
||||
{
|
||||
name: "s3 location",
|
||||
warehouse: "s3://seaweed-iceberg/",
|
||||
wantPrefix: "seaweed-iceberg",
|
||||
wantWarehouse: "s3://seaweed-iceberg",
|
||||
},
|
||||
{
|
||||
name: "bare bucket name",
|
||||
warehouse: "seaweed-iceberg",
|
||||
wantPrefix: "seaweed-iceberg",
|
||||
wantWarehouse: "s3://seaweed-iceberg",
|
||||
},
|
||||
{
|
||||
name: "table bucket ARN",
|
||||
warehouse: "arn:aws:s3tables:us-east-1:admin:bucket/seaweed-iceberg",
|
||||
wantPrefix: "seaweed-iceberg",
|
||||
wantWarehouse: "s3://seaweed-iceberg",
|
||||
},
|
||||
{
|
||||
name: "no warehouse",
|
||||
warehouse: "",
|
||||
},
|
||||
{
|
||||
name: "unusable value",
|
||||
warehouse: "file:///tmp/wh",
|
||||
},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
target := "/v1/config"
|
||||
if tc.warehouse != "" {
|
||||
target += "?" + url.Values{"warehouse": {tc.warehouse}}.Encode()
|
||||
}
|
||||
r := httptest.NewRequest("GET", target, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
(&Server{}).handleConfig(rec, r)
|
||||
|
||||
var config CatalogConfig
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &config); err != nil {
|
||||
t.Fatalf("decode config: %v", err)
|
||||
}
|
||||
if got := config.Overrides["prefix"]; got != tc.wantPrefix {
|
||||
t.Errorf("overrides.prefix = %q, want %q", got, tc.wantPrefix)
|
||||
}
|
||||
if got := config.Defaults["warehouse"]; got != tc.wantWarehouse {
|
||||
t.Errorf("defaults.warehouse = %q, want %q", got, tc.wantWarehouse)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
618
weed/s3api/iceberg/manifest_repair.go
Normal file
618
weed/s3api/iceberg/manifest_repair.go
Normal file
@@ -0,0 +1,618 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/linkedin/goavro/v2"
|
||||
"github.com/seaweedfs/seaweedfs/weed/filer"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/wdclient"
|
||||
)
|
||||
|
||||
// Some engines (ClickHouse's experimental Iceberg writes among them) commit
|
||||
// snapshots whose manifest avro files omit the spec's field-id annotations and
|
||||
// reference files by bucket-relative path. Both make the snapshot unreadable
|
||||
// for other engines even though the data itself is fine. Since the catalog is
|
||||
// the commit chokepoint, add-snapshot updates are repaired here: the manifest
|
||||
// list and manifests are transcoded with the annotations added and the paths
|
||||
// absolutized, and the snapshot is pointed at the repaired copies.
|
||||
|
||||
// avroFieldIDs describes the spec-mandated ids for one avro field and the
|
||||
// fields nested beneath it (through unions, arrays, and records).
|
||||
type avroFieldIDs struct {
|
||||
id int
|
||||
elementID int
|
||||
children map[string]avroFieldIDs
|
||||
}
|
||||
|
||||
// manifestListFieldIDs returns the fixed v2 manifest_file schema ids.
|
||||
func manifestListFieldIDs() map[string]avroFieldIDs {
|
||||
return map[string]avroFieldIDs{
|
||||
"manifest_path": {id: 500},
|
||||
"manifest_length": {id: 501},
|
||||
"partition_spec_id": {id: 502},
|
||||
"content": {id: 517},
|
||||
"sequence_number": {id: 515},
|
||||
"min_sequence_number": {id: 516},
|
||||
"added_snapshot_id": {id: 503},
|
||||
"added_files_count": {id: 504},
|
||||
"existing_files_count": {id: 505},
|
||||
"deleted_files_count": {id: 506},
|
||||
"added_rows_count": {id: 512},
|
||||
"existing_rows_count": {id: 513},
|
||||
"deleted_rows_count": {id: 514},
|
||||
"partitions": {id: 507, elementID: 508, children: map[string]avroFieldIDs{
|
||||
"contains_null": {id: 509},
|
||||
"contains_nan": {id: 518},
|
||||
"lower_bound": {id: 510},
|
||||
"upper_bound": {id: 511},
|
||||
}},
|
||||
"key_metadata": {id: 519},
|
||||
"first_row_id": {id: 520},
|
||||
}
|
||||
}
|
||||
|
||||
// manifestEntryFieldIDs returns the fixed v2 manifest_entry schema ids. The
|
||||
// partition struct is table-specific; its ids come from the partition-spec
|
||||
// JSON embedded in the manifest's OCF metadata.
|
||||
func manifestEntryFieldIDs(partitionIDs map[string]avroFieldIDs) map[string]avroFieldIDs {
|
||||
return map[string]avroFieldIDs{
|
||||
"status": {id: 0},
|
||||
"snapshot_id": {id: 1},
|
||||
"sequence_number": {id: 3},
|
||||
"file_sequence_number": {id: 4},
|
||||
"data_file": {id: 2, children: map[string]avroFieldIDs{
|
||||
"content": {id: 134},
|
||||
"file_path": {id: 100},
|
||||
"file_format": {id: 101},
|
||||
"partition": {id: 102, children: partitionIDs},
|
||||
"record_count": {id: 103},
|
||||
"file_size_in_bytes": {id: 104},
|
||||
"column_sizes": {id: 108, children: kvFieldIDs(117, 118)},
|
||||
"value_counts": {id: 109, children: kvFieldIDs(119, 120)},
|
||||
"null_value_counts": {id: 110, children: kvFieldIDs(121, 122)},
|
||||
"nan_value_counts": {id: 137, children: kvFieldIDs(138, 139)},
|
||||
"lower_bounds": {id: 125, children: kvFieldIDs(126, 127)},
|
||||
"upper_bounds": {id: 128, children: kvFieldIDs(129, 130)},
|
||||
"key_metadata": {id: 131},
|
||||
"split_offsets": {id: 132, elementID: 133},
|
||||
"equality_ids": {id: 135, elementID: 136},
|
||||
"sort_order_id": {id: 140},
|
||||
"first_row_id": {id: 142},
|
||||
"referenced_data_file": {id: 143},
|
||||
"content_offset": {id: 144},
|
||||
"content_size_in_bytes": {id: 145},
|
||||
}},
|
||||
}
|
||||
}
|
||||
|
||||
func kvFieldIDs(keyID, valueID int) map[string]avroFieldIDs {
|
||||
return map[string]avroFieldIDs{
|
||||
"key": {id: keyID},
|
||||
"value": {id: valueID},
|
||||
}
|
||||
}
|
||||
|
||||
// partitionFieldIDsFromSpec extracts name -> field-id from the partition-spec
|
||||
// JSON carried in a manifest's OCF metadata.
|
||||
func partitionFieldIDsFromSpec(specJSON []byte) (map[string]avroFieldIDs, error) {
|
||||
if len(specJSON) == 0 {
|
||||
return map[string]avroFieldIDs{}, nil
|
||||
}
|
||||
var fields []struct {
|
||||
Name string `json:"name"`
|
||||
FieldID int `json:"field-id"`
|
||||
}
|
||||
if err := json.Unmarshal(specJSON, &fields); err != nil {
|
||||
return nil, fmt.Errorf("parse partition-spec metadata: %w", err)
|
||||
}
|
||||
ids := make(map[string]avroFieldIDs, len(fields))
|
||||
for _, f := range fields {
|
||||
ids[f.Name] = avroFieldIDs{id: f.FieldID}
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
// annotateAvroRecordFields adds "field-id" to every field of a parsed avro
|
||||
// record schema, recursing into nested types. Returns whether anything was
|
||||
// added, and ok=false when a field has no known id (the schema is left for an
|
||||
// unknown writer dialect rather than half-annotated).
|
||||
func annotateAvroRecordFields(record map[string]any, ids map[string]avroFieldIDs) (changed, ok bool) {
|
||||
fields, _ := record["fields"].([]any)
|
||||
if len(fields) == 0 {
|
||||
return false, true
|
||||
}
|
||||
for _, f := range fields {
|
||||
field, isMap := f.(map[string]any)
|
||||
if !isMap {
|
||||
return changed, false
|
||||
}
|
||||
name, _ := field["name"].(string)
|
||||
spec, known := ids[name]
|
||||
if !known {
|
||||
return changed, false
|
||||
}
|
||||
if _, has := field["field-id"]; !has {
|
||||
field["field-id"] = spec.id
|
||||
changed = true
|
||||
}
|
||||
typeChanged, typeOK := annotateAvroType(field["type"], spec)
|
||||
changed = changed || typeChanged
|
||||
if !typeOK {
|
||||
return changed, false
|
||||
}
|
||||
}
|
||||
return changed, true
|
||||
}
|
||||
|
||||
// annotateAvroType annotates the schema node of a single field: unions are
|
||||
// followed through their non-null branches, arrays get "element-id", and
|
||||
// nested records recurse with the field's child ids.
|
||||
func annotateAvroType(node any, spec avroFieldIDs) (changed, ok bool) {
|
||||
switch t := node.(type) {
|
||||
case []any: // union
|
||||
for _, branch := range t {
|
||||
branchChanged, branchOK := annotateAvroType(branch, spec)
|
||||
changed = changed || branchChanged
|
||||
if !branchOK {
|
||||
return changed, false
|
||||
}
|
||||
}
|
||||
return changed, true
|
||||
case map[string]any:
|
||||
switch t["type"] {
|
||||
case "record":
|
||||
childIDs := spec.children
|
||||
if childIDs == nil {
|
||||
childIDs = map[string]avroFieldIDs{}
|
||||
}
|
||||
return annotateAvroRecordFields(t, childIDs)
|
||||
case "array":
|
||||
if spec.elementID != 0 {
|
||||
if _, has := t["element-id"]; !has {
|
||||
t["element-id"] = spec.elementID
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
itemChanged, itemOK := annotateAvroType(t["items"], spec)
|
||||
return changed || itemChanged, itemOK
|
||||
case "map":
|
||||
// The spec wants key-id/value-id on genuine avro maps; this repair
|
||||
// only knows the array-of-key_value encoding, so leave the file to
|
||||
// its writer rather than half-annotating it.
|
||||
return changed, false
|
||||
default:
|
||||
return false, true
|
||||
}
|
||||
default: // primitive name or named-type reference
|
||||
return false, true
|
||||
}
|
||||
}
|
||||
|
||||
// transcodeManifestAvro decodes an avro OCF file, annotates its schema with
|
||||
// the given ids, applies mutate to each datum and fixMeta to the OCF metadata,
|
||||
// and re-encodes. When nothing changes, the original bytes are returned
|
||||
// unchanged.
|
||||
func transcodeManifestAvro(raw []byte, ids map[string]avroFieldIDs, mutate func(map[string]any) bool, fixMeta func(map[string][]byte) bool) ([]byte, bool, error) {
|
||||
reader, err := goavro.NewOCFReader(bytes.NewReader(raw))
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("open avro: %w", err)
|
||||
}
|
||||
meta := reader.MetaData()
|
||||
|
||||
outMeta := make(map[string][]byte, len(meta))
|
||||
for k, v := range meta {
|
||||
if k == "avro.schema" || k == "avro.codec" {
|
||||
continue
|
||||
}
|
||||
outMeta[k] = v
|
||||
}
|
||||
metaChanged := false
|
||||
if fixMeta != nil {
|
||||
metaChanged = fixMeta(outMeta)
|
||||
}
|
||||
|
||||
var schema map[string]any
|
||||
if err := json.Unmarshal(meta["avro.schema"], &schema); err != nil {
|
||||
return nil, false, fmt.Errorf("parse avro schema: %w", err)
|
||||
}
|
||||
schemaChanged, ok := annotateAvroRecordFields(schema, ids)
|
||||
if !ok {
|
||||
return nil, false, fmt.Errorf("avro schema has fields outside the Iceberg manifest spec")
|
||||
}
|
||||
|
||||
datumChanged := false
|
||||
var datums []any
|
||||
for reader.Scan() {
|
||||
datum, err := reader.Read()
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("read avro datum: %w", err)
|
||||
}
|
||||
if record, isMap := datum.(map[string]any); isMap && mutate != nil {
|
||||
if mutate(record) {
|
||||
datumChanged = true
|
||||
}
|
||||
}
|
||||
datums = append(datums, datum)
|
||||
}
|
||||
if err := reader.Err(); err != nil {
|
||||
return nil, false, fmt.Errorf("scan avro: %w", err)
|
||||
}
|
||||
|
||||
if !schemaChanged && !datumChanged && !metaChanged {
|
||||
return raw, false, nil
|
||||
}
|
||||
|
||||
annotatedSchema, err := json.Marshal(schema)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
writer, err := goavro.NewOCFWriter(goavro.OCFConfig{
|
||||
W: &buf,
|
||||
Schema: string(annotatedSchema),
|
||||
MetaData: outMeta,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("create avro writer: %w", err)
|
||||
}
|
||||
if err := writer.Append(datums); err != nil {
|
||||
return nil, false, fmt.Errorf("write avro datums: %w", err)
|
||||
}
|
||||
return buf.Bytes(), true, nil
|
||||
}
|
||||
|
||||
// manifestStore abstracts reading and writing files in a table's metadata
|
||||
// directory so the repair logic is testable without a filer.
|
||||
type manifestStore interface {
|
||||
loadFile(ctx context.Context, location string) ([]byte, error)
|
||||
saveFile(ctx context.Context, location string, data []byte) error
|
||||
}
|
||||
|
||||
// absolutizeLocation resolves a bucket-relative path against the bucket of the
|
||||
// table location. Locations that already carry a scheme pass through.
|
||||
func absolutizeLocation(location, tableLocation string) string {
|
||||
if location == "" || strings.Contains(location, "://") {
|
||||
return location
|
||||
}
|
||||
bucket, _, err := parseS3Location(tableLocation)
|
||||
if err != nil {
|
||||
return location
|
||||
}
|
||||
return "s3://" + bucket + "/" + strings.TrimPrefix(location, "/")
|
||||
}
|
||||
|
||||
// metadataFileName returns the file name when location is a direct child of
|
||||
// the table's metadata directory, which is the only place repair will read
|
||||
// from or write to.
|
||||
func metadataFileName(tableLocation, location string) (string, bool) {
|
||||
prefix := strings.TrimSuffix(tableLocation, "/") + "/metadata/"
|
||||
name := strings.TrimPrefix(location, prefix)
|
||||
if name == location || name == "" || strings.Contains(name, "/") {
|
||||
return "", false
|
||||
}
|
||||
return name, true
|
||||
}
|
||||
|
||||
const repairedManifestPrefix = "repaired-"
|
||||
|
||||
// maxRepairableManifestSize bounds how much manifest data the repair path
|
||||
// buffers in memory; anything larger fails the repair and the commit
|
||||
// proceeds with the writer's original files.
|
||||
const maxRepairableManifestSize = 64 << 20
|
||||
|
||||
// hasFieldIDAnnotations is the cheap compliance probe: spec-compliant writers
|
||||
// annotate every manifest avro schema with field-id, and a writer that
|
||||
// annotates the manifest list annotates its manifests too.
|
||||
func hasFieldIDAnnotations(raw []byte) bool {
|
||||
reader, err := goavro.NewOCFReader(bytes.NewReader(raw))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
return bytes.Contains(reader.MetaData()["avro.schema"], []byte(`"field-id"`))
|
||||
}
|
||||
|
||||
// repairManifestList loads a snapshot's manifest list and, when it needs
|
||||
// repair, rewrites it (and the manifests it references) into spec-compliant
|
||||
// copies next to the originals. Returns the manifest list location the
|
||||
// snapshot should reference.
|
||||
func repairManifestList(ctx context.Context, store manifestStore, tableLocation, listLocation string) (string, bool, error) {
|
||||
absListLocation := absolutizeLocation(listLocation, tableLocation)
|
||||
listName, inMetadataDir := metadataFileName(tableLocation, absListLocation)
|
||||
if !inMetadataDir {
|
||||
return listLocation, false, nil
|
||||
}
|
||||
|
||||
listBytes, err := store.loadFile(ctx, absListLocation)
|
||||
if err != nil {
|
||||
return listLocation, false, fmt.Errorf("load manifest list: %w", err)
|
||||
}
|
||||
if absListLocation == listLocation && hasFieldIDAnnotations(listBytes) {
|
||||
return listLocation, false, nil
|
||||
}
|
||||
|
||||
metadataDir := strings.TrimSuffix(tableLocation, "/") + "/metadata/"
|
||||
var manifestErr error
|
||||
repairedList, listChanged, err := transcodeManifestAvro(listBytes, manifestListFieldIDs(), func(record map[string]any) bool {
|
||||
manifestPath, _ := record["manifest_path"].(string)
|
||||
if manifestPath == "" || manifestErr != nil {
|
||||
return false
|
||||
}
|
||||
absManifest := absolutizeLocation(manifestPath, tableLocation)
|
||||
recordChanged := absManifest != manifestPath
|
||||
if recordChanged {
|
||||
record["manifest_path"] = absManifest
|
||||
}
|
||||
|
||||
manifestName, ok := metadataFileName(tableLocation, absManifest)
|
||||
if !ok {
|
||||
return recordChanged
|
||||
}
|
||||
manifestBytes, err := store.loadFile(ctx, absManifest)
|
||||
if err != nil {
|
||||
manifestErr = fmt.Errorf("load manifest %s: %w", absManifest, err)
|
||||
return recordChanged
|
||||
}
|
||||
repaired, changed, err := repairManifest(manifestBytes, tableLocation, manifestContentValue(record))
|
||||
if err != nil {
|
||||
manifestErr = fmt.Errorf("repair manifest %s: %w", absManifest, err)
|
||||
return recordChanged
|
||||
}
|
||||
if !changed {
|
||||
return recordChanged
|
||||
}
|
||||
repairedLocation := metadataDir + repairedManifestPrefix + manifestName
|
||||
if err := store.saveFile(ctx, repairedLocation, repaired); err != nil {
|
||||
manifestErr = fmt.Errorf("save repaired manifest %s: %w", repairedLocation, err)
|
||||
return recordChanged
|
||||
}
|
||||
record["manifest_path"] = repairedLocation
|
||||
record["manifest_length"] = int64(len(repaired))
|
||||
return true
|
||||
}, nil)
|
||||
if err != nil {
|
||||
return listLocation, false, err
|
||||
}
|
||||
if manifestErr != nil {
|
||||
return listLocation, false, manifestErr
|
||||
}
|
||||
if !listChanged && absListLocation == listLocation {
|
||||
return listLocation, false, nil
|
||||
}
|
||||
|
||||
repairedListLocation := metadataDir + repairedManifestPrefix + listName
|
||||
if listChanged {
|
||||
if err := store.saveFile(ctx, repairedListLocation, repairedList); err != nil {
|
||||
return listLocation, false, fmt.Errorf("save repaired manifest list: %w", err)
|
||||
}
|
||||
return repairedListLocation, true, nil
|
||||
}
|
||||
// Only the pointer was relative; the file itself is fine.
|
||||
return absListLocation, true, nil
|
||||
}
|
||||
|
||||
// manifestContentValue reads the manifest-list entry's content field
|
||||
// (0 = data, 1 = deletes); v1 lists have no such field and default to data.
|
||||
// Union-typed values arrive from goavro as a single-branch map.
|
||||
func manifestContentValue(record map[string]any) int {
|
||||
value := record["content"]
|
||||
if union, isUnion := value.(map[string]any); isUnion {
|
||||
for _, branch := range union {
|
||||
value = branch
|
||||
break
|
||||
}
|
||||
}
|
||||
switch v := value.(type) {
|
||||
case int32:
|
||||
return int(v)
|
||||
case int64:
|
||||
return int(v)
|
||||
case int:
|
||||
return v
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// fixManifestOCFMetadata fixes OCF metadata keys the spec requires of
|
||||
// manifests but lax writers omit or contradict: "content" must agree with the
|
||||
// manifest-list entry, which is what readers trust for scan planning, and
|
||||
// "partition-spec-id" (ClickHouse writes the underscore variant).
|
||||
func fixManifestOCFMetadata(listContent int) func(map[string][]byte) bool {
|
||||
return func(meta map[string][]byte) bool {
|
||||
changed := false
|
||||
content := "data"
|
||||
if listContent == 1 {
|
||||
content = "deletes"
|
||||
}
|
||||
if string(meta["content"]) != content {
|
||||
meta["content"] = []byte(content)
|
||||
changed = true
|
||||
}
|
||||
if _, ok := meta["partition-spec-id"]; !ok {
|
||||
if v, ok := meta["partition_spec_id"]; ok {
|
||||
meta["partition-spec-id"] = v
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
return changed
|
||||
}
|
||||
}
|
||||
|
||||
// repairManifest annotates a single manifest's schema and absolutizes the
|
||||
// data file paths inside it. listContent is the content declared by the
|
||||
// manifest-list entry referencing this manifest.
|
||||
func repairManifest(raw []byte, tableLocation string, listContent int) ([]byte, bool, error) {
|
||||
reader, err := goavro.NewOCFReader(bytes.NewReader(raw))
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("open manifest avro: %w", err)
|
||||
}
|
||||
partitionIDs, err := partitionFieldIDsFromSpec(reader.MetaData()["partition-spec"])
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
return transcodeManifestAvro(raw, manifestEntryFieldIDs(partitionIDs), func(record map[string]any) bool {
|
||||
dataFile, _ := record["data_file"].(map[string]any)
|
||||
if dataFile == nil {
|
||||
return false
|
||||
}
|
||||
filePath, _ := dataFile["file_path"].(string)
|
||||
absPath := absolutizeLocation(filePath, tableLocation)
|
||||
if absPath == filePath {
|
||||
return false
|
||||
}
|
||||
dataFile["file_path"] = absPath
|
||||
return true
|
||||
}, fixManifestOCFMetadata(listContent))
|
||||
}
|
||||
|
||||
// repairAddSnapshotUpdates rewrites the manifest-list references of
|
||||
// add-snapshot updates whose manifests need repair. Repair is best effort: on
|
||||
// any error the original update is kept and the commit proceeds as it would
|
||||
// have without repair.
|
||||
func repairAddSnapshotUpdates(ctx context.Context, store manifestStore, tableLocation string, rawUpdates []json.RawMessage) ([]json.RawMessage, bool) {
|
||||
changedAny := false
|
||||
out := make([]json.RawMessage, len(rawUpdates))
|
||||
copy(out, rawUpdates)
|
||||
for i, rawUpdate := range rawUpdates {
|
||||
var probe struct {
|
||||
Action string `json:"action"`
|
||||
}
|
||||
if err := json.Unmarshal(rawUpdate, &probe); err != nil || probe.Action != "add-snapshot" {
|
||||
continue
|
||||
}
|
||||
var update map[string]json.RawMessage
|
||||
if err := json.Unmarshal(rawUpdate, &update); err != nil {
|
||||
continue
|
||||
}
|
||||
var snapshot map[string]json.RawMessage
|
||||
if err := json.Unmarshal(update["snapshot"], &snapshot); err != nil {
|
||||
continue
|
||||
}
|
||||
var listLocation string
|
||||
if err := json.Unmarshal(snapshot["manifest-list"], &listLocation); err != nil || listLocation == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
repairedLocation, changed, err := repairManifestList(ctx, store, tableLocation, listLocation)
|
||||
if err != nil {
|
||||
glog.Warningf("Iceberg: manifest repair for %s skipped: %v", listLocation, err)
|
||||
continue
|
||||
}
|
||||
if !changed {
|
||||
continue
|
||||
}
|
||||
|
||||
locationJSON, err := json.Marshal(repairedLocation)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
snapshot["manifest-list"] = locationJSON
|
||||
snapshotJSON, err := json.Marshal(snapshot)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
update["snapshot"] = snapshotJSON
|
||||
updateJSON, err := json.Marshal(update)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
out[i] = updateJSON
|
||||
changedAny = true
|
||||
glog.V(1).Infof("Iceberg: repaired manifest list %s -> %s", listLocation, repairedLocation)
|
||||
}
|
||||
return out, changedAny
|
||||
}
|
||||
|
||||
// serverManifestStore reads and writes table metadata files through the filer.
|
||||
type serverManifestStore struct {
|
||||
server *Server
|
||||
tableLocation string
|
||||
}
|
||||
|
||||
func (st *serverManifestStore) split(location string) (bucket, tablePath, fileName string, err error) {
|
||||
fileName, ok := metadataFileName(st.tableLocation, location)
|
||||
if !ok {
|
||||
return "", "", "", fmt.Errorf("%s is outside the table metadata directory", location)
|
||||
}
|
||||
bucket, tablePath, err = parseS3Location(st.tableLocation)
|
||||
return bucket, tablePath, fileName, err
|
||||
}
|
||||
|
||||
func (st *serverManifestStore) loadFile(ctx context.Context, location string) ([]byte, error) {
|
||||
bucket, tablePath, fileName, err := st.split(location)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return st.server.loadTableMetadataBlob(ctx, bucket, tablePath, fileName)
|
||||
}
|
||||
|
||||
func (st *serverManifestStore) saveFile(ctx context.Context, location string, data []byte) error {
|
||||
bucket, tablePath, fileName, err := st.split(location)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return st.server.saveMetadataBlob(ctx, bucket, tablePath, fileName, data, "application/avro", false)
|
||||
}
|
||||
|
||||
// repairAddSnapshotManifests is the server entry point used by the commit
|
||||
// handlers.
|
||||
func (s *Server) repairAddSnapshotManifests(ctx context.Context, tableLocation string, rawUpdates []json.RawMessage) ([]json.RawMessage, bool) {
|
||||
store := &serverManifestStore{server: s, tableLocation: tableLocation}
|
||||
return repairAddSnapshotUpdates(ctx, store, tableLocation, rawUpdates)
|
||||
}
|
||||
|
||||
// lookupFileIDAdapter lets filer.StreamContent resolve volume locations
|
||||
// through the iceberg server's filer client.
|
||||
type lookupFileIDAdapter struct {
|
||||
fn wdclient.LookupFileIdFunctionType
|
||||
}
|
||||
|
||||
func (a *lookupFileIDAdapter) GetLookupFileIdFunction() wdclient.LookupFileIdFunctionType {
|
||||
return a.fn
|
||||
}
|
||||
|
||||
// loadTableMetadataBlob reads a file from the table's metadata directory,
|
||||
// following chunks for files that were written through the S3 gateway rather
|
||||
// than inline by the catalog itself.
|
||||
func (s *Server) loadTableMetadataBlob(ctx context.Context, bucketName, tablePath, fileName string) ([]byte, error) {
|
||||
var entry *filer_pb.Entry
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
resp, err := filer_pb.LookupEntry(ctx, client, &filer_pb.LookupDirectoryEntryRequest{
|
||||
Directory: metadataDirPath(bucketName, tablePath),
|
||||
Name: fileName,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
entry = resp.Entry
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if entry == nil {
|
||||
return nil, fmt.Errorf("no entry for %s", fileName)
|
||||
}
|
||||
if len(entry.Content) > 0 || len(entry.GetChunks()) == 0 {
|
||||
return entry.Content, nil
|
||||
}
|
||||
if size := filer.FileSize(entry); size > maxRepairableManifestSize {
|
||||
return nil, fmt.Errorf("%s is %d bytes, larger than the %d byte repair limit", fileName, size, maxRepairableManifestSize)
|
||||
}
|
||||
|
||||
fullClient, ok := s.filerClient.(filer_pb.FilerClient)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("filer client cannot resolve chunk locations")
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
lookup := &lookupFileIDAdapter{fn: filer.LookupFn(fullClient)}
|
||||
if err := filer.StreamContent(lookup, &buf, entry.GetChunks(), 0, int64(filer.FileSize(entry))); err != nil {
|
||||
return nil, fmt.Errorf("read %s: %w", fileName, err)
|
||||
}
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
281
weed/s3api/iceberg/manifest_repair_test.go
Normal file
281
weed/s3api/iceberg/manifest_repair_test.go
Normal file
@@ -0,0 +1,281 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
icebergmanifest "github.com/apache/iceberg-go"
|
||||
"github.com/linkedin/goavro/v2"
|
||||
)
|
||||
|
||||
// The testdata fixtures were written by ClickHouse 25.8's experimental
|
||||
// Iceberg insert against a SeaweedFS table bucket. clickhouse_rel holds the
|
||||
// default output (bucket-relative paths, no field-id annotations);
|
||||
// clickhouse_abs was written with write_full_path_in_iceberg_metadata=1
|
||||
// (absolute paths, still no field-ids).
|
||||
|
||||
type fakeManifestStore struct {
|
||||
files map[string][]byte
|
||||
saved map[string][]byte
|
||||
}
|
||||
|
||||
func newFakeManifestStore() *fakeManifestStore {
|
||||
return &fakeManifestStore{files: map[string][]byte{}, saved: map[string][]byte{}}
|
||||
}
|
||||
|
||||
func (f *fakeManifestStore) loadFile(_ context.Context, location string) ([]byte, error) {
|
||||
if data, ok := f.files[location]; ok {
|
||||
return data, nil
|
||||
}
|
||||
return nil, fmt.Errorf("no file at %s", location)
|
||||
}
|
||||
|
||||
func (f *fakeManifestStore) saveFile(_ context.Context, location string, data []byte) error {
|
||||
f.files[location] = data
|
||||
f.saved[location] = data
|
||||
return nil
|
||||
}
|
||||
|
||||
func loadFixture(t *testing.T, parts ...string) []byte {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(filepath.Join(append([]string{"testdata"}, parts...)...))
|
||||
if err != nil {
|
||||
t.Fatalf("read fixture: %v", err)
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
const (
|
||||
relTableLocation = "s3://iceberg-tables/wprobe/target_rel"
|
||||
relListName = "snap-449670589-2-48b874af-90f6-4e5e-8002-df91e59c8a0c.avro"
|
||||
relManifestName = "690958a2-9936-4a04-8117-d8104c977195.avro"
|
||||
absTableLocation = "s3://iceberg-tables/wprobe/target"
|
||||
absListName = "snap-331611856-2-51798e6c-427b-4fc2-9149-99e75d96e045.avro"
|
||||
absManifestName = "b317f0e5-10f3-41d6-86bc-8b42f01f3e86.avro"
|
||||
)
|
||||
|
||||
// verifyRepairedManifests checks the repaired files with iceberg-go's strict
|
||||
// typed readers: the manifest list parses, references the repaired manifest
|
||||
// with the right length, and the manifest's data file path is absolute.
|
||||
func verifyRepairedManifests(t *testing.T, store *fakeManifestStore, listLocation, tableLocation string) {
|
||||
t.Helper()
|
||||
|
||||
listBytes, ok := store.files[listLocation]
|
||||
if !ok {
|
||||
t.Fatalf("repaired manifest list %s was not saved", listLocation)
|
||||
}
|
||||
if !bytes.Contains(listBytes, []byte(`"field-id"`)) {
|
||||
t.Fatalf("repaired manifest list schema has no field-id annotations")
|
||||
}
|
||||
|
||||
manifests, err := icebergmanifest.ReadManifestList(bytes.NewReader(listBytes))
|
||||
if err != nil {
|
||||
t.Fatalf("iceberg-go cannot read repaired manifest list: %v", err)
|
||||
}
|
||||
if len(manifests) != 1 {
|
||||
t.Fatalf("manifest list has %d entries, want 1", len(manifests))
|
||||
}
|
||||
manifestPath := manifests[0].FilePath()
|
||||
if !strings.HasPrefix(manifestPath, tableLocation+"/metadata/") {
|
||||
t.Fatalf("manifest path %s is not absolute under the table location", manifestPath)
|
||||
}
|
||||
|
||||
manifestBytes, ok := store.files[manifestPath]
|
||||
if !ok {
|
||||
t.Fatalf("manifest %s not present in store", manifestPath)
|
||||
}
|
||||
if manifests[0].Length() != int64(len(manifestBytes)) {
|
||||
t.Fatalf("manifest_length %d != stored size %d", manifests[0].Length(), len(manifestBytes))
|
||||
}
|
||||
if !bytes.Contains(manifestBytes, []byte(`"field-id"`)) {
|
||||
t.Fatalf("repaired manifest schema has no field-id annotations")
|
||||
}
|
||||
|
||||
entries, err := icebergmanifest.ReadManifest(manifests[0], bytes.NewReader(manifestBytes), false)
|
||||
if err != nil {
|
||||
t.Fatalf("iceberg-go cannot read repaired manifest: %v", err)
|
||||
}
|
||||
if len(entries) != 1 {
|
||||
t.Fatalf("manifest has %d entries, want 1", len(entries))
|
||||
}
|
||||
dataFile := entries[0].DataFile()
|
||||
if !strings.HasPrefix(dataFile.FilePath(), "s3://iceberg-tables/") {
|
||||
t.Fatalf("data file path %s is not absolute", dataFile.FilePath())
|
||||
}
|
||||
if dataFile.Count() != 2 {
|
||||
t.Fatalf("record count %d, want 2", dataFile.Count())
|
||||
}
|
||||
if len(dataFile.LowerBoundValues()) == 0 {
|
||||
t.Fatalf("lower bounds were lost in repair")
|
||||
}
|
||||
}
|
||||
|
||||
func addSnapshotUpdateJSON(t *testing.T, manifestList string) []json.RawMessage {
|
||||
t.Helper()
|
||||
update := fmt.Sprintf(`{"action":"add-snapshot","snapshot":{"snapshot-id":449670589,"sequence-number":1,"timestamp-ms":1754625869000,"manifest-list":%q,"summary":{"operation":"append"},"schema-id":0}}`, manifestList)
|
||||
return []json.RawMessage{json.RawMessage(update)}
|
||||
}
|
||||
|
||||
func TestRepairClickHouseRelativeManifests(t *testing.T) {
|
||||
store := newFakeManifestStore()
|
||||
store.files[relTableLocation+"/metadata/"+relListName] = loadFixture(t, "clickhouse_rel", relListName)
|
||||
store.files[relTableLocation+"/metadata/"+relManifestName] = loadFixture(t, "clickhouse_rel", relManifestName)
|
||||
|
||||
rawUpdates := addSnapshotUpdateJSON(t, "wprobe/target_rel/metadata/"+relListName)
|
||||
repaired, changed := repairAddSnapshotUpdates(context.Background(), store, relTableLocation, rawUpdates)
|
||||
if !changed {
|
||||
t.Fatal("expected repair to change the update")
|
||||
}
|
||||
|
||||
var update struct {
|
||||
Snapshot struct {
|
||||
ManifestList string `json:"manifest-list"`
|
||||
SnapshotID int64 `json:"snapshot-id"`
|
||||
Summary struct {
|
||||
Operation string `json:"operation"`
|
||||
} `json:"summary"`
|
||||
} `json:"snapshot"`
|
||||
}
|
||||
if err := json.Unmarshal(repaired[0], &update); err != nil {
|
||||
t.Fatalf("unmarshal repaired update: %v", err)
|
||||
}
|
||||
wantList := relTableLocation + "/metadata/" + repairedManifestPrefix + relListName
|
||||
if update.Snapshot.ManifestList != wantList {
|
||||
t.Fatalf("manifest-list = %s, want %s", update.Snapshot.ManifestList, wantList)
|
||||
}
|
||||
// The rest of the snapshot survives the rewrite.
|
||||
if update.Snapshot.SnapshotID != 449670589 || update.Snapshot.Summary.Operation != "append" {
|
||||
t.Fatalf("snapshot fields were disturbed: %s", repaired[0])
|
||||
}
|
||||
|
||||
verifyRepairedManifests(t, store, wantList, relTableLocation)
|
||||
}
|
||||
|
||||
func TestRepairClickHouseAbsoluteManifests(t *testing.T) {
|
||||
store := newFakeManifestStore()
|
||||
listLocation := absTableLocation + "/metadata/" + absListName
|
||||
store.files[listLocation] = loadFixture(t, "clickhouse_abs", absListName)
|
||||
store.files[absTableLocation+"/metadata/"+absManifestName] = loadFixture(t, "clickhouse_abs", absManifestName)
|
||||
|
||||
repaired, changed, err := repairManifestList(context.Background(), store, absTableLocation, listLocation)
|
||||
if err != nil {
|
||||
t.Fatalf("repairManifestList: %v", err)
|
||||
}
|
||||
if !changed {
|
||||
t.Fatal("expected repair for missing field-ids")
|
||||
}
|
||||
verifyRepairedManifests(t, store, repaired, absTableLocation)
|
||||
}
|
||||
|
||||
func TestRepairedManifestsAreLeftAlone(t *testing.T) {
|
||||
store := newFakeManifestStore()
|
||||
listLocation := absTableLocation + "/metadata/" + absListName
|
||||
store.files[listLocation] = loadFixture(t, "clickhouse_abs", absListName)
|
||||
store.files[absTableLocation+"/metadata/"+absManifestName] = loadFixture(t, "clickhouse_abs", absManifestName)
|
||||
|
||||
firstPass, changed, err := repairManifestList(context.Background(), store, absTableLocation, listLocation)
|
||||
if err != nil || !changed {
|
||||
t.Fatalf("first pass: changed=%v err=%v", changed, err)
|
||||
}
|
||||
|
||||
store.saved = map[string][]byte{}
|
||||
secondPass, changed, err := repairManifestList(context.Background(), store, absTableLocation, firstPass)
|
||||
if err != nil {
|
||||
t.Fatalf("second pass: %v", err)
|
||||
}
|
||||
if changed || secondPass != firstPass {
|
||||
t.Fatalf("second pass rewrote an already-repaired list: changed=%v location=%s", changed, secondPass)
|
||||
}
|
||||
if len(store.saved) != 0 {
|
||||
t.Fatalf("second pass saved files: %v", store.saved)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepairKeepsDeleteManifestContent(t *testing.T) {
|
||||
// The fixture lacks the OCF "content" key; the repaired copy must take the
|
||||
// content declared by the manifest-list entry, not default to data.
|
||||
raw := loadFixture(t, "clickhouse_rel", relManifestName)
|
||||
repaired, changed, err := repairManifest(raw, relTableLocation, 1)
|
||||
if err != nil || !changed {
|
||||
t.Fatalf("repairManifest: changed=%v err=%v", changed, err)
|
||||
}
|
||||
reader, err := goavro.NewOCFReader(bytes.NewReader(repaired))
|
||||
if err != nil {
|
||||
t.Fatalf("read repaired manifest: %v", err)
|
||||
}
|
||||
if got := string(reader.MetaData()["content"]); got != "deletes" {
|
||||
t.Fatalf(`OCF content = %q, want "deletes"`, got)
|
||||
}
|
||||
|
||||
asData, _, err := repairManifest(raw, relTableLocation, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("repairManifest: %v", err)
|
||||
}
|
||||
reader, err = goavro.NewOCFReader(bytes.NewReader(asData))
|
||||
if err != nil {
|
||||
t.Fatalf("read repaired manifest: %v", err)
|
||||
}
|
||||
if got := string(reader.MetaData()["content"]); got != "data" {
|
||||
t.Fatalf(`OCF content = %q, want "data"`, got)
|
||||
}
|
||||
|
||||
// An existing content value that contradicts the manifest-list entry is
|
||||
// corrected, not preserved.
|
||||
relabeled, changed, err := repairManifest(asData, relTableLocation, 1)
|
||||
if err != nil || !changed {
|
||||
t.Fatalf("repairManifest on mismatched content: changed=%v err=%v", changed, err)
|
||||
}
|
||||
reader, err = goavro.NewOCFReader(bytes.NewReader(relabeled))
|
||||
if err != nil {
|
||||
t.Fatalf("read relabeled manifest: %v", err)
|
||||
}
|
||||
if got := string(reader.MetaData()["content"]); got != "deletes" {
|
||||
t.Fatalf(`OCF content after relabel = %q, want "deletes"`, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestContentValue(t *testing.T) {
|
||||
cases := []struct {
|
||||
record map[string]any
|
||||
want int
|
||||
}{
|
||||
{map[string]any{"content": int32(1)}, 1},
|
||||
{map[string]any{"content": int64(1)}, 1},
|
||||
{map[string]any{"content": map[string]any{"int": int32(1)}}, 1},
|
||||
{map[string]any{"content": nil}, 0},
|
||||
{map[string]any{}, 0},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := manifestContentValue(c.record); got != c.want {
|
||||
t.Errorf("manifestContentValue(%v) = %d, want %d", c.record, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepairSkipsLocationsOutsideMetadataDir(t *testing.T) {
|
||||
store := newFakeManifestStore()
|
||||
rawUpdates := addSnapshotUpdateJSON(t, "s3://other-bucket/elsewhere/list.avro")
|
||||
_, changed := repairAddSnapshotUpdates(context.Background(), store, relTableLocation, rawUpdates)
|
||||
if changed {
|
||||
t.Fatal("repair touched a manifest list outside the table metadata directory")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepairLeavesUnreadableListAlone(t *testing.T) {
|
||||
store := newFakeManifestStore()
|
||||
store.files[relTableLocation+"/metadata/junk.avro"] = []byte("not avro")
|
||||
rawUpdates := addSnapshotUpdateJSON(t, relTableLocation+"/metadata/junk.avro")
|
||||
repaired, changed := repairAddSnapshotUpdates(context.Background(), store, relTableLocation, rawUpdates)
|
||||
if changed {
|
||||
t.Fatal("repair claimed to change an unreadable manifest list")
|
||||
}
|
||||
if !bytes.Equal(repaired[0], rawUpdates[0]) {
|
||||
t.Fatal("update was modified despite repair failure")
|
||||
}
|
||||
}
|
||||
132
weed/s3api/iceberg/metadata_compliance.go
Normal file
132
weed/s3api/iceberg/metadata_compliance.go
Normal file
@@ -0,0 +1,132 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
)
|
||||
|
||||
// specRequiredEmptyOrder is iterated in this fixed order so the byte-level
|
||||
// append in ensureMetadataSpecCompliance produces deterministic output.
|
||||
var specRequiredEmptyOrder = []string{
|
||||
"current-snapshot-id",
|
||||
"snapshots",
|
||||
"snapshot-log",
|
||||
"metadata-log",
|
||||
"refs",
|
||||
}
|
||||
|
||||
// specRequiredEmptyDefaults holds the sentinel values Iceberg requires when
|
||||
// the corresponding state is empty.
|
||||
var specRequiredEmptyDefaults = map[string]json.RawMessage{
|
||||
"current-snapshot-id": json.RawMessage("-1"),
|
||||
"snapshots": json.RawMessage("[]"),
|
||||
"snapshot-log": json.RawMessage("[]"),
|
||||
"metadata-log": json.RawMessage("[]"),
|
||||
"refs": json.RawMessage("{}"),
|
||||
}
|
||||
|
||||
// isJSONNull reports whether raw encodes the JSON null literal, ignoring
|
||||
// insignificant whitespace allowed by RFC 8259.
|
||||
func isJSONNull(raw json.RawMessage) bool {
|
||||
return bytes.Equal(bytes.TrimSpace(raw), []byte("null"))
|
||||
}
|
||||
|
||||
// ensureMetadataSpecCompliance backfills Iceberg spec-required fields that
|
||||
// iceberg-go v0.5.0 drops via `omitempty` on optional pointer/slice fields.
|
||||
//
|
||||
// For tables with no snapshots (e.g. freshly created tables), iceberg-go
|
||||
// represents an empty state with nil/empty Go values, then strips those keys
|
||||
// entirely from the JSON output. The Iceberg REST spec, however, requires
|
||||
// these keys to be present even when empty — Java/Spark/Trino clients
|
||||
// fail to parse responses missing them (most visibly with
|
||||
// "Cannot parse missing long current-snapshot-id"). This helper rehydrates
|
||||
// the missing keys without overwriting any present real values.
|
||||
//
|
||||
// Missing keys are spliced in at the byte level just before the closing
|
||||
// brace so iceberg-go's struct-declared key order is preserved (a naive
|
||||
// map remarshal would alphabetize every key in the document). The slower
|
||||
// remarshal path is only used when an explicit JSON null needs replacing,
|
||||
// which iceberg-go itself never emits.
|
||||
//
|
||||
// Returns the original bytes unchanged when parsing fails or no key needs
|
||||
// to be added, to avoid corrupting an otherwise-valid payload.
|
||||
func ensureMetadataSpecCompliance(raw []byte) []byte {
|
||||
if len(raw) == 0 {
|
||||
return raw
|
||||
}
|
||||
var obj map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &obj); err != nil {
|
||||
return raw
|
||||
}
|
||||
// A top-level JSON null leaves obj == nil with no error; writes would
|
||||
// panic on the slow path. It is not a valid TableMetadata document, so
|
||||
// pass it through unchanged.
|
||||
if obj == nil {
|
||||
return raw
|
||||
}
|
||||
|
||||
// A field is "missing" for spec purposes if it's absent OR encoded as
|
||||
// JSON null. Some writers emit `"current-snapshot-id": null` for empty
|
||||
// state instead of omitting; strict clients reject both the same way.
|
||||
var toAppend []string
|
||||
hasExplicitNull := false
|
||||
for _, key := range specRequiredEmptyOrder {
|
||||
v, present := obj[key]
|
||||
switch {
|
||||
case !present:
|
||||
toAppend = append(toAppend, key)
|
||||
case isJSONNull(v):
|
||||
hasExplicitNull = true
|
||||
}
|
||||
}
|
||||
if len(toAppend) == 0 && !hasExplicitNull {
|
||||
return raw
|
||||
}
|
||||
|
||||
if !hasExplicitNull {
|
||||
return appendMissingObjectKeys(raw, toAppend, len(obj) > 0)
|
||||
}
|
||||
|
||||
// Slow path: replace explicit nulls. Rare in practice, so the
|
||||
// alphabetical key reordering inherent to map remarshal is acceptable.
|
||||
for key, v := range obj {
|
||||
if _, required := specRequiredEmptyDefaults[key]; required && isJSONNull(v) {
|
||||
obj[key] = specRequiredEmptyDefaults[key]
|
||||
}
|
||||
}
|
||||
for _, key := range toAppend {
|
||||
obj[key] = specRequiredEmptyDefaults[key]
|
||||
}
|
||||
fixed, err := json.Marshal(obj)
|
||||
if err != nil {
|
||||
return raw
|
||||
}
|
||||
return fixed
|
||||
}
|
||||
|
||||
// appendMissingObjectKeys splices the given keys into raw just before the
|
||||
// closing brace of the top-level JSON object. raw is assumed to be a valid
|
||||
// JSON object (the caller has already unmarshalled it). hasMembers tells us
|
||||
// whether the object had any existing members, so we know whether to emit a
|
||||
// leading comma for the first appended key.
|
||||
func appendMissingObjectKeys(raw []byte, keys []string, hasMembers bool) []byte {
|
||||
closeIdx := bytes.LastIndexByte(raw, '}')
|
||||
if closeIdx < 0 {
|
||||
return raw
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
buf.Grow(len(raw) + len(keys)*48)
|
||||
buf.Write(raw[:closeIdx])
|
||||
for _, key := range keys {
|
||||
if hasMembers {
|
||||
buf.WriteByte(',')
|
||||
}
|
||||
buf.WriteByte('"')
|
||||
buf.WriteString(key)
|
||||
buf.WriteString(`":`)
|
||||
buf.Write(specRequiredEmptyDefaults[key])
|
||||
hasMembers = true
|
||||
}
|
||||
buf.Write(raw[closeIdx:])
|
||||
return buf.Bytes()
|
||||
}
|
||||
162
weed/s3api/iceberg/metadata_compliance_test.go
Normal file
162
weed/s3api/iceberg/metadata_compliance_test.go
Normal file
@@ -0,0 +1,162 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEnsureMetadataSpecCompliance_BackfillsMissingFields(t *testing.T) {
|
||||
// Mirrors the real iceberg-go v0.5.0 output for a freshly created table:
|
||||
// current-snapshot-id, snapshots, snapshot-log, metadata-log, refs all absent.
|
||||
input := []byte(`{
|
||||
"format-version": 2,
|
||||
"table-uuid": "82e3eec4-3aee-414f-a444-94c03c641d20",
|
||||
"location": "s3://s3table/default/t1",
|
||||
"last-sequence-number": 0,
|
||||
"last-updated-ms": 1779866785466,
|
||||
"last-column-id": 2,
|
||||
"current-schema-id": 0,
|
||||
"default-spec-id": 0,
|
||||
"last-partition-id": 999,
|
||||
"default-sort-order-id": 0
|
||||
}`)
|
||||
|
||||
out := ensureMetadataSpecCompliance(input)
|
||||
|
||||
var got map[string]json.RawMessage
|
||||
if err := json.Unmarshal(out, &got); err != nil {
|
||||
t.Fatalf("output is not valid JSON: %v", err)
|
||||
}
|
||||
|
||||
if v, ok := got["current-snapshot-id"]; !ok || string(v) != "-1" {
|
||||
t.Errorf("current-snapshot-id missing or wrong: present=%v value=%s", ok, string(v))
|
||||
}
|
||||
for _, key := range []string{"snapshots", "snapshot-log", "metadata-log"} {
|
||||
v, ok := got[key]
|
||||
if !ok || string(v) != "[]" {
|
||||
t.Errorf("%s missing or wrong: present=%v value=%s", key, ok, string(v))
|
||||
}
|
||||
}
|
||||
if v, ok := got["refs"]; !ok || string(v) != "{}" {
|
||||
t.Errorf("refs missing or wrong: present=%v value=%s", ok, string(v))
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureMetadataSpecCompliance_PreservesExistingFields(t *testing.T) {
|
||||
// Real snapshot id and refs must not be overwritten by sentinels.
|
||||
input := []byte(`{
|
||||
"format-version": 2,
|
||||
"current-snapshot-id": 9876543210,
|
||||
"snapshots": [{"snapshot-id": 9876543210}],
|
||||
"snapshot-log": [{"snapshot-id": 9876543210, "timestamp-ms": 1}],
|
||||
"metadata-log": [{"metadata-file": "v1.metadata.json", "timestamp-ms": 1}],
|
||||
"refs": {"main": {"snapshot-id": 9876543210, "type": "branch"}}
|
||||
}`)
|
||||
|
||||
out := ensureMetadataSpecCompliance(input)
|
||||
|
||||
var got map[string]json.RawMessage
|
||||
if err := json.Unmarshal(out, &got); err != nil {
|
||||
t.Fatalf("output is not valid JSON: %v", err)
|
||||
}
|
||||
if string(got["current-snapshot-id"]) != "9876543210" {
|
||||
t.Errorf("real snapshot id was overwritten: %s", string(got["current-snapshot-id"]))
|
||||
}
|
||||
if string(got["snapshots"]) == "[]" {
|
||||
t.Errorf("non-empty snapshots was overwritten with empty array")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureMetadataSpecCompliance_ReplacesExplicitNullsWithSentinels(t *testing.T) {
|
||||
// Some writers emit explicit JSON null for unset values instead of omitting
|
||||
// the key. Strict Iceberg clients reject these the same way as missing keys.
|
||||
input := []byte(`{
|
||||
"format-version": 2,
|
||||
"current-snapshot-id": null,
|
||||
"snapshots": null,
|
||||
"snapshot-log": null,
|
||||
"metadata-log": null,
|
||||
"refs": null
|
||||
}`)
|
||||
|
||||
out := ensureMetadataSpecCompliance(input)
|
||||
|
||||
var got map[string]json.RawMessage
|
||||
if err := json.Unmarshal(out, &got); err != nil {
|
||||
t.Fatalf("output is not valid JSON: %v", err)
|
||||
}
|
||||
if string(got["current-snapshot-id"]) != "-1" {
|
||||
t.Errorf("current-snapshot-id should be replaced with -1, got %s", string(got["current-snapshot-id"]))
|
||||
}
|
||||
for _, key := range []string{"snapshots", "snapshot-log", "metadata-log"} {
|
||||
if string(got[key]) != "[]" {
|
||||
t.Errorf("%s should be replaced with [], got %s", key, string(got[key]))
|
||||
}
|
||||
}
|
||||
if string(got["refs"]) != "{}" {
|
||||
t.Errorf("refs should be replaced with {}, got %s", string(got["refs"]))
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureMetadataSpecCompliance_InvalidJSONReturnedUnchanged(t *testing.T) {
|
||||
input := []byte(`{not valid json`)
|
||||
out := ensureMetadataSpecCompliance(input)
|
||||
if string(out) != string(input) {
|
||||
t.Errorf("invalid JSON should be returned unchanged; got %s", string(out))
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureMetadataSpecCompliance_EmptyInputReturnedUnchanged(t *testing.T) {
|
||||
if out := ensureMetadataSpecCompliance(nil); out != nil {
|
||||
t.Errorf("nil input should be returned unchanged, got %v", out)
|
||||
}
|
||||
if out := ensureMetadataSpecCompliance([]byte{}); len(out) != 0 {
|
||||
t.Errorf("empty input should be returned unchanged, got %v", out)
|
||||
}
|
||||
// A top-level JSON null literal must not panic on the slow path.
|
||||
if out := ensureMetadataSpecCompliance([]byte("null")); string(out) != "null" {
|
||||
t.Errorf("top-level null should be returned unchanged, got %s", string(out))
|
||||
}
|
||||
}
|
||||
|
||||
// Original iceberg-go key ordering must survive the backfill: appended
|
||||
// sentinels go at the end without disturbing prior fields. A map-based
|
||||
// remarshal would have sorted everything alphabetically.
|
||||
func TestEnsureMetadataSpecCompliance_PreservesOriginalKeyOrder(t *testing.T) {
|
||||
// Compact JSON, keys in struct-declared order from iceberg-go.
|
||||
input := []byte(`{"format-version":2,"table-uuid":"82e3eec4-3aee-414f-a444-94c03c641d20","location":"s3://x/t","last-sequence-number":0,"last-updated-ms":1,"last-column-id":2,"current-schema-id":0,"default-spec-id":0,"last-partition-id":999,"default-sort-order-id":0}`)
|
||||
|
||||
out := ensureMetadataSpecCompliance(input)
|
||||
|
||||
// Prior keys keep their order, sentinels appended at the end.
|
||||
want := `{"format-version":2,"table-uuid":"82e3eec4-3aee-414f-a444-94c03c641d20","location":"s3://x/t","last-sequence-number":0,"last-updated-ms":1,"last-column-id":2,"current-schema-id":0,"default-spec-id":0,"last-partition-id":999,"default-sort-order-id":0,"current-snapshot-id":-1,"snapshots":[],"snapshot-log":[],"metadata-log":[],"refs":{}}`
|
||||
if string(out) != want {
|
||||
t.Errorf("unexpected output\n got: %s\nwant: %s", string(out), want)
|
||||
}
|
||||
|
||||
// Sanity: still valid JSON.
|
||||
var parsed map[string]json.RawMessage
|
||||
if err := json.Unmarshal(out, &parsed); err != nil {
|
||||
t.Fatalf("output is not valid JSON: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// An empty object {} must round-trip to a valid JSON object containing
|
||||
// only the spec sentinels (no leading comma).
|
||||
func TestEnsureMetadataSpecCompliance_EmptyObjectBackfilled(t *testing.T) {
|
||||
out := ensureMetadataSpecCompliance([]byte(`{}`))
|
||||
want := `{"current-snapshot-id":-1,"snapshots":[],"snapshot-log":[],"metadata-log":[],"refs":{}}`
|
||||
if string(out) != want {
|
||||
t.Errorf("unexpected output\n got: %s\nwant: %s", string(out), want)
|
||||
}
|
||||
}
|
||||
|
||||
// When all fields are already present, the original bytes must be returned
|
||||
// untouched (no whitespace normalization, no key reordering).
|
||||
func TestEnsureMetadataSpecCompliance_AllPresentReturnsSameBytes(t *testing.T) {
|
||||
input := []byte("{\n \"current-snapshot-id\": 1,\n \"snapshots\": [],\n \"snapshot-log\": [],\n \"metadata-log\": [],\n \"refs\": {}\n}")
|
||||
out := ensureMetadataSpecCompliance(input)
|
||||
if string(out) != string(input) {
|
||||
t.Errorf("expected original bytes returned unchanged\n got: %q\nwant: %q", string(out), string(input))
|
||||
}
|
||||
}
|
||||
171
weed/s3api/iceberg/metadata_files.go
Normal file
171
weed/s3api/iceberg/metadata_files.go
Normal file
@@ -0,0 +1,171 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// metadataDirPath returns the filer directory holding a table's metadata files.
|
||||
func metadataDirPath(bucketName, tablePath string) string {
|
||||
dir := path.Join(s3tables.TablesPath, bucketName)
|
||||
if tablePath != "" {
|
||||
dir = path.Join(dir, tablePath)
|
||||
}
|
||||
return path.Join(dir, "metadata")
|
||||
}
|
||||
|
||||
// saveMetadataFile saves the Iceberg metadata JSON file to the filer.
|
||||
// It constructs the filer path from the S3 location components. With exclusive
|
||||
// set the write fails with filer_pb.ErrEntryAlreadyExists rather than replacing
|
||||
// a file that is already there: two commits racing off the same base pick the
|
||||
// same v{N} name, and letting the second overwrite the first would leave the
|
||||
// winner's catalog pointer aimed at the loser's metadata.
|
||||
func (s *Server) saveMetadataFile(ctx context.Context, bucketName, tablePath, metadataFileName string, content []byte, exclusive bool) error {
|
||||
return s.saveMetadataBlob(ctx, bucketName, tablePath, metadataFileName, content, "application/json", exclusive)
|
||||
}
|
||||
|
||||
// saveMetadataBlob saves a file into the table's metadata directory.
|
||||
func (s *Server) saveMetadataBlob(ctx context.Context, bucketName, tablePath, metadataFileName string, content []byte, mimeType string, exclusive bool) error {
|
||||
|
||||
// Create context with timeout for file operations
|
||||
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
bucketsPath := s3tables.TablesPath
|
||||
|
||||
ensureDir := func(parent, name, errorContext string) error {
|
||||
_, err := filer_pb.LookupEntry(opCtx, client, &filer_pb.LookupDirectoryEntryRequest{
|
||||
Directory: parent,
|
||||
Name: name,
|
||||
})
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if err != filer_pb.ErrNotFound {
|
||||
return fmt.Errorf("lookup %s failed: %w", errorContext, err)
|
||||
}
|
||||
|
||||
// If lookup fails with ErrNotFound, try to create the directory.
|
||||
resp, createErr := client.CreateEntry(opCtx, &filer_pb.CreateEntryRequest{
|
||||
Directory: parent,
|
||||
Entry: &filer_pb.Entry{
|
||||
Name: name,
|
||||
IsDirectory: true,
|
||||
Attributes: &filer_pb.FuseAttributes{
|
||||
Mtime: time.Now().Unix(),
|
||||
Crtime: time.Now().Unix(),
|
||||
FileMode: uint32(0755 | os.ModeDir),
|
||||
},
|
||||
},
|
||||
})
|
||||
if createErr != nil {
|
||||
return fmt.Errorf("failed to create %s: %w", errorContext, createErr)
|
||||
}
|
||||
if resp.ErrorCode != filer_pb.FilerError_OK && resp.ErrorCode != filer_pb.FilerError_ENTRY_ALREADY_EXISTS {
|
||||
return fmt.Errorf("failed to create %s: %s", errorContext, resp.Error)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
bucketDir := path.Join(bucketsPath, bucketName)
|
||||
// 1. Ensure bucket directory exists: <bucketsPath>/<bucket>
|
||||
if err := ensureDir(bucketsPath, bucketName, "bucket directory"); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 2. Ensure table path exists under the bucket directory
|
||||
tableDir := bucketDir
|
||||
if tablePath != "" {
|
||||
segments := strings.Split(tablePath, "/")
|
||||
for _, segment := range segments {
|
||||
if segment == "" {
|
||||
continue
|
||||
}
|
||||
if err := ensureDir(tableDir, segment, "table directory"); err != nil {
|
||||
return err
|
||||
}
|
||||
tableDir = path.Join(tableDir, segment)
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Ensure metadata directory exists: <bucketsPath>/<bucket>/<tablePath>/metadata
|
||||
metadataDir := path.Join(tableDir, "metadata")
|
||||
if err := ensureDir(tableDir, "metadata", "metadata directory"); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 4. Write the file
|
||||
resp, err := client.CreateEntry(opCtx, &filer_pb.CreateEntryRequest{
|
||||
Directory: metadataDir,
|
||||
OExcl: exclusive,
|
||||
Entry: &filer_pb.Entry{
|
||||
Name: metadataFileName,
|
||||
Attributes: &filer_pb.FuseAttributes{
|
||||
Mtime: time.Now().Unix(),
|
||||
Crtime: time.Now().Unix(),
|
||||
FileMode: uint32(0644),
|
||||
FileSize: uint64(len(content)),
|
||||
},
|
||||
Content: content,
|
||||
Extended: map[string][]byte{
|
||||
"Mime-Type": []byte(mimeType),
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to write metadata file: %w", err)
|
||||
}
|
||||
if resp.ErrorCode != filer_pb.FilerError_OK {
|
||||
if sentinel := filer_pb.FilerErrorToSentinel(resp.ErrorCode); sentinel != nil {
|
||||
return fmt.Errorf("failed to write metadata file: %w", sentinel)
|
||||
}
|
||||
return fmt.Errorf("failed to write metadata file: code=%v %s", resp.ErrorCode, resp.Error)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) deleteMetadataFile(ctx context.Context, bucketName, tablePath, metadataFileName string) error {
|
||||
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
metadataDir := metadataDirPath(bucketName, tablePath)
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return filer_pb.DoRemove(opCtx, client, metadataDir, metadataFileName, true, false, true, false, nil)
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) loadMetadataFile(ctx context.Context, bucketName, tablePath, metadataFileName string) ([]byte, error) {
|
||||
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
metadataDir := metadataDirPath(bucketName, tablePath)
|
||||
|
||||
var content []byte
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
resp, err := filer_pb.LookupEntry(opCtx, client, &filer_pb.LookupDirectoryEntryRequest{
|
||||
Directory: metadataDir,
|
||||
Name: metadataFileName,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp == nil || resp.Entry == nil {
|
||||
return fmt.Errorf("lookup returned nil entry for %s/%s", metadataDir, metadataFileName)
|
||||
}
|
||||
content = append([]byte(nil), resp.Entry.Content...)
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return content, nil
|
||||
}
|
||||
128
weed/s3api/iceberg/metadata_files_test.go
Normal file
128
weed/s3api/iceberg/metadata_files_test.go
Normal file
@@ -0,0 +1,128 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"google.golang.org/grpc"
|
||||
)
|
||||
|
||||
// stubFilerClient answers only the calls the metadata writers make, tracking
|
||||
// which paths exist so exclusive creates can be exercised.
|
||||
type stubFilerClient struct {
|
||||
filer_pb.SeaweedFilerClient
|
||||
entries map[string][]byte
|
||||
}
|
||||
|
||||
func newStubFilerClient() *stubFilerClient {
|
||||
return &stubFilerClient{entries: make(map[string][]byte)}
|
||||
}
|
||||
|
||||
func (c *stubFilerClient) WithFilerClient(_ bool, fn func(client filer_pb.SeaweedFilerClient) error) error {
|
||||
return fn(c)
|
||||
}
|
||||
|
||||
func (c *stubFilerClient) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest, _ ...grpc.CallOption) (*filer_pb.LookupDirectoryEntryResponse, error) {
|
||||
key := path.Join(req.Directory, req.Name)
|
||||
if _, ok := c.entries[key]; !ok {
|
||||
return nil, filer_pb.ErrNotFound
|
||||
}
|
||||
return &filer_pb.LookupDirectoryEntryResponse{Entry: &filer_pb.Entry{Name: req.Name}}, nil
|
||||
}
|
||||
|
||||
func (c *stubFilerClient) CreateEntry(_ context.Context, req *filer_pb.CreateEntryRequest, _ ...grpc.CallOption) (*filer_pb.CreateEntryResponse, error) {
|
||||
key := path.Join(req.Directory, req.Entry.Name)
|
||||
if _, exists := c.entries[key]; exists && req.OExcl {
|
||||
return &filer_pb.CreateEntryResponse{
|
||||
Error: "entry already exists",
|
||||
ErrorCode: filer_pb.FilerError_ENTRY_ALREADY_EXISTS,
|
||||
}, nil
|
||||
}
|
||||
c.entries[key] = req.Entry.Content
|
||||
return &filer_pb.CreateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
// Two commits racing off the same base metadata pick the same v{N} file name.
|
||||
// The loser must be told, not allowed to replace the winner's metadata.
|
||||
func TestSaveMetadataFileExclusiveRefusesToOverwrite(t *testing.T) {
|
||||
client := newStubFilerClient()
|
||||
s := &Server{filerClient: client}
|
||||
ctx := context.Background()
|
||||
|
||||
if err := s.saveMetadataFile(ctx, "bkt", "ns/tbl", "v2.metadata.json", []byte(`{"winner":true}`), true); err != nil {
|
||||
t.Fatalf("first write failed: %v", err)
|
||||
}
|
||||
|
||||
err := s.saveMetadataFile(ctx, "bkt", "ns/tbl", "v2.metadata.json", []byte(`{"loser":true}`), true)
|
||||
if !errors.Is(err, filer_pb.ErrEntryAlreadyExists) {
|
||||
t.Fatalf("second write err = %v, want ErrEntryAlreadyExists", err)
|
||||
}
|
||||
|
||||
stored := string(client.entries[path.Join(metadataDirPath("bkt", "ns/tbl"), "v2.metadata.json")])
|
||||
if stored != `{"winner":true}` {
|
||||
t.Errorf("stored metadata = %s, want the first writer's content", stored)
|
||||
}
|
||||
}
|
||||
|
||||
// A metadata file left behind by an interrupted commit must not wedge the
|
||||
// table: the next commit stages under a unique name rather than failing or
|
||||
// overwriting, and the catalog pointer still decides the winner.
|
||||
func TestStageCommitMetadataFallsBackToAUniqueName(t *testing.T) {
|
||||
client := newStubFilerClient()
|
||||
s := &Server{filerClient: client}
|
||||
ctx := context.Background()
|
||||
|
||||
name, location, err := s.stageCommitMetadata(ctx, "bkt", "ns/tbl", "s3://bkt/ns/tbl", "v2.metadata.json", []byte(`{"orphan":true}`))
|
||||
if err != nil {
|
||||
t.Fatalf("first stage failed: %v", err)
|
||||
}
|
||||
if name != "v2.metadata.json" {
|
||||
t.Errorf("first stage used %q, want the plain versioned name", name)
|
||||
}
|
||||
if location != "s3://bkt/ns/tbl/metadata/v2.metadata.json" {
|
||||
t.Errorf("location = %q", location)
|
||||
}
|
||||
|
||||
name, location, err = s.stageCommitMetadata(ctx, "bkt", "ns/tbl", "s3://bkt/ns/tbl", "v2.metadata.json", []byte(`{"second":true}`))
|
||||
if err != nil {
|
||||
t.Fatalf("second stage failed: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(name, "v2-") || !strings.HasSuffix(name, ".metadata.json") {
|
||||
t.Errorf("second stage used %q, want a unique v2-* name", name)
|
||||
}
|
||||
if location != "s3://bkt/ns/tbl/metadata/"+name {
|
||||
t.Errorf("location = %q, want it to match the staged name", location)
|
||||
}
|
||||
|
||||
stored := string(client.entries[path.Join(metadataDirPath("bkt", "ns/tbl"), "v2.metadata.json")])
|
||||
if stored != `{"orphan":true}` {
|
||||
t.Errorf("the first file was overwritten: %s", stored)
|
||||
}
|
||||
if fallback := string(client.entries[path.Join(metadataDirPath("bkt", "ns/tbl"), name)]); fallback != `{"second":true}` {
|
||||
t.Errorf("fallback file holds %s, want the second writer's metadata", fallback)
|
||||
}
|
||||
}
|
||||
|
||||
// Paths that legitimately rewrite a file, such as manifest repair, keep the
|
||||
// overwriting behaviour.
|
||||
func TestSaveMetadataFileOverwrites(t *testing.T) {
|
||||
client := newStubFilerClient()
|
||||
s := &Server{filerClient: client}
|
||||
ctx := context.Background()
|
||||
|
||||
if err := s.saveMetadataFile(ctx, "bkt", "ns/tbl", "v2.metadata.json", []byte(`{"first":true}`), false); err != nil {
|
||||
t.Fatalf("first write failed: %v", err)
|
||||
}
|
||||
if err := s.saveMetadataFile(ctx, "bkt", "ns/tbl", "v2.metadata.json", []byte(`{"second":true}`), false); err != nil {
|
||||
t.Fatalf("second write failed: %v", err)
|
||||
}
|
||||
|
||||
stored := string(client.entries[path.Join(metadataDirPath("bkt", "ns/tbl"), "v2.metadata.json")])
|
||||
if stored != `{"second":true}` {
|
||||
t.Errorf("stored metadata = %s, want the second write", stored)
|
||||
}
|
||||
}
|
||||
142
weed/s3api/iceberg/name_mapping.go
Normal file
142
weed/s3api/iceberg/name_mapping.go
Normal file
@@ -0,0 +1,142 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/apache/iceberg-go/table"
|
||||
)
|
||||
|
||||
// nameMappingFromSchema derives the spec's field-id-to-names mapping from a
|
||||
// table schema, mirroring Java's MappingUtil.create. List elements and map
|
||||
// keys/values use the canonical "element"/"key"/"value" names.
|
||||
func nameMappingFromSchema(schema *iceberg.Schema) iceberg.NameMapping {
|
||||
return mappedFieldsOf(schema.Fields())
|
||||
}
|
||||
|
||||
func mappedFieldsOf(fields []iceberg.NestedField) []iceberg.MappedField {
|
||||
mapped := make([]iceberg.MappedField, 0, len(fields))
|
||||
for _, field := range fields {
|
||||
id := field.ID
|
||||
mapped = append(mapped, iceberg.MappedField{
|
||||
Names: []string{field.Name},
|
||||
FieldID: &id,
|
||||
Fields: mappedFieldsOfType(field.Type),
|
||||
})
|
||||
}
|
||||
return mapped
|
||||
}
|
||||
|
||||
func mappedFieldsOfType(typ iceberg.Type) []iceberg.MappedField {
|
||||
switch t := typ.(type) {
|
||||
case *iceberg.StructType:
|
||||
return mappedFieldsOf(t.FieldList)
|
||||
case *iceberg.ListType:
|
||||
return mappedFieldsOf([]iceberg.NestedField{t.ElementField()})
|
||||
case *iceberg.MapType:
|
||||
return mappedFieldsOf([]iceberg.NestedField{t.KeyField(), t.ValueField()})
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// refreshDefaultNameMapping keeps schema.name-mapping.default in sync with
|
||||
// the current schema across commits, so schema evolution does not strand
|
||||
// readers of field-id-less data files on the creation-time mapping. The
|
||||
// stored mapping is merged rather than replaced: per field-id, names already
|
||||
// present are kept alongside the current schema name, so files written under
|
||||
// a renamed column's old name (and user-added aliases) stay resolvable.
|
||||
// Returns the (possibly patched) serialized metadata.
|
||||
func refreshDefaultNameMapping(raw []byte, updated table.Metadata) []byte {
|
||||
if updated == nil {
|
||||
return raw
|
||||
}
|
||||
updatedSchema := updated.CurrentSchema()
|
||||
if updatedSchema == nil || len(updatedSchema.Fields()) == 0 {
|
||||
return raw
|
||||
}
|
||||
mapping := nameMappingFromSchema(updatedSchema)
|
||||
|
||||
existing := updated.Properties()[table.DefaultNameMappingKey]
|
||||
if existing != "" {
|
||||
var existingMapping iceberg.NameMapping
|
||||
if err := json.Unmarshal([]byte(existing), &existingMapping); err != nil {
|
||||
// Not a mapping we can merge; leave whatever the user stored.
|
||||
return raw
|
||||
}
|
||||
mapping = mergeMappedFields(existingMapping, mapping)
|
||||
}
|
||||
wantJSON, err := json.Marshal(mapping)
|
||||
if err != nil {
|
||||
return raw
|
||||
}
|
||||
want := string(wantJSON)
|
||||
if existing == want {
|
||||
return raw
|
||||
}
|
||||
|
||||
var metadata map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &metadata); err != nil {
|
||||
return raw
|
||||
}
|
||||
properties := map[string]json.RawMessage{}
|
||||
if rawProperties, ok := metadata["properties"]; ok {
|
||||
if err := json.Unmarshal(rawProperties, &properties); err != nil {
|
||||
return raw
|
||||
}
|
||||
}
|
||||
valueJSON, err := json.Marshal(want)
|
||||
if err != nil {
|
||||
return raw
|
||||
}
|
||||
properties[table.DefaultNameMappingKey] = valueJSON
|
||||
propertiesJSON, err := json.Marshal(properties)
|
||||
if err != nil {
|
||||
return raw
|
||||
}
|
||||
metadata["properties"] = propertiesJSON
|
||||
patched, err := json.Marshal(metadata)
|
||||
if err != nil {
|
||||
return raw
|
||||
}
|
||||
return patched
|
||||
}
|
||||
|
||||
// mergeMappedFields folds the names of an existing mapping into the mapping
|
||||
// derived from the current schema. Fields are matched by field-id per nesting
|
||||
// level; entries whose ids left the schema are dropped, since current readers
|
||||
// cannot project those fields anyway. A name can belong to only one field per
|
||||
// level (Java readers reject ambiguous mappings), so a historical name that
|
||||
// the current schema assigns to a different field-id stays with its new owner
|
||||
// instead of being duplicated onto the old one — which also keeps mappings
|
||||
// clean when a commit replaces the schema with reassigned field ids.
|
||||
func mergeMappedFields(existing, derived []iceberg.MappedField) []iceberg.MappedField {
|
||||
byID := make(map[int]*iceberg.MappedField, len(existing))
|
||||
for i := range existing {
|
||||
byID[existing[i].ID()] = &existing[i]
|
||||
}
|
||||
nameOwner := make(map[string]int)
|
||||
for _, d := range derived {
|
||||
for _, name := range d.Names {
|
||||
nameOwner[name] = d.ID()
|
||||
}
|
||||
}
|
||||
for i := range derived {
|
||||
prior, ok := byID[derived[i].ID()]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
for _, name := range prior.Names {
|
||||
if ownerID, taken := nameOwner[name]; taken && ownerID != derived[i].ID() {
|
||||
continue
|
||||
}
|
||||
if !slices.Contains(derived[i].Names, name) {
|
||||
derived[i].Names = append(derived[i].Names, name)
|
||||
nameOwner[name] = derived[i].ID()
|
||||
}
|
||||
}
|
||||
derived[i].Fields = mergeMappedFields(prior.Fields, derived[i].Fields)
|
||||
}
|
||||
return derived
|
||||
}
|
||||
213
weed/s3api/iceberg/name_mapping_test.go
Normal file
213
weed/s3api/iceberg/name_mapping_test.go
Normal file
@@ -0,0 +1,213 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func TestNameMappingFromSchema(t *testing.T) {
|
||||
schema := iceberg.NewSchema(0,
|
||||
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
|
||||
iceberg.NestedField{ID: 2, Name: "label", Type: iceberg.PrimitiveTypes.String},
|
||||
iceberg.NestedField{ID: 3, Name: "tags", Type: &iceberg.ListType{
|
||||
ElementID: 4, Element: iceberg.PrimitiveTypes.String,
|
||||
}},
|
||||
iceberg.NestedField{ID: 5, Name: "attrs", Type: &iceberg.MapType{
|
||||
KeyID: 6, KeyType: iceberg.PrimitiveTypes.String,
|
||||
ValueID: 7, ValueType: &iceberg.StructType{FieldList: []iceberg.NestedField{
|
||||
{ID: 8, Name: "inner", Type: iceberg.PrimitiveTypes.Int32},
|
||||
}},
|
||||
}},
|
||||
)
|
||||
|
||||
mapping := nameMappingFromSchema(schema)
|
||||
got, err := json.Marshal(mapping)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal mapping: %v", err)
|
||||
}
|
||||
|
||||
want := `[{"names":["id"],"field-id":1},{"names":["label"],"field-id":2},` +
|
||||
`{"names":["tags"],"field-id":3,"fields":[{"names":["element"],"field-id":4}]},` +
|
||||
`{"names":["attrs"],"field-id":5,"fields":[{"names":["key"],"field-id":6},` +
|
||||
`{"names":["value"],"field-id":7,"fields":[{"names":["inner"],"field-id":8}]}]}]`
|
||||
if string(got) != want {
|
||||
t.Fatalf("mapping = %s\nwant %s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefreshDefaultNameMappingDropsReassignedNames(t *testing.T) {
|
||||
// The stored mapping was derived from a schema whose ids were later
|
||||
// reassigned (id<->name swapped). The merged mapping must follow the
|
||||
// current schema exactly: duplicating names across ids makes Java readers
|
||||
// reject the whole mapping.
|
||||
schema := iceberg.NewSchema(0,
|
||||
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
|
||||
iceberg.NestedField{ID: 2, Name: "name", Type: iceberg.PrimitiveTypes.String},
|
||||
)
|
||||
metadata, err := newTableMetadata(uuid.New(), "s3://bucket/ns/tbl", schema, nil, nil,
|
||||
iceberg.Properties{table.DefaultNameMappingKey: `[{"names":["name"],"field-id":1},{"names":["id"],"field-id":2}]`})
|
||||
if err != nil {
|
||||
t.Fatalf("newTableMetadata: %v", err)
|
||||
}
|
||||
raw, err := json.Marshal(metadata)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal metadata: %v", err)
|
||||
}
|
||||
|
||||
refreshed := refreshDefaultNameMapping(raw, metadata)
|
||||
final, err := table.ParseMetadataBytes(refreshed)
|
||||
if err != nil {
|
||||
t.Fatalf("parse refreshed metadata: %v", err)
|
||||
}
|
||||
var mapping iceberg.NameMapping
|
||||
if err := json.Unmarshal([]byte(final.Properties()[table.DefaultNameMappingKey]), &mapping); err != nil {
|
||||
t.Fatalf("parse refreshed mapping: %v", err)
|
||||
}
|
||||
seen := map[string]int{}
|
||||
for _, field := range mapping {
|
||||
for _, name := range field.Names {
|
||||
if prior, dup := seen[name]; dup {
|
||||
t.Fatalf("name %q mapped to both field %d and field %d: %s",
|
||||
name, prior, field.ID(), final.Properties()[table.DefaultNameMappingKey])
|
||||
}
|
||||
seen[name] = field.ID()
|
||||
}
|
||||
}
|
||||
if seen["id"] != 1 || seen["name"] != 2 {
|
||||
t.Fatalf("current schema names must win: %s", final.Properties()[table.DefaultNameMappingKey])
|
||||
}
|
||||
}
|
||||
|
||||
// evolveMetadataSchema returns serialized metadata whose schema renamed
|
||||
// "label" to "tag" and gained an "extra" column while the properties still
|
||||
// carry the creation-time mapping, simulating a schema-evolution commit.
|
||||
func evolveMetadataSchema(t *testing.T, base table.Metadata) []byte {
|
||||
t.Helper()
|
||||
|
||||
raw, err := json.Marshal(base)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal base metadata: %v", err)
|
||||
}
|
||||
evolved := iceberg.NewSchema(0,
|
||||
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
|
||||
iceberg.NestedField{ID: 2, Name: "tag", Type: iceberg.PrimitiveTypes.String},
|
||||
iceberg.NestedField{ID: 3, Name: "extra", Type: iceberg.PrimitiveTypes.String},
|
||||
)
|
||||
schemaJSON, err := json.Marshal(evolved)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal evolved schema: %v", err)
|
||||
}
|
||||
var metadata map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &metadata); err != nil {
|
||||
t.Fatalf("unmarshal metadata: %v", err)
|
||||
}
|
||||
metadata["schemas"] = json.RawMessage("[" + string(schemaJSON) + "]")
|
||||
edited, err := json.Marshal(metadata)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal edited metadata: %v", err)
|
||||
}
|
||||
return edited
|
||||
}
|
||||
|
||||
func TestRefreshDefaultNameMappingOnSchemaEvolution(t *testing.T) {
|
||||
schema := iceberg.NewSchema(0,
|
||||
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
|
||||
iceberg.NestedField{ID: 2, Name: "label", Type: iceberg.PrimitiveTypes.String},
|
||||
)
|
||||
base, err := newTableMetadata(uuid.New(), "s3://bucket/ns/tbl", schema, nil, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("newTableMetadata: %v", err)
|
||||
}
|
||||
|
||||
edited := evolveMetadataSchema(t, base)
|
||||
updated, err := table.ParseMetadataBytes(edited)
|
||||
if err != nil {
|
||||
t.Fatalf("parse evolved metadata: %v", err)
|
||||
}
|
||||
|
||||
refreshed := refreshDefaultNameMapping(edited, updated)
|
||||
final, err := table.ParseMetadataBytes(refreshed)
|
||||
if err != nil {
|
||||
t.Fatalf("parse refreshed metadata: %v", err)
|
||||
}
|
||||
var mapping iceberg.NameMapping
|
||||
if err := json.Unmarshal([]byte(final.Properties()[table.DefaultNameMappingKey]), &mapping); err != nil {
|
||||
t.Fatalf("parse refreshed mapping: %v", err)
|
||||
}
|
||||
if len(mapping) != 3 || mapping[2].ID() != 3 || mapping[2].Names[0] != "extra" {
|
||||
t.Fatalf("mapping was not refreshed for the evolved schema: %s", final.Properties()[table.DefaultNameMappingKey])
|
||||
}
|
||||
// The renamed column keeps its historical name so files written under the
|
||||
// old physical name stay resolvable.
|
||||
if mapping[1].ID() != 2 || !slices.Contains(mapping[1].Names, "tag") || !slices.Contains(mapping[1].Names, "label") {
|
||||
t.Fatalf("renamed column lost a name: %s", final.Properties()[table.DefaultNameMappingKey])
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefreshDefaultNameMappingKeepsUserNames(t *testing.T) {
|
||||
schema := iceberg.NewSchema(0,
|
||||
iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
|
||||
iceberg.NestedField{ID: 2, Name: "label", Type: iceberg.PrimitiveTypes.String},
|
||||
)
|
||||
base, err := newTableMetadata(uuid.New(), "s3://bucket/ns/tbl", schema, nil, nil,
|
||||
iceberg.Properties{table.DefaultNameMappingKey: `[{"names":["custom"],"field-id":1}]`})
|
||||
if err != nil {
|
||||
t.Fatalf("newTableMetadata: %v", err)
|
||||
}
|
||||
|
||||
edited := evolveMetadataSchema(t, base)
|
||||
updated, err := table.ParseMetadataBytes(edited)
|
||||
if err != nil {
|
||||
t.Fatalf("parse evolved metadata: %v", err)
|
||||
}
|
||||
|
||||
refreshed := refreshDefaultNameMapping(edited, updated)
|
||||
final, err := table.ParseMetadataBytes(refreshed)
|
||||
if err != nil {
|
||||
t.Fatalf("parse refreshed metadata: %v", err)
|
||||
}
|
||||
var mapping iceberg.NameMapping
|
||||
if err := json.Unmarshal([]byte(final.Properties()[table.DefaultNameMappingKey]), &mapping); err != nil {
|
||||
t.Fatalf("parse refreshed mapping: %v", err)
|
||||
}
|
||||
// The user's alias survives the merge alongside the schema name.
|
||||
if mapping[0].ID() != 1 || !slices.Contains(mapping[0].Names, "custom") || !slices.Contains(mapping[0].Names, "id") {
|
||||
t.Fatalf("user alias was dropped: %s", final.Properties()[table.DefaultNameMappingKey])
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateTableSetsDefaultNameMapping(t *testing.T) {
|
||||
// Request ids are deliberately non-sequential: the metadata constructor
|
||||
// reassigns fresh ids, and the stamped mapping must follow the final
|
||||
// schema, not the request.
|
||||
schema := iceberg.NewSchema(0,
|
||||
iceberg.NestedField{ID: 7, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
|
||||
iceberg.NestedField{ID: 3, Name: "label", Type: iceberg.PrimitiveTypes.String},
|
||||
)
|
||||
metadata, err := newTableMetadata(uuid.New(), "s3://bucket/ns/tbl", schema, nil, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("newTableMetadata: %v", err)
|
||||
}
|
||||
mappingJSON, ok := metadata.Properties()[table.DefaultNameMappingKey]
|
||||
if !ok {
|
||||
t.Fatalf("newTableMetadata did not set %s: %v", table.DefaultNameMappingKey, metadata.Properties())
|
||||
}
|
||||
var mapping iceberg.NameMapping
|
||||
if err := json.Unmarshal([]byte(mappingJSON), &mapping); err != nil {
|
||||
t.Fatalf("parse stamped mapping: %v", err)
|
||||
}
|
||||
finalSchema := metadata.CurrentSchema()
|
||||
if len(mapping) != len(finalSchema.Fields()) {
|
||||
t.Fatalf("mapping has %d entries, schema has %d", len(mapping), len(finalSchema.Fields()))
|
||||
}
|
||||
for i, field := range finalSchema.Fields() {
|
||||
if mapping[i].ID() != field.ID || mapping[i].Names[0] != field.Name {
|
||||
t.Fatalf("mapping entry %d = %v, want id %d name %s", i, mapping[i], field.ID, field.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
104
weed/s3api/iceberg/path_validation.go
Normal file
104
weed/s3api/iceberg/path_validation.go
Normal file
@@ -0,0 +1,104 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
)
|
||||
|
||||
// validateRequestPath rejects Iceberg REST requests whose captured
|
||||
// {prefix}/{namespace}/{table} mux vars would produce a parent-directory
|
||||
// traversal when joined into a filer path. The iceberg router runs with
|
||||
// SkipClean(true), so `..` survives routing; downstream path.Join calls
|
||||
// (stageCreateMarkerDir, location builders, etc.) then collapse it and
|
||||
// escape the table-bucket directory.
|
||||
//
|
||||
// {prefix} maps to a table-bucket name; {table} is a single path segment;
|
||||
// {namespace} is unit-separator (0x1F) joined parts that get flattened into
|
||||
// a single dotted name for the on-disk layout — each part is validated
|
||||
// individually.
|
||||
func validateRequestPath(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
// Use the comma-ok form so vars only checked when the matched route
|
||||
// actually captures them; when captured, an empty value is itself a
|
||||
// rejection because downstream path.Join would collapse it.
|
||||
if prefix, ok := vars["prefix"]; ok {
|
||||
if prefix == "" || !s3_constants.IsValidBucketName(prefix) {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "invalid prefix")
|
||||
return
|
||||
}
|
||||
}
|
||||
if table, ok := vars["table"]; ok {
|
||||
if table == "" || !isValidNameSegment(table) {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "invalid table name")
|
||||
return
|
||||
}
|
||||
}
|
||||
if ns, ok := vars["namespace"]; ok {
|
||||
if ns == "" {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "invalid namespace")
|
||||
return
|
||||
}
|
||||
// Reject leading/trailing/consecutive unit separators so distinct
|
||||
// inputs cannot collapse to the same parsed namespace via
|
||||
// parseNamespace's empty-part filter.
|
||||
for _, part := range strings.Split(ns, "\x1F") {
|
||||
if part == "" || !isValidNameSegment(part) {
|
||||
writeError(w, http.StatusBadRequest, "BadRequest", "invalid namespace")
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// isValidTablePath reports whether a "/"-separated table path (the part below
|
||||
// the bucket) is free of segments that path.Join would collapse to escape the
|
||||
// bucket directory. Empty segments (from leading/duplicate slashes) are
|
||||
// ignored, but the path must resolve to at least one real segment so it does
|
||||
// not collapse to the bucket-level metadata directory.
|
||||
func isValidTablePath(tablePath string) bool {
|
||||
hasSegment := false
|
||||
for _, segment := range strings.Split(tablePath, "/") {
|
||||
if segment == "" {
|
||||
continue
|
||||
}
|
||||
if !isValidNameSegment(segment) {
|
||||
return false
|
||||
}
|
||||
hasSegment = true
|
||||
}
|
||||
return hasSegment
|
||||
}
|
||||
|
||||
// isValidNameSegment rejects a single path-segment value (bucket prefix slot,
|
||||
// table name, or one namespace part) that would be unsafe to embed in a filer
|
||||
// path: `.`, `..`, embedded slash/backslash, or NUL.
|
||||
func isValidNameSegment(s string) bool {
|
||||
if s == "" {
|
||||
return true
|
||||
}
|
||||
if s == "." || s == ".." {
|
||||
return false
|
||||
}
|
||||
return !strings.ContainsAny(s, "/\\\x00")
|
||||
}
|
||||
|
||||
// confineMetadataLocation checks that a parsed s3 location stays within the
|
||||
// authorized table bucket and rejects traversal segments that path.Join in
|
||||
// saveMetadataBlob would collapse to escape it. The table path must resolve
|
||||
// to at least one real segment so its metadata directory is table-specific.
|
||||
func confineMetadataLocation(metadataBucket, metadataPath, bucketName string) error {
|
||||
if metadataBucket != bucketName {
|
||||
return fmt.Errorf("table location must be within bucket %s", bucketName)
|
||||
}
|
||||
if !isValidTablePath(metadataPath) {
|
||||
return fmt.Errorf("invalid table location path")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
139
weed/s3api/iceberg/path_validation_test.go
Normal file
139
weed/s3api/iceberg/path_validation_test.go
Normal file
@@ -0,0 +1,139 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
)
|
||||
|
||||
func TestIsValidTablePath(t *testing.T) {
|
||||
tests := []struct {
|
||||
tablePath string
|
||||
want bool
|
||||
}{
|
||||
{"sales.orders", true},
|
||||
{"ns/table", true},
|
||||
{"ns//table", true},
|
||||
{"", false},
|
||||
{"/", false},
|
||||
{"//", false},
|
||||
{"../other", false},
|
||||
{"ns/../../etc", false},
|
||||
{"ns/./x", false},
|
||||
{`ns\..\x`, false},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
if got := isValidTablePath(tt.tablePath); got != tt.want {
|
||||
t.Errorf("isValidTablePath(%q) = %v, want %v", tt.tablePath, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRequestPath_RejectsTraversal(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
rawPath string
|
||||
wantCode int
|
||||
}{
|
||||
{"clean namespace+table passes", "/v1/namespaces/sales/tables/orders", http.StatusOK},
|
||||
{"clean prefixed passes", "/v1/wh/namespaces/sales/tables/orders", http.StatusOK},
|
||||
{"clean namespace only passes", "/v1/namespaces/sales", http.StatusOK},
|
||||
|
||||
// SkipClean(true) means raw `..` survives routing — these are the
|
||||
// realistic traversal shapes the middleware must catch.
|
||||
{"dotdot as prefix var rejected", "/v1/../namespaces/sales", http.StatusBadRequest},
|
||||
{"dotdot as namespace var rejected", "/v1/namespaces/..", http.StatusBadRequest},
|
||||
{"dotdot as namespace var prefixed rejected", "/v1/wh/namespaces/..", http.StatusBadRequest},
|
||||
{"dotdot as table var rejected", "/v1/namespaces/sales/tables/..", http.StatusBadRequest},
|
||||
{"dot as table var rejected", "/v1/namespaces/sales/tables/.", http.StatusBadRequest},
|
||||
// Iceberg clients send the 0x1F unit separator percent-encoded; mux
|
||||
// decodes it before the middleware sees the namespace var.
|
||||
{"unit-sep namespace with dotdot part rejected", "/v1/namespaces/sales%1F..%1Fevil", http.StatusBadRequest},
|
||||
{"leading unit-sep namespace rejected", "/v1/namespaces/%1Fsales", http.StatusBadRequest},
|
||||
{"trailing unit-sep namespace rejected", "/v1/namespaces/sales%1F", http.StatusBadRequest},
|
||||
{"consecutive unit-sep namespace rejected", "/v1/namespaces/sales%1F%1Fevil", http.StatusBadRequest},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
router := mux.NewRouter().SkipClean(true)
|
||||
router.Use(validateRequestPath)
|
||||
handlerCalled := false
|
||||
pass := func(w http.ResponseWriter, r *http.Request) {
|
||||
handlerCalled = true
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
router.HandleFunc("/v1/namespaces/{namespace}", pass)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", pass)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", pass)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", pass)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, tt.rawPath, nil)
|
||||
rr := httptest.NewRecorder()
|
||||
router.ServeHTTP(rr, req)
|
||||
|
||||
if rr.Code != tt.wantCode {
|
||||
t.Fatalf("path %q: got status %d, want %d (body=%q)", tt.rawPath, rr.Code, tt.wantCode, rr.Body.String())
|
||||
}
|
||||
if tt.wantCode == http.StatusBadRequest && handlerCalled {
|
||||
t.Fatalf("path %q: inner handler reached despite rejection", tt.rawPath)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Defense-in-depth: if a future route or middleware ever leaves one of the
|
||||
// captured vars empty, the middleware must still reject the request. The
|
||||
// default mux regex won't normally allow this.
|
||||
func TestValidateRequestPath_RejectsEmptyCapturedVars(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
vars map[string]string
|
||||
}{
|
||||
{"empty prefix", map[string]string{"prefix": "", "namespace": "ns"}},
|
||||
{"empty table", map[string]string{"namespace": "ns", "table": ""}},
|
||||
{"empty namespace", map[string]string{"namespace": ""}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
handlerCalled := false
|
||||
h := validateRequestPath(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
handlerCalled = true
|
||||
}))
|
||||
req := mux.SetURLVars(httptest.NewRequest(http.MethodGet, "/", nil), tt.vars)
|
||||
rr := httptest.NewRecorder()
|
||||
h.ServeHTTP(rr, req)
|
||||
if handlerCalled {
|
||||
t.Fatalf("vars %v: inner handler reached despite empty capture", tt.vars)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsValidNameSegment(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
want bool
|
||||
}{
|
||||
{"empty ok", "", true},
|
||||
{"plain", "orders", true},
|
||||
{"with dot inside", "my.table", true},
|
||||
{"hidden", ".hidden", true},
|
||||
|
||||
{"bare dot", ".", false},
|
||||
{"bare dotdot", "..", false},
|
||||
{"contains slash", "foo/bar", false},
|
||||
{"contains backslash", "foo\\bar", false},
|
||||
{"contains nul", "foo\x00bar", false},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := isValidNameSegment(tt.input); got != tt.want {
|
||||
t.Errorf("isValidNameSegment(%q) = %v, want %v", tt.input, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
55
weed/s3api/iceberg/requirements.go
Normal file
55
weed/s3api/iceberg/requirements.go
Normal file
@@ -0,0 +1,55 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
)
|
||||
|
||||
// normalizeRequirements repairs requirement JSON that iceberg-go's parser
|
||||
// rejects but the Iceberg REST spec permits.
|
||||
//
|
||||
// iceberg-go's assert-ref-snapshot-id parser uses a nullableInt64 that tracks
|
||||
// whether the "snapshot-id" field was present in the JSON. An absent field
|
||||
// (set == false) is rejected as "missing required field", even though the spec
|
||||
// makes snapshot-id optional — null means the ref must not already exist.
|
||||
// iceberg-go v0.6.0 used a plain *int64, so absent was nil and accepted.
|
||||
//
|
||||
// Clients like ClickHouse send assert-ref-snapshot-id without snapshot-id when
|
||||
// asserting a branch does not yet exist. Splicing in an explicit null before
|
||||
// unmarshaling restores the v0.6.0 behavior across both iceberg-go versions.
|
||||
func normalizeRequirements(raw json.RawMessage) (json.RawMessage, error) {
|
||||
if len(raw) == 0 {
|
||||
return raw, nil
|
||||
}
|
||||
var reqs []json.RawMessage
|
||||
if err := json.Unmarshal(raw, &reqs); err != nil {
|
||||
return raw, err
|
||||
}
|
||||
changed := false
|
||||
for i, req := range reqs {
|
||||
var obj map[string]json.RawMessage
|
||||
if err := json.Unmarshal(req, &obj); err != nil {
|
||||
continue // let the downstream parser report the error
|
||||
}
|
||||
var typ string
|
||||
if rawType, ok := obj["type"]; ok {
|
||||
_ = json.Unmarshal(rawType, &typ)
|
||||
}
|
||||
if typ != "assert-ref-snapshot-id" {
|
||||
continue
|
||||
}
|
||||
if _, hasSnapshotID := obj["snapshot-id"]; hasSnapshotID {
|
||||
continue
|
||||
}
|
||||
obj["snapshot-id"] = json.RawMessage("null")
|
||||
fixed, err := json.Marshal(obj)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
reqs[i] = fixed
|
||||
changed = true
|
||||
}
|
||||
if !changed {
|
||||
return raw, nil
|
||||
}
|
||||
return json.Marshal(reqs)
|
||||
}
|
||||
94
weed/s3api/iceberg/requirements_test.go
Normal file
94
weed/s3api/iceberg/requirements_test.go
Normal file
@@ -0,0 +1,94 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/apache/iceberg-go/table"
|
||||
)
|
||||
|
||||
func TestNormalizeRequirementsAddsNullSnapshotID(t *testing.T) {
|
||||
// ClickHouse sends assert-ref-snapshot-id without snapshot-id when
|
||||
// asserting a branch does not yet exist. iceberg-go's parser rejects
|
||||
// an absent field; normalizeRequirements splices in an explicit null.
|
||||
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main"}]`)
|
||||
|
||||
// Without normalization, iceberg-go rejects the requirement.
|
||||
var reqs table.Requirements
|
||||
if err := json.Unmarshal(raw, &reqs); err == nil {
|
||||
t.Fatal("iceberg-go accepted an assert-ref-snapshot-id without snapshot-id; expected an error")
|
||||
}
|
||||
|
||||
normalized, err := normalizeRequirements(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("normalizeRequirements: %v", err)
|
||||
}
|
||||
|
||||
if err := json.Unmarshal(normalized, &reqs); err != nil {
|
||||
t.Fatalf("unmarshal normalized requirements: %v", err)
|
||||
}
|
||||
if len(reqs) != 1 {
|
||||
t.Fatalf("got %d requirements, want 1", len(reqs))
|
||||
}
|
||||
if reqs[0].GetType() != "assert-ref-snapshot-id" {
|
||||
t.Fatalf("requirement type = %s, want assert-ref-snapshot-id", reqs[0].GetType())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeRequirementsPreservesPresentSnapshotID(t *testing.T) {
|
||||
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main","snapshot-id":42}]`)
|
||||
normalized, err := normalizeRequirements(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("normalizeRequirements: %v", err)
|
||||
}
|
||||
|
||||
var reqs table.Requirements
|
||||
if err := json.Unmarshal(normalized, &reqs); err != nil {
|
||||
t.Fatalf("unmarshal: %v", err)
|
||||
}
|
||||
if len(reqs) != 1 {
|
||||
t.Fatalf("got %d requirements, want 1", len(reqs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeRequirementsPreservesExplicitNullSnapshotID(t *testing.T) {
|
||||
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main","snapshot-id":null}]`)
|
||||
normalized, err := normalizeRequirements(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("normalizeRequirements: %v", err)
|
||||
}
|
||||
|
||||
var reqs table.Requirements
|
||||
if err := json.Unmarshal(normalized, &reqs); err != nil {
|
||||
t.Fatalf("unmarshal: %v", err)
|
||||
}
|
||||
if len(reqs) != 1 {
|
||||
t.Fatalf("got %d requirements, want 1", len(reqs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeRequirementsLeavesOtherTypesUnchanged(t *testing.T) {
|
||||
raw := json.RawMessage(`[{"type":"assert-create"},{"type":"assert-table-uuid","uuid":"00000000-0000-0000-0000-000000000000"}]`)
|
||||
normalized, err := normalizeRequirements(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("normalizeRequirements: %v", err)
|
||||
}
|
||||
|
||||
var reqs table.Requirements
|
||||
if err := json.Unmarshal(normalized, &reqs); err != nil {
|
||||
t.Fatalf("unmarshal: %v", err)
|
||||
}
|
||||
if len(reqs) != 2 {
|
||||
t.Fatalf("got %d requirements, want 2", len(reqs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeRequirementsEmptyInput(t *testing.T) {
|
||||
normalized, err := normalizeRequirements(nil)
|
||||
if err != nil {
|
||||
t.Fatalf("normalizeRequirements(nil): %v", err)
|
||||
}
|
||||
if normalized != nil {
|
||||
t.Fatalf("expected nil output for nil input, got %s", normalized)
|
||||
}
|
||||
}
|
||||
287
weed/s3api/iceberg/server.go
Normal file
287
weed/s3api/iceberg/server.go
Normal file
@@ -0,0 +1,287 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// FilerClient provides access to the filer for storage operations.
|
||||
type FilerClient interface {
|
||||
WithFilerClient(streamingMode bool, fn func(client filer_pb.SeaweedFilerClient) error) error
|
||||
}
|
||||
|
||||
type S3Authenticator interface {
|
||||
AuthenticateRequest(r *http.Request) (string, interface{}, s3err.ErrorCode)
|
||||
DefaultAllow() bool
|
||||
}
|
||||
|
||||
// CredentialValidator validates S3 access key / secret key pairs
|
||||
// and provides credential lookup for OAuth token verification.
|
||||
type CredentialValidator interface {
|
||||
// ValidateS3Credential checks if the access key and secret key are valid.
|
||||
// Returns the identity name and identity object on success.
|
||||
ValidateS3Credential(accessKey, secretKey string) (identityName string, identity interface{}, err error)
|
||||
// GetCredentialByAccessKey looks up a credential by access key.
|
||||
// Returns the identity name, identity object, and secret key.
|
||||
// Used for verifying Bearer tokens signed with a specific credential.
|
||||
GetCredentialByAccessKey(accessKey string) (identityName string, identity interface{}, secretKey string, err error)
|
||||
}
|
||||
|
||||
// VendedCredentials are short-lived S3 credentials scoped to one table.
|
||||
type VendedCredentials struct {
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
SessionToken string
|
||||
Expiration time.Time
|
||||
}
|
||||
|
||||
// CredentialVendor mints credentials limited to a single table's prefix for a
|
||||
// caller the catalog has already authenticated and authorized. A nil result
|
||||
// with no error means the deployment has vending switched off.
|
||||
type CredentialVendor interface {
|
||||
VendTableCredentials(ctx context.Context, principal, bucket, prefix string) (*VendedCredentials, error)
|
||||
}
|
||||
|
||||
// Server implements the Iceberg REST Catalog API.
|
||||
type Server struct {
|
||||
filerClient FilerClient
|
||||
tablesManager *s3tables.Manager
|
||||
prefix string // optional prefix for routes
|
||||
authenticator S3Authenticator
|
||||
credentialValidator CredentialValidator
|
||||
credentialVendor CredentialVendor
|
||||
s3Endpoint string // http(s):// URL advertised in LoadTable FileIO config
|
||||
}
|
||||
|
||||
// NewServer creates a new Iceberg REST Catalog server.
|
||||
func NewServer(filerClient FilerClient, authenticator S3Authenticator) *Server {
|
||||
manager := s3tables.NewManager()
|
||||
// Mirror the S3 port: fall open by default only when the gateway itself is
|
||||
// open. With auth configured, an authenticated catalog caller must pass the
|
||||
// normal permission check instead of being allowed because no policy denied
|
||||
// it — even if the full identity struct ever fails to reach the handler.
|
||||
if authenticator != nil {
|
||||
manager.SetDefaultAllow(authenticator.DefaultAllow())
|
||||
}
|
||||
return &Server{
|
||||
filerClient: filerClient,
|
||||
tablesManager: manager,
|
||||
prefix: "",
|
||||
authenticator: authenticator,
|
||||
}
|
||||
}
|
||||
|
||||
// SetCredentialVendor enables credential vending for clients that ask for it
|
||||
// with X-Iceberg-Access-Delegation: vended-credentials.
|
||||
func (s *Server) SetCredentialVendor(vendor CredentialVendor) {
|
||||
s.credentialVendor = vendor
|
||||
}
|
||||
|
||||
// SetCredentialValidator sets the credential validator for OAuth token support.
|
||||
func (s *Server) SetCredentialValidator(cv CredentialValidator) {
|
||||
s.credentialValidator = cv
|
||||
}
|
||||
|
||||
// SetS3Endpoint configures the S3 endpoint URL to vend to clients as part of
|
||||
// the LoadTable FileIO config, so they can read table data files directly
|
||||
// without separately discovering the S3 API address. See issue #9103.
|
||||
func (s *Server) SetS3Endpoint(endpoint string) {
|
||||
s.s3Endpoint = endpoint
|
||||
}
|
||||
|
||||
// RegisterRoutes registers Iceberg REST API routes on the provided router.
|
||||
func (s *Server) RegisterRoutes(router *mux.Router) {
|
||||
// Add middleware to log all requests/responses
|
||||
router.Use(loggingMiddleware)
|
||||
|
||||
// Reject `..`/`.`/NUL in {prefix}/{namespace}/{table} vars before any
|
||||
// handler runs. The router uses SkipClean(true), so traversal segments
|
||||
// would otherwise reach path.Join in stage-marker / location builders.
|
||||
// Registered after loggingMiddleware so rejected requests still get
|
||||
// audit-logged.
|
||||
router.Use(validateRequestPath)
|
||||
|
||||
// Configuration endpoint - no auth needed for config
|
||||
router.HandleFunc("/v1/config", s.handleConfig).Methods(http.MethodGet)
|
||||
|
||||
// OAuth2 token endpoint - no auth needed (this IS the auth endpoint)
|
||||
router.HandleFunc("/v1/oauth/tokens", s.handleOAuthTokens).Methods(http.MethodPost)
|
||||
|
||||
// Namespace endpoints - wrapped with Auth middleware
|
||||
router.HandleFunc("/v1/namespaces", s.Auth(s.handleListNamespaces)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces", s.Auth(s.handleCreateNamespace)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}", s.Auth(s.handleGetNamespace)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}", s.Auth(s.handleNamespaceExists)).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}", s.Auth(s.handleDropNamespace)).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/properties", s.Auth(s.handleUpdateNamespaceProperties)).Methods(http.MethodPost)
|
||||
|
||||
// Table endpoints - wrapped with Auth middleware
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables", s.Auth(s.handleListTables)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables", s.Auth(s.handleCreateTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/register", s.Auth(s.handleRegisterTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.Auth(s.handleLoadTable)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.Auth(s.handleTableExists)).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.Auth(s.handleDropTable)).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}", s.Auth(s.handleUpdateTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/tables/rename", s.Auth(s.handleRenameTable)).Methods(http.MethodPost)
|
||||
|
||||
// View endpoints - wrapped with Auth middleware
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/views", s.Auth(s.handleListViews)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/views", s.Auth(s.handleCreateView)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/views/{view}", s.Auth(s.handleLoadView)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/views/{view}", s.Auth(s.handleViewExists)).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/views/{view}", s.Auth(s.handleDropView)).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/views/{view}", s.Auth(s.handleUpdateView)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/views/rename", s.Auth(s.handleRenameView)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespaces/{namespace}/tables/{table}/metrics", s.Auth(s.handleReportMetrics)).Methods(http.MethodPost)
|
||||
|
||||
// Multi-table transaction commit - wrapped with Auth middleware
|
||||
router.HandleFunc("/v1/transactions/commit", s.Auth(s.handleCommitTransaction)).Methods(http.MethodPost)
|
||||
|
||||
// With prefix support - wrapped with Auth middleware
|
||||
router.HandleFunc("/v1/{prefix}/namespaces", s.Auth(s.handleListNamespaces)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces", s.Auth(s.handleCreateNamespace)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.Auth(s.handleGetNamespace)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.Auth(s.handleNamespaceExists)).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}", s.Auth(s.handleDropNamespace)).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/properties", s.Auth(s.handleUpdateNamespaceProperties)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables", s.Auth(s.handleListTables)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables", s.Auth(s.handleCreateTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/register", s.Auth(s.handleRegisterTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.Auth(s.handleLoadTable)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.Auth(s.handleTableExists)).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.Auth(s.handleDropTable)).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}", s.Auth(s.handleUpdateTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/tables/rename", s.Auth(s.handleRenameTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views", s.Auth(s.handleListViews)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views", s.Auth(s.handleCreateView)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views/{view}", s.Auth(s.handleLoadView)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views/{view}", s.Auth(s.handleViewExists)).Methods(http.MethodHead)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views/{view}", s.Auth(s.handleDropView)).Methods(http.MethodDelete)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/views/{view}", s.Auth(s.handleUpdateView)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/views/rename", s.Auth(s.handleRenameView)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics", s.Auth(s.handleReportMetrics)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/{prefix}/transactions/commit", s.Auth(s.handleCommitTransaction)).Methods(http.MethodPost)
|
||||
|
||||
// Catch-all for debugging
|
||||
router.PathPrefix("/").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
glog.V(2).Infof("Catch-all route hit: %s %s", r.Method, r.RequestURI)
|
||||
writeError(w, http.StatusNotFound, "NotFound", "Path not found")
|
||||
})
|
||||
|
||||
glog.V(2).Infof("Registered Iceberg REST Catalog routes")
|
||||
}
|
||||
|
||||
func loggingMiddleware(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
glog.V(2).Infof("Iceberg REST request: %s %s from %s", r.Method, r.RequestURI, r.RemoteAddr)
|
||||
|
||||
// Log all headers for debugging
|
||||
glog.V(2).Infof("Iceberg REST headers:")
|
||||
for name, values := range r.Header {
|
||||
for _, value := range values {
|
||||
// Redact sensitive headers
|
||||
if name == "Authorization" && len(value) > 20 {
|
||||
glog.V(2).Infof(" %s: %s...%s", name, value[:20], value[len(value)-10:])
|
||||
} else {
|
||||
glog.V(2).Infof(" %s: %s", name, value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Create a response writer that captures the status code
|
||||
wrapped := &responseWriter{ResponseWriter: w}
|
||||
next.ServeHTTP(wrapped, r)
|
||||
|
||||
glog.V(2).Infof("Iceberg REST response: %s %s -> %d", r.Method, r.RequestURI, wrapped.statusCode)
|
||||
})
|
||||
}
|
||||
|
||||
type responseWriter struct {
|
||||
http.ResponseWriter
|
||||
statusCode int
|
||||
}
|
||||
|
||||
func (w *responseWriter) WriteHeader(code int) {
|
||||
w.statusCode = code
|
||||
w.ResponseWriter.WriteHeader(code)
|
||||
}
|
||||
|
||||
func (s *Server) Auth(handler http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
// A request carrying a Bearer token is an Iceberg REST client. When
|
||||
// the token is invalid or expired, answer 401 immediately so clients
|
||||
// (Iceberg Java OAuth2Manager, pyiceberg) refresh their token and
|
||||
// retry. Falling through to the S3 authenticator instead would parse
|
||||
// the Authorization header as SigV4, fail with NotImplemented (501),
|
||||
// and clients would retry the dead token forever.
|
||||
// The auth scheme is case-insensitive (RFC 7235), matching
|
||||
// authenticateBearer below.
|
||||
if strings.HasPrefix(strings.ToLower(r.Header.Get("Authorization")), "bearer ") {
|
||||
if identityName, identity, ok := s.authenticateBearer(r); ok {
|
||||
ctx := r.Context()
|
||||
ctx = s3_constants.SetIdentityNameInContext(ctx, identityName)
|
||||
if identity != nil {
|
||||
ctx = s3_constants.SetIdentityInContext(ctx, identity)
|
||||
}
|
||||
r = r.WithContext(ctx)
|
||||
handler(w, r)
|
||||
return
|
||||
}
|
||||
// RFC 6750 / Iceberg REST spec: 401 is the refresh signal.
|
||||
w.Header().Set("WWW-Authenticate", "Bearer")
|
||||
writeError(w, http.StatusUnauthorized, "NotAuthorizedException", "Bearer token is invalid or expired")
|
||||
return
|
||||
}
|
||||
|
||||
if s.authenticator == nil {
|
||||
writeError(w, http.StatusUnauthorized, "NotAuthorizedException", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
identityName, identity, errCode := s.authenticator.AuthenticateRequest(r)
|
||||
if errCode != s3err.ErrNone {
|
||||
// If authentication failed but DefaultAllow is enabled, proceed without identity
|
||||
if s.authenticator.DefaultAllow() {
|
||||
glog.V(2).Infof("Iceberg: AuthenticateRequest failed (%v), but DefaultAllow is true, proceeding", errCode)
|
||||
} else {
|
||||
apiErr := s3err.GetAPIError(errCode)
|
||||
errorType := "RESTException"
|
||||
switch apiErr.HTTPStatusCode {
|
||||
case http.StatusForbidden:
|
||||
errorType = "ForbiddenException"
|
||||
case http.StatusUnauthorized:
|
||||
errorType = "NotAuthorizedException"
|
||||
case http.StatusBadRequest:
|
||||
errorType = "BadRequestException"
|
||||
case http.StatusInternalServerError:
|
||||
errorType = "InternalServerError"
|
||||
}
|
||||
writeError(w, apiErr.HTTPStatusCode, errorType, apiErr.Description)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if identityName != "" || identity != nil {
|
||||
ctx := r.Context()
|
||||
if identityName != "" {
|
||||
ctx = s3_constants.SetIdentityNameInContext(ctx, identityName)
|
||||
}
|
||||
if identity != nil {
|
||||
ctx = s3_constants.SetIdentityInContext(ctx, identity)
|
||||
}
|
||||
r = r.WithContext(ctx)
|
||||
}
|
||||
|
||||
handler(w, r)
|
||||
}
|
||||
}
|
||||
231
weed/s3api/iceberg/server_auth_test.go
Normal file
231
weed/s3api/iceberg/server_auth_test.go
Normal file
@@ -0,0 +1,231 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
jwt "github.com/golang-jwt/jwt/v5"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
||||
)
|
||||
|
||||
type mockS3Authenticator struct {
|
||||
called bool
|
||||
errCode s3err.ErrorCode
|
||||
}
|
||||
|
||||
func (m *mockS3Authenticator) AuthenticateRequest(r *http.Request) (string, interface{}, s3err.ErrorCode) {
|
||||
m.called = true
|
||||
if m.errCode != s3err.ErrNone {
|
||||
return "", nil, m.errCode
|
||||
}
|
||||
return "s3user", nil, s3err.ErrNone
|
||||
}
|
||||
|
||||
func (m *mockS3Authenticator) DefaultAllow() bool { return false }
|
||||
|
||||
func mintTestToken(t *testing.T, accessKey, secret string, issuedAt, expiresAt time.Time) string {
|
||||
t.Helper()
|
||||
key := deriveSigningKey(accessKey, secret)
|
||||
claims := IcebergClaims{
|
||||
IdentityName: "testuser",
|
||||
AccessKey: accessKey,
|
||||
RegisteredClaims: jwt.RegisteredClaims{
|
||||
IssuedAt: jwt.NewNumericDate(issuedAt),
|
||||
ExpiresAt: jwt.NewNumericDate(expiresAt),
|
||||
Issuer: "seaweedfs-iceberg",
|
||||
},
|
||||
}
|
||||
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
|
||||
signed, err := token.SignedString(key)
|
||||
if err != nil {
|
||||
t.Fatalf("sign token: %v", err)
|
||||
}
|
||||
return signed
|
||||
}
|
||||
|
||||
func newAuthTestServer() (*Server, *mockS3Authenticator) {
|
||||
s := newTestServerWithOAuth()
|
||||
auth := &mockS3Authenticator{errCode: s3err.ErrNotImplemented}
|
||||
s.authenticator = auth
|
||||
return s, auth
|
||||
}
|
||||
|
||||
// TestAuthExpiredBearerReturns401: an expired Bearer token must get 401
|
||||
// (the refresh signal for Iceberg clients), never fall through to the S3
|
||||
// authenticator and surface as 501 NotImplemented.
|
||||
func TestAuthExpiredBearerReturns401(t *testing.T) {
|
||||
s, auth := newAuthTestServer()
|
||||
now := time.Now()
|
||||
expired := mintTestToken(t, "AKID123", "secret456", now.Add(-2*time.Hour), now.Add(-1*time.Hour))
|
||||
|
||||
var handlerCalled bool
|
||||
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
|
||||
handlerCalled = true
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+expired)
|
||||
rec := httptest.NewRecorder()
|
||||
handler(rec, req)
|
||||
|
||||
if rec.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("expired Bearer: status = %d, want 401 (previously fell through to 501)", rec.Code)
|
||||
}
|
||||
if auth.called {
|
||||
t.Fatalf("expired Bearer must not fall through to the S3 authenticator")
|
||||
}
|
||||
if handlerCalled {
|
||||
t.Fatalf("handler must not run for an expired token")
|
||||
}
|
||||
if wa := rec.Header().Get("WWW-Authenticate"); wa != "Bearer" {
|
||||
t.Fatalf("WWW-Authenticate = %q, want Bearer", wa)
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), "NotAuthorizedException") {
|
||||
t.Fatalf("error type must be NotAuthorizedException, got: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthCaseInsensitiveBearerScheme: RFC 7235 auth schemes are
|
||||
// case-insensitive; a lowercase or mixed-case "bearer" must take the same
|
||||
// Bearer path (401 refresh signal) instead of falling through to S3 auth.
|
||||
func TestAuthCaseInsensitiveBearerScheme(t *testing.T) {
|
||||
s, auth := newAuthTestServer()
|
||||
now := time.Now()
|
||||
expired := mintTestToken(t, "AKID123", "secret456", now.Add(-2*time.Hour), now.Add(-1*time.Hour))
|
||||
|
||||
for _, scheme := range []string{"bearer", "BEARER", "BeArEr"} {
|
||||
var handlerCalled bool
|
||||
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
|
||||
handlerCalled = true
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
req.Header.Set("Authorization", scheme+" "+expired)
|
||||
rec := httptest.NewRecorder()
|
||||
handler(rec, req)
|
||||
|
||||
if rec.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("scheme %q: status = %d, want 401", scheme, rec.Code)
|
||||
}
|
||||
if auth.called {
|
||||
t.Fatalf("scheme %q: must not fall through to the S3 authenticator", scheme)
|
||||
}
|
||||
if handlerCalled {
|
||||
t.Fatalf("scheme %q: handler must not run for an expired token", scheme)
|
||||
}
|
||||
if wa := rec.Header().Get("WWW-Authenticate"); wa != "Bearer" {
|
||||
t.Fatalf("scheme %q: WWW-Authenticate = %q, want Bearer", scheme, wa)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthGarbageBearerReturns401 pins the same contract for malformed tokens.
|
||||
func TestAuthGarbageBearerReturns401(t *testing.T) {
|
||||
s, auth := newAuthTestServer()
|
||||
|
||||
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
req.Header.Set("Authorization", "Bearer not-a-jwt")
|
||||
rec := httptest.NewRecorder()
|
||||
handler(rec, req)
|
||||
|
||||
if rec.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("garbage Bearer: status = %d, want 401", rec.Code)
|
||||
}
|
||||
if auth.called {
|
||||
t.Fatalf("garbage Bearer must not fall through to the S3 authenticator")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthFreshBearerRunsHandler verifies the happy path still authenticates
|
||||
// and hands the identity to the handler.
|
||||
func TestAuthFreshBearerRunsHandler(t *testing.T) {
|
||||
s, auth := newAuthTestServer()
|
||||
now := time.Now()
|
||||
fresh := mintTestToken(t, "AKID123", "secret456", now, now.Add(time.Hour))
|
||||
|
||||
var gotIdentity string
|
||||
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotIdentity = s3_constants.GetIdentityNameFromContext(r)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+fresh)
|
||||
rec := httptest.NewRecorder()
|
||||
handler(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("fresh Bearer: status = %d, want 200", rec.Code)
|
||||
}
|
||||
if auth.called {
|
||||
t.Fatalf("fresh Bearer should not need the S3 authenticator")
|
||||
}
|
||||
if gotIdentity != "testuser" {
|
||||
t.Fatalf("identity in context = %q, want testuser", gotIdentity)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthNoBearerStillUsesS3Authenticator keeps the non-Bearer path intact:
|
||||
// SigV4 requests keep flowing through the S3 authenticator as before.
|
||||
func TestAuthNoBearerStillUsesS3Authenticator(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
auth := &mockS3Authenticator{errCode: s3err.ErrNone}
|
||||
s.authenticator = auth
|
||||
|
||||
var handlerCalled bool
|
||||
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
|
||||
handlerCalled = true
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
handler(rec, req)
|
||||
|
||||
if !auth.called {
|
||||
t.Fatalf("request without Bearer header must use the S3 authenticator")
|
||||
}
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 after successful S3 auth", rec.Code)
|
||||
}
|
||||
if !handlerCalled {
|
||||
t.Fatalf("handler must run after S3 auth succeeds")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAuthNonBearerAuthorizationHeaderUsesS3Authenticator pins that other
|
||||
// Authorization schemes (e.g. SigV4) never take the Bearer fast path.
|
||||
func TestAuthNonBearerAuthorizationHeaderUsesS3Authenticator(t *testing.T) {
|
||||
s := newTestServerWithOAuth()
|
||||
auth := &mockS3Authenticator{errCode: s3err.ErrNone}
|
||||
s.authenticator = auth
|
||||
|
||||
var handlerCalled bool
|
||||
handler := s.Auth(func(w http.ResponseWriter, r *http.Request) {
|
||||
handlerCalled = true
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/namespaces", nil)
|
||||
req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential=AKID123/...")
|
||||
rec := httptest.NewRecorder()
|
||||
handler(rec, req)
|
||||
|
||||
if !auth.called {
|
||||
t.Fatalf("SigV4 Authorization header must use the S3 authenticator")
|
||||
}
|
||||
if !handlerCalled {
|
||||
t.Fatalf("handler must run after S3 auth succeeds")
|
||||
}
|
||||
}
|
||||
41
weed/s3api/iceberg/server_routes_test.go
Normal file
41
weed/s3api/iceberg/server_routes_test.go
Normal file
@@ -0,0 +1,41 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
)
|
||||
|
||||
// The catch-all answers anything unrouted, so a path that reaches it is a
|
||||
// route that does not exist.
|
||||
func TestRegisteredRoutesReachAHandler(t *testing.T) {
|
||||
router := mux.NewRouter().SkipClean(true)
|
||||
(&Server{}).RegisterRoutes(router)
|
||||
|
||||
cases := []struct {
|
||||
method string
|
||||
target string
|
||||
}{
|
||||
{http.MethodPost, "/v1/views/rename"},
|
||||
{http.MethodPost, "/v1/warehouse/views/rename"},
|
||||
{http.MethodPost, "/v1/namespaces/ns/tables/t/metrics"},
|
||||
{http.MethodPost, "/v1/warehouse/namespaces/ns/tables/t/metrics"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
var match mux.RouteMatch
|
||||
if !router.Match(httptest.NewRequest(tc.method, tc.target, nil), &match) {
|
||||
t.Errorf("%s %s matched no route", tc.method, tc.target)
|
||||
continue
|
||||
}
|
||||
if match.MatchErr != nil {
|
||||
t.Errorf("%s %s: %v", tc.method, tc.target, match.MatchErr)
|
||||
continue
|
||||
}
|
||||
if tmpl, err := match.Route.GetPathTemplate(); err == nil && tmpl == "/" {
|
||||
t.Errorf("%s %s fell through to the catch-all", tc.method, tc.target)
|
||||
}
|
||||
}
|
||||
}
|
||||
276
weed/s3api/iceberg/stage_create.go
Normal file
276
weed/s3api/iceberg/stage_create.go
Normal file
@@ -0,0 +1,276 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"path"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
const (
|
||||
stageCreateMarkerDirName = ".iceberg_staged"
|
||||
// Keep staged markers long enough to avoid deleting in-progress create transactions.
|
||||
stageCreateMarkerTTL = 72 * time.Hour
|
||||
)
|
||||
|
||||
type stageCreateMarker struct {
|
||||
TableUUID string `json:"table_uuid"`
|
||||
Location string `json:"location"`
|
||||
StagedMetadataLocation string `json:"staged_metadata_location,omitempty"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
ExpiresAt string `json:"expires_at"`
|
||||
}
|
||||
|
||||
func stageCreateMarkerNamespaceKey(namespace []string) string {
|
||||
return url.PathEscape(encodeNamespace(namespace))
|
||||
}
|
||||
|
||||
func stageCreateMarkerDir(bucketName string, namespace []string, tableName string) string {
|
||||
return path.Join(s3tables.TablesPath, bucketName, stageCreateMarkerDirName, stageCreateMarkerNamespaceKey(namespace), tableName)
|
||||
}
|
||||
|
||||
func stageCreateStagedTablePath(namespace []string, tableName string, tableUUID uuid.UUID) string {
|
||||
return path.Join(stageCreateMarkerDirName, stageCreateMarkerNamespaceKey(namespace), tableName, tableUUID.String())
|
||||
}
|
||||
|
||||
func (s *Server) pruneExpiredStageCreateMarkers(ctx context.Context, bucketName string, namespace []string, tableName string) error {
|
||||
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
markerDir := stageCreateMarkerDir(bucketName, namespace, tableName)
|
||||
now := time.Now().UTC()
|
||||
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
stream, err := client.ListEntries(opCtx, &filer_pb.ListEntriesRequest{
|
||||
Directory: markerDir,
|
||||
Limit: 1024,
|
||||
})
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) || strings.Contains(strings.ToLower(err.Error()), "not found") {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
for {
|
||||
resp, recvErr := stream.Recv()
|
||||
if recvErr == io.EOF {
|
||||
return nil
|
||||
}
|
||||
if recvErr != nil {
|
||||
return recvErr
|
||||
}
|
||||
if resp.Entry == nil || resp.Entry.IsDirectory || len(resp.Entry.Content) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
var marker stageCreateMarker
|
||||
if err := json.Unmarshal(resp.Entry.Content, &marker); err != nil {
|
||||
continue
|
||||
}
|
||||
expiresAt, err := time.Parse(time.RFC3339Nano, marker.ExpiresAt)
|
||||
if err != nil || !expiresAt.Before(now) {
|
||||
continue
|
||||
}
|
||||
|
||||
if err := filer_pb.DoRemove(opCtx, client, markerDir, resp.Entry.Name, true, false, true, false, nil); err != nil && !errors.Is(err, filer_pb.ErrNotFound) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) loadLatestStageCreateMarker(ctx context.Context, bucketName string, namespace []string, tableName string) (*stageCreateMarker, error) {
|
||||
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
markerDir := stageCreateMarkerDir(bucketName, namespace, tableName)
|
||||
now := time.Now().UTC()
|
||||
|
||||
var latestMarker *stageCreateMarker
|
||||
var latestCreatedAt time.Time
|
||||
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
stream, err := client.ListEntries(opCtx, &filer_pb.ListEntriesRequest{
|
||||
Directory: markerDir,
|
||||
Limit: 1024,
|
||||
})
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) || strings.Contains(strings.ToLower(err.Error()), "not found") {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
for {
|
||||
resp, recvErr := stream.Recv()
|
||||
if recvErr == io.EOF {
|
||||
return nil
|
||||
}
|
||||
if recvErr != nil {
|
||||
return recvErr
|
||||
}
|
||||
if resp.Entry == nil || resp.Entry.IsDirectory || len(resp.Entry.Content) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
var marker stageCreateMarker
|
||||
if err := json.Unmarshal(resp.Entry.Content, &marker); err != nil {
|
||||
continue
|
||||
}
|
||||
expiresAt, err := time.Parse(time.RFC3339Nano, marker.ExpiresAt)
|
||||
if err != nil || !expiresAt.After(now) {
|
||||
continue
|
||||
}
|
||||
|
||||
createdAt, err := time.Parse(time.RFC3339Nano, marker.CreatedAt)
|
||||
if err != nil {
|
||||
createdAt = time.Time{}
|
||||
}
|
||||
if latestMarker == nil || createdAt.After(latestCreatedAt) {
|
||||
candidate := marker
|
||||
latestMarker = &candidate
|
||||
latestCreatedAt = createdAt
|
||||
}
|
||||
}
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return latestMarker, nil
|
||||
}
|
||||
|
||||
func (s *Server) writeStageCreateMarker(ctx context.Context, bucketName string, namespace []string, tableName string, tableUUID uuid.UUID, location string, stagedMetadataLocation string) error {
|
||||
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
marker := stageCreateMarker{
|
||||
TableUUID: tableUUID.String(),
|
||||
Location: location,
|
||||
StagedMetadataLocation: stagedMetadataLocation,
|
||||
CreatedAt: time.Now().UTC().Format(time.RFC3339Nano),
|
||||
ExpiresAt: time.Now().UTC().Add(stageCreateMarkerTTL).Format(time.RFC3339Nano),
|
||||
}
|
||||
content, err := json.Marshal(marker)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := s.pruneExpiredStageCreateMarkers(ctx, bucketName, namespace, tableName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
ensureDir := func(parent, name, errorContext string) error {
|
||||
_, lookupErr := filer_pb.LookupEntry(opCtx, client, &filer_pb.LookupDirectoryEntryRequest{
|
||||
Directory: parent,
|
||||
Name: name,
|
||||
})
|
||||
if lookupErr == nil {
|
||||
return nil
|
||||
}
|
||||
if lookupErr != filer_pb.ErrNotFound {
|
||||
return fmt.Errorf("lookup %s failed: %w", errorContext, lookupErr)
|
||||
}
|
||||
|
||||
resp, createErr := client.CreateEntry(opCtx, &filer_pb.CreateEntryRequest{
|
||||
Directory: parent,
|
||||
Entry: &filer_pb.Entry{
|
||||
Name: name,
|
||||
IsDirectory: true,
|
||||
Attributes: &filer_pb.FuseAttributes{
|
||||
Mtime: time.Now().Unix(),
|
||||
Crtime: time.Now().Unix(),
|
||||
FileMode: uint32(0755 | os.ModeDir),
|
||||
},
|
||||
},
|
||||
})
|
||||
if createErr != nil {
|
||||
return fmt.Errorf("failed to create %s: %w", errorContext, createErr)
|
||||
}
|
||||
if resp.ErrorCode != filer_pb.FilerError_OK && resp.ErrorCode != filer_pb.FilerError_ENTRY_ALREADY_EXISTS {
|
||||
return fmt.Errorf("failed to create %s: %s", errorContext, resp.Error)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
segments := []string{bucketName, stageCreateMarkerDirName, stageCreateMarkerNamespaceKey(namespace), tableName}
|
||||
currentDir := s3tables.TablesPath
|
||||
for _, segment := range segments {
|
||||
if segment == "" {
|
||||
continue
|
||||
}
|
||||
if err := ensureDir(currentDir, segment, "stage-create marker directory"); err != nil {
|
||||
return err
|
||||
}
|
||||
currentDir = path.Join(currentDir, segment)
|
||||
}
|
||||
|
||||
entryName := tableUUID.String() + ".json"
|
||||
resp, createErr := client.CreateEntry(opCtx, &filer_pb.CreateEntryRequest{
|
||||
Directory: currentDir,
|
||||
Entry: &filer_pb.Entry{
|
||||
Name: entryName,
|
||||
Attributes: &filer_pb.FuseAttributes{
|
||||
Mtime: time.Now().Unix(),
|
||||
Crtime: time.Now().Unix(),
|
||||
FileMode: uint32(0644),
|
||||
FileSize: uint64(len(content)),
|
||||
},
|
||||
Content: content,
|
||||
Extended: map[string][]byte{
|
||||
"Mime-Type": []byte("application/json"),
|
||||
},
|
||||
},
|
||||
})
|
||||
if createErr != nil {
|
||||
return createErr
|
||||
}
|
||||
if resp.ErrorCode != filer_pb.FilerError_OK {
|
||||
if sentinel := filer_pb.FilerErrorToSentinel(resp.ErrorCode); sentinel != nil {
|
||||
return fmt.Errorf("create stage marker: %w", sentinel)
|
||||
}
|
||||
return fmt.Errorf("create stage marker: code=%v %s", resp.ErrorCode, resp.Error)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) deleteStageCreateMarkers(ctx context.Context, bucketName string, namespace []string, tableName string) error {
|
||||
opCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
parentDir := path.Dir(stageCreateMarkerDir(bucketName, namespace, tableName))
|
||||
targetName := path.Base(stageCreateMarkerDir(bucketName, namespace, tableName))
|
||||
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
err := filer_pb.DoRemove(opCtx, client, parentDir, targetName, true, true, true, false, nil)
|
||||
if errors.Is(err, filer_pb.ErrNotFound) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
func isStageCreateEnabled() bool {
|
||||
value := strings.TrimSpace(strings.ToLower(os.Getenv("ICEBERG_ENABLE_STAGE_CREATE")))
|
||||
if value == "" {
|
||||
return true
|
||||
}
|
||||
switch value {
|
||||
case "0", "false", "no", "off":
|
||||
return false
|
||||
default:
|
||||
return true
|
||||
}
|
||||
}
|
||||
BIN
weed/s3api/iceberg/testdata/clickhouse_abs/b317f0e5-10f3-41d6-86bc-8b42f01f3e86.avro
vendored
Normal file
BIN
weed/s3api/iceberg/testdata/clickhouse_abs/b317f0e5-10f3-41d6-86bc-8b42f01f3e86.avro
vendored
Normal file
Binary file not shown.
Binary file not shown.
BIN
weed/s3api/iceberg/testdata/clickhouse_rel/690958a2-9936-4a04-8117-d8104c977195.avro
vendored
Normal file
BIN
weed/s3api/iceberg/testdata/clickhouse_rel/690958a2-9936-4a04-8117-d8104c977195.avro
vendored
Normal file
Binary file not shown.
Binary file not shown.
289
weed/s3api/iceberg/types.go
Normal file
289
weed/s3api/iceberg/types.go
Normal file
@@ -0,0 +1,289 @@
|
||||
// Package iceberg defines types for the Iceberg REST Catalog API.
|
||||
// This package uses types from github.com/apache/iceberg-go for spec compliance.
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/apache/iceberg-go/table"
|
||||
"github.com/apache/iceberg-go/view"
|
||||
)
|
||||
|
||||
// CatalogConfig is returned by GET /v1/config.
|
||||
type CatalogConfig struct {
|
||||
Defaults map[string]string `json:"defaults"`
|
||||
Overrides map[string]string `json:"overrides"`
|
||||
Endpoints []string `json:"endpoints,omitempty"`
|
||||
}
|
||||
|
||||
// ErrorModel represents an Iceberg error.
|
||||
type ErrorModel struct {
|
||||
Message string `json:"message"`
|
||||
Type string `json:"type"`
|
||||
Code int `json:"code"`
|
||||
Stack []string `json:"stack,omitempty"`
|
||||
}
|
||||
|
||||
// ErrorResponse wraps an error in the standard Iceberg format.
|
||||
type ErrorResponse struct {
|
||||
Error ErrorModel `json:"error"`
|
||||
}
|
||||
|
||||
// Namespace is a reference to one or more levels of a namespace.
|
||||
type Namespace []string
|
||||
|
||||
// TableIdentifier identifies a table within a namespace.
|
||||
type TableIdentifier struct {
|
||||
Namespace Namespace `json:"namespace"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// ListNamespacesResponse is returned by GET /v1/namespaces.
|
||||
type ListNamespacesResponse struct {
|
||||
NextPageToken string `json:"next-page-token,omitempty"`
|
||||
Namespaces []Namespace `json:"namespaces"`
|
||||
}
|
||||
|
||||
// CreateNamespaceRequest is sent to POST /v1/namespaces.
|
||||
type CreateNamespaceRequest struct {
|
||||
Namespace Namespace `json:"namespace"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
// CreateNamespaceResponse is returned by POST /v1/namespaces.
|
||||
type CreateNamespaceResponse struct {
|
||||
Namespace Namespace `json:"namespace"`
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
// GetNamespaceResponse is returned by GET /v1/namespaces/{namespace}.
|
||||
type GetNamespaceResponse struct {
|
||||
Namespace Namespace `json:"namespace"`
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
// UpdateNamespacePropertiesRequest is sent to POST /v1/namespaces/{namespace}/properties.
|
||||
type UpdateNamespacePropertiesRequest struct {
|
||||
Removals []string `json:"removals,omitempty"`
|
||||
Updates map[string]string `json:"updates,omitempty"`
|
||||
}
|
||||
|
||||
// UpdateNamespacePropertiesResponse is returned by POST /v1/namespaces/{namespace}/properties.
|
||||
type UpdateNamespacePropertiesResponse struct {
|
||||
Removed []string `json:"removed"`
|
||||
Updated []string `json:"updated"`
|
||||
Missing []string `json:"missing"`
|
||||
}
|
||||
|
||||
// ListTablesResponse is returned by GET /v1/namespaces/{namespace}/tables.
|
||||
type ListTablesResponse struct {
|
||||
NextPageToken string `json:"next-page-token,omitempty"`
|
||||
Identifiers []TableIdentifier `json:"identifiers"`
|
||||
}
|
||||
|
||||
// CreateTableRequest is sent to POST /v1/namespaces/{namespace}/tables.
|
||||
// Uses iceberg-go types for Schema, PartitionSpec, and SortOrder.
|
||||
type CreateTableRequest struct {
|
||||
Name string `json:"name"`
|
||||
Location string `json:"location,omitempty"`
|
||||
Schema *iceberg.Schema `json:"schema,omitempty"`
|
||||
PartitionSpec *iceberg.PartitionSpec `json:"partition-spec,omitempty"`
|
||||
WriteOrder *table.SortOrder `json:"write-order,omitempty"`
|
||||
StageCreate bool `json:"stage-create,omitempty"`
|
||||
Properties iceberg.Properties `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
// RegisterTableRequest is sent to POST /v1/namespaces/{namespace}/register.
|
||||
type RegisterTableRequest struct {
|
||||
Name string `json:"name"`
|
||||
MetadataLocation string `json:"metadata-location"`
|
||||
}
|
||||
|
||||
type LoadTableResult struct {
|
||||
MetadataLocation string `json:"metadata-location,omitempty"`
|
||||
Metadata table.Metadata `json:"metadata"`
|
||||
Config iceberg.Properties `json:"config"`
|
||||
StorageCredentials []StorageCredential `json:"storage-credentials,omitempty"`
|
||||
}
|
||||
|
||||
// StorageCredential is one set of credentials scoped to a location prefix, as
|
||||
// returned to clients that asked for credential vending.
|
||||
type StorageCredential struct {
|
||||
Prefix string `json:"prefix"`
|
||||
Config iceberg.Properties `json:"config"`
|
||||
}
|
||||
|
||||
// loadTableResultAlias is used for custom JSON unmarshaling.
|
||||
type loadTableResultAlias struct {
|
||||
MetadataLocation string `json:"metadata-location,omitempty"`
|
||||
RawMetadata json.RawMessage `json:"metadata"`
|
||||
Config iceberg.Properties `json:"config,omitempty"`
|
||||
StorageCredentials []StorageCredential `json:"storage-credentials,omitempty"`
|
||||
}
|
||||
|
||||
// MarshalJSON serializes LoadTableResult while backfilling spec-required
|
||||
// metadata fields that iceberg-go v0.5.0 drops via `omitempty`. Without this,
|
||||
// strict Iceberg REST clients (Java/Spark/Trino) reject otherwise valid
|
||||
// responses for empty tables with errors like
|
||||
// "Cannot parse missing long current-snapshot-id".
|
||||
func (r LoadTableResult) MarshalJSON() ([]byte, error) {
|
||||
metaBytes, err := json.Marshal(r.Metadata)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
metaBytes = ensureMetadataSpecCompliance(metaBytes)
|
||||
|
||||
return json.Marshal(struct {
|
||||
MetadataLocation string `json:"metadata-location,omitempty"`
|
||||
Metadata json.RawMessage `json:"metadata"`
|
||||
Config iceberg.Properties `json:"config"`
|
||||
StorageCredentials []StorageCredential `json:"storage-credentials,omitempty"`
|
||||
}{
|
||||
MetadataLocation: r.MetadataLocation,
|
||||
Metadata: metaBytes,
|
||||
Config: r.Config,
|
||||
StorageCredentials: r.StorageCredentials,
|
||||
})
|
||||
}
|
||||
|
||||
// UnmarshalJSON implements custom unmarshaling for LoadTableResult
|
||||
// to properly parse table.Metadata using iceberg-go's parser.
|
||||
func (r *LoadTableResult) UnmarshalJSON(data []byte) error {
|
||||
var alias loadTableResultAlias
|
||||
if err := json.Unmarshal(data, &alias); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
r.MetadataLocation = alias.MetadataLocation
|
||||
r.Config = alias.Config
|
||||
r.StorageCredentials = alias.StorageCredentials
|
||||
|
||||
if len(alias.RawMetadata) > 0 {
|
||||
metadata, err := table.ParseMetadataBytes(alias.RawMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
r.Metadata = metadata
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// RenameTableRequest is sent to POST /v1/tables/rename.
|
||||
type RenameTableRequest struct {
|
||||
Source TableIdentifier `json:"source"`
|
||||
Destination TableIdentifier `json:"destination"`
|
||||
}
|
||||
|
||||
// CommitTableRequest is sent to POST /v1/namespaces/{namespace}/tables/{table}.
|
||||
type CommitTableRequest struct {
|
||||
Identifier *TableIdentifier `json:"identifier,omitempty"`
|
||||
Requirements table.Requirements `json:"requirements"`
|
||||
Updates table.Updates `json:"updates"`
|
||||
}
|
||||
|
||||
// CommitTableResponse is returned by POST table commit operations.
|
||||
type CommitTableResponse struct {
|
||||
MetadataLocation string `json:"metadata-location"`
|
||||
Metadata table.Metadata `json:"metadata"`
|
||||
}
|
||||
|
||||
// commitTableResponseAlias is used for custom JSON unmarshaling.
|
||||
type commitTableResponseAlias struct {
|
||||
MetadataLocation string `json:"metadata-location"`
|
||||
RawMetadata json.RawMessage `json:"metadata"`
|
||||
}
|
||||
|
||||
// MarshalJSON mirrors LoadTableResult.MarshalJSON: it backfills spec-required
|
||||
// fields that iceberg-go v0.5.0 omits, so commit responses also parse cleanly
|
||||
// on strict Iceberg clients.
|
||||
func (r CommitTableResponse) MarshalJSON() ([]byte, error) {
|
||||
metaBytes, err := json.Marshal(r.Metadata)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
metaBytes = ensureMetadataSpecCompliance(metaBytes)
|
||||
|
||||
return json.Marshal(struct {
|
||||
MetadataLocation string `json:"metadata-location"`
|
||||
Metadata json.RawMessage `json:"metadata"`
|
||||
}{
|
||||
MetadataLocation: r.MetadataLocation,
|
||||
Metadata: metaBytes,
|
||||
})
|
||||
}
|
||||
|
||||
// UnmarshalJSON implements custom unmarshaling for CommitTableResponse.
|
||||
func (r *CommitTableResponse) UnmarshalJSON(data []byte) error {
|
||||
var alias commitTableResponseAlias
|
||||
if err := json.Unmarshal(data, &alias); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
r.MetadataLocation = alias.MetadataLocation
|
||||
|
||||
if len(alias.RawMetadata) > 0 {
|
||||
metadata, err := table.ParseMetadataBytes(alias.RawMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
r.Metadata = metadata
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ListViewsResponse is returned by GET /v1/namespaces/{namespace}/views.
|
||||
type ListViewsResponse struct {
|
||||
NextPageToken string `json:"next-page-token,omitempty"`
|
||||
Identifiers []TableIdentifier `json:"identifiers"`
|
||||
}
|
||||
|
||||
// CreateViewRequest is sent to POST /v1/namespaces/{namespace}/views.
|
||||
type CreateViewRequest struct {
|
||||
Name string `json:"name"`
|
||||
Schema *iceberg.Schema `json:"schema"`
|
||||
Location string `json:"location,omitempty"`
|
||||
Properties iceberg.Properties `json:"properties,omitempty"`
|
||||
ViewVersion *view.Version `json:"view-version"`
|
||||
}
|
||||
|
||||
// ViewResponse is returned by view create/update and load operations.
|
||||
type ViewResponse struct {
|
||||
MetadataLocation string `json:"metadata-location"`
|
||||
Metadata view.Metadata `json:"metadata"`
|
||||
Config iceberg.Properties `json:"config"`
|
||||
}
|
||||
|
||||
// viewResponseAlias is used for custom JSON unmarshaling.
|
||||
type viewResponseAlias struct {
|
||||
MetadataLocation string `json:"metadata-location"`
|
||||
RawMetadata json.RawMessage `json:"metadata"`
|
||||
Config iceberg.Properties `json:"config,omitempty"`
|
||||
}
|
||||
|
||||
// UnmarshalJSON parses view.Metadata using iceberg-go's view parser.
|
||||
func (r *ViewResponse) UnmarshalJSON(data []byte) error {
|
||||
var alias viewResponseAlias
|
||||
if err := json.Unmarshal(data, &alias); err != nil {
|
||||
return err
|
||||
}
|
||||
r.MetadataLocation = alias.MetadataLocation
|
||||
r.Config = alias.Config
|
||||
if len(alias.RawMetadata) > 0 {
|
||||
metadata, err := view.ParseMetadataBytes(alias.RawMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
r.Metadata = metadata
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// UpdateViewRequest is sent to POST /v1/namespaces/{namespace}/views/{view}.
|
||||
type UpdateViewRequest struct {
|
||||
Identifier *TableIdentifier `json:"identifier,omitempty"`
|
||||
Requirements view.Requirements `json:"requirements"`
|
||||
Updates view.Updates `json:"updates"`
|
||||
}
|
||||
353
weed/s3api/iceberg/utils.go
Normal file
353
weed/s3api/iceberg/utils.go
Normal file
@@ -0,0 +1,353 @@
|
||||
package iceberg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// parseNamespace parses the namespace from path parameter.
|
||||
// Iceberg uses unit separator (0x1F) for multi-level namespaces.
|
||||
// Note: mux already decodes URL-encoded path parameters, so we only split by unit separator.
|
||||
func parseNamespace(encoded string) []string {
|
||||
if encoded == "" {
|
||||
return nil
|
||||
}
|
||||
parts := strings.Split(encoded, "\x1F")
|
||||
// Filter empty parts
|
||||
result := make([]string, 0, len(parts))
|
||||
for _, p := range parts {
|
||||
if p != "" {
|
||||
result = append(result, p)
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// encodeNamespace encodes namespace parts using the Iceberg REST protocol's
|
||||
// unit separator (0x1F) convention. This is only appropriate for protocol-level
|
||||
// encoding (e.g. URL path parameters), NOT for filesystem/S3 paths.
|
||||
func encodeNamespace(parts []string) string {
|
||||
return strings.Join(parts, "\x1F")
|
||||
}
|
||||
|
||||
// flattenNamespacePath joins namespace parts with "." for use in S3 location
|
||||
// and filer paths, matching the S3 Tables storage layer convention.
|
||||
func flattenNamespacePath(parts []string) string {
|
||||
return strings.Join(parts, ".")
|
||||
}
|
||||
|
||||
func parseS3Location(location string) (bucketName, tablePath string, err error) {
|
||||
if !strings.HasPrefix(location, "s3://") {
|
||||
return "", "", fmt.Errorf("unsupported location: %s", location)
|
||||
}
|
||||
trimmed := strings.TrimPrefix(location, "s3://")
|
||||
trimmed = strings.TrimSuffix(trimmed, "/")
|
||||
if trimmed == "" {
|
||||
return "", "", fmt.Errorf("invalid location: %s", location)
|
||||
}
|
||||
parts := strings.SplitN(trimmed, "/", 2)
|
||||
bucketName = parts[0]
|
||||
if bucketName == "" {
|
||||
return "", "", fmt.Errorf("invalid location bucket: %s", location)
|
||||
}
|
||||
if len(parts) == 2 {
|
||||
tablePath = parts[1]
|
||||
}
|
||||
return bucketName, tablePath, nil
|
||||
}
|
||||
|
||||
func tableLocationFromMetadataLocation(metadataLocation string) string {
|
||||
trimmed := strings.TrimSuffix(metadataLocation, "/")
|
||||
if idx := strings.LastIndex(trimmed, "/metadata/"); idx != -1 {
|
||||
return trimmed[:idx]
|
||||
}
|
||||
return trimmed
|
||||
}
|
||||
|
||||
// accessDelegationHeader is how a client asks the catalog to hand back storage
|
||||
// credentials along with the table metadata.
|
||||
const accessDelegationHeader = "X-Iceberg-Access-Delegation"
|
||||
|
||||
// wantsVendedCredentials reports whether the client asked for vended
|
||||
// credentials. The header carries a comma-separated list of mechanisms.
|
||||
func wantsVendedCredentials(r *http.Request) bool {
|
||||
if r == nil {
|
||||
return false
|
||||
}
|
||||
for _, value := range r.Header.Values(accessDelegationHeader) {
|
||||
for _, mechanism := range strings.Split(value, ",") {
|
||||
if strings.EqualFold(strings.TrimSpace(mechanism), "vended-credentials") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// writeLoadResult writes a table or view load response. The FileIO config it
|
||||
// carries depends on the delegation the client asked for, so a cache between
|
||||
// us and the client must key on that header and not on the URL alone.
|
||||
func writeLoadResult(w http.ResponseWriter, status int, v interface{}) {
|
||||
w.Header().Set("Vary", accessDelegationHeader)
|
||||
writeJSON(w, status, v)
|
||||
}
|
||||
|
||||
// writeJSON writes a JSON response.
|
||||
func writeJSON(w http.ResponseWriter, status int, v interface{}) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
if v != nil {
|
||||
data, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
glog.Errorf("Iceberg: failed to encode response: %v", err)
|
||||
return
|
||||
}
|
||||
w.Write(data)
|
||||
}
|
||||
}
|
||||
|
||||
// writeError writes an Iceberg error response.
|
||||
func writeError(w http.ResponseWriter, status int, errType, message string) {
|
||||
resp := ErrorResponse{
|
||||
Error: ErrorModel{
|
||||
Message: message,
|
||||
Type: errType,
|
||||
Code: status,
|
||||
},
|
||||
}
|
||||
writeJSON(w, status, resp)
|
||||
}
|
||||
|
||||
// nameValidationError reports whether err is an S3 Tables namespace or table
|
||||
// name validation failure. Such names are client input (the S3 Tables charset
|
||||
// is stricter than the Iceberg REST spec), so the catalog answers 400 rather
|
||||
// than treating it as a server fault.
|
||||
func nameValidationError(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
// Match the validator's own phrasings rather than a bare "namespace name"/
|
||||
// "table name" so unrelated faults (e.g. "failed to resolve table name")
|
||||
// aren't misreported as client errors. Lowercased for resilience to
|
||||
// capitalization changes.
|
||||
msg := strings.ToLower(err.Error())
|
||||
for _, marker := range []string{
|
||||
"invalid namespace name", "namespace name must", "namespace name cannot",
|
||||
"invalid table name", "table name must", "table name cannot",
|
||||
} {
|
||||
if strings.Contains(msg, marker) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// writeManagerError maps a residual error to a response: name validation
|
||||
// failures and rejected schemas come from the request body, so they are client
|
||||
// errors (400) and must carry the reason -- a variant field without
|
||||
// format-version 3 otherwise reads as a bare 500 with "variant is not supported
|
||||
// until v3" only in the log. Anything else is a server fault (500).
|
||||
func writeManagerError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case nameValidationError(err),
|
||||
errors.Is(err, iceberg.ErrInvalidSchema),
|
||||
errors.Is(err, iceberg.ErrInvalidPartitionSpec),
|
||||
errors.Is(err, iceberg.ErrInvalidTypeString),
|
||||
errors.Is(err, iceberg.ErrInvalidTransform),
|
||||
errors.Is(err, iceberg.ErrInvalidArgument):
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
// Storage-layer failures are mostly the client's, not the server's. Reporting
|
||||
// a missing namespace or a name conflict as a 500 makes the catalog look
|
||||
// broken and gives the client nothing to act on.
|
||||
var tableErr *s3tables.S3TablesError
|
||||
if errors.As(err, &tableErr) {
|
||||
switch tableErr.Type {
|
||||
case s3tables.ErrCodeNoSuchBucket:
|
||||
// The storage-layer message names the resolved bucket, which for a
|
||||
// client that sent no warehouse at all is the default one it never
|
||||
// asked for, so say how to select a real table bucket.
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException",
|
||||
fmt.Sprintf("%s: each table bucket is a separate catalog, select one with warehouse=s3://<table-bucket>/ or /v1/<table-bucket>/", tableErr.Message))
|
||||
case s3tables.ErrCodeNoSuchNamespace:
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", tableErr.Message)
|
||||
case s3tables.ErrCodeNoSuchTable:
|
||||
writeError(w, http.StatusNotFound, "NoSuchTableException", tableErr.Message)
|
||||
case s3tables.ErrCodeNoSuchView:
|
||||
writeError(w, http.StatusNotFound, "NoSuchViewException", tableErr.Message)
|
||||
case s3tables.ErrCodeNamespaceAlreadyExists, s3tables.ErrCodeBucketAlreadyExists,
|
||||
s3tables.ErrCodeTableAlreadyExists, s3tables.ErrCodeViewAlreadyExists:
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message)
|
||||
case s3tables.ErrCodeNamespaceNotEmpty, s3tables.ErrCodeBucketNotEmpty:
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message)
|
||||
case s3tables.ErrCodeConflict:
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", tableErr.Message)
|
||||
case s3tables.ErrCodeAccessDenied:
|
||||
writeError(w, http.StatusForbidden, "ForbiddenException", tableErr.Message)
|
||||
case s3tables.ErrCodeInvalidRequest, s3tables.ErrCodeInvalidIcebergLayout:
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", tableErr.Message)
|
||||
default:
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
}
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
}
|
||||
|
||||
// resolveWarehouseBucket maps a client-supplied warehouse value to a table
|
||||
// bucket name. Clients spell the warehouse three ways: the s3://<bucket>/
|
||||
// location this catalog advertises, the table bucket ARN that AWS S3 Tables
|
||||
// uses, and the bare bucket name. Accepting only the first sends every other
|
||||
// spelling to the default bucket, where the request fails naming a bucket the
|
||||
// client never asked for. Returns "" when the value names no usable bucket so
|
||||
// the caller keeps its own default.
|
||||
func resolveWarehouseBucket(warehouse string) string {
|
||||
warehouse = strings.TrimSpace(warehouse)
|
||||
var bucket string
|
||||
switch {
|
||||
case warehouse == "":
|
||||
return ""
|
||||
case strings.HasPrefix(warehouse, "s3://"):
|
||||
parsed, _, err := parseS3Location(warehouse)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
bucket = parsed
|
||||
case strings.HasPrefix(warehouse, "arn:"):
|
||||
parsed, err := s3tables.ParseBucketNameFromARN(warehouse)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
bucket = parsed
|
||||
default:
|
||||
// Bare name, possibly with a sub-path that bucket-scoped routing ignores.
|
||||
bucket, _, _ = strings.Cut(strings.TrimSuffix(warehouse, "/"), "/")
|
||||
}
|
||||
if !s3tables.IsValidBucketName(bucket) {
|
||||
return ""
|
||||
}
|
||||
return bucket
|
||||
}
|
||||
|
||||
// getBucketFromPrefix extracts table bucket name from prefix parameter.
|
||||
// For now, we use the prefix as the table bucket name.
|
||||
//
|
||||
// The Iceberg REST spec lets clients identify a catalog either by embedding
|
||||
// its prefix in the URL (/v1/{prefix}/...) or by passing ?warehouse= as a
|
||||
// query parameter. Clients that skip the /v1/config handshake (or ignore its
|
||||
// overrides) still routinely send the warehouse parameter on every request,
|
||||
// so honor it as a fallback before the env-var default.
|
||||
func getBucketFromPrefix(r *http.Request) string {
|
||||
vars := mux.Vars(r)
|
||||
if prefix := vars["prefix"]; prefix != "" {
|
||||
return prefix
|
||||
}
|
||||
if bucket := resolveWarehouseBucket(r.URL.Query().Get("warehouse")); bucket != "" {
|
||||
return bucket
|
||||
}
|
||||
if bucket := os.Getenv("S3TABLES_DEFAULT_BUCKET"); bucket != "" {
|
||||
return bucket
|
||||
}
|
||||
// Some writers commit to the unprefixed path even though their reads
|
||||
// honor the /v1/config prefix; the deployment's first table bucket is
|
||||
// where those commits belong.
|
||||
for _, name := range strings.Split(os.Getenv("S3_TABLE_BUCKET"), ",") {
|
||||
if name = strings.TrimSpace(name); name != "" {
|
||||
return name
|
||||
}
|
||||
}
|
||||
// Default bucket if no prefix - use "warehouse" for Iceberg
|
||||
return "warehouse"
|
||||
}
|
||||
|
||||
// buildTableBucketARN builds an ARN for a table bucket.
|
||||
func buildTableBucketARN(bucketName string) string {
|
||||
arn, _ := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3_constants.AccountAdminId, bucketName)
|
||||
return arn
|
||||
}
|
||||
|
||||
const (
|
||||
defaultListPageSize = 1000
|
||||
maxListPageSize = 1000
|
||||
)
|
||||
|
||||
func getPaginationQueryParam(r *http.Request, primary, fallback string) string {
|
||||
if v := strings.TrimSpace(r.URL.Query().Get(primary)); v != "" {
|
||||
return v
|
||||
}
|
||||
return strings.TrimSpace(r.URL.Query().Get(fallback))
|
||||
}
|
||||
|
||||
func parsePagination(r *http.Request) (pageToken string, pageSize int, err error) {
|
||||
pageToken = getPaginationQueryParam(r, "pageToken", "page-token")
|
||||
pageSize = defaultListPageSize
|
||||
|
||||
pageSizeValue := getPaginationQueryParam(r, "pageSize", "page-size")
|
||||
if pageSizeValue == "" {
|
||||
return pageToken, pageSize, nil
|
||||
}
|
||||
|
||||
parsedPageSize, parseErr := strconv.Atoi(pageSizeValue)
|
||||
if parseErr != nil || parsedPageSize <= 0 {
|
||||
return pageToken, 0, fmt.Errorf("invalid pageSize %q: must be a positive integer", pageSizeValue)
|
||||
}
|
||||
if parsedPageSize > maxListPageSize {
|
||||
return pageToken, 0, fmt.Errorf("invalid pageSize %q: must be <= %d", pageSizeValue, maxListPageSize)
|
||||
}
|
||||
|
||||
return pageToken, parsedPageSize, nil
|
||||
}
|
||||
|
||||
func normalizeNamespaceProperties(properties map[string]string) map[string]string {
|
||||
if properties == nil {
|
||||
return map[string]string{}
|
||||
}
|
||||
return properties
|
||||
}
|
||||
|
||||
// namespaceLocationProperty is the well-known Iceberg key used to advertise a
|
||||
// default base path for tables created in a namespace.
|
||||
const namespaceLocationProperty = "location"
|
||||
|
||||
// defaultNamespaceLocation returns the s3 path under which tables in the
|
||||
// namespace should be stored when the namespace itself does not carry an
|
||||
// explicit "location" property. The flattened namespace path mirrors the on-
|
||||
// disk layout used by handleCreateTable so a deferred client-side commit ends
|
||||
// up at the same place a server-computed one would.
|
||||
func defaultNamespaceLocation(bucket string, namespace []string) string {
|
||||
if bucket == "" || len(namespace) == 0 {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf("s3://%s/%s", bucket, flattenNamespacePath(namespace))
|
||||
}
|
||||
|
||||
// withDefaultNamespaceLocation populates the Iceberg "location" property on a
|
||||
// namespace response when the storage layer does not carry one. Trino's REST
|
||||
// catalog falls back to an eager createTable code path when the namespace
|
||||
// has no location, which races our metadata write against Trino's emptiness
|
||||
// check and surfaces as a "Cannot create a table on a non-empty location"
|
||||
// error on the very first CREATE TABLE. Advertising a default location lets
|
||||
// Trino take the deferred-transaction path and pick a unique per-table path.
|
||||
// See issue #9074.
|
||||
func withDefaultNamespaceLocation(properties map[string]string, bucket string, namespace []string) map[string]string {
|
||||
properties = normalizeNamespaceProperties(properties)
|
||||
if _, ok := properties[namespaceLocationProperty]; ok {
|
||||
return properties
|
||||
}
|
||||
if def := defaultNamespaceLocation(bucket, namespace); def != "" {
|
||||
properties[namespaceLocationProperty] = def
|
||||
}
|
||||
return properties
|
||||
}
|
||||
Reference in New Issue
Block a user