upstream-seaweedfs/test/s3/remote_cache

Remote Object Cache Integration Tests

This directory contains integration tests for the remote object caching feature with singleflight deduplication.

Test Flow

Each test follows this pattern:

1. Write to local - Upload data to primary SeaweedFS (local storage)
2. Uncache - Push data to remote storage and remove local chunks
3. Read - Read data (triggers caching from remote back to local)

This tests the full remote caching workflow including singleflight deduplication.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Test Client                               │
│                                                                  │
│    1. PUT data to primary SeaweedFS                             │
│    2. remote.cache.uncache (push to remote, purge local)        │
│    3. GET data (triggers caching from remote)                   │
│    4. Verify singleflight deduplication                         │
└──────────────────────────────────┬──────────────────────────────┘
                                   │
                 ┌─────────────────┴─────────────────┐
                 ▼                                   ▼
┌────────────────────────────────────┐   ┌────────────────────────────────┐
│     Primary SeaweedFS              │   │     Remote SeaweedFS           │
│        (port 8333)                 │   │        (port 8334)             │
│                                    │   │                                │
│  - Being tested                    │   │  - Acts as "remote" S3         │
│  - Has remote storage mounted      │──▶│  - Receives uncached data      │
│  - Caches remote objects           │   │  - Serves data for caching     │
│  - Singleflight deduplication      │   │                                │
└────────────────────────────────────┘   └────────────────────────────────┘

What's Being Tested

Test Files and Coverage

Test File	Commands Tested	Test Count	Description
remote_cache_test.go	Basic caching	6 tests	Original caching workflow and singleflight tests, plus a mount with -cacheWait=0 reading straight from the remote
remote_cache_copy_test.go	S3 CopyObject / UploadPartCopy from a remote-only source	2 tests	Source object lives only in remote storage; CopyObject and UploadPartCopy must cache it locally before persisting the destination so the result is readable
command_remote_configure_test.go	remote.configure	6 tests	Configuration management
command_remote_mount_test.go	remote.mount, remote.unmount, remote.mount.buckets	10 tests	Mount operations
command_remote_cache_test.go	remote.cache, remote.uncache	13 tests	Cache/uncache with filters
command_remote_copy_local_test.go	remote.copy.local	12 tests	NEW in PR #8033 - Local to remote copy
command_remote_meta_sync_test.go	remote.meta.sync	8 tests	Metadata synchronization
command_edge_cases_test.go	All commands	11 tests	Edge cases and stress tests

Total: 68 test cases covering 8 weed shell commands and the S3 copy paths for remote-only sources

Commands Tested

1. remote.configure - Configure remote storage backends
2. remote.mount - Mount remote storage to local directory
3. remote.unmount - Unmount remote storage
4. remote.mount.buckets - Mount all buckets from remote
5. remote.cache - Cache remote files locally
6. remote.uncache - Remove local cache, keep metadata
7. remote.copy.local - Copy local files to remote (NEW in PR #8033)
8. remote.meta.sync - Sync metadata from remote

Test Coverage

Basic Operations:

• Basic caching workflow (Write → Uncache → Read)
• Singleflight deduplication (concurrent reads trigger ONE cache operation)
• Large object caching (5MB-100MB files)
• Range requests (partial reads)
• Not found handling

File Filtering:

• Include patterns (*.pdf, *.txt, etc.)
• Exclude patterns
• Size filters (-minSize, -maxSize)
• Age filters (-minAge, -maxAge)
• Combined filters

Command Options:

• Dry run mode (-dryRun=true)
• Concurrency settings (-concurrent=N)
• Force update (-forceUpdate=true)
• Non-empty directory mounting (-nonempty=true)

Edge Cases:

• Empty directories
• Nested directory hierarchies
• Special characters in filenames
• Very large files (100MB+)
• Many small files (100+)
• Rapid cache/uncache cycles
• Concurrent command execution
• Invalid paths
• Zero-byte files

Running Tests

Run All Tests

# Full automated workflow
make test-with-server

# Or manually
go test -v ./...

Run Specific Test Files

# Test remote.configure command
go test -v -run TestRemoteConfigure

# Test remote.mount/unmount commands
go test -v -run TestRemoteMount
go test -v -run TestRemoteUnmount

# Test remote.cache/uncache commands  
go test -v -run TestRemoteCache
go test -v -run TestRemoteUncache

# Test remote.copy.local command (PR #8033)
go test -v -run TestRemoteCopyLocal

# Test remote.meta.sync command
go test -v -run TestRemoteMetaSync

# Test edge cases
go test -v -run TestEdgeCase

Quick Start

Run Full Test Suite (Recommended)

# Build SeaweedFS, start both servers, run tests, stop servers
make test-with-server

Manual Steps

# 1. Build SeaweedFS binary
make build-weed

# 2. Start remote SeaweedFS (acts as "remote" storage)
make start-remote

# 3. Start primary SeaweedFS (the one being tested)
make start-primary

# 4. Configure remote storage mount
make setup-remote

# 5. Run tests
make test

# 6. Clean up
make clean

Configuration

Primary SeaweedFS (Being Tested)

Service	Port
S3 API	8333
Filer	8888
Master	9333
Volume	8080

Remote SeaweedFS (Remote Storage)

Service	Port
S3 API	8334
Filer	8889
Master	9334
Volume	8081

Makefile Targets

make help           # Show all available targets
make build-weed     # Build SeaweedFS binary
make start-remote   # Start remote SeaweedFS
make start-primary  # Start primary SeaweedFS
make setup-remote   # Configure remote storage mount
make test           # Run tests
make test-with-server  # Full automated test workflow
make logs           # Show server logs
make health         # Check server status
make clean          # Stop servers and clean up

Test Details

TestRemoteCacheBasic

Basic workflow test:

1. Write object to primary (local)
2. Uncache (push to remote, remove local chunks)
3. Read (triggers caching from remote)
4. Read again (from local cache - should be faster)

TestRemoteCacheConcurrent

Singleflight deduplication test:

1. Write 1MB object
2. Uncache to remote
3. Launch 10 concurrent reads
4. All should succeed with correct data
5. Only ONE caching operation should run (singleflight)

TestRemoteCacheLargeObject

Large file test (5MB) to verify chunked transfer works correctly.

TestRemoteCacheRangeRequest

Tests HTTP range requests work correctly after caching.

TestRemoteCacheNotFound

Tests proper error handling for non-existent objects.

TestRemoteCacheWaitZero

Remounts with remote.mount -cacheWait=0 and checks that a read of a remote-only object is served from the remote and leaves the object uncached, while the default size-based wait caches it.

Troubleshooting

View logs

make logs           # Show recent logs from both servers
make logs-primary   # Follow primary logs in real-time
make logs-remote    # Follow remote logs in real-time

Check server health

make health

Clean up and retry

make clean
make test-with-server