snapshot of https://github.com/seaweedfs/seaweedfs.git (master, 2026-09-16; blobs > 131072B omitted)

2026-09-16 03:23:42 +05:00
commit 89f5be005d
3997 changed files with 961970 additions and 0 deletions
--- a/test/s3/parquet/README.md
+++ b/test/s3/parquet/README.md
@@ -0,0 +1,312 @@
+# PyArrow Parquet S3 Compatibility Tests
+
+This directory contains tests for PyArrow Parquet compatibility with SeaweedFS S3 API, including the implicit directory detection fix.
+
+## Overview
+
+**Status**: ✅ **All PyArrow methods work correctly with SeaweedFS**
+
+SeaweedFS implements implicit directory detection to improve compatibility with s3fs and PyArrow. When PyArrow writes datasets using `write_dataset()`, it may create directory markers that can confuse s3fs. SeaweedFS now handles these correctly by returning 404 for HEAD requests on implicit directories (directories with children), forcing s3fs to use LIST-based discovery.
+
+## Quick Start
+
+### Running the Example Script
+
+```bash
+# Start SeaweedFS server
+make start-seaweedfs-ci
+
+# Run the example script
+python3 example_pyarrow_native.py
+
+# Or with uv (if available)
+uv run example_pyarrow_native.py
+
+# Stop the server when done
+make stop-seaweedfs-safe
+```
+
+### Running Tests
+
+```bash
+# Setup Python environment
+make setup-python
+
+# Run all tests with server (small and large files)
+make test-with-server
+
+# Run quick tests with small files only (faster for development)
+make test-quick
+
+# Run implicit directory fix tests
+make test-implicit-dir-with-server
+
+# Run PyArrow native S3 filesystem tests
+make test-native-s3-with-server
+
+# Run cross-filesystem compatibility tests (s3fs ↔ PyArrow native)
+make test-cross-fs-with-server
+
+# Run SSE-S3 encryption tests
+make test-sse-s3-compat
+
+# Clean up
+make clean
+```
+
+### Using PyArrow with SeaweedFS
+
+#### Option 1: Using s3fs (recommended for compatibility)
+
+```python
+import pyarrow as pa
+import pyarrow.parquet as pq
+import pyarrow.dataset as pads
+import s3fs
+
+# Configure s3fs
+fs = s3fs.S3FileSystem(
+    key='your_access_key',
+    secret='your_secret_key',
+    endpoint_url='http://localhost:8333',
+    use_ssl=False
+)
+
+# Write dataset (creates directory structure)
+table = pa.table({'id': [1, 2, 3], 'value': ['a', 'b', 'c']})
+pads.write_dataset(table, 'bucket/dataset', filesystem=fs)
+
+# Read dataset (all methods work!)
+dataset = pads.dataset('bucket/dataset', filesystem=fs)  # ✅
+table = pq.read_table('bucket/dataset', filesystem=fs)   # ✅
+dataset = pq.ParquetDataset('bucket/dataset', filesystem=fs)  # ✅
+```
+
+#### Option 2: Using PyArrow's native S3 filesystem (pure PyArrow)
+
+```python
+import pyarrow as pa
+import pyarrow.parquet as pq
+import pyarrow.dataset as pads
+import pyarrow.fs as pafs
+
+# Configure PyArrow's native S3 filesystem
+s3 = pafs.S3FileSystem(
+    access_key='your_access_key',
+    secret_key='your_secret_key',
+    endpoint_override='localhost:8333',
+    scheme='http',
+    allow_bucket_creation=True,
+    allow_bucket_deletion=True
+)
+
+# Write dataset
+table = pa.table({'id': [1, 2, 3], 'value': ['a', 'b', 'c']})
+pads.write_dataset(table, 'bucket/dataset', filesystem=s3)
+
+# Read dataset (all methods work!)
+table = pq.read_table('bucket/dataset', filesystem=s3)  # ✅
+dataset = pq.ParquetDataset('bucket/dataset', filesystem=s3)  # ✅
+dataset = pads.dataset('bucket/dataset', filesystem=s3)  # ✅
+```
+
+## Test Files
+
+### Main Test Suite
+- **`s3_parquet_test.py`** - Comprehensive PyArrow test suite
+  - Tests 2 write methods × 5 read methods × 2 dataset sizes = 20 combinations
+  - Uses s3fs library for S3 operations
+  - All tests pass with the implicit directory fix ✅
+
+### PyArrow Native S3 Tests
+- **`test_pyarrow_native_s3.py`** - PyArrow's native S3 filesystem tests
+  - Tests PyArrow's built-in S3FileSystem (pyarrow.fs.S3FileSystem)
+  - Pure PyArrow solution without s3fs dependency
+  - Tests 3 read methods × 2 dataset sizes = 6 scenarios
+  - All tests pass ✅
+
+- **`test_sse_s3_compatibility.py`** - SSE-S3 encryption compatibility tests
+  - Tests PyArrow native S3 with SSE-S3 server-side encryption
+  - Tests 5 different file sizes (10 to 500,000 rows)
+  - Verifies multipart upload encryption works correctly
+  - All tests pass ✅
+
+### Cross-Filesystem Compatibility Tests
+- **`test_cross_filesystem_compatibility.py`** - Verifies cross-compatibility between s3fs and PyArrow native S3
+  - Tests write with s3fs → read with PyArrow native S3
+  - Tests write with PyArrow native S3 → read with s3fs
+  - Tests 2 directions × 3 read methods × 2 dataset sizes = 12 scenarios
+  - Validates that files written by either filesystem can be read by the other
+  - **All tests pass** ✅
+  - See **`CROSS_FILESYSTEM_COMPATIBILITY.md`** for detailed test results and analysis
+
+### Implicit Directory Tests
+- **`test_implicit_directory_fix.py`** - Specific tests for the implicit directory fix
+  - Tests HEAD request behavior
+  - Tests s3fs directory detection
+  - Tests PyArrow dataset reading
+  - All 6 tests pass ✅
+
+### Examples
+- **`example_pyarrow_native.py`** - Simple standalone example
+  - Demonstrates PyArrow's native S3 filesystem usage
+  - Can be run with `uv run` or regular Python
+  - Minimal dependencies (pyarrow, boto3)
+
+### Configuration
+- **`Makefile`** - Build and test automation
+- **`requirements.txt`** - Python dependencies (pyarrow, s3fs, boto3)
+- **`.gitignore`** - Ignore patterns for test artifacts
+
+## Documentation
+
+### Technical Documentation
+- **`TEST_COVERAGE.md`** - Comprehensive test coverage documentation
+  - Unit tests (Go): 17 test cases
+  - Integration tests (Python): 6 test cases
+  - End-to-end tests (Python): 20 test cases
+
+- **`FINAL_ROOT_CAUSE_ANALYSIS.md`** - Deep technical analysis
+  - Root cause of the s3fs compatibility issue
+  - How the implicit directory fix works
+  - Performance considerations
+
+- **`CROSS_FILESYSTEM_COMPATIBILITY.md`** - Cross-filesystem compatibility test results ✅ **NEW**
+  - Validates s3fs ↔ PyArrow native S3 interoperability
+  - Confirms files written by either can be read by the other
+  - Test methodology and detailed results
+
+- **`MINIO_DIRECTORY_HANDLING.md`** - Comparison with MinIO
+  - How MinIO handles directory markers
+  - Differences in implementation approaches
+
+## The Implicit Directory Fix
+
+### Problem
+When PyArrow writes datasets with `write_dataset()`, it may create 0-byte directory markers. s3fs's `info()` method calls HEAD on these paths, and if HEAD returns 200 with size=0, s3fs incorrectly reports them as files instead of directories. This causes PyArrow to fail with "Parquet file size is 0 bytes".
+
+### Solution
+SeaweedFS now returns 404 for HEAD requests on implicit directories (0-byte objects or directories with children, when requested without a trailing slash). This forces s3fs to fall back to LIST-based discovery, which correctly identifies directories by checking for children.
+
+### Implementation
+The fix is implemented in `weed/s3api/s3api_object_handlers.go`:
+- `HeadObjectHandler` - Returns 404 for implicit directories
+- `hasChildren` - Helper function to check if a path has children
+
+See the source code for detailed inline documentation.
+
+### Test Coverage
+- **Unit tests** (Go): `weed/s3api/s3api_implicit_directory_test.go`
+  - Run: `cd weed/s3api && go test -v -run TestImplicitDirectory`
+  
+- **Integration tests** (Python): `test_implicit_directory_fix.py`
+  - Run: `cd test/s3/parquet && make test-implicit-dir-with-server`
+
+- **End-to-end tests** (Python): `s3_parquet_test.py`
+  - Run: `cd test/s3/parquet && make test-with-server`
+
+## Makefile Targets
+
+```bash
+# Setup
+make setup-python          # Create Python virtual environment and install dependencies
+make build-weed           # Build SeaweedFS binary
+
+# Testing
+make test                 # Run full tests (assumes server is already running)
+make test-with-server     # Run full PyArrow test suite with server (small + large files)
+make test-quick           # Run quick tests with small files only (assumes server is running)
+make test-implicit-dir-with-server  # Run implicit directory tests with server
+make test-native-s3       # Run PyArrow native S3 tests (assumes server is running)
+make test-native-s3-with-server  # Run PyArrow native S3 tests with server management
+make test-cross-fs        # Run cross-filesystem compatibility tests (assumes server is running)
+make test-cross-fs-with-server  # Run cross-filesystem compatibility tests with server management
+make test-sse-s3-compat   # Run comprehensive SSE-S3 encryption compatibility tests
+
+# Server Management
+make start-seaweedfs-ci   # Start SeaweedFS in background (CI mode)
+make stop-seaweedfs-safe  # Stop SeaweedFS gracefully
+make clean                # Clean up all test artifacts
+
+# Development
+make help                 # Show all available targets
+```
+
+## Continuous Integration
+
+The tests are automatically run in GitHub Actions on every push/PR that affects S3 or filer code:
+
+**Workflow**: `.github/workflows/s3-parquet-tests.yml`
+
+**Test Matrix**:
+- Python versions: 3.9, 3.11, 3.12
+- PyArrow integration tests (s3fs): 20 test combinations
+- PyArrow native S3 tests: 6 test scenarios ✅
+- Cross-filesystem compatibility tests: 12 test scenarios ✅ **NEW**
+- SSE-S3 encryption tests: 5 file sizes ✅
+- Implicit directory fix tests: 6 test scenarios
+- Go unit tests: 17 test cases
+
+**Test Steps** (run for each Python version):
+1. Build SeaweedFS
+2. Run PyArrow Parquet integration tests (`make test-with-server`)
+3. Run implicit directory fix tests (`make test-implicit-dir-with-server`)
+4. Run PyArrow native S3 filesystem tests (`make test-native-s3-with-server`)
+5. Run cross-filesystem compatibility tests (`make test-cross-fs-with-server`) ✅ **NEW**
+6. Run SSE-S3 encryption compatibility tests (`make test-sse-s3-compat`)
+7. Run Go unit tests for implicit directory handling
+
+**Triggers**:
+- Push/PR to master (when `weed/s3api/**` or `weed/filer/**` changes)
+- Manual trigger via GitHub UI (workflow_dispatch)
+
+## Requirements
+
+- Python 3.8+
+- PyArrow 22.0.0+
+- s3fs 2024.12.0+
+- boto3 1.40.0+
+- SeaweedFS (latest)
+
+## AWS S3 Compatibility
+
+The implicit directory fix makes SeaweedFS behavior more compatible with AWS S3:
+- AWS S3 typically doesn't create directory markers for implicit directories
+- HEAD on "dataset" (when only "dataset/file.txt" exists) returns 404 on AWS
+- SeaweedFS now matches this behavior for implicit directories with children
+
+## Edge Cases Handled
+
+✅ **Implicit directories with children** → 404 (forces LIST-based discovery)  
+✅ **Empty files (0-byte, no children)** → 200 (legitimate empty file)  
+✅ **Empty directories (no children)** → 200 (legitimate empty directory)  
+✅ **Explicit directory requests (trailing slash)** → 200 (normal directory behavior)  
+✅ **Versioned buckets** → Skip implicit directory check (versioned semantics)  
+✅ **Regular files** → 200 (normal file behavior)
+
+## Performance
+
+The implicit directory check adds minimal overhead:
+- Only triggered for 0-byte objects or directories without trailing slash
+- Cost: One LIST operation with Limit=1 (~1-5ms)
+- No impact on regular file operations
+
+## Contributing
+
+When adding new tests:
+1. Add test cases to the appropriate test file
+2. Update TEST_COVERAGE.md
+3. Run the full test suite to ensure no regressions
+4. Update this README if adding new functionality
+
+## References
+
+- [PyArrow Documentation](https://arrow.apache.org/docs/python/parquet.html)
+- [s3fs Documentation](https://s3fs.readthedocs.io/)
+- [SeaweedFS S3 API](https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API)
+- [AWS S3 API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/)
+
+---
+
+**Last Updated**: November 19, 2025  
+**Status**: All tests passing ✅