posixlake

Cross-platform POSIX and ACID compliant directory-based database built with Rust

A columnar database engine where UNIX tools (cat, grep, awk, wc, head, tail, sort, cut, echo >>, sed -i, vim, mkdir, mv, cp, rmdir, rm) trigger Delta Lake operations including MERGE (UPSERT), ACID transactions, and native format storage. Works with local filesystem directories, S3/MinIO, Azure Blob Storage, and Microsoft Fabric OneLake. Built on Apache Arrow, Parquet, and DataFusion for high-performance analytics workloads.

---

What is posixlake?

posixlake is a cross-platform file-system database where POSIX commands trigger Delta Lake operations. Runs natively on Linux, macOS, and Windows. Works with local filesystem directories (/path/to/database or C:\path\to\database), S3/MinIO (s3://bucket/path), Azure Blob Storage (az://<container>), and Microsoft Fabric OneLake (abfss://workspace@onelake.dfs.fabric.microsoft.com/...) - same unified API for all. Mount it as a filesystem via NFS and use cat, grep, awk, sed directly on your data. CSV overwrites (sed -i, vim edits) trigger MERGE (UPSERT) with atomic INSERT/UPDATE/DELETE. Under the hood, everything is stored in native Delta Lake format for full compatibility with Spark, Databricks, Athena, and Microsoft Fabric. Start fresh, or with existing Delta Tables, raw Parquet files, or CSVs with schema inference.

Mount as Filesystem - Query with UNIX Tools

When you mount a Delta Lake database via posixlake's NFS server, it exposes this filesystem structure:

/mnt/data/              (mount point)
├── data/               (directory containing data views)
│   ├── data.csv        (CSV view of all Parquet data - generated on-demand)
│   ├── data.json       (JSON view of all data)
│   ├── data.jsonl      (JSON Lines view)
│   └── *.parquet       (direct access to underlying Parquet files)
├── schema.sql          (table schema as SQL CREATE TABLE)
└── .query              (write SQL queries here to execute them)

Key Concept: CSV Facade

/data/data.csv is a live view of your Delta Lake table:

• Dynamically generated from Parquet data on read (works with local filesystem and S3-backed tables)
• All columns visible as CSV format
• Writes to this file trigger Delta Lake ACID transactions
• Changes are immediately persisted to Parquet files (local or S3)

How It Works Technically:

Read Operations (cat, grep, awk, wc):

1. User runs cat /mnt/data/data/data.csv or grep "pattern" /mnt/data/data/data.csv
2. OS NFS client sends READ request to posixlake NFS server (with offset + size)
3. Server checks two-tier cache (memory → disk)
4. Cache miss: Query all Parquet files → Convert RecordBatch to CSV → Cache full content
5. Cache hit: Return requested byte range (microsecond from memory, millisecond from disk)
6. For large files (≥1MB): Cached content uses memory-mapped I/O for zero-copy access
7. grep behavior: First read generates full CSV and caches it. Subsequent reads (as grep scans) hit cache and return only requested ranges
8. Result: After initial caching, grep on large files is fast (no regeneration, only cache reads)

Write Operations (echo >>, sed -i):

Append (echo >>):

1. User runs echo "1,Alice,30" >> /mnt/data/data/data.csv
2. OS NFS client sends WRITE request with CSV bytes
3. Server parses CSV using Arrow CSV reader → RecordBatch
4. DatabaseOps::insert(batch) triggers Delta Lake ACID transaction
5. Transaction commits to _delta_log/*.json with new Parquet file
6. Cache invalidated (next read regenerates CSV with new data)
7. Result: POSIX append becomes Delta Lake transaction with full ACID guarantees

Overwrite (sed -i, vim, cat > file):

1. User overwrites CSV: sed -i 's/Alice/ALICE/' data.csv or cat modified.csv > data.csv
2. Server compares old CSV (from cache) vs new CSV (from write)
3. Detects changes using ID column (or first Int32/Int64 column):
   • New IDs → INSERT operations
   • Modified rows (same ID, different data) → UPDATE operations
   • Missing IDs (in old, not in new) → DELETE operations
4. Executes MERGE (UPSERT) via DatabaseOps::merge() in single atomic transaction
5. MERGE applies all INSERT/UPDATE/DELETE operations atomically to Delta Lake
6. Transaction commits to _delta_log/*.json (updates as INSERT+DELETE, deletes use deletion vectors)
7. Cache invalidated (next read shows all changes)
8. Result: CSV overwrite becomes atomic MERGE with INSERT/UPDATE/DELETE in one transaction

Delete Operations (via POSIX):

• File deletion (rm /mnt/data/data/data.csv): Supported - Truncates table (deletes all rows)
  • Uses DatabaseOps::delete_rows_where("1=1") with Delta Lake deletion vectors
  • Efficient: marks all rows as deleted without rewriting Parquet files
  • Run OPTIMIZE later to permanently remove deleted rows and reclaim space
• Row-level operations (INSERT/UPDATE/DELETE via CSV overwrite): Powered by MERGE
  • Overwrite CSV with modified content: grep -v "Alice" data.csv > temp && cat temp > data.csv
  • Or: sed -i '/Alice/d' data.csv (deletes), sed -i 's/Alice/ALICE/' data.csv (updates)
  • Or: Edit with vim/nano and save
  • Server detects all changes by comparing old vs new CSV using ID column
  • Executes MERGE (UPSERT) in single atomic Delta Lake transaction:
    • New rows → INSERT
    • Modified rows → UPDATE (implemented as DELETE + INSERT)
    • Missing rows → DELETE (using deletion vectors)
  • All operations applied atomically - either all succeed or all fail
  • Efficient: deletion vectors avoid Parquet rewrites, OPTIMIZE later reclaims space
  • Stress tested: 30M rows (1GB CSV) - see STRESS_TEST_RESULTS.md

Example workflow:

# Read current data
cat data.csv > old.csv

# Make changes (delete Alice, update Bob's age, add Charlie)
grep -v "Alice" old.csv > temp.csv
sed -i 's/Bob,30/Bob,31/' temp.csv
echo "3,Charlie,28" >> temp.csv

# Overwrite triggers MERGE: 1 DELETE + 1 UPDATE + 1 INSERT in single transaction
cat temp.csv > data.csv

Additional Delete Methods (SQL API):

• Programmatic deletion: db.delete_rows_where("id = 5") for direct API access
• Deletion vectors: Uses Delta Lake's deletion vectors (efficient, no Parquet rewrite)
• OPTIMIZE later compacts files and removes deleted rows permanently

Performance for Large Tables:

• Two-tier cache: Memory (moka, microsecond) + Disk (sled, millisecond) for generated CSV
  • Incremental cache: Efficient append-only writes without full CSV regeneration
  • Byte-range diffing: Detects only changed rows for targeted MERGE operations
• Chunked reads: Request-based byte-range reads avoid generating full CSV on each access
• Memory-mapped I/O: Cached files ≥1MB use mmap for zero-copy reads
• grep optimization: First read caches full CSV, subsequent reads (as grep scans) hit cache with byte-range requests
• Trade-off: Initial read generates full CSV (expensive), but all subsequent operations are cache-only (fast)
• LRU eviction: Max 100 mmapped files, automatic eviction of least recently used
• Cache promotion: Disk cache hits promoted to memory cache
• Cache invalidation: Writes invalidate cache, next read regenerates CSV with updated data

Use standard POSIX commands on your database:

# Mount database as filesystem
posixlake-cli mount /path/to/database /mnt/data

# data.csv is a CSV view of the Parquet data
cat /mnt/data/data/data.csv
# id,name,age,city
# 1,Alice,30,NYC
# 2,Bob,25,SF
# 3,Charlie,35,LA

# Grep for specific records
grep "Alice" /mnt/data/data/data.csv
# 1,Alice,30,NYC

# Stream processing with awk
awk -F',' '{print $2}' /mnt/data/data/data.csv | sort
# Alice
# Bob
# Charlie

# Append new data (writes to Delta Lake!)
echo "4,Dave,28,Boston" >> /mnt/data/data/data.csv

# Edit data with any tool - triggers MERGE (UPSERT) under the hood
sed -i 's/Alice,30/Alice,31/' /mnt/data/data/data.csv  # UPDATE
grep -v "Bob" /mnt/data/data/data.csv > temp && cat temp > /mnt/data/data/data.csv  # DELETE
# All changes applied as atomic Delta Lake MERGE transaction (INSERT/UPDATE/DELETE)

# Verify changes persisted
wc -l /mnt/data/data/data.csv
# 4 /mnt/data/data/data.csv (Bob deleted, Alice updated, Dave inserted)

How it works:

• Pure Rust NFS server (no FUSE, no kernel modules)
• OS built-in NFS client (zero installation)
• All writes go through Delta Lake ACID transactions
• Read Parquet files as CSV automatically

Works with S3 too! Mount cloud Delta tables as POSIX filesystem:

# Mount S3-backed Delta table
posixlake-cli mount s3://my-bucket/delta-table /mnt/s3-data

# Now use UNIX tools on cloud data!
grep "pattern" /mnt/s3-data/data/data.csv
cat /mnt/s3-data/data/data.csv | awk -F',' '{sum+=$3} END {print sum}'

# Writes go to S3 with Delta Lake ACID transactions
echo "new,record,123" >> /mnt/s3-data/data/data.csv

Works with Azure Blob Storage / ADLS Gen2 too!

# Mount Azure-backed Delta table
posixlake-cli mount az://my-delta-table /mnt/azure-data \
  --azure-account myaccount --azure-key mykey

# Same UNIX tools, same API, cloud storage
grep "pattern" /mnt/azure-data/data/data.csv

# Writes go to Azure with Delta Lake ACID transactions
echo "new,record,123" >> /mnt/azure-data/data/data.csv

Delta Lake Native Format

Every posixlake database IS a Delta Lake table:

# Create database via posixlake
posixlake-cli create /path/to/database --schema "id:Int32,name:String"

# Data is immediately readable by Spark/Databricks/Athena
# -> Native _delta_log/ transaction log
# -> Standard Parquet data files
# -> No export or translation needed

posixlake vs Traditional Delta Lake:

Feature	posixlake	Traditional Delta Lake
Format	Native Delta Lake	Delta Lake
ACID Transactions	Delta Lake protocol	Delta Lake protocol
S3/Cloud Storage	✓ Native support	Native support
Azure Blob / ADLS Gen2	✓ Native support	Via Spark/Hadoop
Microsoft Fabric OneLake	✓ Native support (Service Principal)	Via Fabric Spark
Storage Abstraction	✓ Unified Local/S3/Azure/OneLake	ObjectStore only
POSIX Interface	✓ Pure Rust NFS server (works with all backends!)	✗ Not available
SQL Queries	✓ DataFusion (embedded)	Spark/Presto (separate)
Mount as Filesystem	✓ mount /mnt/posixlake (local, S3, or Azure)	✗ Not available
UNIX Tools	✓ cat, grep, awk on any backend	✗ Requires Spark/export
Ecosystem	Full Delta Lake compatibility	Full ecosystem
Deployment	✓ Single binary	Requires Spark cluster

Why posixlake:

• POSIX Commands Work: Use cat, grep, awk, sed, echo >> directly on your database
• No Special Tools: Mount via NFS, use standard UNIX commands - that's it
• Delta Lake Native: Every database IS a Delta Lake table - readable by Spark/Databricks/Athena immediately
• ACID Transactions: All POSIX writes go through proper ACID transactions
• Pure Rust NFS Server: Zero dependencies - no FUSE, no kernel modules, works everywhere
• Unified Storage Abstraction: NFS server works with local, S3, Azure, and Fabric OneLake - mount cloud Delta tables as POSIX filesystem!
• S3 Backend: Store Delta Lake tables on S3/MinIO with transparent local caching - use grep on S3 data!
• Azure Backend: Store Delta Lake tables on Azure Blob Storage / ADLS Gen2 with transparent local caching
• Fabric OneLake: Write Delta tables directly to Microsoft Fabric lakehouses via Service Principal auth
• SQL Support: Full SQL via DataFusion when you need it
• Fast: Columnar Parquet with Snappy compression, query pruning, memory-mapped I/O

Core Features:

• Query with UNIX Tools: cat, grep, awk, wc, head, tail, sort, cut all work on mounted database
• Write with Standard Commands: echo >>, sed -i, vim, cat > - all writes persist to Delta Lake via ACID transactions
• File Operations: mkdir, mv, cp, rmdir, rm, stat - full file/directory management
• MERGE (UPSERT) Operations: INSERT, UPDATE, DELETE in single atomic Delta Lake transaction
• Pure Rust NFS Server: Zero external dependencies - OS has NFS client built-in
• Delta Lake Format: Native _delta_log/ transaction logs - Spark/Databricks/Athena compatible
• ACID Everywhere: Even echo >> goes through proper transactions with conflict detection
• Storage Abstraction Layer: Unified backend for Local/S3/Azure/Fabric OneLake - same API, any storage
• S3 + NFS Integration: Mount S3-backed Delta tables as POSIX filesystem - grep works on cloud data!
• Azure + NFS Integration: Mount Azure Blob/ADLS Gen2 Delta tables as POSIX filesystem
• Fabric OneLake: Create and query Delta tables in Microsoft Fabric lakehouses with Service Principal auth
• Schema Evolution: Add columns dynamically - old data gets NULL automatically
• Note: Metadata operations (touch, chmod, chown) are silently ignored as timestamps/permissions reflect Delta Lake commits and database-layer RBAC. Symlinks (ln -s) return NFS3ERR_NOTSUPP - use SQL views instead.

Architecture

System Overview

┌────────────────────────────────────────────────────────┐
│  POSIX Commands: cat, grep, awk, sed, wc, echo >>      │
│  User: mount /mnt/data → cat /mnt/data/data/data.csv   │
└─────────────────────┬──────────────────────────────────┘
                      │
┌─────────────────────▼──────────────────────────────────┐
│  Pure Rust NFS Server (NFSv3, port 12049)              │
│  ┌──────────────────────────────────────────────┐      │
│  │ Filesystem Structure:                        │      │
│  │   /data/data.csv    (CSV facade)             │      │
│  │   /data/data.json   (JSON view)              │      │
│  │   /data/*.parquet   (raw Parquet files)      │      │
│  │   /schema.sql       (schema definition)      │      │
│  │   /.query           (SQL interface)          │      │
│  └──────────────────────────────────────────────┘      │
│  • Reads: Query Parquet → generate/cache CSV (chunked) │
│  • Writes: Parse CSV → MERGE or INSERT transactions    │
│  • Cache: Two-tier (memory → disk), incremental        │
└─────────────────────┬──────────────────────────────────┘
                      │
┌─────────────────────▼──────────────────────────────────┐
│  posixlake Core (DatabaseOps)                               │
│  • insert() / query() / delete_rows_where()            │
│  • DataFusion SQL engine                               │
│  • Storage abstraction (Local/S3/Azure/OneLake)        │
└─────────────────────┬──────────────────────────────────┘
                      │
┌─────────────────────▼──────────────────────────────────┐
│  Delta Lake Protocol (deltalake-rs)                    │
│  • ACID transactions via _delta_log/*.json             │
│  • Snapshot isolation & time travel                    │
│  • Deletion vectors for row-level deletes              │
│  • Checkpointing & protocol versioning                 │
└─────────────────────────┬──────────────────────────────┘
                          │
    ┌─────────────────┬───┴───┬──────────────────┐
    │                 │       │                  │
    ▼                 ▼       ▼                  ▼
┌─────────┐   ┌──────────┐ ┌──────────┐  ┌──────────────┐
│  Local  │   │ S3/MinIO │ │  Azure   │  │   Fabric     │
│Filesys. │   │  Object  │ │   Blob   │  │   OneLake    │
│_delta/  │   │  Store   │ │ Storage  │  │  (abfss://)  │
│*.parquet│   │_delta/   │ │_delta/   │  │  _delta/     │
└─────────┘   └──────────┘ └──────────┘  └──────────────┘

Key Components:

• NFS Server: Pure Rust NFSv3 server for POSIX interface (works with any storage backend)
• DatabaseOps: High-level CRUD and SQL operations
• Delta Lake: Native Delta Lake format for ACID transactions
• Storage Abstraction: Unified ObjectStore backend - Local, S3/MinIO, Azure Blob, or Fabric OneLake
• Key Innovation: NFS + cloud storage = mount cloud Delta tables as POSIX filesystem with cat, grep, awk!

Data Directory Structure (Delta Lake Native Format)

/path/to/database/
├── _delta_log/                              # Delta Lake Transaction Log
│   ├── 00000000000000000000.json           # Version 0: CREATE TABLE (metadata + protocol)
│   ├── 00000000000000000001.json           # Version 1: INSERT (add actions)
│   ├── 00000000000000000002.json           # Version 2: INSERT (add actions)
│   └── 00000000000000000010.checkpoint.parquet  # Checkpoint (every 10 commits)
│
├── part-00000-{uuid}-c000.snappy.parquet   # Delta Lake Parquet data files
├── part-00001-{uuid}-c000.snappy.parquet   # (Standard Delta Lake naming)
└── part-00002-{uuid}-c000.snappy.parquet

# 100% Delta Lake compatible!
# - Apache Spark can read this directory directly
# - Databricks can query without any conversion
# - AWS Athena can access as Delta table
# - All Delta Lake ecosystem tools work immediately

Quick Start

Installation

Rust Library

# Build from source
git clone https://github.com/npiesco/posixlake.git
cd posixlake
cargo build --release

# Binary is at target/release/posixlake

Prerequisites: Rust 1.70+, NFS client (built-in on macOS/Linux/Windows Pro)

Python Bindings

# Install from PyPI (recommended - requires Python 3.11+)
pip install posixlake

# Or build from source (supports Python 3.8+)
cd posixlake
cargo build --release
cargo run --bin uniffi-bindgen -- generate \
    --library target/release/<platform-library> \
    --language python \
    --out-dir ../bindings/python/posixlake
cp target/release/<platform-library> ../bindings/python/posixlake/
pip install -e ../bindings/python/

Use the correct library name for your OS:

• Linux: libposixlake.so
• macOS: libposixlake.dylib
• Windows: posixlake.dll

Both x86_64 and arm64 are supported. As with any native extension, Python and the native library must use the same architecture (x86_64 ↔ x86_64, arm64 ↔ arm64). If you need to target a specific architecture, build Rust explicitly for that target:

• Windows x86_64 Python: cargo build --release --target x86_64-pc-windows-msvc -p posixlake
• Windows ARM64 Python: cargo build --release --target aarch64-pc-windows-msvc -p posixlake

Requirements:

• Python 3.11+ for prebuilt wheels with native library
• Python 3.8+ for building from source

PyPI Package: https://pypi.org/project/posixlake/

Python Quick Start:

from posixlake import DatabaseOps, Schema, Field, NfsServer

# Create database
schema = Schema(fields=[
    Field(name="id", data_type="Int32", nullable=False),
    Field(name="name", data_type="String", nullable=False),
])
db = DatabaseOps.create("/path/to/db", schema)

# Insert data (regular)
db.insert_json('[{"id": 1, "name": "Alice"}]')

# Buffered insert for high performance (10x faster for small batches)
for i in range(100):
    db.insert_buffered_json(f'[{{"id": {i}, "name": "User_{i}"}}]')
db.flush_write_buffer()  # Commit all buffered data

# Query with SQL
results = db.query_json("SELECT * FROM data WHERE id > 0")
print(results)

# Mount as filesystem and use POSIX tools
nfs = NfsServer(db, 12049)
# sudo mount_nfs -o nolocks,vers=3,tcp,port=12049,mountport=12049 localhost:/share /mnt/posixlake
# cat /mnt/posixlake/data/data.csv | grep "Alice" | awk -F',' '{print $2}'

Python test scripts in this repo:

# Comprehensive pip/package integration coverage
python scripts/test_pip_package.py

# Python S3/MinIO integration coverage via UniFFI
python scripts/test_python_s3.py

# Full manual filesystem/NFS test, which now also invokes the Python S3 test
python scripts/full_test.py

If you are using the repository's Python environment, run them with the bindings venv interpreter instead:

bindings/python/.venv/Scripts/python.exe scripts/test_python_s3.py
bindings/python/.venv/Scripts/python.exe scripts/full_test.py

See Python Bindings Documentation for complete API reference.

Creating a Database

CLI

# 1. Create the directory
mkdir storage

# 2. Create a posixlake database with a schema
posixlake-cli create storage --schema "id:Int32,name:String,email:String"

# 3. What gets created (even with no data):
# storage/
# ├── _delta_log/
# │   └── 00000000000000000000.json   # Delta Lake transaction log
# └── (no parquet files yet - empty table)

# 4. Mount database (starts NFS server and mounts via OS NFS client)
posixlake-cli mount storage /mnt/storage --port 12099

# 5. Now use POSIX commands
ls /mnt/storage/data/           # Shows data.csv (empty, just header)
cat /mnt/storage/data/data.csv  # Shows: id,name,email

# 6. Add data via echo/redirect
echo "1,Alice,alice@example.com" >> /mnt/storage/data/data.csv
echo "2,Bob,bob@example.com" >> /mnt/storage/data/data.csv

# 7. Verify
cat /mnt/storage/data/data.csv
# id,name,email
# 1,Alice,alice@example.com
# 2,Bob,bob@example.com

# 8. Unmount when done
posixlake-cli unmount /mnt/storage

Empty database = valid Delta Lake table with schema but 0 rows. The _delta_log/ is created immediately so any Delta Lake reader can open it.

Import from CSV (Auto Schema Inference)

# Create database from CSV - schema is automatically inferred
posixlake-cli create /path/to/db --from-csv data.csv

# Schema inference rules (from first 10 data rows):
# - All values parse as integers → Int64
# - All values parse as decimals → Float64  
# - All values are true/false/1/0 → Boolean
# - Otherwise → String

Import from Parquet

# Create database from single Parquet file
posixlake-cli create /path/to/db --from-parquet data.parquet

# Create from multiple Parquet files (glob pattern)
posixlake-cli create /path/to/db --from-parquet "data/*.parquet"

Python

from posixlake import DatabaseOps, Schema, Field

# 1. Define schema
schema = Schema(fields=[
    Field(name="id", data_type="Int32", nullable=False),
    Field(name="name", data_type="String", nullable=False),
    Field(name="email", data_type="String", nullable=True),
])

# 2. Create database at a path
db = DatabaseOps.create("/path/to/my_database", schema)

# 3. Insert data
db.insert_json('[{"id": 1, "name": "Alice", "email": "alice@example.com"}]')

Import from CSV (auto schema inference):

# Schema is automatically inferred from CSV content
db = DatabaseOps.create_from_csv("/path/to/db", "/path/to/data.csv")

# Query the imported data
results = db.query_json("SELECT * FROM data")

Import from Parquet:

# Schema is read from Parquet metadata
db = DatabaseOps.create_from_parquet("/path/to/db", "/path/to/data.parquet")

# Supports glob patterns for multiple files
db = DatabaseOps.create_from_parquet("/path/to/db", "/data/*.parquet")

Supported data types:

• Int32, Int64
• Float32, Float64
• String
• Boolean
• Date32, Timestamp
• Binary

Opening Existing Delta Lake Tables

# Open any existing Delta Lake table (created by Spark, Databricks, etc.)
db = DatabaseOps.open("/path/to/existing/delta_table")

Directory Behavior

Creating a new database (DatabaseOps.create):

• The directory must not exist or be empty
• If files exist, it will fail to prevent overwriting data

Opening an existing database (DatabaseOps.open):

• Looks for _delta_log/ subdirectory to identify it as a Delta Lake table
• Ignores other files in the directory
• Fails if no valid Delta Lake structure is found

If you want to add a Delta Lake table to a directory with existing files:

# Create in a subdirectory
db = DatabaseOps.create("/path/to/existing_dir/my_delta_table", schema)

Usage - POSIX Interface

The easiest way to use posixlake is to mount it as a filesystem:

# Mount an existing database
./target/release/posixlake-cli mount /path/to/database /mnt/data

# Now use regular UNIX commands!
cat /mnt/data/data/data.csv        # Read all data
grep "pattern" /mnt/data/data/*.csv # Search
awk -F',' '{print $2}' /mnt/data/data/data.csv  # Process
echo "id,name,value" >> /mnt/data/data/data.csv  # Append

# Unmount when done
./target/release/posixlake-cli unmount /mnt/data

All writes persist to Delta Lake with full ACID guarantees. No special tools needed - just standard UNIX commands.

Windows Usage

On Windows, mount to a drive letter and use PowerShell or cmd.exe for CRUD operations:

# 1. Create database from CSV
.\posixlake-cli.exe create test_db --from-csv data.csv

# 2. Mount to drive letter
.\posixlake-cli.exe mount test_db T:

# 3. READ - view data
Get-Content T:\data\data.csv
Get-Content T:\data\data.csv | Select-Object -First 10

# 4. INSERT - append new row
Add-Content -Path T:\data\data.csv -Value "26,New Person,new@example.com,30,Sales,70000"

# 5. UPDATE - overwrite with modified content (triggers MERGE)
Get-Content T:\data\data.csv | ForEach-Object { $_ -replace 'OldValue','NewValue' } | Out-File temp.csv -Encoding ascii
cmd /c "copy /Y temp.csv T:\data\data.csv"

# 6. DELETE - overwrite without the row (triggers MERGE with DELETE)
Get-Content T:\data\data.csv | Where-Object { $_ -notmatch 'RowToDelete' } | Out-File temp.csv -Encoding ascii
cmd /c "copy /Y temp.csv T:\data\data.csv"

# 7. Unmount
.\posixlake-cli.exe unmount T:

# 8. Check status
.\posixlake-cli.exe status T:

Note: Use -Encoding ascii with Out-File and cmd /c copy for overwrites to ensure proper CSV formatting. PowerShell's default encoding can cause issues with CSV parsing.

What Happens on Unmount

When you unmount the NFS filesystem, your data persists because it's stored in the Delta Lake directory, not in the mount:

# Unmount
./target/release/posixlake-cli unmount /mnt/data

# The mount point becomes empty, but data remains at:
# /path/to/database/
# ├── _delta_log/     # Transaction log
# └── *.parquet       # Your data files

The NFS mount is just a view into the Delta Lake - it doesn't own the data.

To access data after unmount:

# Option 1: Re-mount via NFS
nfs = NfsServer(db, port)
# mount again...

# Option 2: Use posixlake API directly (no mount needed)
db = DatabaseOps.open("/path/to/database")
results = db.query_json("SELECT * FROM data")

# Option 3: Use any Delta Lake reader (Spark, DuckDB, etc.)
spark.read.format("delta").load("/path/to/database")

What the NFS Mount Exposes

When mounted, the NFS server exposes a virtual filesystem representing the Delta Lake table:

/mnt/data/
├── data/
│   ├── data.csv      # Virtual CSV view of the table
│   └── *.parquet     # The actual Parquet files
└── schema.sql        # Table schema

It does not expose arbitrary files that exist in the underlying directory - it's a virtual filesystem, not a passthrough to the host filesystem.

Setup MinIO (Optional - for S3 backend)

# Start MinIO using Docker Compose
docker compose up -d

# Verify MinIO is running
curl http://localhost:9000/minio/health/live
# Expected: 200 OK

# MinIO Console (web UI)
open http://localhost:9001
# Login: minioadmin / minioadmin

# Test S3 backend
./target/release/posixlake-cli s3-test "s3://posixlake-test/test_db"

MinIO Configuration:

The project includes .cargo/config.toml with default MinIO settings:

[env]
MINIO_ENDPOINT = "http://localhost:9000"
MINIO_ACCESS_KEY = "minioadmin"
MINIO_SECRET_KEY = "minioadmin"
MINIO_BUCKET = "posixlake-test"

Custom S3 Configuration:

To use AWS S3 or custom MinIO:

# Export environment variables
export MINIO_ENDPOINT="https://s3.amazonaws.com"
export MINIO_ACCESS_KEY="your-access-key"
export MINIO_SECRET_KEY="your-secret-key"
export MINIO_BUCKET="your-bucket"

# Or pass via CLI
./target/release/posixlake-cli s3-test "s3://your-bucket/your-db" \
  --endpoint "https://s3.amazonaws.com" \
  --access-key "your-access-key" \
  --secret-key "your-secret-key"

Basic Usage

Rust API

use posixlake::{Database, Record};
use std::collections::HashMap;

fn main() -> posixlake::Result<()> {
    // Create a new database
    let db = Database::create("/path/to/database")?;
    
    // Insert data
    let mut record = HashMap::new();
    record.insert("name".to_string(), "Alice".to_string());
    record.insert("age".to_string(), "30".to_string());
    record.insert("email".to_string(), "alice@example.com".to_string());
    
    let id = db.insert(Record { data: record })?;
    println!("Inserted record with ID: {}", id);
    
    // Query data
    let result = db.get(&id)?;
    println!("Retrieved: {:?}", result);
    
    Ok(())
}

SQL Queries (DataFusion Integration)

use posixlake::query::PosixLakeTableProvider;
use datafusion::prelude::*;
use arrow::array::{Int32Array, StringArray, RecordBatch};
use arrow::datatypes::{DataType, Field, Schema};
use std::sync::Arc;

#[tokio::main]
async fn main() -> posixlake::Result<()> {
    // Create schema
    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int32, false),
        Field::new("name", DataType::Utf8, false),
        Field::new("email", DataType::Utf8, false),
    ]));
    
    // Create sample data
    let batch = RecordBatch::try_new(
        schema.clone(),
        vec![
            Arc::new(Int32Array::from(vec![1, 2, 3])),
            Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])),
            Arc::new(StringArray::from(vec![
                "alice@example.com",
                "bob@example.com",
                "charlie@example.com"
            ])),
        ],
    )?;
    
    // Register table provider
    let provider = PosixLakeTableProvider::new(schema, vec![batch])?;
    let ctx = SessionContext::new();
    ctx.register_table("users", Arc::new(provider))?;
    
    // Execute SQL queries
    let df = ctx.sql("SELECT name, email FROM users WHERE id > 1").await?;
    let results = df.collect().await?;
    
    println!("Query results: {:?}", results);
    
    Ok(())
}

Transaction Example

use posixlake::transaction::{TransactionManager, TransactionVisibility, FileVersion};

fn main() -> posixlake::Result<()> {
    // Initialize transaction manager
    let txn_manager = TransactionManager::new();
    
    // Begin transaction
    let mut txn = txn_manager.begin_transaction()?;
    println!("Started transaction: {}", txn.id);
    
    // Simulate file operations
    let file = FileVersion {
        path: "data/users_001.parquet".to_string(),
        created_by_txn_id: txn.id,
        created_at_ts: txn.snapshot.timestamp,
        created_committed: false,
        deleted_by_txn_id: None,
        deleted_at_ts: None,
        deleted_committed: None,
    };
    
    // Check visibility (read-your-own-writes)
    let visible = TransactionVisibility::is_visible(&txn.snapshot, &file);
    println!("File visible to transaction: {}", visible);
    
    // Commit transaction
    txn.commit()?;
    println!("Transaction committed");
    
    Ok(())
}

How S3 Storage Works:

posixlake uses Delta Lake native format with full S3 support:

• Delta Lake Format: Native _delta_log/ transaction log in S3
• S3-First Architecture: All data and metadata stored in S3
• Local Cache: Transparent local caching for performance
• Full ACID: Delta Lake provides transaction guarantees
• Ecosystem Compatible: Tables readable by Spark/Databricks/Athena
• MinIO Compatible: Works with S3-compatible object stores

Storage Distribution:

Component	Location	Reason
Delta Transaction Log (_delta_log/)	S3 + Local cache	Delta Lake standard, ACID guarantees
Parquet Files	S3 + Local cache	Large, immutable, columnar data
Local Cache	Local filesystem	Performance optimization

Verification:

After creating a database with S3 backend, you can verify files are in MinIO:

# List databases
docker compose exec minio sh -c "mc alias set myminio http://localhost:9000 minioadmin minioadmin && mc ls myminio/posixlake-test/"

# Inspect Delta Lake transaction log
docker compose exec minio sh -c "mc ls myminio/posixlake-test/your_db/_delta_log/"
# → 00000000000000000000.json (Delta Lake transaction log)
# → 00000000000000000001.json (subsequent transactions)

# Inspect Parquet data files
docker compose exec minio sh -c "mc ls myminio/posixlake-test/your_db/"
# → part-00000-*.snappy.parquet (Delta Lake Parquet files)

# Read Delta transaction log
docker compose exec minio sh -c "mc cat myminio/posixlake-test/your_db/_delta_log/00000000000000000000.json"
# → Delta Lake metadata, protocol, and add actions

Environment Configuration:

# .cargo/config.toml
[env]
MINIO_ENDPOINT = "http://localhost:9000"
MINIO_ACCESS_KEY = "minioadmin"
MINIO_SECRET_KEY = "minioadmin"
MINIO_BUCKET = "posixlake-test"

Tech Stack

Core Libraries:

• Arrow 56.2 - Columnar in-memory format
• Parquet 56.2 - Columnar file format
• DataFusion 50.3 - SQL query engine
• DeltaLake 0.29 - Table storage framework
• Tokio 1.48 - Async runtime
• ObjectStore 0.12 - Storage abstraction
• Moka 0.12 - High-performance in-memory cache (LRU/TTL)
• Sled 0.34 - Embedded database for persistent disk cache
• Memmap2 0.9 - Memory-mapped file I/O for zero-copy reads
• Serde 1.0 - JSON serialization
• Bincode 2.0 - Binary serialization for WAL
• CRC 3.3 - CRC32 checksums
• NFSServe - Pure Rust NFSv3 server implementation

Development:

• TempFile 3.23 - Temporary directories for tests
• Tracing - Structured logging
• Thiserror - Error handling

# Run all tests
cargo test

# S3 integration tests (requires MinIO)
docker compose up -d
cargo test s3_test

# Python bindings test
python3 posixlake/examples/python_example.py

# Python package integration test
python scripts/test_pip_package.py

# Python S3/MinIO integration test via UniFFI
python scripts/test_python_s3.py

# Full manual NFS/POSIX test, also runs the Python S3 integration test
python scripts/full_test.py

# Delta Lake interoperability test (requires PySpark)
python3 posixlake/examples/interop_test_spark_posixlake.py

# Cleanup
docker compose down

Test Suites:

• Full Delta Lake operations (Time Travel, OPTIMIZE, VACUUM, Z-ORDER, Data Skipping)
• S3/MinIO backend tests
• NFS server and POSIX interface tests
• Security and RBAC tests
• Python bindings (complete working example with NFS)
• Delta Lake Interoperability Test: PySpark ↔ posixlake (100% compatible)

For detailed testing documentation, see tests/POSIX_TEST_SETUP.md.

Production Readiness Checklist

If you are evaluating whether posixlake is merely test-green or actually ready for a production rollout, use this checklist.

Baseline validation (developer/CI ready):

cargo fmt --all
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test

Passing the baseline suite means the repo is in a good mergeable state, but it is not by itself sufficient to claim production readiness.

Production-style validation (recommended before rollout):

# Build the optimized artifact you plan to ship
cargo build --release

# Explicit backup / restore validation
cargo test -p posixlake-integration-tests --test backup_restore_test -- --nocapture

# Ignored stress suites (long-running)
cargo test -p posixlake-integration-tests test_stress_100k_rows -- --ignored --nocapture
cargo test -p posixlake-integration-tests test_stress_large_batch_delete -- --ignored --nocapture
cargo test -p posixlake-integration-tests test_stress_1m_rows -- --ignored --nocapture
cargo test -p posixlake-integration-tests test_1gb_csv_stress -- --ignored --nocapture

What this proves:

• normal regression suite is green
• release artifact builds successfully
• backup, restore, and point-in-time restore work
• large Windows NFS workflows remain stable under sustained load
• CSV diff / delete paths complete successfully on very large datasets

What this still does not prove automatically:

• behavior in your exact production hardware / cloud environment
• safe rollout without a canary
• live operational monitoring / alerting coverage

Current Windows validation status used for this README:

• cargo test passed
• cargo build --release passed
• backup_restore_test passed
• ignored NFS stress tests passed at 100K rows, 1M rows, 50K-row batch delete, and 30M-row / ~1GB CSV scale

Practical interpretation: after the full checklist above passes, posixlake is reasonable to call production-ready for a canary rollout, not just “tests passed.”

Features

Delta Lake Integration

• Native Delta Lake _delta_log/ transaction log format
• Full compatibility with Spark, Databricks, and AWS Athena
• No translation or export required
• Standard Delta Lake Parquet file naming

Storage & Performance

• Columnar Parquet storage with Snappy compression
• S3 and MinIO native support
• Two-tier caching system (memory + disk)
• Memory-mapped I/O for large files
• Lazy loading and write buffering

Transaction & Concurrency

• Delta Lake snapshot isolation
• Read-your-own-writes semantics
• Write-write conflict detection
• ACID guarantees via Delta Lake protocol
• Automatic schema evolution with NULL padding

Query Engine

• Full SQL support via DataFusion
• MERGE (UPSERT) operations - INSERT/UPDATE/DELETE in single transaction
  • Explicit primary key support (string or integer columns)
  • Automatic primary key detection if not specified
• Primary key metadata - set and retrieve via set_primary_key() / primary_key()
• Column statistics for query pruning
• Predicate pushdown optimization
• Deletion vectors for efficient row-level deletes
• Complex type support (Struct, List, Map, Decimal, Timestamp)

POSIX Interface

• Pure Rust NFS server (zero dependencies)
• Mount database as filesystem
• Use standard UNIX tools (cat, grep, awk, sed, vim)
• Cross-platform (Linux, macOS, Windows)
• CSV views of all Parquet data

Advanced Features

• User authentication with bcrypt
• Role-based access control (RBAC)
• Audit logging for compliance
• Backup and restore with point-in-time recovery
• Monitoring and health check APIs

Python Bindings

• UniFFI-generated bindings for Python 3.8+ (3.11+ recommended for PyPI wheels)
• Full API Coverage: All core features accessible from Python including MERGE
• NFS Server Support: Mount Delta Lake as filesystem from Python
• Type Safety: Complete type hints and error handling
• Zero-Copy Operations: Efficient Rust-Python interop
• Robust: Comprehensive error handling and async support
• PyPI Available: pip install posixlake (Python 3.11+ for prebuilt wheels)

Example MERGE (UPSERT) operation:

from posixlake import DatabaseOps
import json

db = DatabaseOps.open("my_delta_table")

# Set primary key for automatic MERGE behavior
db.set_primary_key("id")
pk = db.primary_key()  # Returns: "id"

# Explicit MERGE with specified primary key
merge_data = [
    {"id": 1, "name": "Alice", "age": 31, "_op": "UPDATE"},
    {"id": 2, "name": "Bob", "age": 35, "_op": "DELETE"},
    {"id": 3, "name": "Charlie", "age": 28, "_op": "INSERT"}
]
result = db.merge_json(json.dumps(merge_data), "id")
metrics = json.loads(result)  # {"rows_inserted": 1, "rows_updated": 1, "rows_deleted": 1}

# List underlying Parquet files
files = db.list_parquet_files()
print(f"Data files: {files}")

See bindings/python/ for documentation and PYTHON_BINDINGS_COMPLETE.md for implementation details.

Performance

posixlake is designed for high-performance analytics workloads:

• Write latency: 5-50ms (Parquet encoding + Delta Lake commit)
• Read latency: 1-10ms (Parquet decompression with column pruning)
• Write throughput: 1K-10K rows/sec
• Read throughput: 10K-100K rows/sec
• Memory overhead: ~1MB for 100 active files
• Row deletion: 46ms for 10 rows (100-row table), 152ms for 100 rows (10K-row table)
• Stress tested: 30M rows (1GB CSV) with successful deletion - see STRESS_TEST_RESULTS.md

Running Stress Tests

The stress test suite is ignored by default due to long runtimes (~30 minutes). To run:

# Medium / large NFS stress tests
cargo test -p posixlake-integration-tests test_stress_100k_rows -- --ignored --nocapture
cargo test -p posixlake-integration-tests test_stress_large_batch_delete -- --ignored --nocapture
cargo test -p posixlake-integration-tests test_stress_1m_rows -- --ignored --nocapture

# Run the 1GB CSV stress test
cargo test -p posixlake-integration-tests test_1gb_csv_stress -- --ignored --nocapture

# Run all tests including stress tests
cargo test -- --ignored --nocapture

See STRESS_TEST_RESULTS.md for detailed performance benchmarks on extreme-scale datasets.

License

Apache License 2.0

Copyright 2025 posixlake Contributors

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Why Apache 2.0? posixlake builds on Delta Lake (Apache 2.0) and follows Linux Foundation standards for open infrastructure software. This permissive license enables broad adoption while protecting contributors through patent grants.

See LICENSE.md for the full license text.

Contributing

Contributions welcome! Please follow these guidelines:

1. Write tests first - TDD approach for all features
2. Run full suite - Ensure cargo test passes
3. Update documentation - Keep README and docs up to date
4. Commit messages - Use conventional commits (e.g., feat: Add conflict detection)

---

Questions? Open an issue

Like this project? Star the repo and share with your data engineering team!