Overview

Loading…

Overview

Relevant source files

• Cargo.lock
• Cargo.toml
• README.md

Purpose and Scope

This document provides a high-level introduction to the SIMD R Drive codebase, explaining its purpose as a high-performance, append-only storage engine and outlining its major architectural components. For detailed information about specific subsystems, see the corresponding sections: Core Storage Engine, Network Layer and RPC, Python Integration, Performance Optimizations, and Extensions and Utilities.

Sources: README.md:1-40 Cargo.toml:1-21

What is SIMD R Drive?

SIMD R Drive is a high-performance, thread-safe, append-only storage engine designed for zero-copy binary access. It stores arbitrary binary data in a single-file storage container where all payloads are written at fixed 64-byte aligned boundaries , optimizing for SIMD operations and cache-line efficiency.

The system operates schema-less —it treats all stored data as raw bytes (&[u8]) without enforcing serialization formats or endianness. This design provides maximum flexibility for applications requiring high-speed storage and retrieval of structured or unstructured binary data.

Key characteristics:

Sources: README.md:5-87 Cargo.toml:12-21

High-Level System Architecture

The following diagram shows the complete system architecture, mapping high-level concepts to concrete code entities:

Diagram: System Architecture - Mapping Concepts to Code Entities

graph TB
    subgraph "User Interfaces"
        CLI["CLI Application\n(main.rs)"]
PY_BIND["Python Direct Bindings\n(simd-r-drive-py)"]
PY_WS["Python WebSocket Client\n(simd-r-drive-ws-client-py)"]
end
    
    subgraph "Core Storage Engine (simd-r-drive)"
        DS["DataStore\n(data_store/mod.rs)"]
READER["DataStoreReader trait"]
WRITER["DataStoreWriter trait"]
INDEX["KeyIndexer\n(key_indexer.rs)"]
end
    
    subgraph "Entry Abstraction (simd-r-drive-entry-handle)"
        EH["EntryHandle\n(entry_handle.rs)"]
META["EntryMetadata\n(entry_metadata.rs)"]
end
    
    subgraph "Network Layer (Experimental)"
        WS_SERVER["simd-r-drive-ws-server\n(WebSocket Server)"]
WS_CLIENT["simd-r-drive-ws-client\n(Native Rust Client)"]
SERVICE_DEF["simd-r-drive-muxio-service-definition\n(RPC Contract)"]
end
    
    subgraph "Storage Backend"
        MMAP["Memory-Mapped File\n(Arc<Mmap>)"]
FILE["Single Binary File\n(.bin)"]
end
    
    subgraph "Performance Layer"
        SIMD["SIMD Operations\n(simd_copy)"]
XXH3["xxh3_64 Hashing\n(KeyIndexer)"]
end
    
 
   CLI --> DS
 
   PY_BIND --> DS
 
   PY_WS --> WS_CLIENT
 
   WS_CLIENT --> SERVICE_DEF
 
   SERVICE_DEF --> WS_SERVER
 
   WS_SERVER --> DS
    
 
   DS --> READER
 
   DS --> WRITER
 
   DS --> INDEX
 
   DS --> EH
 
   DS --> MMAP
    
 
   EH --> META
 
   EH --> MMAP
 
   MMAP --> FILE
    
 
   DS --> SIMD
 
   INDEX --> XXH3
    
    style DS fill:#f9f9f9,stroke:#333,stroke-width:2px
    style EH fill:#f9f9f9,stroke:#333,stroke-width:2px

This diagram illustrates how user-facing interfaces connect to the core storage engine and supporting subsystems, using actual code entity names.

Sources: Cargo.toml:66-77 README.md:11-40

Repository Structure

The codebase is organized as a Cargo workspace with the following packages:

For detailed information about the repository layout and package relationships, see Repository Structure.

Sources: Cargo.toml:66-77 README.md:259-265

graph TB
    subgraph "Public API"
        DST["DataStore struct\n(data_store/mod.rs)"]
READER_TRAIT["DataStoreReader trait\n(traits.rs)"]
WRITER_TRAIT["DataStoreWriter trait\n(traits.rs)"]
end
    
    subgraph "Indexing Layer"
        KI["KeyIndexer\n(key_indexer.rs)"]
DASHMAP["DashMap<u64, u64>\n(concurrent hash map)"]
XXH3_HASH["xxh3_64\n(key hashing)"]
end
    
    subgraph "Entry Management"
        EH_STRUCT["EntryHandle\n(entry_handle.rs)"]
EM_STRUCT["EntryMetadata\n(entry_metadata.rs)"]
PAYLOAD_ALIGN["PAYLOAD_ALIGNMENT\n(constants.rs)"]
end
    
    subgraph "Storage Backend"
        RWLOCK_FILE["RwLock<File>"]
MUTEX_MMAP["Mutex<Arc<Mmap>>"]
ATOMIC_TAIL["AtomicU64 tail_offset"]
end
    
    subgraph "SIMD Acceleration"
        SIMD_COPY["simd_copy\n(arch-specific impls)"]
AVX2["AVX2 impl (x86_64)"]
NEON["NEON impl (aarch64)"]
end
    
 
   DST --> READER_TRAIT
 
   DST --> WRITER_TRAIT
 
   DST --> KI
 
   DST --> RWLOCK_FILE
 
   DST --> MUTEX_MMAP
 
   DST --> ATOMIC_TAIL
    
 
   KI --> DASHMAP
 
   KI --> XXH3_HASH
    
 
   READER_TRAIT --> EH_STRUCT
 
   EH_STRUCT --> EM_STRUCT
 
   EH_STRUCT --> PAYLOAD_ALIGN
    
 
   WRITER_TRAIT --> SIMD_COPY
 
   SIMD_COPY --> AVX2
 
   SIMD_COPY --> NEON
    
    style DST fill:#f9f9f9,stroke:#333,stroke-width:2px
    style KI fill:#f9f9f9,stroke:#333,stroke-width:2px

Core Storage Components

The following diagram maps storage concepts to their implementing code entities:

Diagram: Core Storage Components - Code Entity Mapping

This diagram shows the relationship between storage concepts and their concrete implementations in the codebase.

Sources: README.md:172-183 Cargo.toml:23-34

Key Features Summary

Storage and Access Patterns

Sources: README.md:43-148

Concurrency Model

For detailed concurrency semantics, see Concurrency and Thread Safety.

Sources: README.md:170-200

Write and Read Modes

Write Modes:

• Single Entry : write() - atomic single key-value write
• Batch Entry : batch_write() - multiple writes with single flush
• Streaming : write_stream() - large entries via Read trait

Read Modes:

• Direct Memory Access : read() - zero-copy via EntryHandle
• Streaming : read_stream() - incremental reads for large entries
• Parallel Iteration : par_iter_entries() - Rayon-powered parallel scanning (requires parallel feature)

For detailed read/write APIs, see DataStore API.

Sources: README.md:208-247

SIMD and Performance Optimizations

SIMD R Drive employs multiple optimization strategies:

For detailed performance information, see Performance Optimizations and SIMD Acceleration.

Sources: README.md:249-257 Cargo.toml34

Multi-Language Support

Native Rust

The core library is implemented in Rust and can be used directly via Cargo:

Sources: Cargo.toml:11-21

Python Bindings

Two experimental Python integration paths are available:

1. Direct Bindings (simd-r-drive-py): PyO3-based bindings for direct access to DataStore
2. WebSocket Client (simd-r-drive-ws-client-py): Remote access via WebSocket RPC

For Python integration details, see Python Integration.

Sources: README.md:262-265 Cargo.toml:74-76

WebSocket RPC (Experimental)

The experimental network layer enables remote access:

• Server : simd-r-drive-ws-server - Exposes DataStore over WebSocket
• Native Client : simd-r-drive-ws-client - Rust client for WebSocket connection
• Service Definition : simd-r-drive-muxio-service-definition - Shared RPC contract using bitcode serialization

For network layer details, see Network Layer and RPC.

Sources: Cargo.toml:70-72 Cargo.toml:85-89

Feature Flags

The core simd-r-drive package supports the following Cargo features:

Sources: Cargo.toml:49-55

Dependencies Overview

Core Dependencies

Network Layer Dependencies (Experimental)

Sources: Cargo.toml:23-34 Cargo.toml:80-112

Development and Testing

The repository includes:

• Unit Tests : Inline tests in each module
• Integration Tests : tests/ directory with full system tests
• Benchmarks : Criterion-based benchmarks in benches/ (see Benchmarking)
• CI/CD : GitHub Actions workflows for cross-platform testing (see CI/CD Pipeline)

Sources: Cargo.toml:36-63

Next Steps

This overview introduces the high-level architecture and key components of SIMD R Drive. For deeper exploration:

• Core Storage Mechanics : See Core Storage Engine for detailed information about DataStore, storage format, and memory management
• API Usage : See DataStore API for method documentation and usage patterns
• Performance Tuning : See Performance Optimizations for SIMD usage, alignment, and benchmarking
• Python Integration : See Python Integration for binding usage and WebSocket client examples
• Building and Testing : See Development Guide for build instructions and contribution guidelines

Sources: README.md:1-285

Dismiss

Refresh this wiki

Enter email to refresh

Core Storage Engine

Loading…

Core Storage Engine

Relevant source files

• Cargo.toml
• README.md
• src/lib.rs
• src/storage_engine.rs
• src/storage_engine/entry_iterator.rs

Purpose and Scope

This document provides an overview of the core storage engine architecture in SIMD R Drive. It covers the fundamental design principles, main components, and data flow patterns that enable high-performance, append-only binary storage.

For detailed information about specific aspects of the storage engine:

Sources: README.md:1-50 src/lib.rs:1-28

System Architecture

The core storage engine is implemented as a single-file, append-only key-value store. It consists of four primary components that work together to provide high-performance binary storage with zero-copy read access.

Diagram: Core Storage Engine Components

graph TB
    subgraph "Public API Layer"
        DS["DataStore"]
DSR["DataStoreReader trait"]
DSW["DataStoreWriter trait"]
end
    
    subgraph "Indexing Layer"
        KI["KeyIndexer"]
DASHMAP["DashMap<u64, u64>"]
XXH3["xxh3_64 hasher"]
end
    
    subgraph "Storage Layer"
        MMAP["Arc<Mmap>"]
FILE["Single binary file"]
RWLOCK["RwLock<File>"]
end
    
    subgraph "Access Layer"
        EH["EntryHandle"]
EI["EntryIterator"]
ES["EntryStream"]
end
    
 
   DS --> DSR
 
   DS --> DSW
 
   DS --> KI
 
   DS --> MMAP
 
   DS --> RWLOCK
    
 
   KI --> DASHMAP
 
   KI --> XXH3
    
 
   DSR --> EH
 
   DSR --> EI
 
   DSW --> RWLOCK
    
 
   EH --> MMAP
 
   EI --> MMAP
 
   ES --> EH
    
 
   MMAP --> FILE
 
   RWLOCK --> FILE

The diagram shows the relationship between the main code entities:

Sources: src/storage_engine.rs:1-25 src/lib.rs:129-136 README.md:5-11

Append-Only Design

The storage engine follows a strict append-only model where data is never modified or deleted in place. All operations result in new entries being appended to the end of the file.

Diagram: Write Path Data Flow

graph LR
    subgraph "Write Operations"
        W["write()"]
BW["batch_write()"]
WS["write_stream()"]
DEL["delete()"]
end
    
    subgraph "Internal Write Path"
        HASH["Calculate xxh3_64 hash"]
ALIGN["Calculate prepad for\n64-byte alignment"]
COPY["simd_copy payload"]
APPEND["Append to file"]
META["Write metadata:\nkey_hash, prev_offset, crc32"]
end
    
    subgraph "File State"
        TAIL["AtomicU64 tail_offset"]
CHAIN["Backward-linked chain"]
end
    
 
   W --> HASH
 
   BW --> HASH
 
   WS --> HASH
 
   DEL --> HASH
    
 
   HASH --> ALIGN
 
   ALIGN --> COPY
 
   COPY --> APPEND
 
   APPEND --> META
 
   META --> TAIL
 
   META --> CHAIN

Key Characteristics

The append-only design provides several benefits:

• Crash safety : Incomplete writes can be detected and discarded
• Simplified concurrency : No in-place modifications to coordinate
• Time-travel : Historical data remains until compaction
• Write performance : Sequential I/O with no seek overhead

Sources: README.md:98-147 src/lib.rs:3-17

Single-File Storage Container

All data is stored in a single binary file with a specific structure designed for efficient access and validation.

Diagram: File Organization and Backward-Linked Chain

graph TB
    subgraph "File Structure"
        START["File Start: offset 0"]
E1["Entry 1\nprepad + payload + metadata"]
E2["Entry 2\nprepad + payload + metadata"]
E3["Entry 3\nprepad + payload + metadata"]
EN["Entry N\nprepad + payload + metadata"]
TAIL["tail_offset\nEnd of valid data"]
end
    
    subgraph "Metadata Chain"
        M1["metadata.prev_offset = 0"]
M2["metadata.prev_offset = end(E1)"]
M3["metadata.prev_offset = end(E2)"]
MN["metadata.prev_offset = end(E3)"]
end
    
 
   START --> E1
 
   E1 --> E2
 
   E2 --> E3
 
   E3 --> EN
 
   EN --> TAIL
    
 
   E1 -.-> M1
 
   E2 -.-> M2
 
   E3 -.-> M3
 
   EN -.-> MN
    
    MN -.backward chain.-> M3
    M3 -.backward chain.-> M2
    M2 -.backward chain.-> M1

Storage Properties

Entry Composition

Each entry consists of three parts:

1. Pre-padding (0-63 bytes): Zero bytes to align payload start to 64-byte boundary
2. Payload (variable length): Raw binary data
3. Metadata (20 bytes): Hash, previous offset, checksum

Tombstone entries (deletions) use a minimal format:

• 1-byte payload (0x00)
• 20-byte metadata

Sources: README.md:104-138 README.md:61-97

DataStore: Primary Interface

DataStore is the main public interface providing all storage operations. It implements the DataStoreReader and DataStoreWriter traits to separate read and write capabilities.

Diagram: DataStore Structure and Methods

graph TB
    subgraph "DataStore struct"
        FILE_LOCK["file: RwLock<File>"]
MMAP_LOCK["mmap: Mutex<Arc<Mmap>>"]
INDEXER["indexer: KeyIndexer"]
TAIL["tail_offset: AtomicU64"]
end
    
    subgraph "DataStoreWriter methods"
        W1["write(key, value)"]
W2["batch_write(entries)"]
W3["write_stream(key, reader)"]
W4["delete(key)"]
end
    
    subgraph "DataStoreReader methods"
        R1["read(key) -> Option<EntryHandle>"]
R2["batch_read(keys)"]
R3["iter_entries() -> EntryIterator"]
R4["contains_key(key) -> bool"]
end
    
    subgraph "Maintenance methods"
        M1["compact() -> Stats"]
M2["estimate_compaction_space()"]
M3["verify_file_integrity()"]
end
    
 
   FILE_LOCK --> W1
 
   FILE_LOCK --> W2
 
   FILE_LOCK --> W3
 
   FILE_LOCK --> W4
    
 
   MMAP_LOCK --> R1
 
   MMAP_LOCK --> R2
 
   MMAP_LOCK --> R3
    
 
   INDEXER --> R1
 
   INDEXER --> R2
 
   INDEXER --> R4
    
 
   TAIL --> W1
 
   TAIL --> W2
 
   TAIL --> W3

Core Fields

The DataStore struct maintains four critical fields:

Sources: src/storage_engine.rs:4-5 src/lib.rs:19-63

graph LR
    subgraph "Read Request"
        KEY["Key bytes"]
HASH_KEY["xxh3_64(key)"]
end
    
    subgraph "Index Lookup"
        DASHMAP_GET["DashMap.get(hash)"]
PACKED["Packed value:\n16-bit tag + 48-bit offset"]
TAG_CHECK["Verify 16-bit tag"]
end
    
    subgraph "Memory Access"
        OFFSET["File offset"]
MMAP_SLICE["Arc<Mmap> slice"]
METADATA_READ["Read EntryMetadata"]
PAYLOAD_RANGE["Calculate payload range"]
end
    
    subgraph "Result"
        EH["EntryHandle\n(zero-copy view)"]
end
    
 
   KEY --> HASH_KEY
 
   HASH_KEY --> DASHMAP_GET
 
   DASHMAP_GET --> PACKED
 
   PACKED --> TAG_CHECK
 
   TAG_CHECK --> OFFSET
 
   OFFSET --> MMAP_SLICE
 
   MMAP_SLICE --> METADATA_READ
 
   METADATA_READ --> PAYLOAD_RANGE
 
   PAYLOAD_RANGE --> EH

Zero-Copy Read Path

Read operations leverage memory-mapped files to provide zero-copy access to stored data without deserialization overhead.

Diagram: Zero-Copy Read Operation Flow

Read Performance Characteristics

EntryHandle

The EntryHandle struct provides a zero-copy view into the memory-mapped file:

Methods like as_slice(), as_bytes(), and streaming conversion allow direct access to payload data without copying.

Sources: README.md:43-50 README.md:228-231 src/storage_engine/entry_iterator.rs:8-21

graph TB
    subgraph "Key to Hash"
        K1["Key: 'user:1234'"]
H1["xxh3_64 hash:\n0xABCDEF0123456789"]
end
    
    subgraph "Hash to Packed Value"
        TAG["Extract 16-bit tag:\n0xABCD"]
OFF["Extract file offset:\n0x00EF0123456789"]
PACKED_VAL["Packed 64-bit value:\ntag:16 / offset:48"]
end
    
    subgraph "DashMap Storage"
        DM["DashMap<u64, u64>"]
ENTRY["hash -> packed_value"]
end
    
    subgraph "Collision Detection"
        LOOKUP["On read: lookup hash"]
VERIFY["Verify 16-bit tag matches"]
RESOLVE["If mismatch: collision detected"]
end
    
 
   K1 --> H1
 
   H1 --> TAG
 
   H1 --> OFF
 
   TAG --> PACKED_VAL
 
   OFF --> PACKED_VAL
 
   PACKED_VAL --> ENTRY
 
   ENTRY --> DM
    
 
   DM --> LOOKUP
 
   LOOKUP --> VERIFY
 
   VERIFY --> RESOLVE

Key Indexing System

The KeyIndexer maintains an in-memory hash index for O(1) key lookups using the xxh3_64 hashing algorithm with SIMD acceleration.

Diagram: Key Indexing and Collision Detection

Index Structure

Hash Algorithm Properties

The xxh3_64 algorithm provides:

• SIMD acceleration : Uses SSE2/AVX2 (x86_64) or NEON (ARM)
• High quality : Low collision probability
• Performance : Optimized for throughput
• Stability : Consistent across platforms

Sources: README.md:158-168 src/storage_engine.rs:13-14

graph TB
    subgraph "Concurrent Reads"
        T1["Thread 1 read()"]
T2["Thread 2 read()"]
T3["Thread 3 read()"]
DM_READ["DashMap lock-free read"]
MMAP_SHARE["Shared Arc<Mmap>"]
end
    
    subgraph "Synchronized Writes"
        T4["Thread 4 write()"]
T5["Thread 5 write()"]
RWLOCK_ACQUIRE["RwLock write lock"]
FILE_WRITE["Exclusive file access"]
INDEX_UPDATE["Update DashMap"]
ATOMIC_UPDATE["AtomicU64 tail_offset"]
end
    
    subgraph "Memory Map Updates"
        REMAP["Remap after write"]
MUTEX_LOCK["Mutex<Arc<Mmap>>"]
NEW_ARC["Create new Arc<Mmap>"]
end
    
 
   T1 --> DM_READ
 
   T2 --> DM_READ
 
   T3 --> DM_READ
 
   DM_READ --> MMAP_SHARE
    
 
   T4 --> RWLOCK_ACQUIRE
 
   T5 --> RWLOCK_ACQUIRE
 
   RWLOCK_ACQUIRE --> FILE_WRITE
 
   FILE_WRITE --> INDEX_UPDATE
 
   FILE_WRITE --> ATOMIC_UPDATE
 
   FILE_WRITE --> REMAP
    
 
   REMAP --> MUTEX_LOCK
 
   MUTEX_LOCK --> NEW_ARC
 
   NEW_ARC --> MMAP_SHARE

Thread Safety Model

The storage engine provides thread-safe concurrent access within a single process using a combination of synchronization primitives.

Diagram: Concurrency Control Mechanisms

Synchronization Primitives

Thread Safety Guarantees

Within single process:

• ✅ Multiple concurrent reads (zero-copy, lock-free)
• ✅ Serialized writes (RwLock ensures ordering)
• ✅ Consistent index updates (DashMap internal locks)
• ✅ Safe memory mapping (Arc reference counting)

Across multiple processes:

• ❌ No cross-process coordination
• ❌ Requires external file locking (e.g., flock)

Sources: README.md:170-206

graph LR
    subgraph "Write Path SIMD"
        PAYLOAD["Payload bytes"]
SIMD_COPY["simd_copy()"]
AVX2["x86_64: AVX2\n256-bit vectors"]
NEON["ARM: NEON\n128-bit vectors"]
BUFFER["Aligned buffer"]
end
    
    subgraph "Hash Path SIMD"
        KEY["Key bytes"]
XXH3_SIMD["xxh3_64 SIMD"]
SSE2["x86_64: SSE2/AVX2"]
NEON2["ARM: NEON"]
HASH_OUT["64-bit hash"]
end
    
 
   PAYLOAD --> SIMD_COPY
 
   SIMD_COPY --> AVX2
 
   SIMD_COPY --> NEON
 
   AVX2 --> BUFFER
 
   NEON --> BUFFER
    
 
   KEY --> XXH3_SIMD
 
   XXH3_SIMD --> SSE2
 
   XXH3_SIMD --> NEON2
 
   SSE2 --> HASH_OUT
 
   NEON2 --> HASH_OUT

Performance Optimizations

The storage engine incorporates several performance optimizations for high-throughput workloads.

SIMD Acceleration

Diagram: SIMD Operations in Write Path

Optimization Features

Alignment Benefits

The 64-byte PAYLOAD_ALIGNMENT constant ensures:

1. Cache efficiency : Payloads align with CPU cache lines
2. SIMD compatibility : Vector loads don’t cross boundaries
3. Predictable performance : Consistent access patterns
4. Type casting safety : Can reinterpret as typed slices

Sources: README.md:51-59 README.md:248-256

Operation Modes

The storage engine supports multiple operation modes optimized for different use cases.

Write Modes

Read Modes

Sources: README.md:208-246 src/lib.rs:20-115

Summary

The core storage engine provides a high-performance, append-only key-value store built on four main components:

1. DataStore : Public API implementing reader and writer traits
2. KeyIndexer : O(1) hash-based index with collision detection
3. Arc : Zero-copy memory-mapped file access
4. EntryHandle : View abstraction for payload data

Key architectural decisions:

• Append-only : Simplifies concurrency and crash recovery
• Single-file : Easy deployment and management
• 64-byte alignment : Optimizes cache and SIMD performance
• Backward chain : Enables fast validation and iteration
• Zero-copy reads : Eliminates deserialization overhead
• Lock-free index : Scales read throughput across threads

For implementation details on specific subsystems, refer to the child pages listed at the beginning of this document.

Sources: README.md:1-50 src/lib.rs:1-28 src/storage_engine.rs:1-25

Dismiss

Refresh this wiki

Enter email to refresh

Repository Structure

Loading…

Repository Structure

Relevant source files

• Cargo.lock
• Cargo.toml

Purpose and Scope

This document describes the organization of the SIMD R Drive repository as a Cargo workspace, detailing the individual packages (crates) that comprise the system, their purposes, and their inter-dependencies. The repository is structured as a monorepo containing a core storage engine, supporting libraries, experimental network components, and language bindings.

For information about the core storage engine architecture and on-disk format, see Storage Architecture. For details on building and testing the codebase, see Building and Testing.

Workspace Organization

The SIMD R Drive repository is organized as a Cargo workspace defined in Cargo.toml:65-78 The workspace uses Cargo’s resolver version 2 and manages multiple interdependent packages with shared versioning and dependencies.

Sources: Cargo.toml:65-78

Workspace Configuration

The workspace defines common package metadata that all member crates inherit:

Sources: Cargo.toml:1-9

Package Structure Overview

Workspace Members

The workspace includes six member crates defined in Cargo.toml:66-73:

1. "." - The root simd-r-drive package
2. "simd-r-drive-entry-handle" - Entry abstraction library
3. "extensions" - Utility extensions
4. "experiments/simd-r-drive-ws-server" - WebSocket server
5. "experiments/simd-r-drive-ws-client" - WebSocket client
6. "experiments/simd-r-drive-muxio-service-definition" - RPC service contract

Excluded Members

Two Python binding packages are excluded from the workspace Cargo.toml:74-77 because they use maturin with separate build systems:

• "experiments/bindings/python" - Direct Rust-Python bindings
• "experiments/bindings/python-ws-client" - Python WebSocket client bindings

Sources: Cargo.toml:65-78

Core Packages

simd-r-drive (Root Package)

Location: Root directory
Cargo Name: simd-r-drive
Description: “SIMD-optimized append-only schema-less storage engine. Key-based binary storage in a single-file storage container.”

This is the main storage engine package providing the DataStore API for append-only key-value storage with SIMD acceleration and memory-mapped file access.

Key Exports:

• DataStore - Main storage interface
• DataStoreReader / DataStoreWriter - Trait-based access patterns
• KeyIndexer - Hash-based key indexing with xxh3_64

Dependencies:

• simd-r-drive-entry-handle (workspace)
• dashmap - Lock-free concurrent hash map
• memmap2 - Memory-mapped file access
• xxhash-rust - Fast hashing with SIMD support
• rayon (optional, with parallel feature)

Features:

• default - No features enabled by default
• expose-internal-api - Exposes internal APIs for testing/extensions
• parallel - Enables parallel iteration with rayon
• arrow - Proxies to simd-r-drive-entry-handle/arrow

Sources: Cargo.toml:11-56

simd-r-drive-entry-handle

Location: simd-r-drive-entry-handle/
Cargo Name: simd-r-drive-entry-handle

Provides the EntryHandle abstraction for zero-copy access to stored entries via memory-mapped files. This package is separated to allow optional Apache Arrow integration without requiring arrow dependencies in the core package.

Key Exports:

• EntryHandle - Zero-copy entry accessor
• EntryMetadata - Entry metadata structure (key_hash, prev_offset, crc32)

Dependencies:

• memmap2 - Memory-mapped file access
• crc32fast - CRC32 checksum validation
• arrow (optional, with arrow feature) - Apache Arrow buffer integration

Features:

• arrow - Enables zero-copy integration with Apache Arrow buffers

Sources: Cargo.toml83 Cargo.lock:1823-1829

simd-r-drive-extensions

Location: extensions/
Cargo Name: simd-r-drive-extensions

Utility functions and helpers built on top of the core storage engine, including alignment utilities, formatting helpers, and namespace hashing.

Key Exports:

• align_or_copy - Memory alignment utilities
• format_bytes - Human-readable byte formatting
• NamespaceHasher - Namespace-based key hashing
• File verification utilities

Dependencies:

• simd-r-drive (workspace)
• bincode - Serialization support
• serde - Serialization framework

Sources: Cargo.toml:66-73 Cargo.lock:1832-1841

Experimental Network Components

simd-r-drive-muxio-service-definition

Location: experiments/simd-r-drive-muxio-service-definition/
Cargo Name: simd-r-drive-muxio-service-definition

Defines the RPC service contract (interface definition) for remote access to the storage engine. This serves as the shared contract between WebSocket clients and servers, ensuring type-safe communication.

Key Exports:

• Service trait definitions for RPC operations
• Request/response message types
• Bitcode serialization schemas

Dependencies:

• bitcode - Compact binary serialization
• muxio-rpc-service - RPC service framework

Sources: Cargo.toml:66-73 Cargo.lock:1844-1849

simd-r-drive-ws-server

Location: experiments/simd-r-drive-ws-server/
Cargo Name: simd-r-drive-ws-server

WebSocket server implementation providing remote RPC access to a DataStore instance via the muxio framework.

Key Exports:

• WebSocket server with RPC endpoint
• Service implementation for simd-r-drive-muxio-service-definition

Dependencies:

• simd-r-drive (workspace)
• simd-r-drive-muxio-service-definition (workspace)
• muxio-tokio-rpc-server - RPC server implementation
• tokio - Async runtime
• clap - CLI argument parsing

Sources: Cargo.toml:66-73 Cargo.lock:1866-1878

simd-r-drive-ws-client

Location: experiments/simd-r-drive-ws-client/
Cargo Name: simd-r-drive-ws-client

Rust WebSocket client for connecting to simd-r-drive-ws-server instances. Provides a native Rust client API matching the core DataStore interface but operating over the network.

Key Exports:

• WsClient - WebSocket client implementation
• Async methods mirroring DataStore API

Dependencies:

• simd-r-drive (workspace)
• simd-r-drive-muxio-service-definition (workspace)
• muxio-tokio-rpc-client - RPC client implementation
• tokio - Async runtime

Sources: Cargo.toml:66-73 Cargo.lock:1852-1863

Python Bindings (External Build System)

experiments/bindings/python

Location: experiments/bindings/python/
Build System: Maturin + PyO3

Direct Python bindings to the core simd-r-drive package using PyO3. Provides a Python API for local (in-process) access to the storage engine. This package is excluded from the Cargo workspace because it uses a separate pyproject.toml build configuration with maturin.

Key Exports:

• Python DataStore class wrapping Rust implementation
• Type stubs (.pyi files) for IDE support

Sources: Cargo.toml:74-77

experiments/bindings/python-ws-client

Location: experiments/bindings/python-ws-client/
Build System: Maturin + PyO3

Python bindings for the simd-r-drive-ws-client, enabling remote access to storage servers from Python via asyncio. Uses pyo3-async-runtimes to bridge Python’s asyncio with Rust’s tokio.

Key Exports:

• DataStoreWsClient - Python async WebSocket client
• Asyncio-compatible API
• Type stubs for Python type checkers

Sources: Cargo.toml:74-77

Dependency Relationships

Sources: Cargo.toml:23-34 Cargo.toml:80-112 Cargo.lock:1795-1878

Workspace Dependency Management

The workspace defines shared dependencies in the [workspace.dependencies] section Cargo.toml:80-112 to ensure version consistency across all member crates:

Intra-Workspace Dependencies

Key External Dependencies

Sources: Cargo.toml:80-112

Feature Flags

The root simd-r-drive package defines three feature flags Cargo.toml:49-55:

default

No features enabled by default. This keeps the core storage engine lightweight with minimal dependencies.

expose-internal-api

Exposes internal APIs that are normally private. Used for extension development and integration testing. Not intended for general use.

parallel

Enables parallel iteration capabilities using the rayon crate. When enabled, operations like iter_entries() can leverage multi-core parallelism for improved throughput on large datasets.

arrow

A proxy feature that enables simd-r-drive-entry-handle/arrow. This provides zero-copy integration with Apache Arrow buffers, allowing EntryHandle instances to be viewed as Arrow arrays without data copying.

Sources: Cargo.toml:49-55

Benchmarks

The root package defines two benchmark suites using Criterion.rs Cargo.toml:57-63:

storage_benchmark

Measures write throughput, read throughput, batch operations, and streaming performance for the core storage engine.

contention_benchmark

Measures performance under concurrent access patterns, testing the effectiveness of the lock-free index and concurrent read scalability.

Both benchmarks use harness = false to integrate with Criterion’s custom benchmark harness.

Sources: Cargo.toml:57-63

Version Management

All workspace members share a common version number 0.15.5-alpha managed through Cargo.toml3 The -alpha suffix indicates this is pre-release software under active development. The workspace uses semantic versioning, where:

• Major version (0): Pre-1.0 indicating API instability
• Minor version (15): Feature releases and API changes
• Patch version (5): Bug fixes and minor improvements
• Suffix (-alpha): Pre-release stability indicator

Sources: Cargo.toml3

File System Layout

The physical repository structure mirrors the logical package organization:

rust-simd-r-drive/
├── Cargo.toml                      # Workspace root
├── Cargo.lock                      # Dependency lock file
├── src/                            # simd-r-drive source
├── benches/                        # Benchmark suites
├── simd-r-drive-entry-handle/      # Entry handle crate
│   ├── Cargo.toml
│   └── src/
├── extensions/                     # Extensions crate
│   ├── Cargo.toml
│   └── src/
└── experiments/
    ├── simd-r-drive-muxio-service-definition/
    │   ├── Cargo.toml
    │   └── src/
    ├── simd-r-drive-ws-server/
    │   ├── Cargo.toml
    │   └── src/
    ├── simd-r-drive-ws-client/
    │   ├── Cargo.toml
    │   └── src/
    └── bindings/
        ├── python/                 # Excluded from workspace
        │   ├── pyproject.toml
        │   └── src/
        └── python-ws-client/       # Excluded from workspace
            ├── pyproject.toml
            └── src/

Sources: Cargo.toml:65-78

Summary Table: All Packages

| Package Name | Location | Type | Dependencies | Purpose | |—|—|—|—| | simd-r-drive | . | Core | simd-r-drive-entry-handle, dashmap, memmap2, xxhash-rust | Main storage engine | | simd-r-drive-entry-handle | simd-r-drive-entry-handle/ | Library | memmap2, crc32fast, arrow (opt) | Entry abstraction | | simd-r-drive-extensions | extensions/ | Library | simd-r-drive, bincode | Utility functions | | simd-r-drive-muxio-service-definition | experiments/... | Library | bitcode, muxio-rpc-service | RPC contract | | simd-r-drive-ws-server | experiments/... | Binary | Core + service-def + muxio-server | WebSocket server | | simd-r-drive-ws-client | experiments/... | Library | Core + service-def + muxio-client | WebSocket client | | Python bindings | experiments/bindings/python | PyO3 | simd-r-drive, pyo3 | Direct Python access | | Python WS client | experiments/bindings/python-ws-client | PyO3 | simd-r-drive-ws-client, pyo3 | Remote Python access |

Sources: Cargo.toml:1-112 Cargo.lock:1795-1878

Dismiss

Refresh this wiki

Enter email to refresh

Storage Architecture

Loading…

Storage Architecture

Relevant source files

• README.md
• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/entry_metadata.rs
• src/storage_engine/data_store.rs

Purpose and Scope

This document describes the on-disk storage format used by SIMD R Drive, including the physical layout of entries, the alignment strategy, the backward-linked chain structure, and the recovery mechanism that ensures data integrity after crashes or incomplete writes.

For information about the in-memory data structures and API, see DataStore API. For details about entry metadata fields, see Entry Structure and Metadata. For information about memory-mapped access patterns, see Memory Management and Zero-Copy Access.

Single-File Storage Container

SIMD R Drive stores all data in a single binary file with an append-only design. The storage engine writes sequentially to minimize disk seeks and maximize throughput. Each write operation appends a new entry to the end of the file, and the file position is tracked using the AtomicU64 tail_offset field in DataStore.

Sources: README.md:61-97 src/storage_engine/data_store.rs:27-33

graph LR
    FILE["Single Binary File\n(*.simd-r-drive)"]
ENTRY1["Entry 1\n(Pre-pad + Payload + Metadata)"]
ENTRY2["Entry 2\n(Pre-pad + Payload + Metadata)"]
ENTRY3["Entry 3\n(Pre-pad + Payload + Metadata)"]
TAIL["tail_offset\n(AtomicU64)"]
FILE --> ENTRY1
 
   ENTRY1 --> ENTRY2
 
   ENTRY2 --> ENTRY3
 
   ENTRY3 --> TAIL

Entry Layout

Each entry in the storage file consists of three components: optional pre-padding for alignment, the payload data, and metadata. The layout differs between non-tombstone entries (data) and tombstone entries (deletion markers).

Non-Tombstone Entry Structure

Non-tombstone entries store actual data and are aligned to PAYLOAD_ALIGNMENT (64 bytes by default). The alignment ensures cache-line efficiency and enables zero-copy access for typed slices.

Physical Layout Table:

graph LR
    PREV_TAIL["Previous\ntail_offset"]
PREPAD["Pre-Pad\n0-63 bytes\n(zero bytes)"]
PAYLOAD["Payload\nVariable Length\n(actual data)"]
METADATA["EntryMetadata\n20 bytes"]
NEXT_TAIL["New\ntail_offset"]
PREV_TAIL --> PREPAD
 
   PREPAD --> PAYLOAD
 
   PAYLOAD --> METADATA
 
   METADATA --> NEXT_TAIL

Where:

• pad = DataStore::prepad_len(prev_tail) computed at write time
• PAYLOAD_ALIGNMENT = 64 (defined in simd-r-drive-entry-handle/src/constants.rs)
• Next entry starts at N + 20

Sources: README.md:112-125 simd-r-drive-entry-handle/src/entry_metadata.rs:11-23 src/storage_engine/data_store.rs:666-673

Entry Metadata Structure

The EntryMetadata struct stores three critical fields in exactly 20 bytes:

Field Purposes:

The metadata is serialized using little-endian encoding via EntryMetadata::serialize() and deserialized via EntryMetadata::deserialize().

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:44-113

Tombstone Entry Structure

Tombstone entries mark deleted keys. They consist of a single null byte (NULL_BYTE[0] = 0x00) followed by metadata, with no pre-padding.

Physical Layout Table:

graph LR
    PREV_TAIL["Previous\ntail_offset"]
NULL["NULL_BYTE\n1 byte\n(0x00)"]
METADATA["EntryMetadata\n20 bytes"]
NEXT_TAIL["New\ntail_offset"]
PREV_TAIL --> NULL
 
   NULL --> METADATA
 
   METADATA --> NEXT_TAIL

Tombstones are written using DataStore::batch_write_with_key_hashes() with the allow_null_bytes parameter set to true. The deletion logic filters existing keys before writing tombstones to avoid unnecessary I/O.

Sources: README.md:126-132 src/storage_engine/data_store.rs:863-897 src/storage_engine/data_store.rs:990-1024

Alignment Strategy

The pre-padding mechanism ensures that every non-tombstone payload starts at a 64-byte aligned boundary. This alignment is critical for:

1. Cache-line efficiency : Payloads align with CPU cache lines (typically 64 bytes)
2. SIMD operations : Vectorized loads/stores can operate without crossing boundaries
3. Zero-copy typed access : Enables safe reinterpretation as typed slices (e.g., &[u64])

Alignment Calculation

The DataStore::prepad_len() function implements this calculation:

fn prepad_len(offset: u64) -> usize {
    let a = PAYLOAD_ALIGNMENT;
    ((a - (offset % a)) & (a - 1)) as usize
}

During writes, the code checks if pre-padding is needed and writes zero bytes before the payload:

• At src/storage_engine/data_store.rs:765-771: Streaming write path
• At src/storage_engine/data_store.rs:909-914: Batch write path

Sources: src/storage_engine/data_store.rs:666-673 README.md:52-59

Backward-Linked Chain Structure

Each entry’s prev_offset field in EntryMetadata points to the absolute file offset of the previous entry’s tail, forming a backward-linked chain. This chain enables:

1. Iteration : Walking entries from end to beginning
2. Recovery : Validating chain integrity
3. Alignment derivation : Computing payload start from previous tail

Chain Traversal During Reads

When reading an entry from the index:

1. Look up key_hash in KeyIndexer to get metadata offset
2. Read EntryMetadata at that offset
3. Extract prev_offset (previous tail)
4. Calculate payload start: prev_offset + DataStore::prepad_len(prev_offset)
5. Payload ends at metadata offset

This logic is implemented in DataStore::read_entry_with_context() at src/storage_engine/data_store.rs:501-565

Chain Traversal Diagram:

Sources: src/storage_engine/data_store.rs:501-565 simd-r-drive-entry-handle/src/entry_metadata.rs:40-43

graph TD
    START["Start at file_len"]
CHECK_SIZE{"file_len <\nMETADATA_SIZE?"}
RETURN_ZERO["Return Ok(0)"]
INIT["cursor = file_len\nbest_valid_offset = None"]
LOOP{"cursor >=\nMETADATA_SIZE?"}
READ_META["Read metadata at\n(cursor - METADATA_SIZE)"]
EXTRACT["Extract prev_offset\nDerive entry_start"]
VALIDATE{"entry_start <\nmetadata_offset?"}
WALK["Walk chain backward\nvia prev_offset"]
CHAIN_VALID{"Chain reaches\noffset 0?"}
SET_BEST["best_valid_offset =\ncursor"]
BREAK["Break loop"]
DECREMENT["cursor -= 1"]
RETURN_BEST["Return\nbest_valid_offset\nor 0"]
START --> CHECK_SIZE
 
   CHECK_SIZE -->|Yes| RETURN_ZERO
 
   CHECK_SIZE -->|No| INIT
 
   INIT --> LOOP
 
   LOOP -->|Yes| READ_META
 
   LOOP -->|No| RETURN_BEST
 
   READ_META --> EXTRACT
 
   EXTRACT --> VALIDATE
 
   VALIDATE -->|No| DECREMENT
 
   VALIDATE -->|Yes| WALK
 
   WALK --> CHAIN_VALID
 
   CHAIN_VALID -->|Yes| SET_BEST
 
   CHAIN_VALID -->|No| DECREMENT
 
   SET_BEST --> BREAK
 
   BREAK --> RETURN_BEST
 
   DECREMENT --> LOOP

Recovery Mechanism

The DataStore::recover_valid_chain() function validates chain integrity when opening a file. It scans backward from the file end to find the deepest valid chain that reaches offset 0, automatically recovering from incomplete writes.

Recovery Algorithm

Recovery Process Steps

1. Initial Check : If file is smaller than METADATA_SIZE (20 bytes), return offset 0
2. Backward Scan : Start from file_len and scan backward by 1 byte at a time
3. Metadata Read : At each position, attempt to read metadata at cursor - METADATA_SIZE
4. Entry Validation :
   • Extract prev_offset from metadata
   • Calculate expected entry start using DataStore::prepad_len(prev_offset)
   • Handle tombstone special case (single null byte without pre-pad)
   • Verify entry_start < metadata_offset
5. Chain Walk : For valid entry, walk entire chain backward:
   • Follow prev_offset links
   • Validate each link points to earlier offset
   • Track total chain size
   • Stop when prev_offset = 0 (chain start)
6. Validation : Chain is valid if:
   • All links point backward (no cycles)
   • Chain reaches offset 0
   • Total chain size ≤ file length
7. Result : Return the first valid chain found (deepest chain)

Tombstone Handling in Recovery

Tombstones have special handling during recovery because they lack pre-padding:

if entry_end > prev_tail
    && entry_end - prev_tail == 1
    && mmap[prev_tail..entry_end] == NULL_BYTE
{
    entry_start = prev_tail  // No pre-pad for tombstone
} else {
    entry_start = prev_tail + prepad_len(prev_tail)
}

This logic appears at:

• src/storage_engine/data_store.rs:404-416: Recovery path
• src/storage_engine/data_store.rs:447-454: Chain walking

Sources: src/storage_engine/data_store.rs:363-482 README.md:139-148

Recovery on File Open

The DataStore::open() function performs recovery automatically:

If recovery detects corruption (incomplete chain), the file is truncated to the last valid offset and reopened. This ensures the storage is always in a consistent state.

Sources: src/storage_engine/data_store.rs:84-117 README.md:150-156

Entry Size Calculation

Each entry’s total file size includes pre-padding, payload, and metadata. The calculation depends on entry type:

Non-Tombstone Entry:

total_size = prepad_len(prev_tail) + payload.len() + METADATA_SIZE

Tombstone Entry:

total_size = 1 + METADATA_SIZE  // No pre-pad

The EntryHandle::file_size() method computes this from the entry’s range and metadata. For iteration and compaction, this allows precise tracking of storage usage.

Sources: README.md:112-132 src/storage_engine/data_store.rs:705-749

File Growth and Tail Tracking

The tail_offset field tracks the current end of valid data:

The atomic store uses Ordering::Release to ensure visibility across threads. Writers acquire tail_offset with Ordering::Acquire before computing pre-padding.

Sources: src/storage_engine/data_store.rs256 src/storage_engine/data_store.rs763 src/storage_engine/data_store.rs858

Dismiss

Refresh this wiki

Enter email to refresh

DataStore API

Loading…

DataStore API

Relevant source files

• README.md
• src/lib.rs
• src/storage_engine.rs
• src/storage_engine/data_store.rs
• src/storage_engine/entry_iterator.rs

This page documents the public API of the DataStore struct and its associated traits DataStoreReader and DataStoreWriter. These interfaces provide the primary methods for interacting with the storage engine, including write, read, delete, batch operations, and streaming methods.

Scope : This page covers the application-level API methods available to users of the storage engine. For details on the underlying storage format, see Entry Structure and Metadata. For implementation details of concurrency mechanisms, see Concurrency and Thread Safety. For key hashing internals, see Key Indexing and Hashing.

API Architecture

The DataStore API is organized around a core DataStore struct with two trait-based interfaces:

Sources : src/storage_engine/data_store.rs:26-33 src/storage_engine/traits.rs src/storage_engine.rs21

graph TB
    subgraph "Public API"
        DS["DataStore"]
DSR["DataStoreReader trait"]
DSW["DataStoreWriter trait"]
end
    
    subgraph "Core Operations"
        WRITE["Write Operations\nwrite()\nbatch_write()\nwrite_stream()"]
READ["Read Operations\nread()\nbatch_read()\nread_last_entry()"]
DELETE["Delete Operations\ndelete()\nbatch_delete()"]
MANAGE["Management Operations\nrename()\ncopy()\ntransfer()"]
ITER["Iteration\niter_entries()\npar_iter_entries()"]
end
    
    subgraph "Internal Components"
        FILE["Arc<RwLock<BufWriter<File>>>"]
MMAP["Arc<Mutex<Arc<Mmap>>>"]
INDEXER["Arc<RwLock<KeyIndexer>>"]
TAIL["AtomicU64 tail_offset"]
end
    
 
   DS --> DSR
 
   DS --> DSW
    
 
   DSR --> READ
 
   DSR --> ITER
 
   DSW --> WRITE
 
   DSW --> DELETE
 
   DSW --> MANAGE
    
 
   DS --> FILE
 
   DS --> MMAP
 
   DS --> INDEXER
 
   DS --> TAIL
    
 
   WRITE --> FILE
 
   WRITE --> INDEXER
 
   WRITE --> TAIL
 
   READ --> MMAP
 
   READ --> INDEXER
 
   DELETE --> FILE
 
   DELETE --> INDEXER

DataStore Struct

The DataStore struct is the primary interface for interacting with the storage engine. It encapsulates file I/O, memory mapping, key indexing, and concurrency control.

Core Fields

Sources : src/storage_engine/data_store.rs:26-33

Creation Methods

Sources : src/storage_engine/data_store.rs:84-117 src/storage_engine/data_store.rs:141-144

Opening Storage

Sources : src/storage_engine/data_store.rs:84-117 src/storage_engine/data_store.rs:141-144 src/storage_engine/data_store.rs:53-64

Write Operations

Write operations are defined by the DataStoreWriter trait and implemented for DataStore. All write methods return Result<u64> where the u64 is the new tail offset after writing.

Sources : src/storage_engine/data_store.rs:752-939

graph TB
    subgraph "Write API Methods"
        W1["write(key, payload)"]
W2["batch_write(entries)"]
W3["write_stream(key, reader)"]
W4["write_with_key_hash(hash, payload)"]
W5["batch_write_with_key_hashes(entries)"]
W6["write_stream_with_key_hash(hash, reader)"]
end
    
    subgraph "Internal Write Path"
        LOCK["Acquire RwLock<File>"]
HASH["compute_hash()
or\ncompute_hash_batch()"]
ALIGN["Calculate prepad_len()"]
BUFFER["Buffer construction"]
SIMD["simd_copy()
for payload"]
META["EntryMetadata serialization"]
FLUSH["file.flush()"]
REINDEX["reindex()"]
end
    
 
   W1 --> HASH
 
   W2 --> HASH
 
   W3 --> HASH
 
   HASH --> W4
 
   HASH --> W5
 
   HASH --> W6
    
 
   W4 --> LOCK
 
   W5 --> LOCK
 
   W6 --> LOCK
    
 
   LOCK --> ALIGN
 
   ALIGN --> BUFFER
 
   BUFFER --> SIMD
 
   SIMD --> META
 
   META --> FLUSH
 
   FLUSH --> REINDEX
    
 
   REINDEX --> MMAP_UPDATE["Update mmap Arc"]
REINDEX --> INDEX_UPDATE["Update KeyIndexer"]
REINDEX --> TAIL_UPDATE["Update AtomicU64"]

Single Entry Write

Method : write(key: &[u8], payload: &[u8]) -> Result<u64>

Writes a single key-value pair atomically. The write is immediately flushed to disk.

Implementation details :

• Computes XXH3 hash of key using compute_hash()
• Delegates to write_with_key_hash()
• Internally uses batch_write_with_key_hashes() with single entry
• Calculates 64-byte alignment padding via prepad_len()
• Uses simd_copy() for payload transfer
• Appends EntryMetadata (20 bytes)
• Calls reindex() to update mmap and index

Sources : src/storage_engine/data_store.rs:827-834

Batch Write

Method : batch_write(entries: &[(&[u8], &[u8])]) -> Result<u64>

Writes multiple key-value pairs in a single locked operation. Reduces disk I/O overhead by buffering all entries and flushing once at the end.

Process :

1. Computes hashes for all keys via compute_hash_batch()
2. Acquires write lock once for entire batch
3. Builds buffer with aligned entries
4. Writes buffer to file with single write_all()
5. Updates index with all new mappings atomically

Sources : src/storage_engine/data_store.rs:838-843 src/storage_engine/data_store.rs:847-939

Streaming Write

Method : write_stream<R: Read>(key: &[u8], reader: &mut R) -> Result<u64>

Writes data from a Read source without requiring full in-memory allocation. Suitable for large payloads that exceed available memory.

Characteristics :

• Uses fixed 8KB buffer (WRITE_STREAM_BUFFER_SIZE)
• Reads chunks incrementally from source
• Computes CRC32C checksum while streaming
• Validates that payload is non-empty and not null-only
• Immediately flushes after completion

Sources : src/storage_engine/data_store.rs:753-825

Pre-hashed Write Methods

For performance optimization when keys are reused, the API provides methods accepting pre-computed hashes:

These skip the hashing step and proceed directly to storage operations.

Sources : src/storage_engine/data_store.rs:832-834 src/storage_engine/data_store.rs:847-939 src/storage_engine/data_store.rs:758-825

graph TB
    subgraph "Read API Methods"
        R1["read(key)"]
R2["batch_read(keys)"]
R3["read_last_entry()"]
R4["read_with_key_hash(hash)"]
R5["batch_read_hashed_keys(hashes)"]
R6["read_metadata(key)"]
R7["exists(key)"]
end
    
    subgraph "Internal Read Path"
        HASH_KEY["compute_hash()
or\ncompute_hash_batch()"]
INDEXER_READ["key_indexer.read()"]
MMAP_CLONE["get_mmap_arc()"]
UNPACK["KeyIndexer::unpack(packed)"]
TAG_CHECK["Verify 16-bit tag"]
BOUNDS["Bounds checking"]
PREPAD["Derive entry_start from\nprev_offset + prepad_len()"]
HANDLE["Construct EntryHandle"]
end
    
 
   R1 --> HASH_KEY
 
   R2 --> HASH_KEY
 
   HASH_KEY --> R4
 
   HASH_KEY --> R5
    
 
   R4 --> INDEXER_READ
 
   R5 --> INDEXER_READ
 
   R7 --> R1
    
 
   INDEXER_READ --> MMAP_CLONE
 
   MMAP_CLONE --> UNPACK
 
   UNPACK --> TAG_CHECK
 
   TAG_CHECK --> BOUNDS
 
   BOUNDS --> PREPAD
 
   PREPAD --> HANDLE
    
 
   R3 --> MMAP_CLONE
 
   R6 --> R1

Read Operations

Read operations are defined by the DataStoreReader trait. All reads are zero-copy when possible, returning EntryHandle references to memory-mapped regions.

Sources : src/storage_engine/data_store.rs:1027-1182

Single Entry Read

Method : read(key: &[u8]) -> Result<Option<EntryHandle>>

Retrieves a single entry by key. Returns None if key does not exist or is deleted (tombstone).

Implementation :

• Computes key hash via compute_hash()
• Acquires read lock on key_indexer
• Looks up packed (tag, offset) value
• Verifies 16-bit tag to detect hash collisions
• Derives entry boundaries from prev_offset and prepad_len()
• Returns EntryHandle with zero-copy access to payload

Sources : src/storage_engine/data_store.rs:1040-1049 src/storage_engine/data_store.rs:501-565

Batch Read

Method : batch_read(keys: &[&[u8]]) -> Result<Vec<Option<EntryHandle>>>

Reads multiple entries in a single index lock acquisition. More efficient than individual reads when processing multiple keys.

Process :

1. Computes all key hashes via compute_hash_batch()
2. Acquires single read lock on indexer
3. Performs lookup for each hash
4. Verifies tags for collision detection
5. Returns vector preserving input order

Sources : src/storage_engine/data_store.rs:1105-1109 src/storage_engine/data_store.rs:1111-1158

Read Last Entry

Method : read_last_entry() -> Result<Option<EntryHandle>>

Retrieves the most recently written entry without requiring a key lookup. Uses tail_offset to locate the last metadata block.

Use case : Useful for sequential processing or determining the latest state.

Sources : src/storage_engine/data_store.rs:1061-1103

Metadata Read

Method : read_metadata(key: &[u8]) -> Result<Option<EntryMetadata>>

Retrieves only the metadata (key hash, previous offset, checksum) without accessing the payload. More efficient when only metadata is needed.

Sources : src/storage_engine/data_store.rs:1160-1162

Existence Check

Method : exists(key: &[u8]) -> Result<bool>

Checks if a key exists without retrieving the full entry. Lightweight operation that only performs index lookup and tag verification.

Sources : src/storage_engine/data_store.rs:1030-1032

Read Operations Summary

Sources : src/storage_engine/data_store.rs:1027-1182

graph LR
 
   DELETE_API["delete(key)\nbatch_delete(keys)"] --> HASH["compute_hash_batch()"]
HASH --> CHECK_EXISTS["Filter existing keys\nvia key_indexer.read()"]
CHECK_EXISTS --> TOMBSTONE["Create (hash, NULL_BYTE)\npairs"]
TOMBSTONE --> BATCH_WRITE["batch_write_with_key_hashes()\nwith allow_null_bytes=true"]
BATCH_WRITE --> UPDATE_INDEX["reindex()
removes\nkeys from index"]

Delete Operations

Delete operations write tombstone entries (single null byte + metadata) to mark keys as deleted. The append-only model means deletions do not reclaim space until compaction.

Sources : src/storage_engine/data_store.rs:986-1024

Single Delete

Method : delete(key: &[u8]) -> Result<u64>

Deletes a single key by writing a tombstone entry. Internally delegates to batch_delete() with a single key.

Sources : src/storage_engine/data_store.rs:986-988

Batch Delete

Method : batch_delete(keys: &[&[u8]]) -> Result<u64>

Deletes multiple keys in a single operation. Optimized to skip keys that don’t exist, avoiding unnecessary tombstone writes.

Process :

1. Hashes all keys via compute_hash_batch()
2. Filters to only keys present in index
3. Constructs tombstone entries (NULL_BYTE + metadata)
4. Calls batch_write_with_key_hashes() with allow_null_bytes=true
5. Index updated to remove deleted keys

Sources : src/storage_engine/data_store.rs:990-1024

Pre-hashed Delete

Method : batch_delete_key_hashes(prehashed_keys: &[u64]) -> Result<u64>

Deletes keys using pre-computed hashes. Useful when hashes are already available from previous operations.

Sources : src/storage_engine/data_store.rs:995-1024

Entry Management Operations

These operations combine read, write, and delete to provide higher-level functionality for managing entries across storage instances.

Rename

Method : rename(old_key: &[u8], new_key: &[u8]) -> Result<u64>

Renames a key by:

1. Reading the entry at old_key
2. Creating an EntryStream from it
3. Writing to new_key via write_stream()
4. Deleting old_key

Constraint : old_key must exist and must differ from new_key.

Sources : src/storage_engine/data_store.rs:941-958

Copy

Method : copy(key: &[u8], target: &DataStore) -> Result<u64>

Copies an entry from the current storage to a different DataStore instance. The source entry remains unchanged.

Process :

1. Reads entry from source
2. Extracts payload and metadata
3. Writes to target using write_stream_with_key_hash()
4. Preserves original key hash

Constraint : Source and target must be different storage files.

Sources : src/storage_engine/data_store.rs:960-979 src/storage_engine/data_store.rs:587-590

Transfer

Method : transfer(key: &[u8], target: &DataStore) -> Result<u64>

Moves an entry from the current storage to a different instance by copying then deleting from source.

Equivalent to : copy() followed by delete()

Sources : src/storage_engine/data_store.rs:981-984

Entry Management Summary

Sources : src/storage_engine/data_store.rs:941-984

graph TB
    subgraph "Iteration Methods"
        ITER_OWNED["into_iter()\n(consumes DataStore)"]
ITER_REF["iter_entries()\n(borrows DataStore)"]
PAR_ITER["par_iter_entries()\n(parallel, requires 'parallel' feature)"]
end
    
    subgraph "EntryIterator Implementation"
        CURSOR["cursor: u64 = tail_offset"]
SEEN_KEYS["seen_keys: HashSet<u64>"]
NEXT["next()
method"]
METADATA["Read EntryMetadata"]
PREPAD_CALC["Derive entry_start from\nprev_offset + prepad_len()"]
SKIP_DUPE["Skip if key_hash in seen_keys"]
SKIP_TOMB["Skip if entry is NULL_BYTE"]
EMIT["Emit EntryHandle"]
end
    
    subgraph "Parallel Iterator"
        COLLECT["Collect key_indexer offsets"]
PAR_MAP["Rayon par_iter()"]
FILTER_MAP["filter_map constructs\nEntryHandle per thread"]
end
    
 
   ITER_OWNED --> ITER_REF
 
   ITER_REF --> CURSOR
 
   ITER_REF --> SEEN_KEYS
    
 
   CURSOR --> NEXT
 
   NEXT --> METADATA
 
   METADATA --> PREPAD_CALC
 
   PREPAD_CALC --> SKIP_DUPE
 
   SKIP_DUPE --> SKIP_TOMB
 
   SKIP_TOMB --> EMIT
    
 
   PAR_ITER --> COLLECT
 
   COLLECT --> PAR_MAP
 
   PAR_MAP --> FILTER_MAP

Iteration and Traversal

The DataStore provides multiple methods for iterating over all valid entries in the storage.

Sources : src/storage_engine/data_store.rs:269-361 src/storage_engine/entry_iterator.rs:8-127

Sequential Iteration

Method : iter_entries() -> EntryIterator

Returns an iterator that traverses all valid entries sequentially. The iterator:

• Starts at tail_offset and walks backward via prev_offset chain
• Tracks seen key hashes to ensure only latest versions are returned
• Filters out tombstone entries automatically
• Returns EntryHandle objects with zero-copy access

Sources : src/storage_engine/data_store.rs:276-280 src/storage_engine/entry_iterator.rs:41-47

Consuming Iteration

Trait : impl IntoIterator for DataStore

Allows consuming a DataStore instance to produce an iterator:

Internally delegates to iter_entries().

Sources : src/storage_engine/data_store.rs:44-50

Parallel Iteration

Method : par_iter_entries() -> impl ParallelIterator<Item = EntryHandle>

Feature gate : Requires parallel feature flag.

Provides Rayon-powered parallel iteration for high-throughput processing on multi-core systems.

Implementation strategy :

1. Acquires read lock on key_indexer briefly
2. Collects all packed offset values into a Vec<u64>
3. Releases lock immediately
4. Creates parallel iterator over collected offsets
5. Constructs EntryHandle objects in parallel threads

Performance : Ideal for bulk operations like analytics, caching, or transformation pipelines.

Sources : src/storage_engine/data_store.rs:296-361

Iteration Comparison

Sources : src/storage_engine/data_store.rs:44-50 src/storage_engine/data_store.rs:276-280 src/storage_engine/data_store.rs:296-361

Utility and Maintenance Methods

File Information

Sources : src/storage_engine/data_store.rs:1164-1181 src/storage_engine/data_store.rs:265-267

Compaction

Method : compact(&mut self) -> Result<()>

Reclaims disk space by creating a new storage file containing only the latest version of each key. Tombstone entries are excluded.

Process :

1. Creates temporary backup file with .bk extension
2. Iterates through iter_entries() (which returns only latest versions)
3. Copies each entry via copy_handle()
4. Swaps temporary file with original via std::fs::rename()

Thread safety warning : Should only be called when no other threads are accessing the storage. The &mut self requirement prevents concurrent mutations but does not prevent reads if the instance is wrapped in Arc<DataStore>.

Sources : src/storage_engine/data_store.rs:706-749

Compaction Estimation

Method : estimate_compaction_savings() -> u64

Calculates potential space savings from compaction without performing the operation. Returns the difference between total file size and the size needed for unique entries only.

Sources : src/storage_engine/data_store.rs:605-616

Internal Support Methods

These methods support the public API but are not directly exposed to users.

Reindexing

Method : reindex()

Called after every write operation to:

1. Re-map the file via init_mmap() to include new data
2. Update key_indexer with new key-to-offset mappings
3. Update tail_offset atomically

Acquires locks on both mmap and key_indexer to ensure consistency.

Sources : src/storage_engine/data_store.rs:224-259

Entry Context Reading

Method : read_entry_with_context()

Internal helper centralizing read logic for both read() and batch_read(). Parameters include the key hash, mmap reference, and indexer guard. Performs:

• Index lookup
• Tag verification (if original key provided)
• Bounds checking
• Tombstone detection
• EntryHandle construction

Sources : src/storage_engine/data_store.rs:501-565

Recovery Chain Validation

Method : recover_valid_chain()

Called during open() to validate storage file integrity. Walks backward through the file following prev_offset chains until reaching offset 0. Truncates file if incomplete write detected.

Sources : src/storage_engine/data_store.rs:383-482

Alignment Calculation

Method : prepad_len(offset: u64) -> usize

Computes padding bytes required to align offset to PAYLOAD_ALIGNMENT (64 bytes). Uses bitwise operations for efficiency:

pad = (A - (offset % A)) & (A - 1)

Sources : src/storage_engine/data_store.rs:669-673

API Patterns and Conventions

Return Value Pattern

Most write operations return Result<u64> where the u64 is the new tail_offset after the operation. This allows chaining operations or validating expected file growth.

Error Handling

The API uses std::io::Result<T> consistently. Common error cases:

• InvalidInput: Empty payloads, null-byte-only payloads, invalid rename
• NotFound: Key does not exist (for operations requiring existing keys)
• Lock poisoning errors (converted to std::io::Error)

Pre-hashed Key Methods

Many operations offer both standard and pre-hashed variants:

• Standard: write(key, payload) - computes hash internally
• Pre-hashed: write_with_key_hash(hash, payload) - uses provided hash

Pre-hashed methods enable optimization when keys are reused across multiple operations.

Batch Operations Benefit

Batch methods acquire locks once for the entire batch, significantly reducing overhead:

• batch_write() vs multiple write() calls
• batch_read() vs multiple read() calls
• batch_delete() vs multiple delete() calls

Sources : src/storage_engine/data_store.rs:752-1182

Trait Implementations

DataStoreReader Trait

Defines read-only operations. Associated type EntryHandleType allows flexibility in handle implementation.

Implementors : DataStore

Key methods : read(), batch_read(), read_last_entry(), read_metadata(), exists(), len(), is_empty(), file_size()

Sources : src/storage_engine/traits.rs

DataStoreWriter Trait

Defines mutating operations. All methods take &self (not &mut self) because internal synchronization via RwLock enables safe concurrent access.

Implementors : DataStore

Key methods : write(), batch_write(), write_stream(), delete(), batch_delete(), rename(), copy(), transfer()

Sources : src/storage_engine/traits.rs

From Trait

Convenience constructor that panics on failure:

Sources : src/storage_engine/data_store.rs:53-64

IntoIterator Trait

Allows consuming iteration over storage entries. Returns EntryIterator as the iterator type.

Sources : src/storage_engine/data_store.rs:44-50

Dismiss

Refresh this wiki

Enter email to refresh

On this page

• DataStore API
• API Architecture
• DataStore Struct
• Core Fields
• Creation Methods
• Opening Storage
• Write Operations
• Single Entry Write
• Batch Write
• Streaming Write
• Pre-hashed Write Methods
• Read Operations
• Single Entry Read
• Batch Read
• Read Last Entry
• Metadata Read
• Existence Check
• Read Operations Summary
• Delete Operations
• Single Delete
• Batch Delete
• Pre-hashed Delete
• Entry Management Operations
• Rename
• Copy
• Transfer
• Entry Management Summary
• Iteration and Traversal
• Sequential Iteration
• Consuming Iteration
• Parallel Iteration
• Iteration Comparison
• Utility and Maintenance Methods
• File Information
• Compaction
• Compaction Estimation
• Internal Support Methods
• Reindexing
• Entry Context Reading
• Recovery Chain Validation
• Alignment Calculation
• API Patterns and Conventions
• Return Value Pattern
• Error Handling
• Pre-hashed Key Methods
• Batch Operations Benefit
• Trait Implementations
• DataStoreReader Trait
• DataStoreWriter Trait
• From Trait
• IntoIterator Trait

Entry Structure and Metadata

Loading…

Entry Structure and Metadata

Relevant source files

• README.md
• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/constants.rs
• simd-r-drive-entry-handle/src/entry_metadata.rs
• simd-r-drive-entry-handle/src/lib.rs
• src/utils/align_or_copy.rs

Purpose and Scope

This document details the on-disk binary layout of entries in the SIMD R Drive storage engine. It covers the structure of aligned entries, tombstones, metadata fields, and the alignment strategy that enables zero-copy access.

For information about how entries are read and accessed in memory, see Memory Management and Zero-Copy Access. For details on the validation chain and recovery mechanisms, see Storage Architecture.

On-Disk Entry Layout Overview

Every entry written to the storage file consists of three components:

1. Pre-Pad Bytes (optional, 0-63 bytes) - Zero bytes inserted to ensure the payload starts at a 64-byte boundary
2. Payload - Variable-length binary data
3. Metadata - Fixed 20-byte structure containing key hash, previous offset, and checksum

The exception is tombstones (deletion markers), which use a minimal 1-byte payload with no pre-padding.

Sources: README.md:104-137 simd-r-drive-entry-handle/src/entry_metadata.rs:9-37

Aligned Entry Structure

Entry Layout Table

Where:

• pad = (A - (prev_tail % A)) & (A - 1), with A = PAYLOAD_ALIGNMENT (64 bytes)
• The next entry starts at offset N + 20

Aligned Entry Structure Diagram

Sources: README.md:112-137 simd-r-drive-entry-handle/src/entry_metadata.rs:11-23

Tombstone Structure

Tombstones are special deletion markers that do not require payload alignment. They consist of a single zero byte followed by the standard 20-byte metadata structure.

Tombstone Layout Table

Tombstone Structure Diagram

Sources: README.md:126-131 simd-r-drive-entry-handle/src/entry_metadata.rs:25-30

EntryMetadata Structure

The EntryMetadata struct represents the fixed 20-byte metadata block that follows every payload. It is defined in #[repr(C)] layout to ensure consistent binary representation.

graph TB
    subgraph EntryMetadataStruct["EntryMetadata struct"]
field1["key_hash: u64\n8 bytes\nXXH3_64 hash"]
field2["prev_offset: u64\n8 bytes\nbackward chain link"]
field3["checksum: [u8; 4]\n4 bytes\nCRC32C payload checksum"]
end
    
 
   field1 --> field2
 
   field2 --> field3
    
    note4["Serialized at offset N\nfollowing payload"]
note5["Total: METADATA_SIZE = 20"]
field1 -.-> note4
 
   field3 -.-> note5

Metadata Fields

Field Descriptions

key_hash: u64 (8 bytes, offset N .. N+8)

• 64-bit XXH3 hash of the key
• Used by KeyIndexer for O(1) lookups
• Combined with a tag for collision detection
• Hardware-accelerated via SSE2/AVX2/NEON

prev_offset: u64 (8 bytes, offset N+8 .. N+16)

• Absolute file offset of the previous entry for this key
• Forms a backward-linked chain for version history
• Set to 0 for the first entry of a key
• Used during chain validation and recovery

checksum: [u8; 4] (4 bytes, offset N+16 .. N+20)

• CRC32C checksum of the payload
• Provides fast integrity verification
• Not cryptographically secure
• Used during recovery to detect corruption

Serialization and Deserialization

The EntryMetadata struct provides methods for converting to/from bytes:

• serialize() -> [u8; 20] - Converts metadata to byte array using little-endian encoding
• deserialize(data:&[u8]) -> Self - Reconstructs metadata from byte slice

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:44-113 README.md:114-120

Pre-Padding and Alignment Strategy

Alignment Purpose

All non-tombstone payloads start at a 64-byte aligned address. This alignment ensures:

• Cache-line efficiency - Matches typical CPU cache line size
• SIMD optimization - Enables full-speed AVX2/AVX-512/NEON operations
• Zero-copy typed views - Allows safe reinterpretation as typed slices (&[u16], &[u32], etc.)

graph TD
    Start["Calculate padding needed"]
GetPrevTail["prev_tail = last written offset"]
CalcPad["pad = (PAYLOAD_ALIGNMENT - (prev_tail % PAYLOAD_ALIGNMENT))\n& (PAYLOAD_ALIGNMENT - 1)"]
CheckPad{"pad > 0?"}
WritePad["Write pad zero bytes"]
WritePayload["Write payload at aligned offset"]
Start --> GetPrevTail
 
   GetPrevTail --> CalcPad
 
   CalcPad --> CheckPad
 
   CheckPad -->|Yes| WritePad
 
   CheckPad -->|No| WritePayload
 
   WritePad --> WritePayload

The alignment is configured via PAYLOAD_ALIGNMENT constant (64 bytes as of version 0.15.0).

Pre-Padding Calculation

The formula pad = (A - (prev_tail % A)) & (A - 1) where A = PAYLOAD_ALIGNMENT ensures:

• If prev_tail is already aligned, pad = 0
• Otherwise, pad equals the bytes needed to reach the next aligned boundary
• Maximum padding is A - 1 bytes (63 bytes for 64-byte alignment)

Constants

The alignment is defined in simd-r-drive-entry-handle/src/constants.rs:1-20:

Sources: README.md:51-59 simd-r-drive-entry-handle/src/entry_metadata.rs:22-23 CHANGELOG.md:25-51

Backward Chain Formation

Chain Structure

Each entry’s prev_offset field creates a backward-linked chain that tracks the version history for a given key. This chain is essential for:

• Recovery and validation on file open
• Detecting incomplete writes
• Rebuilding the index

Chain Properties

• Most recent entry is at the end of the file (highest offset)
• Chain traversal moves backward from tail toward offset 0
• First entry for a key has prev_offset = 0
• Valid chain can be walked all the way back to byte 0 without gaps
• Broken chain indicates corruption or incomplete write

Usage in Recovery

During file open, the system:

1. Scans backward from EOF reading metadata
2. Follows prev_offset links to validate chain continuity
3. Verifies checksums at each step
4. Truncates file if corruption is detected
5. Scans forward to rebuild the index

Sources: README.md:139-147 simd-r-drive-entry-handle/src/entry_metadata.rs:41-43

Entry Type Comparison

Aligned Entry vs. Tombstone

When Tombstones Are Used

Tombstones mark key deletions while maintaining chain integrity. They:

• Preserve the backward chain via prev_offset
• Use minimal space (no alignment overhead)
• Are detected during reads and filtered out
• Enable recovery to skip deleted entries

Sources: README.md:112-137 simd-r-drive-entry-handle/src/entry_metadata.rs:9-37

Metadata Serialization Format

Binary Layout in File

Constants for Range Indexing

The simd-r-drive-entry-handle/src/constants.rs:1-20 file defines range constants for metadata field access:

• KEY_HASH_RANGE = 0..8
• PREV_OFFSET_RANGE = 8..16
• CHECKSUM_RANGE = 16..20
• METADATA_SIZE = 20

These ranges are used in EntryMetadata::serialize() and deserialize() methods.

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:62-112

Alignment Evolution and Migration

Version History

v0.14.0-alpha and earlier: Used 16-byte alignment (PAYLOAD_ALIGNMENT = 16)

v0.15.0-alpha onwards: Changed to 64-byte alignment (PAYLOAD_ALIGNMENT = 64)

This change was made to:

• Ensure full cache-line alignment
• Support AVX-512 and future SIMD extensions
• Improve zero-copy performance across modern hardware

Migration Considerations

Storage files created with different alignment values are not compatible :

• v0.14.x readers cannot correctly parse v0.15.x stores
• v0.15.x readers may misinterpret v0.14.x padding

To migrate between versions:

1. Read all entries using the old version binary
2. Write entries to a new store using the new version binary
3. Replace the old file after verification

In multi-service environments, deploy reader upgrades before writer upgrades to avoid mixed-version issues.

Sources: CHANGELOG.md:25-82 README.md:51-59

Debug Assertions for Alignment

Runtime Validation

The codebase includes debug-only alignment assertions that validate both pointer and offset alignment:

debug_assert_aligned(ptr: *const u8, align: usize) - Validates pointer alignment

• Active in debug and test builds
• Zero cost in release/bench builds
• Ensures buffer base address is properly aligned

debug_assert_aligned_offset(off: u64) - Validates file offset alignment

• Checks that derived payload start offset is at PAYLOAD_ALIGNMENT boundary
• Used during entry handle creation
• Defined in simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-88

These assertions help catch alignment issues during development without imposing runtime overhead in production.

Sources: simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-88 CHANGELOG.md:33-41

Summary

The SIMD R Drive entry structure uses a carefully designed binary layout that balances efficiency, integrity, and flexibility:

• Fixed 64-byte alignment ensures cache-friendly, SIMD-optimized access
• 20-byte fixed metadata provides fast integrity checks and chain traversal
• Variable pre-padding maintains alignment without complex calculations
• Minimal tombstones mark deletions efficiently
• Backward-linked chain enables robust recovery and validation

This design enables zero-copy reads, high write throughput, and automatic crash recovery while maintaining a simple, append-only storage model.

Dismiss

Refresh this wiki

Enter email to refresh

Memory Management and Zero-Copy Access

Loading…

Memory Management and Zero-Copy Access

Relevant source files

• README.md
• src/lib.rs
• src/storage_engine.rs
• src/storage_engine/data_store.rs
• src/storage_engine/entry_iterator.rs

Purpose and Scope

This document describes the memory management strategy used by SIMD R Drive’s core storage engine, focusing on memory-mapped file access and zero-copy read patterns. It covers the memmap2 crate integration, the Arc<Mmap> shared reference architecture, and how EntryHandle provides zero-copy views into stored data.

For details on entry structure and metadata organization, see Entry Structure and Metadata. For concurrency mechanisms that protect memory-mapped access, see Concurrency and Thread Safety.

Memory-Mapped File Architecture

Core mmap Integration

The storage engine uses the memmap2 crate to memory-map the entire storage file, allowing direct access to file contents without explicit read system calls. The memory-mapped region is managed through a layered reference-counting structure:

Arc<Mutex<Arc<Mmap>>>
    │
    ├─ Outer Arc: Shared across DataStore clones
    ├─ Mutex: Serializes remapping operations
    └─ Inner Arc<Mmap>: Shared across readers

Sources: src/storage_engine/data_store.rs:1-30

DataStore mmap Field Structure

The DataStore struct maintains the memory map using nested Arc wrappers:

This structure enables:

• Multiple readers to hold Arc<Mmap> references simultaneously
• Safe remapping after writes without invalidating existing reader references
• Lock-free reads once an Arc<Mmap> is obtained

Sources: src/storage_engine/data_store.rs:26-33 README.md:174-183

Memory Map Initialization and Remapping

graph TB
    Open["DataStore::open()"]
OpenFile["open_file_in_append_mode()"]
InitMmap["init_mmap()"]
UnsafeMap["unsafe memmap2::MmapOptions::new().map()"]
ArcWrap["Arc::new(mmap)"]
Open --> OpenFile
 
   OpenFile --> InitMmap
 
   InitMmap --> UnsafeMap
 
   UnsafeMap --> ArcWrap
    
    OpenFile -.returns.-> File
    UnsafeMap -.returns.-> Mmap
    ArcWrap -.stored in.-> DataStore

Initial Mapping

When a DataStore is opened, the storage file is memory-mapped using unsafe code that delegates to the OS:

Diagram: Initial memory map creation flow

The init_mmap function wraps the unsafe memmap2::MmapOptions::new().map() call, which asks the OS to map the file into the process address space. The resulting Mmap is immediately wrapped in an Arc for shared access.

Sources: src/storage_engine/data_store.rs:172-174 src/storage_engine/data_store.rs:84-117

sequenceDiagram
    participant Writer as "Write Operation"
    participant File as "RwLock<BufWriter<File>>"
    participant Reindex as "reindex()"
    participant MmapMutex as "Mutex<Arc<Mmap>>"
    participant Indexer as "RwLock<KeyIndexer>"
    
    Writer->>File: Acquire write lock
    Writer->>File: Append data + metadata
    Writer->>File: flush()
    Writer->>Reindex: reindex(&write_guard, offsets, tail)
    
    Reindex->>File: init_mmap(&write_guard)
    Note over Reindex,File: Create new Mmap from flushed file
    
    Reindex->>MmapMutex: lock()
    Reindex->>MmapMutex: *guard = Arc::new(new_mmap)
    Note over MmapMutex: Old Arc<Mmap> still valid for readers
    
    Reindex->>Indexer: write().insert(key_hash, offset)
    Reindex->>Indexer: Release lock
    
    Reindex->>MmapMutex: Release lock
    
    Note over Writer: New reads see updated mmap

Remapping After Writes

After write operations extend the file, the memory map must be refreshed to make new data visible. The reindex method handles this critical operation:

Diagram: Memory map remapping sequence during writes

The reindex method performs three synchronized updates:

1. Creates a new Mmap from the extended file
2. Atomically replaces the Arc<Mmap> in the mutex
3. Updates the key indexer with new offsets

Sources: src/storage_engine/data_store.rs:224-259 src/storage_engine/data_store.rs:176-186

Zero-Copy Read Patterns

graph LR
    subgraph "DataStore"
        MmapContainer["Mutex<Arc<Mmap>>"]
end
    
    subgraph "EntryHandle"
        MmapRef["Arc<Mmap>"]
Range["range: Range<usize>"]
Metadata["metadata: EntryMetadata"]
end
    
    subgraph "User Code"
        Slice["&[u8] payload slice"]
end
    
 
   MmapContainer -->|get_mmap_arc| MmapRef
 
   MmapRef -->|&mmap[range]| Slice
    Range -.defines region.-> Slice
    
    Note1["Zero-copy: slice points\ndirectly into mmap"]
Slice -.-> Note1

EntryHandle Architecture

EntryHandle is the primary abstraction for zero-copy reads. It holds an Arc<Mmap> reference and a byte range, providing direct slice access without copying:

Diagram: EntryHandle zero-copy architecture

When EntryHandle::as_slice() is called, it returns &self.mmap_arc[self.range.clone()], which is a direct reference into the memory-mapped region. No data is copied; the slice is a view into the OS page cache.

Sources: [simd-r-drive-entry-handle crate](https://github.com/jzombie/rust-simd-r-drive/blob/0299fd5d/simd-r-drive-entry-handle crate) src/storage_engine/data_store.rs:560-565

graph TB
    Read["read(key)"]
ComputeHash["compute_hash(key)"]
GetMmap["get_mmap_arc()"]
LockIndex["key_indexer.read()"]
ReadContext["read_entry_with_context()"]
IndexLookup["key_indexer.get_packed(key_hash)"]
Unpack["KeyIndexer::unpack(packed)"]
CreateHandle["EntryHandle { mmap_arc, range, metadata }"]
AsSlice["entry.as_slice()"]
DirectRef["&mmap[range]"]
Read --> ComputeHash
 
   Read --> GetMmap
 
   Read --> LockIndex
 
   ComputeHash --> ReadContext
 
   GetMmap --> ReadContext
 
   LockIndex --> ReadContext
 
   ReadContext --> IndexLookup
 
   IndexLookup --> Unpack
 
   Unpack --> CreateHandle
 
   CreateHandle --> AsSlice
 
   AsSlice --> DirectRef
    
    DirectRef -.zero-copy.-> OSPageCache["OS Page Cache"]

Read Operation Flow

The zero-copy read flow demonstrates how data moves from disk to user code without intermediate buffers:

Diagram: Zero-copy read operation flow from key lookup to slice access

Key points:

• get_mmap_arc() obtains an Arc<Mmap> clone (cheap atomic increment)
• Index lookup finds the file offset
• EntryHandle is constructed with the Arc<Mmap> and byte range
• as_slice() returns a reference directly into the mapped memory

Sources: src/storage_engine/data_store.rs:1040-1049 src/storage_engine/data_store.rs:502-565 src/storage_engine/data_store.rs:658-663

Shared Access with Arc

Thread-Safe Reference Counting

The Arc<Mmap> enables multiple threads to hold references to the same memory-mapped region simultaneously. Each clone increments an atomic reference count:

When a writer remaps the file, it replaces the Arc<Mmap> inside the mutex. Old Arc<Mmap> references remain valid until all readers drop them, at which point the OS automatically unmaps the old region.

Sources: src/storage_engine/data_store.rs:658-663 README.md:174-183

Clone Semantics in Iteration

EntryIterator demonstrates efficient Arc<Mmap> usage. The iterator holds one Arc<Mmap> and clones it for each EntryHandle it yields:

Diagram: Arc cloning pattern in EntryIterator

graph TB
    IterNew["EntryIterator::new(mmap_arc, tail)"]
IterField["EntryIterator { mmap: Arc<Mmap>, ... }"]
Next["next()
called"]
CreateHandle["EntryHandle { mmap_arc: Arc::clone(&self.mmap), ... }"]
UserCode["User processes EntryHandle"]
Drop["EntryHandle dropped"]
IterNew --> IterField
 
   IterField --> Next
 
   Next --> CreateHandle
    CreateHandle -.cheap clone.-> UserCode
 
   UserCode --> Drop
    Drop -.atomic decrement.-> RefCount["Reference count"]
Note["Iterator holds 1 Arc\nEach EntryHandle clones it\nAll point to same Mmap"]
IterField -.-> Note

This design allows the iterator and all yielded handles to coexist safely. The cloning overhead is minimal—just an atomic operation—while providing complete memory safety.

Sources: src/storage_engine/entry_iterator.rs:21-47 src/storage_engine/entry_iterator.rs:121-125

Memory Management Flow

graph TB
    subgraph "Initialization"
        OpenFile["open_file_in_append_mode()"]
InitMmap1["init_mmap(&file)"]
Recovery["recover_valid_chain()"]
ReinitMmap["Remap if truncation needed"]
BuildIndex["KeyIndexer::build()"]
StoreMmap["Store Arc<Mutex<Arc<Mmap>>>"]
end
    
    subgraph "Read Path"
        GetArc["get_mmap_arc()"]
ReadLock["key_indexer.read()"]
Lookup["Index lookup"]
ConstructHandle["EntryHandle { Arc::clone(mmap_arc), range, ... }"]
AsSlice["as_slice() → &mmap[range]"]
end
    
    subgraph "Write Path"
        WriteLock["file.write()"]
AppendData["Append payload + metadata"]
Flush["flush()"]
Reindex["reindex()"]
NewMmap["init_mmap() → new Mmap"]
SwapMmap["Mutex: *guard = Arc::new(new_mmap)"]
UpdateIndex["KeyIndexer: insert offsets"]
end
    
    subgraph "Iterator Path"
        IterCreate["iter_entries()"]
CloneMmap["get_mmap_arc()"]
IterNew["EntryIterator::new(mmap_arc, tail)"]
IterNext["next() → EntryHandle"]
end
    
 
   OpenFile --> InitMmap1
 
   InitMmap1 --> Recovery
 
   Recovery --> ReinitMmap
 
   ReinitMmap --> BuildIndex
 
   BuildIndex --> StoreMmap
    
    StoreMmap -.available for.-> GetArc
 
   GetArc --> ReadLock
 
   ReadLock --> Lookup
 
   Lookup --> ConstructHandle
 
   ConstructHandle --> AsSlice
    
    StoreMmap -.available for.-> WriteLock
 
   WriteLock --> AppendData
 
   AppendData --> Flush
 
   Flush --> Reindex
 
   Reindex --> NewMmap
 
   NewMmap --> SwapMmap
 
   SwapMmap --> UpdateIndex
    
    StoreMmap -.available for.-> IterCreate
 
   IterCreate --> CloneMmap
 
   CloneMmap --> IterNew
 
   IterNew --> IterNext

Complete Lifecycle

The following diagram maps the complete lifecycle of memory-mapped access, from initial file open through reads and writes to iterator cleanup:

Diagram: Complete memory management lifecycle

Sources: src/storage_engine/data_store.rs:84-117 src/storage_engine/data_store.rs:1040-1049 src/storage_engine/data_store.rs:752-825 src/storage_engine/data_store.rs:276-280

Code Entity Mapping

The following table maps high-level concepts to specific code entities:

Sources: src/storage_engine/data_store.rs:1-33 src/storage_engine/entry_iterator.rs:21-25

Safety Considerations

OS Page Cache Integration

The memory-mapped approach delegates memory management to the OS page cache:

Diagram: OS page cache interaction with memory-mapped region

Key benefits:

• Pages loaded on-demand (lazy loading)
• OS handles eviction when memory is tight
• Multiple processes can share the same page cache entries
• No explicit memory allocation in application code

Sources: README.md:43-50 README.md174

Large File Handling

The system is designed to handle datasets larger than available RAM. The memory mapping does not load the entire file into RAM:

Available RAM| Only accessed pages cached| OS loads pages on-demand

Available RAM| LRU page eviction active| Older pages evicted as needed

When iterating or reading, only the accessed byte ranges are loaded into physical memory. The OS automatically evicts least-recently-used pages under memory pressure.

Sources: README.md:45-50

Unsafe Code Boundaries

Memory mapping inherently requires unsafe code:

DataStore::init_mmap()
    └─> unsafe { memmap2::MmapOptions::new().map(file) }

The memmap2 crate provides safe abstractions over this unsafe operation, ensuring:

• The file descriptor remains valid while mapped
• The mapped region respects file size boundaries
• Concurrent modifications to the file (outside the mmap) are handled correctly

SIMD R Drive’s architecture ensures safety by:

• Never resizing the file while an mmap exists
• Remapping after writes extend the file
• Using Arc<Mmap> to prevent use-after-unmap bugs

Sources: src/storage_engine/data_store.rs:172-174 src/lib.rs:123-124

Thread Safety Guarantees

The nested Arc<Mutex<Arc<Mmap>>> structure provides these guarantees:

The key insight is that remapping creates a new Arc<Mmap> without invalidating existing references. Readers holding old Arc<Mmap> instances continue accessing the old mapping until they drop their references.

Sources: src/storage_engine/data_store.rs:26-33 README.md:174-183 README.md:196-206

Memory Pressure and Resource Management

Automatic Resource Cleanup

When memory pressure increases, the OS automatically evicts pages from the page cache. However, the Mmap object itself is small—it only holds file descriptor information and address space pointers. The actual memory is managed by the kernel.

Arc<Mmap> ensures that:

• The file is not unmapped while any thread holds a reference
• When the last Arc is dropped, the Mmap destructor unmaps the region
• The OS then reclaims the virtual address space

Sources: src/storage_engine/data_store.rs:658-663

Testing Hooks

For validation and testing, the system exposes mmap internals in debug builds:

These methods allow tests to verify zero-copy behavior by comparing pointer addresses and validating that slices point directly into the mapped region.

Sources: src/storage_engine/data_store.rs:631-656

Dismiss

Refresh this wiki

Enter email to refresh

Concurrency and Thread Safety

Loading…

Concurrency and Thread Safety

Relevant source files

• README.md
• benches/storage_benchmark.rs
• src/main.rs
• src/storage_engine/data_store.rs
• src/utils/format_bytes.rs
• tests/concurrency_tests.rs

Purpose and Scope

This document describes the concurrency model and thread safety guarantees of the SIMD R Drive storage engine. It covers the synchronization primitives used to enable safe multi-threaded access within a single process, including lock strategies for reads and writes, atomic operations, and memory map management.

For information about the core storage architecture and data structures, see Storage Architecture. For details on memory-mapped file usage, see Memory Management and Zero-Copy Access.

Key Limitation : The concurrency mechanisms described here apply only to single-process, multi-threaded environments. Multiple processes accessing the same storage file simultaneously are not supported and require external file locking mechanisms.

Concurrency Model Overview

The DataStore structure uses a combination of read-write locks, atomic operations, and mutexes to enable safe concurrent access across multiple threads while maintaining data consistency.

Diagram: DataStore Synchronization Architecture

graph TB
    subgraph "DataStore Synchronization Primitives"
        FILE["Arc<RwLock<BufWriter<File>>>\nfile"]
MMAP["Arc<Mutex<Arc<Mmap>>>\nmmap"]
TAIL["AtomicU64\ntail_offset"]
INDEX["Arc<RwLock<KeyIndexer>>\nkey_indexer"]
end
    
    subgraph "Write Operations"
        W_STREAM["write_stream"]
W_SINGLE["write"]
W_BATCH["batch_write"]
end
    
    subgraph "Read Operations"
        R_SINGLE["read"]
R_BATCH["batch_read"]
R_ITER["iter_entries"]
end
    
 
   W_STREAM --> FILE
 
   W_SINGLE --> FILE
 
   W_BATCH --> FILE
    
 
   W_STREAM --> TAIL
 
   W_SINGLE --> TAIL
 
   W_BATCH --> TAIL
    
    W_STREAM -.updates.-> MMAP
    W_SINGLE -.updates.-> MMAP
    W_BATCH -.updates.-> MMAP
    
    W_STREAM -.updates.-> INDEX
    W_SINGLE -.updates.-> INDEX
    W_BATCH -.updates.-> INDEX
    
 
   R_SINGLE --> INDEX
 
   R_BATCH --> INDEX
 
   R_ITER --> MMAP
    
 
   R_SINGLE --> MMAP
 
   R_BATCH --> MMAP

Sources: src/storage_engine/data_store.rs:27-33 README.md:172-183

Synchronization Primitives

DataStore Field Overview

The DataStore struct contains four primary fields that implement concurrency control:

Sources: src/storage_engine/data_store.rs:27-33

RwLock for File Writes

All write operations acquire an exclusive write lock on the file handle to prevent concurrent modifications.

Diagram: Write Lock Serialization

sequenceDiagram
    participant T1 as "Thread 1"
    participant T2 as "Thread 2"
    participant FILE as "RwLock<File>"
    
    T1->>FILE: write.lock() - acquire
    Note over T1,FILE: Thread 1 holds write lock
    T2->>FILE: write.lock() - blocks
    Note over T2: Thread 2 waits
    T1->>FILE: write data + flush
    T1->>FILE: release lock
    Note over FILE: Lock released
    T2->>FILE: acquire lock
    Note over T2,FILE: Thread 2 now writes
    T2->>FILE: write data + flush
    T2->>FILE: release lock

Write Lock Acquisition

Write operations acquire the lock at the start of the write process:

This pattern appears in:

• write_stream_with_key_hash: src/storage_engine/data_store.rs:759-762
• batch_write_with_key_hashes: src/storage_engine/data_store.rs:852-855

The write lock ensures that only one thread can append data to the file at any given time, preventing:

• Race conditions on file position
• Interleaved writes corrupting the append-only chain
• Inconsistent metadata ordering

Sources: src/storage_engine/data_store.rs:752-825 src/storage_engine/data_store.rs:847-945 README.md176

AtomicU64 for Tail Offset

The tail_offset field tracks the current end of the valid data in the storage file using atomic operations, enabling lock-free reads of the current file position.

Atomic Operations Used

Load Operation

Reads use Acquire ordering to ensure they see all previous writes:

Examples:

• iter_entries: src/storage_engine/data_store.rs278
• Write operations reading previous tail: src/storage_engine/data_store.rs763 src/storage_engine/data_store.rs858

Store Operation

Writes use Release ordering to ensure all previous writes are visible:

Location: src/storage_engine/data_store.rs256

This atomic coordination ensures that:

• Readers always see a consistent tail offset
• Writers update the tail only after data is flushed
• No locks are needed for reading the tail position

Sources: src/storage_engine/data_store.rs30 src/storage_engine/data_store.rs256 src/storage_engine/data_store.rs278 README.md182

graph LR
    subgraph "Memory Map Management"
        MUTEX["Mutex<Arc<Mmap>>"]
MMAP1["Arc<Mmap> v1"]
MMAP2["Arc<Mmap> v2"]
end
    
    subgraph "Readers"
        R1["Reader Thread 1"]
R2["Reader Thread 2"]
end
    
    subgraph "Writer"
        W["Writer Thread"]
end
    
    R1 -.clones.-> MMAP1
    R2 -.clones.-> MMAP1
 
   W -->|1. Lock mutex| MUTEX
 
   W -->|2. Create new| MMAP2
 
   W -->|3. Replace| MUTEX
 
   W -->|4. Release| MUTEX
    
    MMAP1 -.remains valid.-> R1
    MMAP1 -.remains valid.-> R2

Mutex for Memory Map

The memory-mapped file reference is protected by a Mutex<Arc<Mmap>> to prevent concurrent remapping during reads.

Diagram: Memory Map Arc Cloning Pattern

Accessing the Memory Map

Read operations clone the Arc<Mmap> to obtain a stable reference:

Source: src/storage_engine/data_store.rs:658-663

This pattern ensures:

• Readers hold a reference to a specific memory map version
• Writers can create a new memory map without invalidating existing readers
• The Arc reference counting prevents premature deallocation
• The mutex is held only briefly during the clone operation

Remapping After Writes

After writing and flushing data, the reindex function creates a new memory map:

Source: src/storage_engine/data_store.rs:231-255

Sources: src/storage_engine/data_store.rs29 src/storage_engine/data_store.rs:224-259 src/storage_engine/data_store.rs:658-663 README.md180

RwLock for Key Index

The KeyIndexer is protected by a read-write lock, allowing multiple concurrent readers but exclusive writers.

Read Access Pattern

Multiple threads can acquire read locks simultaneously for lookups:

Example: src/storage_engine/data_store.rs509

Write Access Pattern

Index updates require exclusive write access:

Source: src/storage_engine/data_store.rs:233-253

Parallel Iterator Lock Strategy

The parallel iterator minimizes lock holding time by collecting offsets first:

Source: src/storage_engine/data_store.rs:300-302

Sources: src/storage_engine/data_store.rs31 src/storage_engine/data_store.rs:233-253 src/storage_engine/data_store.rs:300-302 README.md178

sequenceDiagram
    participant R1 as "Reader 1"
    participant R2 as "Reader 2"
    participant R3 as "Reader 3"
    participant INDEX as "RwLock<KeyIndexer>"
    participant MMAP as "Arc<Mmap>"
    
    par Concurrent Reads
        R1->>INDEX: read().lock() - acquire
        R2->>INDEX: read().lock() - acquire
        R3->>INDEX: read().lock() - acquire
    end
    
    par Index Lookups
        R1->>INDEX: get_packed(key_hash_1)
        R2->>INDEX: get_packed(key_hash_2)
        R3->>INDEX: get_packed(key_hash_3)
    end
    
    Note over R1,R3: All readers release index lock
    
    par Zero-Copy Access
        R1->>MMAP: Access offset_1
        R2->>MMAP: Access offset_2
        R3->>MMAP: Access offset_3
    end
    
    Note over R1,MMAP: No locks during data access

Lock-Free Read Operations

Read operations achieve lock-free access through memory-mapped files and atomic operations.

Diagram: Concurrent Lock-Free Read Pattern

Zero-Copy Read Implementation

Once the offset is obtained from the index, data access is lock-free:

Source: src/storage_engine/data_store.rs:502-565

Benefits of Lock-Free Reads

1. No Read Contention : Multiple readers access different memory regions simultaneously
2. Zero-Copy : Data is accessed directly from the memory map without copying
3. Scalability : Read throughput scales linearly with CPU cores
4. Low Latency : No lock acquisition overhead after index lookup

Sources: README.md174 src/storage_engine/data_store.rs:502-565 tests/concurrency_tests.rs:163-229

graph TB
    subgraph "Write Operation Phases"
        ACQUIRE["1. Acquire RwLock<File>"]
LOAD["2. Load tail_offset\n(Atomic)"]
CALC["3. Calculate pre-padding"]
WRITE["4. Write payload + metadata"]
FLUSH["5. Flush to disk"]
REMAP["6. Remap file (Mutex)"]
UPDATE["7. Update index (RwLock)"]
STORE["8. Store new tail_offset\n(Atomic)"]
RELEASE["9. Release file lock"]
end
    
 
   ACQUIRE --> LOAD
 
   LOAD --> CALC
 
   CALC --> WRITE
 
   WRITE --> FLUSH
 
   FLUSH --> REMAP
 
   REMAP --> UPDATE
 
   UPDATE --> STORE
 
   STORE --> RELEASE

Write Synchronization

Write operations are fully serialized through the file lock, ensuring consistency.

Diagram: Write Operation Synchronization Flow

Single Write Flow

Source: src/storage_engine/data_store.rs:758-825

Batch Write Optimization

Batch writes hold the lock once for multiple entries:

Source: src/storage_engine/data_store.rs:847-945

Sources: src/storage_engine/data_store.rs:752-825 src/storage_engine/data_store.rs:847-945 README.md176

Thread Safety Guarantees

Thread Safety Matrix

The following table summarizes thread safety guarantees for different environments:

Source: README.md:196-200

graph TB
    subgraph "Thread Safety Properties"
        P1["Atomic Tail Updates\nNo torn reads/writes"]
P2["Serialized File Writes\nNo interleaved data"]
P3["Consistent Index View\nRwLock guarantees"]
P4["Valid Memory Maps\nArc prevents premature free"]
P5["Backward Chain Integrity\nSequential offsets"]
end
    
 
   P1 --> SAFE["Thread-Safe\nMulti-Reader/Single-Writer"]
P2 --> SAFE
 
   P3 --> SAFE
 
   P4 --> SAFE
 
   P5 --> SAFE

Safe Concurrency Properties

The design ensures the following properties in single-process, multi-threaded environments:

Diagram: Thread Safety Property Dependencies

1. Atomicity : All operations on shared state are atomic or properly locked
2. Visibility : Changes made by one thread are visible to others through Release/Acquire semantics
3. Ordering : The append-only design ensures writes happen in a strict sequence
4. Isolation : Readers see a consistent snapshot via Arc<Mmap> cloning

Sources: README.md:172-206 src/storage_engine/data_store.rs:27-33

Single-Process vs Multi-Process

Single-Process Multi-Threaded (Supported)

All synchronization primitives work correctly within a single process:

Diagram: Single-Process Shared State

Example from concurrency tests:

Source: tests/concurrency_tests.rs:117-137

graph TB
    subgraph "Process 1"
        P1_DS["DataStore"]
P1_INDEX["KeyIndexer\n(separate)"]
P1_MMAP["Mmap\n(separate)"]
end
    
    subgraph "Process 2"
        P2_DS["DataStore"]
P2_INDEX["KeyIndexer\n(separate)"]
P2_MMAP["Mmap\n(separate)"]
end
    
    FILE["storage.bin\n(shared file)"]
P1_DS --> P1_INDEX
 
   P1_DS --> P1_MMAP
 
   P2_DS --> P2_INDEX
 
   P2_DS --> P2_MMAP
    
    P1_MMAP -.unsafe.-> FILE
    P2_MMAP -.unsafe.-> FILE

Multi-Process (Not Supported)

Multiple processes have separate address spaces and cannot share the in-memory synchronization primitives:

Diagram: Multi-Process Unsafe Access

Why Multi-Process is Unsafe

1. Separate Index State : Each process has its own KeyIndexer in memory
2. Independent Mmap Views : Memory maps are not synchronized across processes
3. No Lock Coordination : RwLock and Mutex are process-local, not system-wide
4. Race Conditions : Concurrent writes can corrupt the file structure

Recommendation : Use external file locking (e.g., flock, advisory locks) if multi-process access is required.

Sources: README.md:186-206 README.md:189-191

Testing Concurrency

The test suite validates concurrent access patterns to ensure thread safety guarantees.

Concurrent Write Test

Tests multiple threads writing simultaneously:

Source: tests/concurrency_tests.rs:111-161

Interleaved Read-Write Test

Tests read-after-write consistency with coordinated threads:

Source: tests/concurrency_tests.rs:163-229

Concurrent Streamed Write Test

Tests slow, streaming writes that hold the lock for extended periods:

Source: tests/concurrency_tests.rs:14-109

Sources: tests/concurrency_tests.rs:1-230

Summary

The SIMD R Drive concurrency model provides thread-safe access through a carefully coordinated set of synchronization primitives:

• RwLock : Serializes file writes while allowing concurrent reads of the lock
• AtomicU64 : Provides lock-free tail offset tracking
• Mutex : Protects memory map updates without blocking existing readers
• RwLock : Enables highly concurrent index reads with exclusive write access

This design achieves:

• Zero-copy concurrent reads via memory mapping
• Serialized writes preventing data corruption
• Linear read scalability across CPU cores
• Consistent snapshots through atomic operations

However, these guarantees apply only within a single process. Multi-process access requires external coordination mechanisms.

Sources: README.md:170-206 src/storage_engine/data_store.rs:26-33 tests/concurrency_tests.rs:1-230

Dismiss

Refresh this wiki

Enter email to refresh

Key Indexing and Hashing

Loading…

Key Indexing and Hashing

Relevant source files

• README.md
• experiments/bindings/python-ws-client/pyproject.toml
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/init.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi
• src/storage_engine/data_store.rs
• src/storage_engine/key_indexer.rs

Purpose and Scope

This page documents the key indexing system and hashing mechanisms used in SIMD R Drive’s storage engine. It covers the KeyIndexer data structure, the XXH3 hashing algorithm, tag-based collision detection, and hardware acceleration features.

For information about how the index is accessed in concurrent operations, see Concurrency and Thread Safety. For details on how metadata is stored alongside payloads, see Entry Structure and Metadata.

Overview

The SIMD R Drive storage engine maintains an in-memory index that maps key hashes to file offsets, enabling O(1) lookup performance for stored entries. This index is critical for avoiding full file scans when retrieving data.

The indexing system consists of three main components:

1. KeyIndexer : A concurrent hash map that stores packed values containing both a collision-detection tag and a file offset
2. XXH3_64 hashing : A fast, hardware-accelerated hashing algorithm that generates 64-bit hashes from arbitrary keys
3. Tag-based verification : A secondary collision detection mechanism that validates lookups to prevent hash collision errors

Sources: src/storage_engine/data_store.rs:1-33 README.md:158-168

KeyIndexer Structure

The KeyIndexer struct is defined in src/storage_engine/key_indexer.rs:56-59 and manages the in-memory hash index. It wraps a HashMap<u64, u64, Xxh3BuildHasher> where keys are XXH3 key hashes and values are packed 64-bit integers containing both a collision-detection tag and file offset.

graph TB
    subgraph DataStore["DataStore struct"]
KeyIndexerField["key_indexer: Arc<RwLock<KeyIndexer>>"]
end
    
    subgraph KeyIndexer["KeyIndexer struct"]
IndexField["index: HashMap<u64, u64, Xxh3BuildHasher>"]
MapKey["Key: u64\n(key_hash from compute_hash)"]
MapValue["Value: u64\n(packed tag / offset)"]
end
    
    subgraph Constants["Key Constants"]
TagBits["TAG_BITS = 16"]
OffsetMask["OFFSET_MASK = (1 << 48) - 1"]
end
    
    subgraph Methods["Public Methods"]
TagFromHash["tag_from_hash(key_hash) -> u16"]
TagFromKey["tag_from_key(key) -> u16"]
Pack["pack(tag, offset) -> u64"]
Unpack["unpack(packed) -> (u16, u64)"]
Build["build(mmap, tail_offset) -> Self"]
Insert["insert(key_hash, offset) -> Result"]
GetPacked["get_packed(&key_hash) -> Option<&u64>"]
GetOffset["get_offset(&key_hash) -> Option<u64>"]
Remove["remove(&key_hash) -> Option<u64>"]
end
    
 
   KeyIndexerField --> IndexField
 
   IndexField --> MapKey
 
   IndexField --> MapValue
 
   MapValue --> TagBits
 
   MapValue --> OffsetMask
    
    Pack -.uses.-> TagBits
    Pack -.uses.-> OffsetMask
    Unpack -.uses.-> TagBits
    Unpack -.uses.-> OffsetMask

Packed Value Format

The KeyIndexer stores a compact 64-bit packed value for each hash. This value is constructed by the pack function src/storage_engine/key_indexer.rs:79-85 and decoded by unpack src/storage_engine/key_indexer.rs:88-93:

The packing formula is: packed = (tag << (64 - TAG_BITS)) | offset

The unpacking extracts: tag = (packed >> (64 - TAG_BITS)) as u16 and offset = packed & OFFSET_MASK

Maximum Addressable File Size : The 48-bit offset field supports files up to 256 TiB (2^48 bytes). Attempting to use larger offsets will panic in debug builds due to debug_assert! checks src/storage_engine/key_indexer.rs:80-83

Sources: src/storage_engine/key_indexer.rs:9-15 src/storage_engine/key_indexer.rs:56-59 src/storage_engine/key_indexer.rs:79-93 src/storage_engine/data_store.rs31

Hashing Algorithm: XXH3_64

SIMD R Drive uses the XXH3_64 hashing algorithm from the xxhash-rust crate src/storage_engine/digest.rs XXH3 is optimized for speed and provides automatic hardware acceleration through SIMD instructions.

graph TB
    subgraph DigestModule["storage_engine::digest module"]
ComputeHash["compute_hash(key: &[u8]) -> u64"]
ComputeHashBatch["compute_hash_batch(keys: &[&[u8]]) -> Vec<u64>"]
ComputeChecksum["compute_checksum(payload: &[u8]) -> [u8; 4]"]
Xxh3BuildHasher["Xxh3BuildHasher struct"]
end
    
    subgraph Callers["Usage in DataStore"]
WriteMethod["write(key, payload)"]
BatchWrite["batch_write(entries)"]
KeyIndexerHashMap["HashMap<u64, u64, Xxh3BuildHasher>"]
end
    
 
   WriteMethod -->|calls| ComputeHash
 
   BatchWrite -->|calls| ComputeHashBatch
 
   WriteMethod -->|calls| ComputeChecksum
 
   KeyIndexerHashMap -->|uses hasher| Xxh3BuildHasher
    
 
   ComputeHash -->|produces| KeyHash["key_hash: u64"]
ComputeHashBatch -->|produces| KeyHashes["Vec<u64>"]

Hash Function API

The digest module exports the following functions used throughout the codebase:

The compute_hash_batch function leverages Rayon for parallel hashing when processing multiple keys simultaneously src/storage_engine/data_store.rs:839-842

graph LR
    KeyInput["Input: &[u8]"]
subgraph XXH3Crate["xxhash-rust crate"]
CPUDetect["Runtime CPU\nFeature Detection"]
subgraph x86Implementation["x86_64 Implementation"]
SSE2Path["SSE2 Path\n(always available)"]
AVX2Path["AVX2 Path\n(if cpuid detects)"]
end
        
        subgraph ARMImplementation["aarch64 Implementation"]
NEONPath["NEON Path\n(default on ARM)"]
end
        
        subgraph FallbackImplementation["Fallback"]
ScalarPath["Scalar Operations"]
end
    end
    
    Output["Output: u64 hash"]
KeyInput --> CPUDetect
 
   CPUDetect --> SSE2Path
 
   CPUDetect --> AVX2Path
 
   CPUDetect --> NEONPath
 
   CPUDetect --> ScalarPath
    
 
   SSE2Path --> Output
 
   AVX2Path --> Output
 
   NEONPath --> Output
 
   ScalarPath --> Output

Hardware Acceleration

The XXH3_64 implementation automatically detects and utilizes CPU-specific SIMD instructions at runtime:

The hardware acceleration is transparent to the application code. The compute_hash function signature remains the same regardless of which SIMD path is taken README.md:160-165

Sources: src/storage_engine/digest.rs src/storage_engine/data_store.rs:2-4 src/storage_engine/key_indexer.rs2 README.md:158-168

Tag-Based Collision Detection

While XXH3_64 produces high-quality 64-bit hashes, the system implements an additional collision detection layer using 16-bit tags. The tag is derived from the upper 16 bits of the key hash src/storage_engine/key_indexer.rs:64-66

Tag Computation Methods

Two methods generate tags for collision detection:

The tag_from_hash function extracts the tag: (key_hash >> (64 - TAG_BITS)) as u16

The tag_from_key function computes: tag_from_hash(compute_hash(key))

graph TB
    subgraph WriteFlow["Write Operation Flow"]
Key["key: &[u8]"]
ComputeHash["compute_hash(key)"]
KeyHash["key_hash: u64"]
TagFromHash["KeyIndexer::tag_from_hash(key_hash)"]
Tag["tag: u16"]
WriteData["Write payload + metadata to file"]
Offset["metadata_offset: u64"]
Pack["KeyIndexer::pack(tag, offset)"]
PackedValue["packed: u64"]
Insert["key_indexer.insert(key_hash, offset)"]
end
    
 
   Key --> ComputeHash
 
   ComputeHash --> KeyHash
 
   KeyHash --> TagFromHash
 
   TagFromHash --> Tag
 
   WriteData --> Offset
 
   Tag --> Pack
 
   Offset --> Pack
 
   Pack --> PackedValue
 
   KeyHash --> Insert
 
   PackedValue --> Insert

Write-Time Tag Storage

During write operations, the tag is packed with the offset before insertion into the index:

The insert method in KeyIndexer src/storage_engine/key_indexer.rs:135-160 performs collision detection at write time by verifying that the new tag matches any existing tag for the same key hash.

Read-Time Tag Verification

The read_entry_with_context method src/storage_engine/data_store.rs:501-565 implements tag verification during reads:

The verification logic src/storage_engine/data_store.rs:513-521:

Collision Probability Analysis

The dual-layer verification provides strong collision resistance:

With 2^16 = 65,536 possible tag values, the tag check provides sufficient discrimination for practical workloads. The KeyIndexer documentation src/storage_engine/key_indexer.rs:20-56 notes this can distinguish over 4 billion keys with ~50% collision probability (birthday bound).

Sources: src/storage_engine/key_indexer.rs:9-72 src/storage_engine/key_indexer.rs:135-160 src/storage_engine/data_store.rs:501-565

Write-Time Collision Rejection

The KeyIndexer::insert method src/storage_engine/key_indexer.rs:135-160 enforces collision detection at write time:

If a collision is detected (same hash, different tag), the write operation fails with an error src/storage_engine/data_store.rs:245-251 This prevents index corruption and ensures data integrity.

Sources: src/storage_engine/key_indexer.rs:135-160 src/storage_engine/data_store.rs:238-252

Index Building and Maintenance

Index Construction on Open

When DataStore::open src/storage_engine/data_store.rs:84-117 is called, the KeyIndexer is constructed by the static build method src/storage_engine/key_indexer.rs:98-124 which scans backward through the validated storage file:

The backward scan ensures only the most recent version of each key is indexed. Keys seen earlier in the scan (which represent newer entries) are added to the seen HashSet to skip older versions src/storage_engine/key_indexer.rs:108-111

Sources: src/storage_engine/data_store.rs:84-117 src/storage_engine/key_indexer.rs:98-124

Index Updates During Writes

After each write operation, the reindex method src/storage_engine/data_store.rs:224-259 updates the in-memory index with new key mappings:

Critical : The file must be flushed before calling reindex src/storage_engine/data_store.rs814 to ensure newly written data is visible in the new memory-mapped view. The flush guarantees that the OS has persisted the data to disk before the mmap is recreated.

The key_indexer.insert call may return an error if a hash collision is detected at write time src/storage_engine/data_store.rs:246-250 In this case, the entire batch operation is aborted to prevent an inconsistent index state.

Sources: src/storage_engine/data_store.rs:224-259 src/storage_engine/data_store.rs:818-824

Concurrent Access and Locking

The KeyIndexer is protected by an Arc<RwLock<KeyIndexer>> wrapper src/storage_engine/data_store.rs31 enabling multiple concurrent readers while ensuring exclusive access for writers.

graph TB
    subgraph DataStoreField["DataStore struct field"]
KeyIndexerArc["key_indexer: Arc<RwLock<KeyIndexer>>"]
end
    
    subgraph ReadOperations["Read Operations (Shared Lock)"]
ReadOp["read(key)"]
BatchReadOp["batch_read(keys)"]
IterEntries["iter_entries()"]
ParIterEntries["par_iter_entries() (parallel feature)"]
end
    
    subgraph WriteOperations["Write Operations (Exclusive Lock)"]
WriteOp["write(key, payload)"]
BatchWriteOp["batch_write(entries)"]
DeleteOp["delete(key)"]
ReindexOp["reindex() (internal)"]
end
    
    subgraph LockAcquisition["Lock Acquisition"]
ReadLock["key_indexer.read().unwrap()"]
WriteLock["key_indexer.write().map_err(...)"]
end
    
 
   ReadOp --> ReadLock
 
   BatchReadOp --> ReadLock
 
   IterEntries --> ReadLock
 
   ParIterEntries --> ReadLock
    
 
   WriteOp --> WriteLock
 
   BatchWriteOp --> WriteLock
 
   DeleteOp --> WriteLock
 
   ReindexOp --> WriteLock
    
 
   ReadLock -.-> KeyIndexerArc
 
   WriteLock -.-> KeyIndexerArc

Lock Granularity

The reindex method acquires both the mmap mutex src/storage_engine/data_store.rs232 and the key_indexer write lock src/storage_engine/data_store.rs:233-236 to atomically update both structures after a write operation.

Parallel Iteration : When the parallel feature is enabled, par_iter_entries src/storage_engine/data_store.rs:296-361 clones all packed values under a read lock, then releases the lock before parallel processing. This allows concurrent reads during parallel iteration.

Sources: src/storage_engine/data_store.rs31 src/storage_engine/data_store.rs:224-259 src/storage_engine/data_store.rs:296-361

Performance Characteristics

Lookup Performance

The KeyIndexer provides O(1) average-case lookup performance using the HashMap src/storage_engine/key_indexer.rs58 The get_packed method src/storage_engine/key_indexer.rs:163-166 performs a single hash table lookup.

Empirical performance from README README.md:166-167:

• 1 million random seeks (8-byte entries): typically < 1 second
• Hash computation overhead : negligible due to SIMD acceleration
• Tag verification overhead : minimal (one bit shift + one comparison)

Memory Overhead

Each index entry in the HashMap<u64, u64, Xxh3BuildHasher> consumes:

For a dataset with 1 million unique keys , the KeyIndexer occupies approximately 32-40 MB of RAM. This is a small fraction of typical system memory, enabling efficient indexing even for large datasets.

Batch Operation Performance

The compute_hash_batch function src/storage_engine/data_store.rs:839-842 leverages Rayon for parallel hashing:

This parallel hashing provides near-linear speedup with CPU core count for large batches, as each key hash is computed independently.

Hardware Acceleration Impact

SIMD acceleration in XXH3_64 provides measurable performance improvements for hash-intensive workloads:

Performance gains are most significant for:

• batch_write operations with many keys
• compute_hash_batch calls processing large key sets
• Workloads with small payload sizes where hashing dominates

Sources: src/storage_engine/key_indexer.rs:56-59 src/storage_engine/data_store.rs:838-843 README.md:160-167

Error Handling and Collision Management

Collision Probability Analysis

The dual-layer verification system (64-bit hash + 16-bit tag) provides strong collision resistance as documented in src/storage_engine/key_indexer.rs:17-56:

The 16-bit tag provides 65,536 distinct values. According to the birthday paradox, this supports over 4 billion keys with ~50% collision probability src/storage_engine/key_indexer.rs:48-49

Write-Time Collision Handling

The KeyIndexer::insert method src/storage_engine/key_indexer.rs:135-160 enforces strict collision detection during writes. If a tag mismatch occurs, the insert returns Err:

The reindex method propagates this error and aborts the entire batch operation src/storage_engine/data_store.rs:246-250:

This fail-fast approach ensures:

• No partial writes that could corrupt the index
• Deterministic error handling (writes either fully succeed or fully fail)
• Index consistency is maintained across all operations

Read-Time Collision Handling

The read_entry_with_context method src/storage_engine/data_store.rs:501-565 detects collisions during reads when the original key is provided for verification src/storage_engine/data_store.rs:513-521:

When a read-time collision is detected:

1. A warning is logged to help diagnose the issue
2. None is returned to the caller (key not found)
3. The index remains unchanged (reads do not modify state)

Read operations without the original key (e.g., when using pre-hashed keys) cannot perform tag verification and may return incorrect results if a hash collision exists. This is a tradeoff for performance in batch operations.

Sources: src/storage_engine/key_indexer.rs:17-56 src/storage_engine/key_indexer.rs:135-160 src/storage_engine/data_store.rs:238-252 src/storage_engine/data_store.rs:501-565

Summary

The key indexing and hashing system in SIMD R Drive provides:

1. Fast lookups : O(1) hash-based access to entries
2. Hardware acceleration : Automatic SIMD optimization on SSE2, AVX2, and NEON platforms
3. Collision resistance : Dual-layer verification with 64-bit hashes and 16-bit tags
4. Thread safety : Concurrent reads with exclusive writes via RwLock
5. Low memory overhead : 16 bytes per unique key

This design enables efficient storage operations even for datasets with millions of entries, while maintaining data integrity through robust collision detection.

Sources: src/storage_engine/data_store.rs:1-1183 README.md:158-168

Dismiss

Refresh this wiki

Enter email to refresh

Compaction and Maintenance

Loading…

Compaction and Maintenance

Relevant source files

• README.md
• src/storage_engine/data_store.rs

Purpose and Scope

This page documents the maintenance operations available in the DataStore, focusing on compaction for space reclamation and automatic file recovery mechanisms. These operations ensure the storage engine remains efficient and resilient despite its append-only architecture.

For information about the underlying append-only storage model, see Storage Architecture. For details on entry structure that affects compaction, see Entry Structure and Metadata.

Sources: src/storage_engine/data_store.rs:1-1183

Compaction Process

Overview

The compaction process eliminates space waste caused by outdated entry versions. In the append-only model, updating a key creates a new entry while leaving the old version in the file. Compaction creates a new file containing only the latest version of each key, then atomically swaps it with the original.

graph TB
    subgraph "Original DataStore"
        DS["DataStore\n(self)"]
FILE["file: RwLock<BufWriter<File>>"]
MMAP["mmap: Mutex<Arc<Mmap>>"]
IDX["key_indexer: RwLock<KeyIndexer>"]
PATH["path: PathBuf"]
end
    
    subgraph "Compaction Process"
        COMPACT["compact()"]
ITER["iter_entries()"]
COPY["copy_handle()"]
end
    
    subgraph "Temporary DataStore"
        TEMP_DS["DataStore\n(compacted_storage)"]
TEMP_FILE["file (path + .bk)"]
TEMP_PATH["compacted_path"]
end
    
    subgraph "Final Operation"
        RENAME["std::fs::rename()"]
SWAP["Atomic File Swap"]
end
    
 
   DS --> COMPACT
 
   COMPACT --> TEMP_PATH
 
   TEMP_PATH --> TEMP_DS
 
   DS --> ITER
 
   ITER --> COPY
 
   COPY --> TEMP_DS
 
   TEMP_DS --> TEMP_FILE
 
   TEMP_FILE --> RENAME
 
   RENAME --> SWAP
 
   SWAP --> PATH
    
    style COMPACT fill:#f9f9f9
    style TEMP_DS fill:#f9f9f9
    style SWAP fill:#f9f9f9

Architecture

Sources: src/storage_engine/data_store.rs:706-749

Implementation Details

The compact() method at src/storage_engine/data_store.rs:706-749 performs the following sequence:

Sources: src/storage_engine/data_store.rs:706-749 src/storage_engine/data_store.rs:587-590 src/storage_engine/data_store.rs:269-280

Key Implementation Characteristics

Sources: src/storage_engine/data_store.rs:706-749

Thread Safety Considerations

The compact() method has critical thread safety limitations documented at src/storage_engine/data_store.rs:681-693:

Important Constraints:

• Requires&mut self: Prevents concurrent mutations but does NOT prevent concurrent reads
• Arc-wrapped risk : If DataStore is wrapped in Arc<DataStore>, other threads may hold read references during compaction
• Recommendation : Only compact when you have exclusive access (single thread or external synchronization)
• No automatic locking : External synchronization must be enforced by the caller

Sources: src/storage_engine/data_store.rs:679-705

EntryHandle Copying Mechanism

The copy_handle() method at src/storage_engine/data_store.rs:587-590 creates an EntryStream from the entry’s memory-mapped data and writes it to the target store using the original key hash. This preserves the entry’s identity while creating a new physical copy.

Sources: src/storage_engine/data_store.rs:567-590

Space Reclamation

Estimating Compaction Savings

The estimate_compaction_savings() method at src/storage_engine/data_store.rs:605-616 calculates potential space savings without performing actual compaction:

Algorithm:

1. Get total file size via file_size()
2. Iterate through all valid entries (latest versions only)
3. Track seen keys using HashSet<u64> with Xxh3BuildHasher
4. Sum the file_size() of each unique entry
5. Return total_size - unique_entry_size

Note: The iter_entries() method already filters to latest versions, so this calculation represents the minimum achievable size through compaction.

Sources: src/storage_engine/data_store.rs:592-616

When to Compact

Compaction should be considered when:

The compaction process at src/storage_engine/data_store.rs:727-741 includes logic to skip static index generation if it wouldn’t save space, indicating an awareness of space optimization trade-offs.

Sources: src/storage_engine/data_store.rs:605-616 src/storage_engine/data_store.rs:727-741

File Recovery

Chain Validation

The recover_valid_chain() method at src/storage_engine/data_store.rs:383-482 ensures data integrity by validating the backward-linked chain of entries:

Sources: src/storage_engine/data_store.rs:363-482

Validation Algorithm Details

The recovery process validates multiple constraints:

Sources: src/storage_engine/data_store.rs:383-482

Corruption Detection and Handling

When DataStore::open() is called at src/storage_engine/data_store.rs:84-117 it automatically invokes recovery:

Sources: src/storage_engine/data_store.rs:84-117

Automatic Recovery Steps

The recovery process at src/storage_engine/data_store.rs:91-103:

1. Detection : recover_valid_chain() returns final_len < file_len
2. Warning : Logs truncation message with offsets
3. Cleanup : Drops mmap and file handles
4. Truncation : Re-opens file and calls set_len(final_len)
5. Sync : Forces sync_all() to persist truncation
6. Retry : Recursively calls open() with clean file

This ensures corrupted tail data is removed and the file is left in a valid state.

Sources: src/storage_engine/data_store.rs:89-103

Maintenance Operations

Re-indexing After Writes

The reindex() method at src/storage_engine/data_store.rs:224-259 updates internal structures after writes:

Key Operations:

1. Re-map file : Creates new Mmap reflecting written data
2. Update index : Inserts or removes key-hash-to-offset mappings
3. Collision check : insert() returns error on hash collision (tag mismatch)
4. Atomic update : Stores new tail_offset with Release ordering
5. Visibility : New mmap makes written data visible to readers

Sources: src/storage_engine/data_store.rs:176-259

File Path and Metadata Access

These methods provide essential metadata for maintenance decision-making.

Sources: src/storage_engine/data_store.rs:261-267 src/storage_engine/data_store.rs:1164-1181

Iterators for Maintenance

The storage provides multiple iteration mechanisms useful for maintenance:

Iteration Characteristics:

• Sequential : iter_entries() at src/storage_engine/data_store.rs:269-280
• Parallel : par_iter_entries() at src/storage_engine/data_store.rs:283-361 (requires parallel feature)
• Latest Only : Both return only the most recent version of each key
• Zero-Copy : Return EntryHandle with memory-mapped data references

Sources: src/storage_engine/data_store.rs:269-361

Summary

The maintenance system provides:

All operations maintain the append-only guarantee while ensuring data integrity and efficient space utilization.

Sources: src/storage_engine/data_store.rs:1-1183

Dismiss

Refresh this wiki

Enter email to refresh

Network Layer and RPC

Loading…

Network Layer and RPC

Relevant source files

• Cargo.lock
• experiments/simd-r-drive-muxio-service-definition/Cargo.toml
• experiments/simd-r-drive-ws-client/Cargo.toml
• experiments/simd-r-drive-ws-server/Cargo.toml

Purpose and Scope

This document describes the network layer and Remote Procedure Call (RPC) system that enables remote access to the DataStore over WebSocket connections. The system is built on the Muxio framework and provides a standardized interface for clients in any language to communicate with a DataStore server.

For information about the core storage engine that the network layer wraps, see Core Storage Engine. For details on language-specific client implementations, see Python Integration and Native Rust Client.

Overview

The network layer provides a WebSocket-based RPC protocol that allows remote clients to perform DataStore operations. The architecture consists of three main components:

The system uses bitcode for efficient binary serialization and tokio-tungstenite for WebSocket transport. The Muxio framework handles message framing, multiplexing, and concurrent request processing over a single WebSocket connection.

Sources: experiments/simd-r-drive-muxio-service-definition/Cargo.toml:1-17 experiments/simd-r-drive-ws-server/Cargo.toml:1-23 experiments/simd-r-drive-ws-client/Cargo.toml:1-22

Architecture Overview

Diagram: Network Layer Component Architecture

The client and server both depend on simd-r-drive-muxio-service-definition to ensure they share the same RPC contract. The Muxio framework components (muxio-rpc-service-caller and muxio-rpc-service-endpoint) handle the RPC mechanics, while muxio-tokio-rpc-client and muxio-tokio-rpc-server provide the WebSocket transport layer.

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:13-21 experiments/simd-r-drive-ws-server/Cargo.toml:13-22 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:13-16

Service Definition Layer

The simd-r-drive-muxio-service-definition crate defines the RPC contract shared between client and server. This contract specifies:

• RPC Methods : Operations that can be invoked remotely (e.g., read, write, compact)
• Request/Response Types : Data structures for method parameters and return values
• Error Types : Standardized error representations for network and storage errors

Key Components

The service definition uses the muxio-rpc-service framework to define typed RPC interfaces. All data types use bitcode for serialization, which provides:

• Compact Binary Encoding : Efficient wire format optimized for small message sizes
• Zero-Copy Deserialization : Where possible, data is accessed without copying
• Type Safety : Compile-time guarantees that client and server speak the same protocol

Diagram: Service Definition Structure

The service trait defines method signatures, while request/response structs define the data exchanged. The bitcode derive macros generate serialization code automatically.

Sources: experiments/simd-r-drive-muxio-service-definition/Cargo.toml:14-15 experiments/bindings/python-ws-client/Cargo.lock:133-143

Transport Layer

WebSocket Protocol

The network layer uses WebSocket as the transport protocol for several reasons:

The transport is implemented using tokio-tungstenite, which provides async WebSocket support integrated with the Tokio runtime.

Sources: experiments/bindings/python-ws-client/Cargo.lock:1213-1222 experiments/bindings/python-ws-client/Cargo.lock:1303-1317

Muxio Framework

The Muxio framework provides the RPC layer on top of WebSocket:

Diagram: RPC Message Flow Through Muxio

The caller assigns a unique request_id to each RPC call, enabling multiplexing : multiple concurrent requests can be in flight over the same WebSocket connection. The endpoint matches responses to requests using these IDs.

Sources: experiments/bindings/python-ws-client/Cargo.lock:659-682 experiments/bindings/python-ws-client/Cargo.lock:685-700

Connection Management

Diagram: WebSocket Connection State Machine

The client maintains connection state and can implement automatic reconnection logic. The Muxio layer handles connection interruptions gracefully, returning errors for in-flight requests when the connection drops.

Sources: experiments/bindings/python-ws-client/Cargo.lock:685-700

Concurrency Model

The network layer is fully asynchronous and built on Tokio:

Client-Side Concurrency

The muxio-rpc-service-caller manages request IDs and matches responses to the correct awaiting future, enabling concurrent operations without blocking.

Server-Side Concurrency

The server spawns a new Tokio task for each incoming RPC request, allowing concurrent processing of multiple client requests. The underlying DataStore handles concurrent reads efficiently through its shared memory-mapped file access pattern (see Concurrency and Thread Safety).

Sources: experiments/simd-r-drive-ws-client/Cargo.toml19 experiments/simd-r-drive-ws-server/Cargo.toml18 experiments/bindings/python-ws-client/Cargo.lock:1184-1199

Error Handling

The RPC layer distinguishes between different error categories:

Diagram: Error Type Hierarchy

Each error category provides different information to help diagnose issues:

• Transport Errors : Indicate network connectivity problems
• Protocol Errors : Suggest version mismatches or implementation bugs
• Application Errors : Represent normal error conditions from DataStore operations

Sources: experiments/bindings/python-ws-client/Cargo.lock:659-669

Security Considerations

The current implementation provides:

For production deployments, consider:

1. Use WSS (WebSocket Secure) : Implement TLS encryption by placing the server behind a reverse proxy (nginx, Caddy, etc.)
2. Implement Authentication : Add token-based auth in the WebSocket handshake
3. Add Authorization : Implement per-key access controls in the server handler
4. Rate Limiting : Protect against DoS by limiting request rates per client

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:1-23

Performance Characteristics

Serialization Overhead

The bitcode serialization format is optimized for performance:

• Small Message Size : Typically 30-50% smaller than JSON
• Fast Encoding : Zero-copy for many types, SIMD-optimized where applicable
• Predictable Layout : Fixed-size types don’t require length prefixes

Network Latency

RPC call latency consists of:

Total Latency = Serialization + Network RTT + Deserialization + Processing

For typical operations:

• Serialization/Deserialization : <1ms for small payloads
• Network RTT : Depends on network conditions (LAN: <1ms, WAN: 10-100ms)
• Processing : Varies by operation (read: <1ms, write: ~1-5ms with flush)

Connection Multiplexing

A single WebSocket connection can handle thousands of concurrent RPC calls, limited only by:

• Available memory for buffering requests
• Server processing capacity
• Network bandwidth

Sources: experiments/simd-r-drive-muxio-service-definition/Cargo.toml14 experiments/bindings/python-ws-client/Cargo.lock:133-143

Dependency Graph

Diagram: Complete Dependency Graph for Network Layer

The layered architecture ensures clean separation of concerns:

• Application Layer : Client and server applications
• simd-r-drive Components : Project-specific RPC wrappers and service definitions
• Muxio Framework : Generic RPC infrastructure
• Transport & Runtime: Low-level WebSocket and async runtime

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:13-21 experiments/simd-r-drive-ws-server/Cargo.toml:13-22 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:13-16 experiments/bindings/python-ws-client/Cargo.lock:648-700

Dismiss

Refresh this wiki

Enter email to refresh

WebSocket Server

Loading…

WebSocket Server

Relevant source files

• Cargo.lock
• experiments/simd-r-drive-muxio-service-definition/Cargo.toml
• experiments/simd-r-drive-ws-client/Cargo.toml
• experiments/simd-r-drive-ws-server/Cargo.toml

Purpose and Scope

The WebSocket Server provides remote network access to the SIMD R Drive storage engine through an RPC-based interface. This experimental component enables clients to perform storage operations over WebSocket connections using the muxio RPC framework with bitcode serialization.

This page covers the server implementation, configuration, and connection handling. For information about the RPC protocol and service definitions, see Muxio RPC Framework. For client-side implementation, see Native Rust Client. For the underlying storage operations, see DataStore API.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:1-23

Server Architecture

The WebSocket Server acts as a bridge between remote clients and the local storage engine, handling WebSocket connections, deserializing RPC requests, executing storage operations, and returning serialized responses.

graph TB
    subgraph "simd-r-drive-ws-server"
        MAIN["main.rs\nServer Entry Point"]
CLI["clap Parser\nCLI Arguments"]
SERVER["muxio-tokio-rpc-server\nWebSocket Server"]
SERVICE["Service Implementation\nRPC Handler"]
end
    
    subgraph "Dependencies"
        DEFINITION["simd-r-drive-muxio-service-definition\nService Contract"]
CORE["simd-r-drive\nDataStore"]
TOKIO["tokio Runtime\nAsync Executor"]
TRACING["tracing-subscriber\nLogging"]
end
    
    subgraph "Network"
        WS_CONN["WebSocket Connection\ntokio-tungstenite"]
CLIENT["Remote Client"]
end
    
 
   MAIN --> CLI
 
   MAIN --> TRACING
 
   MAIN --> SERVER
 
   SERVER --> SERVICE
 
   SERVICE --> DEFINITION
 
   SERVICE --> CORE
 
   SERVER --> TOKIO
 
   SERVER --> WS_CONN
 
   WS_CONN --> CLIENT
    
    style SERVER fill:#f9f9f9
    style SERVICE fill:#f9f9f9
    style CORE fill:#e8e8e8

Component Overview

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:13-23

Server Implementation

Package Structure

The simd-r-drive-ws-server package implements the WebSocket server as an experimental component in the workspace. It provides a binary that can be executed to start the server.

Service Implementation Flow

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:13-18

Configuration and Startup

Command-Line Arguments

The server uses clap for CLI argument parsing with the derive feature, providing a structured interface for configuration.

Expected Configuration Options

Initialization Sequence

The server initialization creates a local DataStore instance which is then accessed by the service handler for all RPC operations. The tracing-subscriber dependency with env-filter feature allows runtime configuration of logging levels.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:19-22

Connection Handling

WebSocket Lifecycle

The muxio-tokio-rpc-server package handles the WebSocket protocol details, including:

1. Connection Upgrade : HTTP to WebSocket protocol upgrade
2. Message Framing : Binary message framing over WebSocket
3. Multiplexing : Multiple concurrent RPC calls over a single connection
4. Error Handling : Connection errors and RPC-level errors

Connection State Management

Each WebSocket connection is handled by a separate tokio task, allowing concurrent client connections without blocking.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:16-18

Service Implementation Details

RPC Service Interface

The service implementation must implement the interface defined in simd-r-drive-muxio-service-definition. This shared contract ensures type-safe communication between client and server.

Service Methods

Based on the DataStore API and typical RPC patterns, the service likely implements these methods:

Service Handler Structure

The service handler wraps a shared reference to the DataStore (likely Arc<DataStore>) to allow concurrent read access across multiple RPC calls while serializing writes through the DataStore’s internal concurrency control.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:13-17 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:1-17

Dependencies and Runtime

Core Dependencies

muxio-tokio-rpc-server

The muxio-tokio-rpc-server package provides the WebSocket server implementation built on top of:

• axum for HTTP/WebSocket handling
• tokio for async runtime
• muxio-rpc-service-endpoint for RPC dispatch

graph TB
    subgraph "tokio Runtime"
        MULTI["Multi-threaded\nExecutor"]
REACTOR["Reactor\nI/O Events"]
TIMER["Timer\nTimeouts"]
end
    
    subgraph "Server Tasks"
        LISTENER["Listener Task\nAccept Connections"]
CONN1["Connection Task 1\nClient 1"]
CONN2["Connection Task 2\nClient 2"]
CONNN["Connection Task N\nClient N"]
end
    
 
   MULTI --> LISTENER
 
   MULTI --> CONN1
 
   MULTI --> CONN2
 
   MULTI --> CONNN
    
 
   REACTOR --> LISTENER
 
   REACTOR --> CONN1
 
   REACTOR --> CONN2
 
   REACTOR --> CONNN

Serialization

The service uses bitcode for binary serialization, shared through the simd-r-drive-muxio-service-definition package. Bitcode provides compact binary encoding with zero-copy deserialization where possible.

Async Runtime

The server runs on the tokio multi-threaded runtime, with each WebSocket connection handled by an independent task. This allows efficient concurrent handling of multiple clients.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:16-20

Logging and Observability

tracing Integration

The server uses tracing with tracing-subscriber for structured logging. The env-filter feature allows configuration via environment variables:

Log Levels

Key Trace Points

The server likely includes trace instrumentation at:

• Server initialization and configuration
• WebSocket connection establishment
• RPC method dispatch
• DataStore operation execution
• Error conditions and recovery

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:19-20

Building and Running

Build Command

Run Command Example

Development Mode

For development with detailed logging:

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:1-23

Security Considerations

Network Exposure

The WebSocket server exposes the DataStore over the network. Important considerations:

1. Authentication : The current implementation does not include authentication (experimental status)
2. Encryption : WebSocket connections are not TLS-encrypted by default
3. Access Control : No per-key or per-operation access control
4. Network Binding : Binding to 0.0.0.0 exposes the server to all network interfaces

Recommended Production Practices

For production deployment, additional layers would be needed:

• TLS/SSL termination (via reverse proxy or native support)
• Authentication middleware
• Rate limiting
• Request validation
• Network-level access control (firewall rules)

Note: This is an experimental component and should not be used in production without additional security hardening.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:1-11

Integration with Core Storage

DataStore Access Pattern

The server maintains a single DataStore instance that is shared across all RPC handlers. Write operations serialize through the DataStore’s internal RwLock, while read operations can proceed concurrently through the lock-free DashMap index.

Thread Safety

The server implementation relies on the DataStore’s thread-safe design:

• Multiple concurrent reads via DashMap
• Serialized writes via RwLock
• Atomic tail offset tracking via AtomicU64

This allows multiple WebSocket connections to safely access the same DataStore instance without additional synchronization.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:14-15

Dismiss

Refresh this wiki

Enter email to refresh

Muxio RPC Framework

Loading…

Muxio RPC Framework

Relevant source files

• Cargo.lock
• experiments/simd-r-drive-muxio-service-definition/Cargo.toml
• experiments/simd-r-drive-ws-client/Cargo.toml
• experiments/simd-r-drive-ws-server/Cargo.toml

Purpose and Scope

This document describes the Muxio RPC (Remote Procedure Call) framework as implemented in SIMD R Drive for remote storage access over WebSocket connections. The framework provides a type-safe, multiplexed communication protocol using bitcode serialization for efficient binary data transfer.

For information about the WebSocket server implementation, see WebSocket Server. For the native Rust client implementation, see Native Rust Client. For Python client integration, see Python WebSocket Client API.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:1-23 experiments/simd-r-drive-ws-client/Cargo.toml:1-22 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:1-17

Architecture Overview

The Muxio RPC framework consists of multiple layers that work together to provide remote procedure calls over WebSocket connections:

Muxio RPC Framework Layer Architecture

graph TB
    subgraph "Client Application Layer"
        App["Application Code"]
end
    
    subgraph "Client RPC Stack"
        Caller["muxio-rpc-service-caller\nMethod Invocation"]
ClientRuntime["muxio-tokio-rpc-client\nWebSocket Client Runtime"]
end
    
    subgraph "Shared Contract"
        ServiceDef["simd-r-drive-muxio-service-definition\nService Interface Contract\nMethod Signatures"]
Bitcode["bitcode\nBinary Serialization"]
end
    
    subgraph "Server RPC Stack"
        ServerRuntime["muxio-tokio-rpc-server\nWebSocket Server Runtime"]
Endpoint["muxio-rpc-service-endpoint\nRequest Router"]
end
    
    subgraph "Server Application Layer"
        Impl["DataStore Implementation"]
end
    
    subgraph "Core Framework"
        Core["muxio-rpc-service\nBase RPC Traits & Types"]
end
    
 
   App --> Caller
 
   Caller --> ClientRuntime
 
   ClientRuntime --> ServiceDef
 
   ClientRuntime --> Bitcode
 
   ClientRuntime --> Core
    
 
   ServiceDef --> Bitcode
 
   ServiceDef --> Core
    
 
   ServerRuntime --> ServiceDef
 
   ServerRuntime --> Bitcode
 
   ServerRuntime --> Core
 
   ServerRuntime --> Endpoint
 
   Endpoint --> Impl
    
    style ServiceDef fill:#f9f9f9,stroke:#333,stroke-width:2px

The framework is organized into distinct layers:

Sources: Cargo.lock:1250-1336 experiments/simd-r-drive-ws-server/Cargo.toml:14-17 experiments/simd-r-drive-ws-client/Cargo.toml:14-21

Core Framework Components

muxio-rpc-service

The muxio-rpc-service crate provides the foundational abstractions for the RPC system. This crate defines the core traits and types that both client and server components build upon.

Core RPC Framework Message Structure and Dependencies

graph TB
    subgraph "muxio-rpc-service Crate"
        RpcService["#[async_trait]\nRpcService Trait"]
Request["RpcRequest\nStruct"]
Response["RpcResponse\nStruct"]
ServiceDef["Service Definition\nInfrastructure"]
end
    
    subgraph "RpcRequest Fields"
        ReqID["request_id: u64\n(unique per call)"]
MethodID["method_id: u64\n(xxhash-rust XXH3)"]
Payload["payload: Vec<u8>\n(bitcode serialized)"]
end
    
    subgraph "RpcResponse Fields"
        RespID["request_id: u64\n(matches request)"]
Result["result: Result<Vec<u8>, Error>\n(bitcode serialized)"]
end
    
    subgraph "Dependencies"
        AsyncTrait["async-trait"]
Futures["futures"]
NumEnum["num_enum"]
XXHash["xxhash-rust"]
end
    
 
   RpcService -->|defines| ServiceDef
 
   Request -->|contains| ReqID
 
   Request -->|contains| MethodID
 
   Request -->|contains| Payload
 
   Response -->|contains| RespID
 
   Response -->|contains| Result
    
    RpcService -.uses.- AsyncTrait
    MethodID -.hashed with.- XXHash

The muxio-rpc-service crate provides:

The framework uses async-trait to enable async methods in traits, and XXH3 hashing (via xxhash-rust) for method identification, allowing fast O(1) method dispatch without string comparisons.

Sources: Cargo.lock:1261-1272 experiments/simd-r-drive-muxio-service-definition/Cargo.toml15

Service Definition Layer

simd-r-drive-muxio-service-definition

The simd-r-drive-muxio-service-definition crate serves as the shared RPC contract between clients and servers. This crate is compiled into both client and server binaries, ensuring type-safe method signatures on both sides.

Service Definition Compilation Model

graph TB
    subgraph "simd-r-drive-muxio-service-definition"
        Contract["RPC Service Contract"]
Methods["Method Signatures"]
Types["Shared Types"]
end
    
    subgraph "Client Binary"
        ClientStub["Generated Client Stubs"]
end
    
    subgraph "Server Binary"
        ServerImpl["Generated Server Handlers"]
end
    
 
   Contract --> Methods
 
   Contract --> Types
 
   Methods -->|compiled into| ClientStub
 
   Methods -->|compiled into| ServerImpl
 
   Types -->|used by| ClientStub
 
   Types -->|used by| ServerImpl
    
 
   ClientStub -->|invokes via| WS["WebSocket"]
WS -->|routes to| ServerImpl

The service definition provides the RPC interface contract. Both client and server depend on this crate, which defines:

Method ID Generation

Each RPC method is identified by a stable method_id computed as the XXH3 hash of its signature string. This enables O(1) method routing:

Method ID Computation and Routing with Code Entities

flowchart LR
    Sig["Method Signature\n'write(key: &[u8], value: &[u8])\n-> Result<u64>'"]
XXH3["xxhash_rust::xxh3\nxxh3_64(sig.as_bytes())"]
ID["method_id: u64\ne.g., 0x1a2b3c4d5e6f7890"]
HashMap["HashMap<u64,\nBox<dyn Fn>>\nin RpcServiceEndpoint"]
Lookup["HashMap::get\n(&method_id)"]
Handler["async fn handler\n(decoded args)"]
Sig -->|hash at compile time| XXH3
 
   XXH3 --> ID
 
   ID -->|stored in| HashMap
 
   HashMap -->|O 1 lookup| Lookup
 
   Lookup --> Handler

The XXH3 hash (via xxhash-rust crate) ensures:

The xxhash-rust dependency provides the xxh3_64 function used by muxio-rpc-service for method ID generation. The server’s RpcServiceEndpoint struct maintains the HashMap<u64, Box<dyn Fn>> dispatcher.

Sources: Cargo.lock:1261-1272 Cargo.lock:1905-1915 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:1-17

Bitcode Serialization

The framework uses the bitcode crate (version 0.6.6) for efficient binary serialization with the following characteristics:

graph LR
    subgraph "Bitcode Serialization Pipeline"
        RustType["Rust Type\n#[derive(Encode, Decode)]"]
Encode["bitcode::encode\n<T>(&value)"]
Binary["Vec<u8>\nCompact Binary"]
Decode["bitcode::decode\n<T>(&bytes)"]
RustType2["Rust Type\nReconstructed"]
end
    
    subgraph "bitcode Dependencies"
        BitcodeDerve["bitcode_derive\nproc macros"]
Bytemuck["bytemuck\nzero-copy casts"]
Arrayvec["arrayvec\nstack arrays"]
Glam["glam\nSIMD vectors"]
end
    
 
   RustType -->|serialize| Encode
 
   Encode --> Binary
 
   Binary -->|deserialize| Decode
 
   Decode --> RustType2
    
    Encode -.uses.- BitcodeDerve
    Encode -.uses.- Bytemuck
    Decode -.uses.- BitcodeDerve
    Decode -.uses.- Bytemuck

Serialization Features

Bitcode Encoding/Decoding Pipeline with Dependencies

Integration with RPC

The serialization is integrated at multiple points:

The use of #[derive(Encode, Decode)] on request/response types ensures compile-time validation of serialization compatibility.

Sources: Cargo.lock:392-414 experiments/simd-r-drive-muxio-service-definition/Cargo.toml14

Client-Side Components

flowchart TB
    subgraph "Client Call Flow"
        ClientApp["Client Application"]
Caller["RpcServiceCaller\nStruct"]
GenID["Generate request_id\n(AtomicU64::fetch_add)"]
Request["Create RpcRequest\nStruct"]
Serialize["bitcode::encode\n(method args)"]
Send["Send via\ntokio::sync::mpsc"]
Await["tokio::sync::oneshot\nawait response"]
Deserialize["bitcode::decode\n(response payload)"]
Return["Return Result\nto caller"]
end
    
 
   ClientApp -->|async fn call| Caller
 
   Caller --> GenID
 
   GenID --> Request
 
   Request --> Serialize
 
   Serialize --> Send
 
   Send --> Await
 
   Await --> Deserialize
 
   Deserialize --> Return
 
   Return --> ClientApp

muxio-rpc-service-caller

The muxio-rpc-service-caller crate provides the client-side method invocation interface:

Client Method Invocation Flow with tokio Primitives

Key responsibilities and implementation:

The caller uses tokio’s async primitives to coordinate between the application thread and the WebSocket send/receive loops.

Sources: Cargo.lock:1274-1285 experiments/simd-r-drive-ws-client/Cargo.toml18

graph TB
    subgraph "muxio-tokio-rpc-client Crate"
        Client["RpcClient\nStruct"]
SendLoop["send_loop\ntokio::task::spawn"]
RecvLoop["recv_loop\ntokio::task::spawn"]
PendingMap["Arc<DashMap<u64,\noneshot::Sender<Result>>>\nShared state"]
ReqChan["mpsc::Receiver\n<RpcRequest>"]
end
    
    subgraph "tokio-tungstenite Integration"
        WS["WebSocketStream\n<MaybeTlsStream>"]
Split["ws.split()"]
WSRead["SplitStream\n(read half)"]
WSWrite["SplitSink\n(write half)"]
end
    
    subgraph "Application Layer"
        AppCall["async fn call()"]
Future["impl Future\n<Output=Result>"]
end
    
 
   AppCall -->|1. create oneshot| Client
 
   Client -->|2. insert into| PendingMap
 
   Client -->|3. mpsc::send| ReqChan
 
   ReqChan -->|4. recv request| SendLoop
 
   SendLoop -->|5. bitcode::encode| SendLoop
 
   SendLoop -->|6. send binary| WSWrite
 
   WSRead -->|7. next binary| RecvLoop
 
   RecvLoop -->|8. bitcode::decode| RecvLoop
 
   RecvLoop -->|9. lookup by id| PendingMap
 
   PendingMap -->|10. oneshot::send| Future
 
   Future -->|11. return| AppCall
    
 
   WS --> Split
 
   Split --> WSRead
 
   Split --> WSWrite

muxio-tokio-rpc-client

The muxio-tokio-rpc-client crate implements the WebSocket client runtime with request multiplexing and response routing:

Client Runtime Request Multiplexing with tokio and tungstenite

Implementation details:

The multiplexing architecture uses DashMap for lock-free concurrent access to pending requests. The WebSocket stream is split into read and write halves, allowing the send_loop and recv_loop tasks to operate independently. Each request gets a unique request_id, and the recv_loop task matches responses back to waiting callers via oneshot channels.

Sources: Cargo.lock:1302-1318 experiments/simd-r-drive-ws-client/Cargo.toml16 Cargo.lock:681-693

Server-Side Components

graph TB
    subgraph "muxio-tokio-rpc-server Crate"
        Server["RpcServer\nStruct"]
AxumApp["axum::Router\nwith WebSocket route"]
AcceptLoop["tokio::spawn\n(per connection)"]
ConnHandler["handle_connection\nasync fn"]
Dispatcher["RpcServiceEndpoint\n<ServiceImpl>"]
end
    
    subgraph "axum WebSocket Integration"
        Route["GET /ws\nWebSocket upgrade"]
WSUpgrade["axum::extract::ws\nWebSocketUpgrade"]
WSStream["axum::extract::ws\nWebSocket"]
end
    
    subgraph "Service Implementation"
        ServiceImpl["Arc<ServiceImpl>\n(e.g., DataStore)"]
Methods["#[async_trait]\nRpcService methods"]
end
    
    subgraph "Method Dispatch"
        MethodMap["HashMap<u64,\nBox<dyn Fn>>\n(method_id → handler)"]
end
    
 
   AxumApp -->|upgrade| WSUpgrade
 
   WSUpgrade -->|on_upgrade| WSStream
 
   WSStream -->|tokio::spawn| AcceptLoop
 
   AcceptLoop --> ConnHandler
 
   ConnHandler -->|recv Message::Binary| ConnHandler
 
   ConnHandler -->|bitcode::decode| ConnHandler
 
   ConnHandler -->|dispatch by id| MethodMap
 
   MethodMap -->|invoke| Methods
    Methods -.implemented by.- ServiceImpl
 
   Methods -->|return Result| ConnHandler
 
   ConnHandler -->|bitcode::encode| ConnHandler
 
   ConnHandler -->|send Message::Binary| WSStream
    
 
   Dispatcher -->|owns| MethodMap
 
   Dispatcher -->|holds Arc| ServiceImpl

muxio-tokio-rpc-server

The muxio-tokio-rpc-server crate implements the WebSocket server runtime with connection management and request dispatching:

Server Runtime with axum WebSocket Integration

The server runtime architecture:

Request Processing Pipeline

Each incoming request follows this pipeline:

Server Request Processing Pipeline with Code Entities

The dispatcher performs O(1) method lookup using the method_id hash from the HashMap, then invokes the corresponding service implementation. All service methods use #[async_trait], allowing concurrent request handling. The use of Arc<ServiceImpl> enables safe sharing of the DataStore across multiple client connections.

Sources: Cargo.lock:1320-1336 experiments/simd-r-drive-ws-server/Cargo.toml16 Cargo.lock:305-340

Request/Response Flow

Complete RPC Call Sequence

End-to-End RPC Call Flow

Message Format

The Muxio RPC wire protocol uses WebSocket binary frames with bitcode-encoded messages. The exact frame structure is managed by the muxio framework, but the logical message structure is:

The use of WebSocket binary frames and bitcode serialization provides:

• Compact encoding : Smaller than JSON or MessagePack
• Zero-copy potential : bitcode can deserialize without copying
• Type safety : Compile-time verification of message structure

Sources: Cargo.lock:133-143 Cargo.lock:648-656 Cargo.lock:1213-1222

Error Handling

The framework provides comprehensive error handling across the RPC boundary:

RPC Error Classification and Propagation

Error Categories

Error Recovery

The framework implements several recovery strategies:

• Connection loss : Client automatically attempts reconnection
• Request timeout : Client cancels pending request after configured duration
• Serialization failure : Error logged and generic error returned
• Invalid method ID : Server returns “method not found” error

Sources: Cargo.lock:1261-1336

Performance Characteristics

The Muxio RPC framework is optimized for high-performance remote storage access:

The use of bitcode serialization and WebSocket binary frames minimizes overhead compared to text-based protocols like JSON over HTTP. The multiplexed architecture allows clients to issue multiple concurrent requests without blocking, essential for high-performance batch operations.

Sources: Cargo.lock:392-414 Cargo.lock:1250-1336

Dismiss

Refresh this wiki

Enter email to refresh

Native Rust Client

Loading…

Native Rust Client

Relevant source files

• Cargo.lock
• experiments/simd-r-drive-muxio-service-definition/Cargo.toml
• experiments/simd-r-drive-ws-client/Cargo.toml
• experiments/simd-r-drive-ws-server/Cargo.toml

Purpose and Scope

The simd-r-drive-ws-client crate provides a native Rust client library for remote access to the SIMD R Drive storage engine via WebSocket connections. This client enables Rust applications to interact with a remote DataStore instance through the Muxio RPC framework with bitcode serialization.

This document covers the native Rust client implementation. For the WebSocket server that this client connects to, see WebSocket Server. For the Muxio RPC protocol details, see Muxio RPC Framework. For the Python bindings that wrap this client, see Python WebSocket Client API.

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:1-22

Architecture Overview

The native Rust client is structured as a thin wrapper around the Muxio RPC client infrastructure, providing type-safe access to remote DataStore operations.

Key Components:

graph TB
    subgraph "Application Layer"
        UserApp["User Application\nRust Code"]
end
    
    subgraph "simd-r-drive-ws-client Crate"
        ClientAPI["Client API\nDataStoreReader/Writer Traits"]
WsClient["WebSocket Client\nConnection Management"]
end
    
    subgraph "RPC Infrastructure"
        ServiceCaller["muxio-rpc-service-caller\nMethod Invocation"]
TokioRpcClient["muxio-tokio-rpc-client\nTransport Layer"]
end
    
    subgraph "Shared Contract"
        ServiceDef["simd-r-drive-muxio-service-definition\nRPC Interface"]
Bitcode["bitcode\nSerialization"]
end
    
    subgraph "Network"
        WsConnection["WebSocket Connection\ntokio-tungstenite"]
end
    
 
   UserApp --> ClientAPI
 
   ClientAPI --> WsClient
 
   WsClient --> ServiceCaller
 
   ServiceCaller --> TokioRpcClient
 
   ServiceCaller --> ServiceDef
 
   TokioRpcClient --> Bitcode
 
   TokioRpcClient --> WsConnection
    
    style ClientAPI fill:#f9f9f9,stroke:#333,stroke-width:2px
    style ServiceDef fill:#f9f9f9,stroke:#333,stroke-width:2px

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:13-21 Cargo.lock:1302-1318

Client API Structure

The client implements the same DataStoreReader and DataStoreWriter traits as the local DataStore, enabling transparent remote access with minimal API differences.

Core Traits:

graph LR
    subgraph "Trait Implementations"
        Reader["DataStoreReader\nread()\nexists()\nbatch_read()"]
Writer["DataStoreWriter\nwrite()\ndelete()\nbatch_write()"]
end
    
    subgraph "Client Implementation"
        WsClient["WebSocket Client\nAsync Methods"]
ConnState["Connection State\nURL, Options"]
end
    
    subgraph "RPC Layer"
        Serializer["Request Serialization\nbitcode"]
Caller["Service Caller\nCall Routing"]
Deserializer["Response Deserialization\nbitcode"]
end
    
 
   Reader --> WsClient
 
   Writer --> WsClient
 
   WsClient --> ConnState
 
   WsClient --> Serializer
 
   Serializer --> Caller
 
   Caller --> Deserializer
    
    style Reader fill:#f9f9f9,stroke:#333,stroke-width:2px
    style Writer fill:#f9f9f9,stroke:#333,stroke-width:2px

• DataStoreReader : Read-only operations (read, exists, batch_read, iteration)
• DataStoreWriter : Write operations (write, delete, batch_write)
• async-trait : All methods are asynchronous, requiring a Tokio runtime

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:14-21 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:1-16

Connection Management

The client manages persistent WebSocket connections to the remote server with automatic reconnection and error handling.

Connection Lifecycle:

sequenceDiagram
    participant App as "Application"
    participant Client as "WebSocket Client"
    participant Transport as "muxio-tokio-rpc-client"
    participant Server as "Remote Server"
    
    Note over App,Server: Connection Establishment
    App->>Client: connect(url)
    Client->>Transport: create WebSocket connection
    Transport->>Server: WebSocket handshake
    Server-->>Transport: connection established
    Transport-->>Client: client ready
    Client-->>App: connected client
    
    Note over App,Server: Normal Operation
    App->>Client: read(key)
    Client->>Transport: serialize request
    Transport->>Server: send via WebSocket
    Server-->>Transport: response data
    Transport-->>Client: deserialize response
    Client-->>App: return result
    
    Note over App,Server: Error Handling
    Server-->>Transport: connection lost
    Transport-->>Client: connection error
    Client->>Transport: reconnection attempt
    Transport->>Server: reconnect

1. Initialization : Client connects to server URL with connection options
2. Authentication : Optional authentication via Muxio RPC mechanisms
3. Active State : Client maintains persistent WebSocket connection
4. Error Recovery : Automatic reconnection on transient failures
5. Shutdown : Graceful connection termination

Sources: Cargo.lock:1302-1318 experiments/simd-r-drive-ws-client/Cargo.toml:16-19

graph TB
    subgraph "Client Side"
        Method["Client Method Call\nread/write/delete"]
ReqBuilder["Request Builder\nCreate RPC Request"]
Serializer["bitcode Serialization\nBinary Encoding"]
Sender["WebSocket Send\nBinary Frame"]
end
    
    subgraph "Network"
        WsFrame["WebSocket Frame\nBinary Message"]
end
    
    subgraph "Server Side"
        Receiver["WebSocket Receive\nBinary Frame"]
Deserializer["bitcode Deserialization\nBinary Decoding"]
Handler["Request Handler\nExecute DataStore Operation"]
Response["Response Builder\nCreate RPC Response"]
end
    
 
   Method --> ReqBuilder
 
   ReqBuilder --> Serializer
 
   Serializer --> Sender
 
   Sender --> WsFrame
 
   WsFrame --> Receiver
 
   Receiver --> Deserializer
 
   Deserializer --> Handler
 
   Handler --> Response
 
   Response --> Serializer
    
    style Method fill:#f9f9f9,stroke:#333,stroke-width:2px
    style Handler fill:#f9f9f9,stroke:#333,stroke-width:2px

Request-Response Flow

All client operations follow a standardized request-response pattern through the Muxio RPC framework.

Request Structure:

Response Structure:

Sources: experiments/simd-r-drive-muxio-service-definition/Cargo.toml:14-15 Cargo.lock:392-402

Async Runtime Requirements

The client requires a Tokio async runtime for all operations. The async-trait crate enables async methods in trait implementations.

Runtime Configuration:

graph TB
    subgraph "Application"
        Main["#[tokio::main]\nasync fn main()"]
UserCode["User Code\nawait client.read()"]
end
    
    subgraph "Client"
        AsyncMethods["async-trait Methods\nDataStoreReader/Writer"]
TokioTasks["Tokio Tasks\nNetwork I/O"]
end
    
    subgraph "Tokio Runtime"
        Executor["Task Executor\nWork Stealing Scheduler"]
Reactor["I/O Reactor\nepoll/kqueue/IOCP"]
end
    
 
   Main --> UserCode
 
   UserCode --> AsyncMethods
 
   AsyncMethods --> TokioTasks
 
   TokioTasks --> Executor
 
   TokioTasks --> Reactor
    
    style AsyncMethods fill:#f9f9f9,stroke:#333,stroke-width:2px
    style Executor fill:#f9f9f9,stroke:#333,stroke-width:2px

• Multi-threaded Runtime : Default for concurrent operations
• Current-thread Runtime : Available for single-threaded use cases
• Feature Flags : Requires tokio with rt-multi-thread and net features

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:19-21 Cargo.lock:279-287

Error Handling

The client propagates errors from multiple layers of the stack, providing detailed error information for debugging and recovery.

Error Types:

Error Propagation Flow:

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:17-18 Cargo.lock:1261-1271

Usage Patterns

Basic Connection and Operations

The client follows standard Rust async patterns for initialization and operation:

Concurrent Operations

The client supports concurrent operations through standard Tokio concurrency primitives:

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:19-21

graph TB
    subgraph "Service Definition"
        Methods["Method Definitions\nread, write, delete, etc."]
Types["Request/Response Types\nbitcode derive macros"]
MethodHash["Method ID Hashing\nXXH3 of method names"]
end
    
    subgraph "Client Usage"
        ClientImpl["Client Implementation\nUses defined methods"]
TypeSafety["Type Safety\nCompile-time checking"]
end
    
    subgraph "Server Usage"
        ServerImpl["Server Implementation\nHandles defined methods"]
Routing["Request Routing\nHash-based dispatch"]
end
    
 
   Methods --> ClientImpl
 
   Methods --> ServerImpl
 
   Types --> ClientImpl
 
   Types --> ServerImpl
 
   MethodHash --> Routing
 
   ClientImpl --> TypeSafety
    
    style Methods fill:#f9f9f9,stroke:#333,stroke-width:2px
    style Types fill:#f9f9f9,stroke:#333,stroke-width:2px

Integration with Service Definition

The client relies on the shared service definition crate for type-safe RPC communication.

Shared Contract Benefits:

• Type Safety : Compile-time verification of request/response types
• Version Compatibility : Client and server must use compatible service definitions
• Method Resolution : XXH3 hash-based method identification
• Serialization Schema : Consistent bitcode encoding across client and server

Sources: experiments/simd-r-drive-ws-client/Cargo.toml15 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:1-16

Performance Considerations

The native Rust client provides several performance advantages over alternative approaches:

Performance Characteristics:

Memory Efficiency:

• Zero-copy where possible through bitcode and WebSocket frames
• Efficient buffer reuse in Tokio’s I/O layer
• Minimal allocation overhead compared to HTTP-based protocols

Throughput:

• Supports request pipelining for high-throughput workloads
• Concurrent operations through Tokio’s work-stealing scheduler
• Batch operations reduce round-trip overhead

Sources: Cargo.lock:392-402 Cargo.lock:1302-1318

Comparison with Direct Access

The WebSocket client provides remote access with different tradeoffs compared to direct DataStore usage:

When to Use the Client:

• Remote access to centralized storage
• Microservice architectures requiring shared state
• Language interoperability (via Python bindings)
• Isolation of storage from compute workloads

When to Use Direct Access:

• Single-machine deployments
• Latency-critical applications
• Maximum throughput requirements
• Zero-copy read performance needs

Sources: experiments/simd-r-drive-ws-client/Cargo.toml14

Logging and Debugging

The client uses the tracing crate for structured logging and diagnostics.

Logging Levels:

• TRACE : Detailed RPC message contents and serialization
• DEBUG : Connection state changes, request/response flow
• INFO : Connection establishment, disconnection events
• WARN : Recoverable errors, retry attempts
• ERROR : Unrecoverable errors, connection failures

Diagnostic Information:

• Request/response timing
• Serialized message sizes
• Connection state transitions
• Error context and stack traces

Sources: experiments/simd-r-drive-ws-client/Cargo.toml20 Cargo.lock:279-287

Dismiss

Refresh this wiki

Enter email to refresh

Python Integration

Loading…

Python Integration

Relevant source files

• experiments/bindings/python-ws-client/Cargo.lock
• experiments/bindings/python-ws-client/README.md
• experiments/bindings/python-ws-client/extract_readme_tests.py
• experiments/bindings/python-ws-client/integration_test.sh
• experiments/bindings/python-ws-client/pyproject.toml
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/init.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi
• experiments/bindings/python-ws-client/uv.lock
• src/storage_engine/key_indexer.rs

This page provides an overview of the Python bindings for SIMD R Drive. The system offers two approaches for Python integration:

1. Modern WebSocket Client (simd-r-drive-ws-client-py): Communicates with a remote simd-r-drive-ws-server via WebSocket RPC. This is the primary, recommended approach documented in this section.
2. Legacy Direct Bindings (simd-r-drive-py): Directly embeds the Rust storage engine into Python. This approach is deprecated and not covered in detail here.

The WebSocket client bindings are implemented in Rust using PyO3, compiled to native Python extension modules (.so/.pyd), and distributed as platform-specific wheels via Maturin. The package is published as simd-r-drive-ws-client on PyPI.

Related Pages:

• Python WebSocket Client API - Detailed API reference
• Building Python Bindings - Build and packaging instructions
• Integration Testing - Testing infrastructure and workflows

Sources: experiments/bindings/python-ws-client/README.md:1-60 experiments/bindings/python-ws-client/pyproject.toml:1-6

Architecture Overview

The WebSocket client bindings use a layered architecture that bridges Python user code to the native Rust WebSocket client implementation. The package consists of pure Python wrapper code, PyO3-compiled Rust bindings, and the underlying simd-r-drive-ws-client Rust crate.

Diagram: Python Binding Architecture with Code Entities

graph TB
    subgraph "Python_Layer"
        UserCode["user_script.py"]
Import["from simd_r_drive_ws_client import DataStoreWsClient"]
end
    
    subgraph "Package_simd_r_drive_ws_client"
        InitPy["__init__.py"]
DataStoreWsClientPy["data_store_ws_client.py::DataStoreWsClient"]
TypeStubs["data_store_ws_client.pyi"]
end
    
    subgraph "PyO3_Native_Extension"
        BinaryModule["simd_r_drive_ws_client.so / .pyd"]
BaseDataStoreWsClient["BaseDataStoreWsClient"]
NamespaceHasher["NamespaceHasher"]
end
    
    subgraph "Rust_Dependencies"
        WsClient["simd-r-drive-ws-client crate"]
MuxioClient["muxio-tokio-rpc-client"]
ServiceDef["simd-r-drive-muxio-service-definition"]
end
    
 
   UserCode --> Import
 
   Import --> InitPy
 
   InitPy --> DataStoreWsClientPy
    DataStoreWsClientPy -.inherits.-> BaseDataStoreWsClient
    DataStoreWsClientPy -.types.-> TypeStubs
    
 
   BaseDataStoreWsClient --> WsClient
 
   NamespaceHasher --> WsClient
 
   BinaryModule --> BaseDataStoreWsClient
 
   BinaryModule --> NamespaceHasher
    
 
   WsClient --> MuxioClient
 
   WsClient --> ServiceDef

Architecture Layers

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/init.py:1-14 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:1-63 experiments/bindings/python-ws-client/README.md:12-15

Python API Surface

The simd-r-drive-ws-client package exposes two primary classes:

1. DataStoreWsClient - Main client for read/write operations
2. NamespaceHasher - Utility for generating collision-free namespaced keys

graph TB
    subgraph "Python_Wrapper"
        DSWsClient["data_store_ws_client.py::DataStoreWsClient"]
NSHasher["NamespaceHasher"]
end
    
    subgraph "PyO3_Bindings"
        BaseClient["BaseDataStoreWsClient"]
NSHasherImpl["NamespaceHasher_impl"]
end
    
    subgraph "Method_Sources"
        RustMethods["write()\nbatch_write()\ndelete()\nread()\nbatch_read()\nexists()\n__len__()\n__contains__()\nis_empty()\nfile_size()"]
PythonMethods["batch_read_structured()"]
end
    
 
   DSWsClient -->|inherits| BaseClient
 
   NSHasher -->|exposed via PyO3| NSHasherImpl
    
 
   BaseClient --> RustMethods
 
   DSWsClient --> PythonMethods
    
    RustMethods -.implemented in.-> WsClientCrate["simd-r-drive-ws-client"]

The API is implemented through a combination of Rust PyO3 bindings (BaseDataStoreWsClient) and Python wrapper code that adds convenience methods.

Diagram: Class Hierarchy and Method Implementation

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:11-63 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:8-219

Core Operations

DataStoreWsClient provides operations organized by implementation layer:

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:27-168 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:11-63

Python-Rust Method Mapping

Diagram: Method Call Flow from Python to Rust

The batch_read_structured() method demonstrates the hybrid approach:

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:12-62 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:109-129

PyO3 Binding Architecture

PyO3 provides the FFI layer that exposes Rust structs and methods as Python classes. The binding layer uses PyO3 procedural macros (#[pyclass], #[pymethods]) to generate Python-compatible wrappers around Rust types.

Diagram: PyO3 Macro Transformation Pipeline

graph TB
    subgraph "Rust_Source_Code"
        StructDef["#[pyclass]\nstruct BaseDataStoreWsClient"]
MethodsDef["#[pymethods]\nimpl BaseDataStoreWsClient"]
NSStruct["#[pyclass]\nstruct NamespaceHasher"]
NSMethods["#[pymethods]\nimpl NamespaceHasher"]
end
    
    subgraph "PyO3_Macro_Expansion"
        PyClassTrait["PyClass trait\nPyTypeInfo\nPyObjectProtocol"]
PyMethodsWrap["Method wrappers\nPyArg extraction\nResult conversion"]
end
    
    subgraph "Python_Extension_Module"
        PythonClass["BaseDataStoreWsClient\nwrite()\nread()\nbatch_write()"]
PythonNS["NamespaceHasher\n__init__()\nnamespace()"]
end
    
 
   StructDef --> PyClassTrait
 
   MethodsDef --> PyMethodsWrap
 
   NSStruct --> PyClassTrait
 
   NSMethods --> PyMethodsWrap
    
 
   PyClassTrait --> PythonClass
 
   PyMethodsWrap --> PythonClass
 
   PyClassTrait --> PythonNS
 
   PyMethodsWrap --> PythonNS

PyO3 Macro Functions

Sources: experiments/bindings/python-ws-client/Cargo.lock:832-846 experiments/bindings/python-ws-client/Cargo.lock:1096-1108

graph TB
    subgraph "Python_Async_Layer"
        PyAsyncCall["await client.write(key, data)"]
PyEventLoop["asyncio event loop"]
end
    
    subgraph "pyo3_async_runtimes_Bridge"
        Bridge["pyo3_async_runtimes::tokio"]
FutureConv["Future<Output=T> → PyObject"]
LocalSet["LocalSet spawning"]
end
    
    subgraph "Tokio_Runtime"
        TokioFuture["async fn write() → Future"]
TokioExecutor["Tokio thread pool"]
end
    
 
   PyAsyncCall --> PyEventLoop
 
   PyEventLoop --> Bridge
 
   Bridge --> FutureConv
 
   FutureConv --> LocalSet
 
   LocalSet --> TokioFuture
 
   TokioFuture --> TokioExecutor

Async Runtime Bridge

The Python bindings use pyo3-async-runtimes to bridge Python’s async/await model with Rust’s Tokio runtime. This enables Python code to call async Rust methods transparently.

Diagram: Python-Tokio Async Bridge

Runtime Bridge Components

The bridge automatically converts Rust Future<Output=T> values to Python awaitables, handling the differences in execution models between Python’s single-threaded async and Tokio’s work-stealing scheduler.

Sources: experiments/bindings/python-ws-client/Cargo.lock:849-860 experiments/bindings/python-ws-client/Cargo.lock:1287-1308

graph LR
    subgraph "Configuration"
        PyProject["pyproject.toml\n[build-system]\nbuild-backend = maturin"]
CargoToml["Cargo.toml\n[lib]\ncrate-type = ['cdylib']"]
end
    
    subgraph "Build_Process"
        RustcCompile["rustc\n--crate-type=cdylib\nPyO3 linking"]
CreateExtension["simd_r_drive_ws_client.so\nor .pyd"]
PackageWheel["maturin build\nAdd Python files\nAdd metadata"]
end
    
    subgraph "Artifacts"
        Wheel["simd_r_drive_ws_client-0.11.1-cp310-linux_x86_64.whl"]
PyPI["PyPI\npip install simd-r-drive-ws-client"]
end
    
 
   PyProject --> RustcCompile
 
   CargoToml --> RustcCompile
 
   RustcCompile --> CreateExtension
 
   CreateExtension --> PackageWheel
 
   PackageWheel --> Wheel
 
   Wheel --> PyPI

Build and Distribution System

The Python package is built using Maturin, which compiles Rust code to native extensions and packages them as platform-specific wheels. The build process produces binary wheels containing the compiled .so (Linux/macOS) or .pyd (Windows) extension module.

Diagram: Maturin Build and Distribution Pipeline

Sources: experiments/bindings/python-ws-client/pyproject.toml:29-35 experiments/bindings/python-ws-client/README.md:25-38

Build Configuration

pyproject.toml configures the build system and package metadata:

Build Commands

Sources: experiments/bindings/python-ws-client/pyproject.toml:1-47 experiments/bindings/python-ws-client/README.md:31-36

Platform and Python Version Support

The package is distributed as pre-compiled wheels for multiple Python versions and platforms.

Supported Configurations

Wheel Naming Convention

simd_r_drive_ws_client-{version}-{python_tag}-{platform_tag}.whl

Examples:
- simd_r_drive_ws_client-0.11.1-cp310-cp310-manylinux_2_17_x86_64.whl
- simd_r_drive_ws_client-0.11.1-cp312-cp312-macosx_11_0_arm64.whl
- simd_r_drive_ws_client-0.11.1-cp313-cp313-win_amd64.whl

Sources: experiments/bindings/python-ws-client/pyproject.toml:19-27 experiments/bindings/python-ws-client/README.md:18-23

Dependency Management

The package uses uv for Python dependency management and cargo for Rust dependencies. Runtime Python dependencies are zero—all Rust dependencies are statically compiled into the wheel.

Dependency Categories

Key Development Dependencies

Key Rust Dependencies

Sources: experiments/bindings/python-ws-client/pyproject.toml:37-46 experiments/bindings/python-ws-client/uv.lock:1-299 experiments/bindings/python-ws-client/Cargo.lock:1-1380

Type Stubs and IDE Support

The package includes .pyi type stub files that provide complete type information for IDEs and static type checkers like mypy.

Type Stub File:data_store_ws_client.pyi

Type Stub Features

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:1-219

graph LR
    Input1["Namespace prefix\ne.g., b'users'"]
Input2["Key\ne.g., b'user123'"]
Hash1["XXH3 hash\n8 bytes"]
Hash2["XXH3 hash\n8 bytes"]
Output["Namespaced key\n16 bytes total"]
Input1 -->|hash once at init| Hash1
 
   Input2 -->|hash per call| Hash2
 
   Hash1 --> Output
 
   Hash2 --> Output

graph LR
    subgraph "Input"
        Prefix["prefix: bytes\ne.g. b'users'"]
Key["key: bytes\ne.g. b'user123'"]
end
    
    subgraph "Hashing"
        XXH3_Prefix["XXH3(prefix)"]
XXH3_Key["XXH3(key)"]
end
    
    subgraph "Output"
        PrefixHash["8 bytes\nprefix_hash"]
KeyHash["8 bytes\nkey_hash"]
Combined["16 bytes total\nprefix_hash // key_hash"]
end
    
 
   Prefix --> XXH3_Prefix
 
   Key --> XXH3_Key
 
   XXH3_Prefix --> PrefixHash
 
   XXH3_Key --> KeyHash
 
   PrefixHash --> Combined
 
   KeyHash --> Combined

NamespaceHasher Utility

NamespaceHasher provides deterministic key namespacing using XXH3 hashing to prevent key collisions across logical domains.

Diagram: Namespace Key Derivation

Usage Example

Key Properties

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:170-219 src/storage_engine/key_indexer.rs:64-72

graph TB
    subgraph "integration_test.sh"
        Setup["cd experiments/\nBuild if needed"]
StartServer["cargo run --package\nsimd-r-drive-ws-server\n/tmp/simd-r-drive-pytest-storage.bin\n--host 127.0.0.1 --port 34129"]
SetupPython["uv venv\nuv pip install pytest maturin\nuv pip install -e . --group dev"]
ExtractTests["python extract_readme_tests.py"]
RunPytest["pytest -v -s\nTEST_SERVER_HOST=127.0.0.1\nTEST_SERVER_PORT=34129"]
Cleanup["kill -9 $SERVER_PID\nrm /tmp/simd-r-drive-pytest-storage.bin"]
end
    
 
   Setup --> StartServer
 
   StartServer --> SetupPython
 
   SetupPython --> ExtractTests
 
   ExtractTests --> RunPytest
 
   RunPytest --> Cleanup

Integration Test Infrastructure

The Python bindings include comprehensive integration tests that validate the entire stack, from Python client code to the WebSocket server and storage engine.

Diagram: Integration Test Workflow

Sources: experiments/bindings/python-ws-client/integration_test.sh:1-91

Test Components

The test infrastructure consists of multiple components working together:

Sources: experiments/bindings/python-ws-client/integration_test.sh:1-91 experiments/bindings/python-ws-client/extract_readme_tests.py:1-46

graph LR
    subgraph "Input_File"
        README["README.md"]
CodeBlocks["```python\ncode\n```"]
end
    
    subgraph "Extraction_Logic"
        Regex["re.compile(r'```python\\n(.*?)```', re.DOTALL)"]
Extract["pattern.findall(text)"]
Wrap["def test_readme_block_{i}():\n {indented_code}"]
end
    
    subgraph "Output_File"
        TestFile["tests/test_readme_blocks.py"]
TestFunctions["test_readme_block_0()\ntest_readme_block_1()\ntest_readme_block_N()"]
end
    
 
   README --> CodeBlocks
 
   CodeBlocks --> Regex
 
   Regex --> Extract
 
   Extract --> Wrap
 
   Wrap --> TestFile
 
   TestFile --> TestFunctions

README Test Extraction

extract_readme_tests.py automatically extracts Python code blocks from the README and generates pytest test functions, ensuring documentation examples remain accurate.

Diagram: README to Pytest Pipeline

Extraction Process

This ensures documentation examples are automatically validated on every test run, preventing drift between documentation and implementation.

Sources: experiments/bindings/python-ws-client/extract_readme_tests.py:14-45

graph TB
    subgraph "Internal Modules"
        RustBinary["simd_r_drive_ws_client_py.so/.pyd\nBinary compiled module"]
RustSymbols["BaseDataStoreWsClient\nNamespaceHasher\nsetup_logging\ntest_rust_logging"]
PythonWrapper["data_store_ws_client.py\nDataStoreWsClient"]
end
    
    subgraph "Package __init__.py"
        ImportRust["from .simd_r_drive_ws_client import\n setup_logging, test_rust_logging"]
ImportPython["from .data_store_ws_client import\n DataStoreWsClient, NamespaceHasher"]
AllList["__all__ = [\n 'DataStoreWsClient',\n 'NamespaceHasher',\n 'setup_logging',\n 'test_rust_logging'\n]"]
end
    
    subgraph "Public API"
        UserCode["from simd_r_drive_ws_client import DataStoreWsClient"]
end
    
 
   RustBinary --> RustSymbols
 
   RustSymbols --> ImportRust
 
   PythonWrapper --> ImportPython
    
 
   ImportRust --> AllList
 
   ImportPython --> AllList
 
   AllList --> UserCode

Package Exports and Public API

The package’s public API is defined through the __init__.py file, which controls what symbols are available when users import the package.

Export Structure

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/init.py:1-14

The __all__ list explicitly defines the public API surface, preventing internal implementation details from being accidentally imported by users. This follows Python best practices for package design.

Dismiss

Refresh this wiki

Enter email to refresh