Overview

Loading…

Overview

Relevant source files

• CHANGELOG.md
• Cargo.toml
• README.md

SIMD R Drive is a high-performance, thread-safe storage engine designed for zero-copy binary access within a single-file container. It is optimized for SIMD-accelerated workloads and cache-friendly data processing, providing a schema-less, append-only architecture that bridges the gap between raw filesystem performance and structured key-value access README.md:5-9

The system is designed to handle datasets larger than available RAM by leveraging memory mapping (mmap), allowing transparent access to specific file segments while minimizing memory pressure README.md:43-50

Key Design Goals

The architecture of SIMD R Drive is centered around four primary pillars:

• Append-Only Design : High write throughput is achieved by using sequential, append-based writes, which minimizes disk seek overhead and ensures data integrity through a backward-linked validation chain README.md:98-103
• Zero-Copy Access : By memory-mapping the storage file, the engine provides direct access to data payloads via the EntryHandle. This allows applications to use data without the overhead of deserialization or intermediate buffering README.md:43-50 simd-r-drive-entry-handle/src/lib.rs:1-10
• SIMD-Optimized : Payloads are written at fixed 64-byte aligned boundaries (matching typical CPU cachelines). This ensures that SIMD/vector loads (AVX, NEON, etc.) can operate at maximum hardware speed without crossing cacheline boundaries README.md:51-60 CHANGELOG.md:92-97
• Schema-less Flexibility : The engine treats payloads as raw bytes (&[u8]). It makes no assumptions about data format, endianness, or structure, allowing it to store anything from simple strings to nested storage containers or executable binaries README.md:61-88

System Architecture and Code Entities

The following diagram illustrates how the natural language concepts of the storage engine map to specific code entities and traits within the Rust implementation.

Storage Engine Logic Mapping

graph TD
    subgraph "Natural Language Space"
        "Append-Only_Storage"["Append-Only Storage"]
        "Zero-Copy_Handles"["Zero-Copy Handles"]
        "Key_Indexing"["Key Indexing"]
        "Memory_Mapping"["Memory Mapping"]
end

    subgraph "Code Entity Space (simd-r-drive)"
        "Append-Only_Storage" --> "DataStore"["DataStore struct"]
        "DataStore" --> "DataStoreWriter"["DataStoreWriter trait"]
        "DataStore" --> "DataStoreReader"["DataStoreReader trait"]
        
        "Zero-Copy_Handles" --> "EntryHandle"["EntryHandle struct"]
        
        "Key_Indexing" --> "KeyIndexer"["KeyIndexer struct"]
        "KeyIndexer" --> "Xxh3BuildHasher"["Xxh3BuildHasher"]
        
        "Memory_Mapping" --> "Mmap"["memmap2::Mmap"]
        "Mmap" --> "EntryHandle"
    end

    subgraph "Files"
        "DataStore" --- "src_storage_engine_mod"["src/storage_engine/mod.rs"]
        "EntryHandle" --- "entry_handle_src"["simd-r-drive-entry-handle/src/lib.rs"]
        "KeyIndexer" --- "indexer_src"["src/storage_engine/index/key_indexer.rs"]
end

Sources: README.md:5-9 Cargo.toml:80-91 simd-r-drive-entry-handle/src/lib.rs:1-10

Workspace Layout

The project is organized as a Cargo workspace containing the core engine, shared data types, and experimental extensions.

Workspace Entity Relationships

Sources: Cargo.toml:14-27 Cargo.toml:60-63

Navigation

For detailed technical information, please refer to the following sub-sections:

• Getting Started : Instructions for building the project, understanding Cargo features (like parallel, arrow, and expose-internal-api), and using the CLI quick-start. For details, see Getting Started.
• Repository Layout : A comprehensive map of the directory structure, including the core library, entry-handle crate, extensions, experiments, and CI configurations. For details, see Repository Layout.

Sources: README.md:1-112 Cargo.toml:1-112 CHANGELOG.md:1-145

Core Storage Engine

Loading…

Core Storage Engine

Relevant source files

• src/lib.rs
• src/storage_engine/data_store.rs

The core storage engine of rust-simd-r-drive is an append-only, high-performance data store designed for zero-copy reads and SIMD-optimized writes. It is implemented primarily within the src/storage_engine/ directory and revolves around the DataStore struct src/storage_engine/data_store.rs:27-33

The engine utilizes memory-mapped files via the memmap2 crate to provide high-speed access to data while maintaining a simple, immutable-on-disk structure src/lib.rs:7-17 By treating the file as a continuous log, it avoids the complexities of in-place updates and fragmentation, relying on a reverse-traversal iterator and a background compaction process to manage data lifecycle and reclaim space src/storage_engine/data_store.rs:35-51

System Architecture

The storage engine is composed of several interlocking components that manage data from initial write through recovery and eventual compaction.

High-Level Component Interaction

The following diagram illustrates how the core entities interact during standard operations.

Storage Engine Component Map

graph TD
    subgraph "Public API [src/traits.rs]"
        DR["DataStoreReader"]
DW["DataStoreWriter"]
end
    
    subgraph "Core Logic [src/storage_engine/data_store.rs]"
        DS["DataStore Struct"]
DS -.-> |implements| DR
 
       DS -.-> |implements| DW
    end

    subgraph "Indexing & Search"
        KI["KeyIndexer [src/storage_engine/key_indexer.rs]"]
EH["EntryHandle [simd-r-drive-entry-handle]"]
end

    subgraph "Disk & Memory"
        MM["Mmap [memmap2]"]
BW["BufWriter [std::io]"]
FILE["On-Disk .bin File"]
end

 
   DS --> ["Arc<RwLock<BufWriter<File>>>"]
DS --> ["Arc<Mutex<Arc<Mmap>>>"]
DS --> ["AtomicU64 tail_offset"]
DS --> ["Arc<RwLock<KeyIndexer>>"]
    
    ["Arc<RwLock<BufWriter<File>>>"] -->
 FILE
    ["Arc<Mutex<Arc<Mmap>>>"] --> FILE
 
   DR --> EH
 
   EH -.-> |points into| MM

Sources: src/storage_engine/data_store.rs:27-33 src/storage_engine/data_store.rs7 src/storage_engine/key_indexer.rs:15-25

Key Components

DataStore

The DataStore struct is the primary handle for interacting with the storage engine src/storage_engine/data_store.rs:27-33 It manages the file handles, memory maps, and the in-memory key index. It implements the DataStoreReader and DataStoreWriter traits to provide a standardized API for operations like write(), read(), and batch_write() src/storage_engine/data_store.rs:66-117

• For details, see DataStore: Read and Write Operations.

On-Disk Format

Data is stored as a sequence of entries. Each entry consists of an optional pre-pad for alignment, a variable-length payload, and a fixed-size EntryMetadata footer simd-r-drive-entry-handle/src/lib.rs10 To ensure SIMD efficiency, payloads are aligned to 64-byte boundaries (PAYLOAD_ALIGNMENT) src/storage_engine/constants.rs:3-4 The file also maintains a backward-linked validation chain via prev_offset fields in the metadata simd-r-drive-entry-handle/src/lib.rs10

• For details, see On-Disk Storage Format.

Key Indexer and Hashing

To avoid scanning the entire file for every read, the engine maintains an in-memory KeyIndexer src/storage_engine/data_store.rs:107-114 This index maps 64-bit XXH3 hashes of keys to their most recent file offsets src/storage_engine/data_store.rs:2-4 It allows for O(1) lookup performance by keeping the mapping of the latest version of every key in RAM.

• For details, see Key Indexer and Hashing.

Entry Iterator and Compaction

Since the store is append-only, updates to a key result in a new entry being written at the end of the file. The EntryIterator traverses the file in reverse (newest to oldest) using the prev_offset pointers to ensure only the latest version of a key is retrieved src/storage_engine/entry_iterator.rs:24-34 Compaction reclaims space by using this iterator to identify and rewrite only “live” entries into a fresh file src/storage_engine/data_store.rs:555-562

• For details, see Entry Iterator and Compaction.

Concurrency and Thread Safety

The engine supports concurrent access through a layered locking strategy. It uses RwLock<BufWriter<File>> for serialized appends, RwLock<KeyIndexer> for index updates, and Mutex<Arc<Mmap>> to safely manage memory-map remapping during file growth src/storage_engine/data_store.rs:28-31 An AtomicU64 tracks the tail_offset to allow progress checks without acquiring heavy locks src/storage_engine/data_store.rs113

• For details, see Concurrency and Thread Safety.

sequenceDiagram
    participant User
    participant DS as "DataStore::write()"
    participant SC as "simd_copy()"
    participant KI as "KeyIndexer::insert()"
    participant MM as "remap_and_index()"

    User->>DS: key, value
    DS->>DS: compute_hash(key)
    DS->>SC: copy value to BufWriter
    DS->>DS: EntryMetadata::serialize()
    DS->>KI: update index with offset
    DS->>MM: refresh memory map and KeyIndexer
    DS-->>User: Success

Data Flow: Write vs Read

The following diagram bridges the gap between the high-level logic and the specific code entities involved in the data lifecycle.

Logic to Entity Mapping

Sources: src/storage_engine/data_store.rs:176-195 src/storage_engine/data_store.rs:2-5 src/storage_engine/data_store.rs:320-335

Child Pages

Sources: src/lib.rs:1-137 src/storage_engine/data_store.rs:1-117 src/storage_engine/constants.rs:1-7

Getting Started

Loading…

Getting Started

Relevant source files

• .cargo/config.toml.example
• .github/workflows/python-net-release.yml
• .github/workflows/rust-tests.yml
• .gitignore
• Cargo.toml

This page provides the technical details required to build, configure, and run the SIMD R Drive project locally. It covers the workspace architecture, the role of various Cargo features, and a quick-start guide for the Command Line Interface (CLI).

Workspace Structure

The project is organized as a Cargo workspace to separate the core storage engine from its various extensions, handles, and experimental network layers. The workspace uses resolver = "2" for modern dependency resolution Cargo.toml27

Workspace Members

The workspace is defined in the root Cargo.toml and includes the following crates:

The workspace also explicitly excludes Python bindings from the Cargo resolver to allow them to be managed by maturin and uv Cargo.toml:23-26

Sources: Cargo.toml:14-27 Cargo.toml:60-63

Logical Component Diagram

The following diagram illustrates the relationship between the workspace members and how they interact to form the system.

Workspace Entity Map

graph TD
    subgraph "Core_Workspace"
        A["simd-r-drive (Core & CLI)"]
B["simd-r-drive-entry-handle"]
end

    subgraph "Extensions_Crate"
        C["simd-r-drive-extensions"]
end

    subgraph "Network_Experiments"
        D["simd-r-drive-ws-server"]
E["simd-r-drive-ws-client"]
F["simd-r-drive-muxio-service-definition"]
end

 
   A --> B
 
   C --> A
 
   D --> A
 
   D --> F
 
   E --> F
 
   B -.->|Zero-copy view| A

Sources: Cargo.toml:14-22 Cargo.toml88

Cargo Features

The project uses feature gates to manage dependencies and performance characteristics. These can be toggled during compilation using --features.

Sources: Cargo.toml:72-78 Cargo.toml87

Local Configuration

.cargo/config.toml.example

The project provides a .cargo/config.toml.example file. This is particularly useful for developers working on the experimental network layers who may need to patch dependencies like muxio-rpc-service-caller or muxio-tokio-rpc-client to local paths for simultaneous development across repositories .cargo/config.toml.example:1-5

The .gitignore is configured to ignore the actual .cargo/config.toml, allowing developers to maintain local overrides without committing them .gitignore12

Sources: .cargo/config.toml.example:1-5 .gitignore12

CLI Quick-Start

The simd-r-drive CLI is the primary entry point for interacting with the storage engine from the terminal. It uses clap for argument parsing Cargo.toml82 and supports streaming data via stdin/stdout.

Core Commands

The CLI supports various operations including metadata inspection and data manipulation.

Sources: Cargo.toml82

sequenceDiagram
    participant User
    participant CLI as "Cli (clap::Parser)"
    participant EXEC as "execute_command (cli_parser.rs)"
    participant DS as "DataStore (storage_engine)"

    User->>CLI: simd-r-drive data.bin write my_key 'my_value'
    CLI->>EXEC: execute_command(&cli)
    EXEC->>DS: DataStore::open(path)
    alt "Direct Value"
        EXEC->>DS: DataStore::write(key, value)
    else "Piped Stdin"
        EXEC->>DS: DataStore::write_stream(key, &mut stdin)
    end
    DS-->>User: Stored 'my_key'

Data Flow: CLI Execution

The following diagram shows the data flow when executing a write command via the CLI, mapping CLI entities to core storage logic.

CLI Write Data Flow

Sources: Cargo.toml82

Common Usage Examples

1. Writing a value:

2. Piping a file into storage:

3. Reading with a specific buffer size:

4. Compacting storage:

Building and Testing

Compilation

To build the workspace with all targets and specified features:

The CI pipeline validates builds across ubuntu-latest, macos-latest, and windows-latest .github/workflows/rust-tests.yml:17-22

Running Tests

The project includes extensive integration tests. The parallel feature flag enables parallel iteration tests via Rayon Cargo.toml75

The CI also ensures that benchmarks compile using the --no-run flag .github/workflows/rust-tests.yml:65-66

For Python developers, the python-net-release.yml workflow demonstrates how to run integration tests for the WebSocket client using uv and a dedicated shell script .github/workflows/python-net-release.yml:35-45

Sources: .github/workflows/rust-tests.yml:17-66 .github/workflows/python-net-release.yml:35-45 Cargo.toml75

On-Disk Storage Format

Loading…

On-Disk Storage Format

Relevant source files

• README.md
• assets/storage-layout.png
• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/entry_metadata.rs
• src/storage_engine.rs
• src/storage_engine/constants.rs
• src/storage_engine/entry_iterator.rs

SIMD R Drive utilizes a high-performance, append-only binary storage format. The layout is specifically engineered for zero-copy memory-mapped access and SIMD-optimized processing by enforcing strict alignment and minimal metadata overhead.

Binary Layout Overview

The storage file consists of a sequence of entries. Every entry is either an Aligned Entry (containing a payload) or a Tombstone (representing a deletion). To ensure that payloads can be cast to typed slices (e.g., &[u64]) without memory copies, the engine enforces a fixed alignment boundary README.md:51-55

Alignment and Padding

The system uses a PAYLOAD_ALIGNMENT of 64 bytes by default src/storage_engine/constants.rs1 This matches typical CPU cacheline sizes, preventing SIMD/vector loads from crossing cacheline boundaries README.md:51-53

When a new entry is written, the engine calculates a pre-pad based on the current file tail: pad = (A - (prev_tail % A)) & (A - 1), where A = PAYLOAD_ALIGNMENT simd-r-drive-entry-handle/src/entry_metadata.rs22 This calculation is implemented in EntryIterator::prepad_len to correctly locate payloads during backward traversal src/storage_engine/entry_iterator.rs:50-53

Entry Structures

1. Aligned Entry (Non-Tombstone)

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:11-23 src/storage_engine/entry_iterator.rs:86-88

2. Tombstone (Deletion Marker)

Tombstones are treated as special cases. While modern tombstones follow the alignment rules, the EntryIterator maintains backward compatibility for legacy tombstones that do not include a pre-pad src/storage_engine/entry_iterator.rs:92-98 They use a single NULL_BYTE (0x00) as a payload marker src/storage_engine/constants.rs4

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:25-30 src/storage_engine/constants.rs4 src/storage_engine/entry_iterator.rs:117-119

Metadata Implementation: EntryMetadata

The EntryMetadata struct is the core descriptor for every entry. It is stored immediately following the payload.

Data Structure

The struct uses #[repr(C)] to ensure a stable memory layout for direct deserialization from the memory map simd-r-drive-entry-handle/src/entry_metadata.rs:44-50

Serialization

Metadata is stored using Little-Endian encoding for numeric values simd-r-drive-entry-handle/src/entry_metadata.rs:70-71 The serialize and deserialize functions handle the conversion between the struct and the 20-byte fixed-size array defined by METADATA_SIZE simd-r-drive-entry-handle/src/entry_metadata.rs:75-112

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:44-112

The Backward-Linked Validation Chain

SIMD R Drive uses a backward-linked chain via the prev_offset field. Unlike traditional linked lists that point to the start of an entry, prev_offset stores the absolute offset of the previous tail simd-r-drive-entry-handle/src/entry_metadata.rs:32-34 This allows the reader to derive the current payload’s start by calculating the pre-pad length relative to that previous tail src/storage_engine/entry_iterator.rs:86-87

Data Flow: Writing and Linking

The following diagram illustrates how the DataStore logic calculates alignment and links the new EntryMetadata.

Entry Writing and Alignment Flow

graph TD
    subgraph "DataStore_Logic"
        ["Start write(key, payload)"] --> ["Get current tail_offset"]
        ["Get current tail_offset"] --> ["Calculate pre-pad via EntryIterator::prepad_len"]
        ["Calculate pre-pad via EntryIterator::prepad_len"] --> ["Compute XXH3 key_hash"]
        ["Compute XXH3 key_hash"] --> ["Compute CRC32C checksum"]
        ["Compute CRC32C checksum"] --> ["Set prev_offset = tail_offset"]
        ["Set prev_offset = tail_offset"] --> ["Construct EntryMetadata"]
end

    subgraph "Disk_Layout"
        ["Construct EntryMetadata"] --> ["Write padding bytes (0..63 bytes)"]
        ["Write padding bytes (0..63 bytes)"] --> ["Write payload bytes"]
        ["Write payload bytes"] --> ["Write serialized EntryMetadata (20 bytes)"]
end

    ["Write serialized EntryMetadata (20 bytes)"] --> ["Update KeyIndexer with metadata_offset"]

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:11-36 src/storage_engine/entry_iterator.rs:50-53

Integrity and Checksums

Integrity is maintained through the checksum field in EntryMetadata. This is a CRC32C hash of the raw payload bytes simd-r-drive-entry-handle/src/entry_metadata.rs19

Verification Process

The EntryIterator traverses the file backward from a given tail_offset src/storage_engine/entry_iterator.rs:16-17 For each entry, it:

1. Deserializes the 20-byte metadata block using EntryMetadata::deserialize src/storage_engine/entry_iterator.rs:80-81
2. Derives the entry_start by adding the calculated prepad_len to the prev_offset src/storage_engine/entry_iterator.rs:86-87
3. Identifies the entry_end as the start of the metadata block src/storage_engine/entry_iterator.rs89
4. Returns an EntryHandle which can be used to verify the checksum against the mapped payload src/storage_engine/entry_iterator.rs:121-125

Code Entity Mapping: Storage Logic

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:75-76 src/storage_engine/entry_iterator.rs:21-47 src/storage_engine/entry_iterator.rs:69-126

DataStore: Read and Write Operations

Loading…

DataStore: Read and Write Operations

Relevant source files

• experiments/bindings/python_(old_client)/LICENSE/LICENSE)
• src/storage_engine/data_store.rs
• src/storage_engine/entry_stream.rs
• src/storage_engine/traits/reader.rs
• src/storage_engine/traits/writer.rs
• tests/basic_operations_tests.rs
• tests/batch_ops_tests.rs

The DataStore is the central entity of the rust-simd-r-drive storage engine. It provides a thread-safe, append-only interface for managing data with a focus on zero-copy reads and SIMD-accelerated writes. Operations are primarily defined through the DataStoreReader and DataStoreWriter traits, allowing for high-level abstractions over the underlying memory-mapped file.

Lifecycle and Initialization

A DataStore instance manages a buffered file writer, a memory-mapped view (Mmap), an atomic tail offset, and an in-memory KeyIndexer src/storage_engine/data_store.rs:27-33

Opening a Store

The DataStore::open() function is the primary entry point src/storage_engine/data_store.rs:84-117 It performs the following sequence:

1. File Opening : Opens the file in read/write mode, creating it if it doesn’t exist via open_file_in_append_mode src/storage_engine/data_store.rs:161-170
2. Mmap Initialization : Maps the file into memory via init_mmap src/storage_engine/data_store.rs:172-174
3. Chain Recovery : Validates the backward-linked integrity chain to find the last valid entry via recover_valid_chain src/storage_engine/data_store.rs89
4. Truncation : If corruption is detected (i.e., the valid chain is shorter than the file length), the file is truncated to the last known good state src/storage_engine/data_store.rs:91-104
5. Index Building : The KeyIndexer is populated by scanning the validated file content src/storage_engine/data_store.rs108

Entity Relationship: Initialization

The following diagram illustrates the transition from a file system path to an active DataStore instance.

graph TD
    subgraph "Natural Language Space"
        Path["File System Path"]
Recovery["Integrity Recovery"]
end

    subgraph "Code Entity Space"
        DS_Open["DataStore::open(path)"]
OpenAppend["open_file_in_append_mode()"]
InitMmap["init_mmap()"]
Recover["recover_valid_chain()"]
BuildIdx["KeyIndexer::build()"]
DS_Struct["struct DataStore"]
end

 
   Path --> DS_Open
 
   DS_Open --> OpenAppend
 
   OpenAppend --> InitMmap
 
   InitMmap --> Recover
 
   Recovery -.-> Recover
 
   Recover --> BuildIdx
 
   BuildIdx --> DS_Struct

Title: DataStore Initialization Flow

Sources: src/storage_engine/data_store.rs:66-117 src/storage_engine/data_store.rs:161-174

Write Operations

Write operations are defined in the DataStoreWriter trait src/storage_engine/traits/writer.rs:4-152 All writes are append-only.

Standard and Batch Writes

• write(key, payload) : Computes the key hash and appends the payload followed by EntryMetadata src/storage_engine/traits/writer.rs65
• batch_write(entries) : Writes multiple key-value pairs in a single operation src/storage_engine/traits/writer.rs106 This is more efficient as it acquires the write lock only once src/storage_engine/traits/writer.rs131
• batch_write_with_key_hashes : The low-level implementation for batching. It uses simd_copy for high-performance data movement and performs a single mmap remapping after the batch is complete src/storage_engine/traits/writer.rs:108-138

Streaming Writes

For payloads larger than available RAM, write_stream allows writing data from any source implementing std::io::Read src/storage_engine/traits/writer.rs29

• Buffer Size : Data is read in 64KB chunks (WRITE_STREAM_BUFFER_SIZE) src/storage_engine/traits/writer.rs25
• Incremental Checksum : The CRC32C checksum is updated as chunks are streamed to disk src/storage_engine/traits/writer.rs27

Write Data Flow

Title: Write Operation Data Flow

Sources: src/storage_engine/traits/writer.rs:5-138 src/storage_engine/data_store.rs:176-210

Read Operations

Read operations are defined in the DataStoreReader trait src/storage_engine/traits/reader.rs:4-160

Zero-Copy Access

The read(key) method returns an Option<EntryHandle> src/storage_engine/traits/reader.rs54

1. The key is hashed, and the KeyIndexer provides the file offset src/storage_engine/traits/reader.rs:41-42
2. The EntryHandle contains an Arc<Mmap>, allowing the application to access the payload as a slice (&[u8]) without copying data from the kernel buffer to user space src/storage_engine/traits/reader.rs:89-90

Batch and Hashed Reads

• batch_read(keys) : Vectorized lookup for multiple keys. It takes a read lock on the index once for the entire batch src/storage_engine/traits/reader.rs101
• batch_read_hashed_keys : Optimized version for callers who already have precomputed hashes. It includes an optional verification step to handle hash collisions by comparing the original keys src/storage_engine/traits/reader.rs:134-138

Streaming Reads

The EntryStream struct wraps an EntryHandle to provide a std::io::Read interface src/storage_engine/entry_stream.rs:44-47 Note that while the EntryHandle is zero-copy, the EntryStream::read() method does perform copies into the provided buffer src/storage_engine/entry_stream.rs:76-91

Sources: src/storage_engine/traits/reader.rs:4-138 src/storage_engine/entry_stream.rs:1-92

Management Operations

Deletion and Compaction

• delete(key) : Appends a “tombstone” entry to the file src/storage_engine/traits/writer.rs145 The key is removed from the KeyIndexer, making it immediately invisible to readers src/storage_engine/traits/reader.rs:11-12
• compact() : Reclaims space by creating a new storage file and copying only “live” (non-deleted, latest version) entries into it. This uses the EntryIterator to traverse the store src/storage_engine/data_store.rs:420-475

Copy, Move, and Rename

• copy(key, target_store) : Reads an entry from the current store and writes it to a different DataStore instance src/storage_engine/traits/writer.rs152
• transfer(key, target_store) : Performs a copy to the target store and then deletes the key from the source store (equivalent to a “move”) src/storage_engine/traits/writer.rs152
• rename(old_key, new_key) : Copies an entry to a new key within the same store and deletes the old key src/storage_engine/traits/writer.rs:140-152

Summary of Traits

Sources: src/storage_engine/traits/reader.rs:4-160 src/storage_engine/traits/writer.rs:4-152 src/storage_engine/data_store.rs:420-475

Repository Layout

Loading…

Repository Layout

Relevant source files

• Cargo.lock
• Cargo.toml
• LICENSE
• experiments/bindings/python-ws-client/LICENSE

The SIMD R Drive repository is organized as a Cargo workspace containing a core storage engine, specialized utility crates, high-level extensions, and experimental network-based components. The project emphasizes a clean separation between the low-level binary storage logic and the high-level interfaces (CLI, RPC, and Python bindings) that consume it.

Workspace Structure Overview

The repository follows a modular architecture where the core logic is kept lean, and additional functionality is opted into via sub-crates or feature flags.

Sources: Cargo.toml:14-22 Cargo.toml:60-63

Component Relationship Diagram

The following diagram illustrates how the various crates and directories interact within the workspace.

Workspace Dependency and Data Flow

graph TD
    subgraph "Core_Workspace"
        CORE["simd-r-drive_(src/)"]
HANDLE["simd-r-drive-entry-handle"]
EXT["simd-r-drive-extensions"]
end

    subgraph "Experiments"
        WS_SERV["simd-r-drive-ws-server"]
WS_CLI["simd-r-drive-ws-client"]
DEF["simd-r-drive-muxio-service-definition"]
end

    subgraph "Bindings"
        PY_WS["python-ws-client_(PyO3)"]
end

    EXT -- "uses" --> CORE
    CORE -- "depends_on" --> HANDLE
    WS_SERV -- "wraps" --> CORE
    WS_SERV -- "implements" --> DEF
    WS_CLI -- "calls" --> DEF
    PY_WS -- "wraps" --> WS_CLI

Sources: Cargo.toml:14-22 Cargo.toml:60-63 Cargo.toml88

1. Core Library (src/)

The root directory contains the primary simd-r-drive crate. This is the heart of the project, responsible for the append-only storage format, memory-mapped I/O, and SIMD-accelerated operations.

• src/storage_engine/ : Implementation of DataStore, which manages file handles and the KeyIndexer.
• src/utils/ : Utility functions and performance-critical helpers, including align_or_copy and NamespaceHasher.
• src/lib.rs : The main entry point for the library, exporting core traits like DataStoreReader and DataStoreWriter.
• SIMD Copy : Specialized implementations for x86_64 (AVX2) and aarch64 (NEON) are housed in the storage engine.

Sources: Cargo.toml:2-12 Cargo.toml:80-91

2. Entry Handle Crate (simd-r-drive-entry-handle/)

This is a standalone crate designed to be as lightweight as possible. It defines the EntryHandle type, which provides a zero-copy view into the memory-mapped storage.

• Role : Allows third-party crates to read SIMD R Drive data without pulling in the full storage engine or its heavy dependencies.
• Key Entities : EntryHandle provides the primary interface for accessing payloads and validating checksums.
• Features : Includes an optional arrow feature to provide Apache Arrow buffer compatibility via simd-r-drive-entry-handle/arrow. Cargo.toml78

Sources: Cargo.toml21 Cargo.toml61 Cargo.toml78

3. Extensions Crate (extensions/)

The simd-r-drive-extensions crate provides high-level storage patterns built on top of the base DataStore.

• TTL Support : Adds Time-To-Live metadata to entries via StorageCacheExt.
• Option Storage : Differentiates between a missing key and a key explicitly set to None via StorageOptionExt.
• Filesystem Import : Utilities to recursively import local directories into a storage file via StorageFileImportExt using the walkdir dependency.

Sources: Cargo.toml20 Cargo.toml69

4. Experiments Directory (experiments/)

This directory houses work-in-progress or network-related components that are not part of the core “local-first” storage philosophy.

WebSocket RPC Stack

The system uses muxio for multiplexed asynchronous RPC over WebSockets.

• simd-r-drive-muxio-service-definition : Shared trait definitions and bitcode serialization schemas for RPC calls. Cargo.toml17 Cargo.toml62
• simd-r-drive-ws-server : A tokio-based server that exposes a DataStore instance over a network port. Cargo.toml19 Cargo.toml66
• simd-r-drive-ws-client : An async client implementing RPC-based access to a remote DataStore. Cargo.toml18 Cargo.toml63

Sources: Cargo.toml:17-19 Cargo.toml:62-63

5. Bindings and CI

The repository includes infrastructure for cross-language support and automated quality assurance.

• Python Bindings : Located in experiments/bindings/. These sub-projects (e.g., python-ws-client) are excluded from the main Cargo workspace to avoid mandatory local dependency on Python development headers during standard Rust builds. Cargo.toml:23-26
• CI Configuration : Found in .github/workflows/. It covers Rust linting, multi-platform testing (Ubuntu, macOS, Windows), and various feature flag combinations including parallel (which enables rayon) and expose-internal-api. Cargo.toml:74-75

Sources: Cargo.toml:23-26 Cargo.toml:74-75

Code Entity Mapping

The following diagrams map high-level repository concepts to specific code structures and file locations.

Entity Mapping: Storage and Metadata

classDiagram
    class DataStore {<<src/storage_engine/mod.rs>>\n+write(key, payload)\n+read(key)\n+compact()\n+write_stream(key, reader)}
    class DataStoreReader {<<src/storage_engine/traits.rs>>\n+read(key)\n+exists(key)}
    class EntryHandle {<<simd-r-drive-entry-handle/src/lib.rs>>\n+as_slice()\n+is_valid_checksum()}
    class EntryStream {<<src/storage_engine/entry_stream.rs>>\n+read(buf)\n+from(EntryHandle)}

    DataStore ..|> DataStoreReader : implements
    DataStore ..> EntryHandle : returns_on_read
    EntryStream ..> EntryHandle : wraps_for_io

Sources: Cargo.toml:60-61

Entity Mapping: Utility and Performance

Sources: Cargo.toml70 Cargo.toml91

Key Indexer and Hashing

Loading…

Key Indexer and Hashing

Relevant source files

• experiments/bindings/python-ws-client/pyproject.toml
• src/storage_engine/digest.rs
• src/storage_engine/digest/compute_hash.rs
• src/storage_engine/key_indexer.rs
• src/utils/namespace_hasher.rs
• tests/hash_stability_tests.rs
• tests/namespace_hasher_tests.rs

The KeyIndexer is the primary in-memory data structure used to map 64-bit key hashes to their corresponding physical locations within the append-only data file. To maintain a low memory footprint while providing high reliability, it employs a packed bit scheme that combines file offsets with collision-detection fingerprints.

Hashing Implementation

The system relies on XXH3 , a high-performance non-cryptographic hash function, to generate 64-bit identifiers for keys. XXH3 is chosen for its SIMD optimization (AVX2, NEON) and excellent distribution properties src/storage_engine/digest/compute_hash.rs:3-9

Core Hashing Functions

• compute_hash(key:&[u8]) -> u64: Computes a single 64-bit hash using xxh3_64 src/storage_engine/digest/compute_hash.rs:25-27
• compute_hash_batch(keys:&[&[u8]]) -> Vec<u64>: Optimizes batch operations by pre-allocating the results vector and iterating through keys to minimize high-level API overhead and lock contention src/storage_engine/digest/compute_hash.rs:64-77
• Xxh3BuildHasher : A custom hasher implementation used by the internal HashMap to ensure the index itself benefits from XXH3 performance src/storage_engine/key_indexer.rs58 src/storage_engine/digest/xxh3_build_hasher.rs:21-30
• Xxh3Hasher : Implements the Hasher trait by wrapping xxh3_64 for use in standard library collections src/storage_engine/digest/xxh3_build_hasher.rs:6-18

Hash Stability

The project maintains strict hash stability to ensure that data written to disk remains accessible across dependency updates. Hardcoded mapping tests (e.g., alice → 0x4da10dd61a0116b0) guard against regressions in the xxhash-rust crate or its feature flags tests/hash_stability_tests.rs:1-33 Batch stability is also verified to ensure compute_hash_batch results match individual compute_hash calls tests/hash_stability_tests.rs:58-65

NamespaceHasher

The NamespaceHasher provides a mechanism to scope keys within logical domains (e.g., “users” vs “sessions”). It generates a 16-byte namespaced key by concatenating two 8-byte XXH3 hashes:

1. The hash of the namespace prefix src/utils/namespace_hasher.rs:33-37
2. The hash of the individual key src/utils/namespace_hasher.rs:56-62

This ensures that different namespaces do not generate overlapping keys, even if the underlying keys are identical src/utils/namespace_hasher.rs:45-48 Stability tests enforce specific 16-byte outputs for known namespace/key combinations tests/hash_stability_tests.rs:77-100

Sources: src/storage_engine/digest/compute_hash.rs:1-77 src/utils/namespace_hasher.rs:1-66 src/storage_engine/digest/xxh3_build_hasher.rs:1-30 src/storage_engine/digest.rs:1-8 tests/hash_stability_tests.rs:1-108

KeyIndexer Architecture

The KeyIndexer manages a HashMap<u64, u64, Xxh3BuildHasher> where the key is the 64-bit XXH3 hash and the value is a packed u64 containing both the file offset and a collision tag src/storage_engine/key_indexer.rs:56-59

Packed Value Format

To save memory, the 64-bit value is split into two segments:

• Tag (16 bits) : The upper 16 bits of the original key hash (TAG_BITS = 16). This serves as a fingerprint to detect if two different keys produced the same 64-bit hash src/storage_engine/key_indexer.rs:9-12
• Offset (48 bits) : The lower 48 bits represent the absolute file offset (OFFSET_MASK). This allows for a maximum file size of 256 TiB src/storage_engine/key_indexer.rs:15-45

Collision Detection Logic

During lookups or inserts, the system re-derives the 16-bit tag from the key hash src/storage_engine/key_indexer.rs136 If an entry exists but the stored tag does not match the new tag, a hash collision is detected, and the operation is rejected with an error src/storage_engine/key_indexer.rs:145-148

Data Flow: Key to Offset

The following diagram illustrates how a raw byte key is transformed into a physical file offset through the indexing layer.

Key Translation and Indexing Flow

graph TD
    subgraph "NaturalLanguageSpace"
        A["User Key (e.g., 'my_data')"]
end

    subgraph "CodeEntitySpace:src/storage_engine/"
        B["compute_hash(key)"]
C["KeyIndexer::tag_from_hash(u64)"]
D["KeyIndexer::pack(tag, offset)"]
E["KeyIndexer.index: HashMap<u64, u64, Xxh3BuildHasher>"]
end

 
   A -->|&[u8]| B
 
   B -->|u64 Hash| C
 
   B -->|u64 Hash| E
 
   C -->|u16 Tag| D
 
   D -->|u64 Packed| E
 
   E -->|lookup/insert| F["File Offset (48-bit)"]

Sources: src/storage_engine/key_indexer.rs:9-59 src/storage_engine/key_indexer.rs:135-160

KeyIndexer Operations

Index Initialization

During DataStore::open(), the KeyIndexer::build function performs a reverse scan of the storage file. It starts from the tail_offset and follows the prev_offset pointers in EntryMetadata src/storage_engine/key_indexer.rs:98-105 Because it scans newest-to-oldest, it only indexes the most recent version of any given key hash by tracking seen hashes src/storage_engine/key_indexer.rs:108-115

Insertion and Updates

The insert method handles both new keys and updates to existing ones:

1. Unpack : If the hash exists, it unpacks the stored value to retrieve the tag src/storage_engine/key_indexer.rs:141-142
2. Verify : It compares the new_tag with the stored_tag src/storage_engine/key_indexer.rs145
3. Commit : If they match, the new packed value (new tag + new offset) is stored src/storage_engine/key_indexer.rs151

Mapping Component Relationships

The following diagram maps the logical hashing components to their implementation structs and functions.

Hashing and Indexing Component Map

graph LR
    subgraph "LogicalFunction"
        H["Hashing"]
I["Indexing"]
N["Namespacing"]
end

    subgraph "ImplementationEntities"
        CH["compute_hash()"]
KI["struct KeyIndexer"]
NH["struct NamespaceHasher"]
XBH["struct Xxh3BuildHasher"]
end

    H --- CH
    H --- XBH
    I --- KI
    N --- NH
 
   KI -.->|uses| XBH
 
   NH -.->|calls| CH

Summary of KeyIndexer Methods

Sources: src/storage_engine/key_indexer.rs:61-200 src/utils/namespace_hasher.rs:17-65

Entry Iterator and Compaction

Loading…

Entry Iterator and Compaction

Relevant source files

• experiments/simd-r-drive-ws-server/src/main.rs
• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/entry_metadata.rs
• src/storage_engine.rs
• src/storage_engine/entry_iterator.rs
• tests/compaction_tests.rs
• tests/streaming_tests.rs

This section describes the mechanisms for traversing the append-only storage and reclaiming space. Because the DataStore never modifies data in place, the EntryIterator must navigate a chain of historical versions to find live data, while the compaction process periodically rewrites the file to remove stale entries and tombstones.

Entry Iterator

The EntryIterator is responsible for traversing the memory-mapped storage file in reverse chronological order (newest to oldest). This reverse traversal is critical because it allows the iterator to identify the most recent version of a key first.

Traversal Logic

The iterator follows a backward-linked chain using the prev_offset field stored in each entry’s EntryMetadata src/storage_engine/entry_iterator.rs:12-13

1. Initialization : Starts at the current tail_offset of the file and initializes a HashSet with a high-performance Xxh3BuildHasher src/storage_engine/entry_iterator.rs:41-47
2. Metadata Extraction : At each step, it reads the EntryMetadata located immediately before the current cursor. The metadata size is fixed at METADATA_SIZE (20 bytes) src/storage_engine/entry_iterator.rs:76-81
3. Deduplication : It maintains a seen_keys HashSet to track key hashes already encountered. If a hash is already in the set, the entry is skipped via a recursive call to next(), ensuring only the latest version is yielded src/storage_engine/entry_iterator.rs:110-112
4. Tombstone Handling : If an entry consists of a single NULL_BYTE, it is identified as a deleted record and skipped src/storage_engine/entry_iterator.rs:117-119
5. Termination : The iterator stops when the cursor reaches a position smaller than METADATA_SIZE or when the memory map is empty src/storage_engine/entry_iterator.rs:71-73

Entry Iterator Data Flow

The following diagram illustrates how the EntryIterator transforms raw file offsets into unique EntryHandle items.

Diagram: EntryIterator Traversal Flow

graph TD
    subgraph "File Space (Mmap)"
        [E3_Entry_N_Newest] --> [E2_Entry_N_Minus_1]
        [E2_Entry_N_Minus_1] --> [E1_Entry_N_Minus_2_Oldest]
    end

    subgraph "EntryIterator_Entity [src/storage_engine/entry_iterator.rs]"
        [CURSOR_tail_offset]
        [SEEN_keys_HashSet]
        [NEXT_fn_next]
    end

    [CURSOR_tail_offset] -- "points to tail" --> [E3_Entry_N_Newest]
    [NEXT_fn_next] -- "deserialize_metadata" --> [E3_Entry_N_Newest]
    [E3_Entry_N_Newest] -- "metadata.prev_offset" --> [CURSOR_tail_offset]
    
    [NEXT_fn_next] -- "check metadata.key_hash" --> [SEEN_keys_HashSet]
    [SEEN_keys_HashSet] -- "new hash?" --> [Yield_EntryHandle]
    [SEEN_keys_HashSet] -- "existing hash?" --> [Skip_Recurse_next]

    [Yield_EntryHandle] -- "populate" --> [EntryHandle_Entity]

Sources: src/storage_engine/entry_iterator.rs:21-25 src/storage_engine/entry_iterator.rs:69-126 simd-r-drive-entry-handle/src/entry_metadata.rs:46-50

Compaction Process

Compaction is the mechanism for reclaiming disk space in the append-only architecture. It works by creating a new version of the storage file containing only the “live” (most recent and non-deleted) entries.

Implementation Details

The DataStore::compact() method performs the following steps:

1. Identify Live Entries : It utilizes the EntryIterator to scan the file from the end to the beginning src/storage_engine/entry_iterator.rs:12-13
2. Filter : The iterator automatically filters out older versions of keys and entries marked with tombstones (single NULL_BYTE) src/storage_engine/entry_iterator.rs:18-19
3. Rewrite : The live entries are written to a temporary file. During this process, only the latest version of each key is persisted tests/compaction_tests.rs:136-138
4. Atomic Swap : Once the rewrite is complete, the temporary file replaces the original storage file, and the KeyIndexer is rebuilt to point to the new offsets tests/compaction_tests.rs:152-154

Compaction State Transition

This diagram maps the high-level compaction logic to the internal functions and components involved.

Diagram: Compaction Logic Mapping

graph LR
    subgraph "DataStore_Engine [src/storage_engine/data_store.rs]"
        [compact_fn]
        [EntryIterator_new]
        [DataStoreWriter_write]
    end

    subgraph "Storage_State"
        [OLD_File_Stale_Deleted]
        [NEW_File_Live_Only]
        [KeyIndexer_Entity]
    end

    [compact_fn] -- "initializes" --> [EntryIterator_new]
    [EntryIterator_new] -- "scans backward" --> [OLD_File_Stale_Deleted]
    [EntryIterator_new] -- "yields live EntryHandle" --> [compact_fn]
    [compact_fn] -- "appends to temp" --> [DataStoreWriter_write]
    [DataStoreWriter_write] -- "persists" --> [NEW_File_Live_Only]
    [compact_fn] -- "rebuilds index" --> [KeyIndexer_Entity]
    [KeyIndexer_Entity] -- "maps hashes to new offsets" --> [NEW_File_Live_Only]

Sources: src/storage_engine/entry_iterator.rs:21-25 src/storage_engine.rs:4-5 src/storage_engine/entry_iterator.rs:41-47 tests/compaction_tests.rs:136-138 tests/compaction_tests.rs:152-154

Key Implementation Details

Metadata and Alignment

During iteration, the EntryIterator must account for PAYLOAD_ALIGNMENT (64 bytes) src/storage_engine/entry_iterator.rs:50-51 It uses prepad_len to calculate the padding added during the write phase to ensure the payload was correctly aligned src/storage_engine/entry_iterator.rs:50-53

Tombstones and Legacy Support

A deletion is recorded by appending a new entry with the target key but a payload consisting of a single 0x00 (NULL_BYTE) src/storage_engine/entry_iterator.rs:117-119

The EntryIterator handles two cases for tombstones:

1. Aligned Tombstones : Standard entries following the alignment rules where the payload start is derived from the prev_tail plus padding src/storage_engine/entry_iterator.rs:86-87
2. Legacy Tombstones : Unaligned single-byte entries used in older versions of the format, identified by checking if the entry_end is exactly one byte ahead of prev_tail and contains a NULL_BYTE src/storage_engine/entry_iterator.rs:92-98

Parallel Iteration

When the parallel feature is enabled, the storage engine supports par_iter_entries(), which allows Rayon-based parallel processing of entries while maintaining the same deduplication and tombstone-skipping guarantees as the sequential iterator. This is verified through comprehensive testing of parallel access patterns tests/compaction_tests.rs:1-178

Sources: src/storage_engine/entry_iterator.rs:1-127 simd-r-drive-entry-handle/src/entry_metadata.rs:9-38 tests/compaction_tests.rs:118-120 tests/streaming_tests.rs:1-105

Concurrency and Thread Safety

Loading…

Concurrency and Thread Safety

Relevant source files

• README.md
• benches/storage_benchmark.rs
• src/main.rs
• src/storage_engine/data_store.rs
• src/utils/format_bytes.rs
• tests/concurrency_tests.rs

Purpose and Scope

This document describes the concurrency model and thread safety guarantees of the SIMD R Drive storage engine. It covers the synchronization primitives used to enable safe multi-threaded access within a single process, including lock strategies for reads and writes, atomic operations, and memory map management.

For information about the core storage architecture and data structures, see Storage Architecture. For details on memory-mapped file usage, see Memory Management and Zero-Copy Access.

Key Limitation : The concurrency mechanisms described here apply only to single-process, multi-threaded environments. Multiple processes accessing the same storage file simultaneously are not supported and require external file locking mechanisms.

Concurrency Model Overview

The DataStore structure uses a combination of read-write locks, atomic operations, and mutexes to enable safe concurrent access across multiple threads while maintaining data consistency.

Diagram: DataStore Synchronization Architecture

Sources: src/storage_engine/data_store.rs:27-33 README.md:172-183

Synchronization Primitives

DataStore Field Overview

The DataStore struct contains four primary fields that implement concurrency control:

Sources: src/storage_engine/data_store.rs:27-33

RwLock for File Writes

All write operations acquire an exclusive write lock on the file handle to prevent concurrent modifications.

Diagram: Write Lock Serialization

Write Lock Acquisition

Write operations acquire the lock at the start of the write process:

This pattern appears in:

• write_stream_with_key_hash: src/storage_engine/data_store.rs:759-762
• batch_write_with_key_hashes: src/storage_engine/data_store.rs:852-855

The write lock ensures that only one thread can append data to the file at any given time, preventing:

• Race conditions on file position
• Interleaved writes corrupting the append-only chain
• Inconsistent metadata ordering

Sources: src/storage_engine/data_store.rs:752-825 src/storage_engine/data_store.rs:847-945 README.md176

AtomicU64 for Tail Offset

The tail_offset field tracks the current end of the valid data in the storage file using atomic operations, enabling lock-free reads of the current file position.

Atomic Operations Used

Load Operation

Reads use Acquire ordering to ensure they see all previous writes:

Examples:

• iter_entries: src/storage_engine/data_store.rs278
• Write operations reading previous tail: src/storage_engine/data_store.rs763 src/storage_engine/data_store.rs858

Store Operation

Writes use Release ordering to ensure all previous writes are visible:

Location: src/storage_engine/data_store.rs256

This atomic coordination ensures that:

• Readers always see a consistent tail offset
• Writers update the tail only after data is flushed
• No locks are needed for reading the tail position

Sources: src/storage_engine/data_store.rs30 src/storage_engine/data_store.rs256 src/storage_engine/data_store.rs278 README.md182

Mutex for Memory Map

The memory-mapped file reference is protected by a Mutex<Arc<Mmap>> to prevent concurrent remapping during reads.

Diagram: Memory Map Arc Cloning Pattern

Accessing the Memory Map

Read operations clone the Arc<Mmap> to obtain a stable reference:

Source: src/storage_engine/data_store.rs:658-663

This pattern ensures:

• Readers hold a reference to a specific memory map version
• Writers can create a new memory map without invalidating existing readers
• The Arc reference counting prevents premature deallocation
• The mutex is held only briefly during the clone operation

Remapping After Writes

After writing and flushing data, the reindex function creates a new memory map:

Source: src/storage_engine/data_store.rs:231-255

Sources: src/storage_engine/data_store.rs29 src/storage_engine/data_store.rs:224-259 src/storage_engine/data_store.rs:658-663 README.md180

RwLock for Key Index

The KeyIndexer is protected by a read-write lock, allowing multiple concurrent readers but exclusive writers.

Read Access Pattern

Multiple threads can acquire read locks simultaneously for lookups:

Example: src/storage_engine/data_store.rs509

Write Access Pattern

Index updates require exclusive write access:

Source: src/storage_engine/data_store.rs:233-253

Parallel Iterator Lock Strategy

The parallel iterator minimizes lock holding time by collecting offsets first:

Source: src/storage_engine/data_store.rs:300-302

Sources: src/storage_engine/data_store.rs31 src/storage_engine/data_store.rs:233-253 src/storage_engine/data_store.rs:300-302 README.md178

Lock-Free Read Operations

Read operations achieve lock-free access through memory-mapped files and atomic operations.

Diagram: Concurrent Lock-Free Read Pattern

Zero-Copy Read Implementation

Once the offset is obtained from the index, data access is lock-free:

Source: src/storage_engine/data_store.rs:502-565

Benefits of Lock-Free Reads

1. No Read Contention : Multiple readers access different memory regions simultaneously
2. Zero-Copy : Data is accessed directly from the memory map without copying
3. Scalability : Read throughput scales linearly with CPU cores
4. Low Latency : No lock acquisition overhead after index lookup

Sources: README.md174 src/storage_engine/data_store.rs:502-565 tests/concurrency_tests.rs:163-229

Write Synchronization

Write operations are fully serialized through the file lock, ensuring consistency.

Diagram: Write Operation Synchronization Flow

Single Write Flow

Source: src/storage_engine/data_store.rs:758-825

Batch Write Optimization

Batch writes hold the lock once for multiple entries:

Source: src/storage_engine/data_store.rs:847-945

Sources: src/storage_engine/data_store.rs:752-825 src/storage_engine/data_store.rs:847-945 README.md176

Thread Safety Guarantees

Thread Safety Matrix

The following table summarizes thread safety guarantees for different environments:

Source: README.md:196-200

Safe Concurrency Properties

The design ensures the following properties in single-process, multi-threaded environments:

Diagram: Thread Safety Property Dependencies

1. Atomicity : All operations on shared state are atomic or properly locked
2. Visibility : Changes made by one thread are visible to others through Release/Acquire semantics
3. Ordering : The append-only design ensures writes happen in a strict sequence
4. Isolation : Readers see a consistent snapshot via Arc<Mmap> cloning

Sources: README.md:172-206 src/storage_engine/data_store.rs:27-33

Single-Process vs Multi-Process

Single-Process Multi-Threaded (Supported)

All synchronization primitives work correctly within a single process:

Diagram: Single-Process Shared State

Example from concurrency tests:

Source: tests/concurrency_tests.rs:117-137

Multi-Process (Not Supported)

Multiple processes have separate address spaces and cannot share the in-memory synchronization primitives:

Diagram: Multi-Process Unsafe Access

Why Multi-Process is Unsafe

1. Separate Index State : Each process has its own KeyIndexer in memory
2. Independent Mmap Views : Memory maps are not synchronized across processes
3. No Lock Coordination : RwLock and Mutex are process-local, not system-wide
4. Race Conditions : Concurrent writes can corrupt the file structure

Recommendation : Use external file locking (e.g., flock, advisory locks) if multi-process access is required.

Sources: README.md:186-206 README.md:189-191

Testing Concurrency

The test suite validates concurrent access patterns to ensure thread safety guarantees.

Concurrent Write Test

Tests multiple threads writing simultaneously:

Source: tests/concurrency_tests.rs:111-161

Interleaved Read-Write Test

Tests read-after-write consistency with coordinated threads:

Source: tests/concurrency_tests.rs:163-229

Concurrent Streamed Write Test

Tests slow, streaming writes that hold the lock for extended periods:

Source: tests/concurrency_tests.rs:14-109

Sources: tests/concurrency_tests.rs:1-230

Summary

The SIMD R Drive concurrency model provides thread-safe access through a carefully coordinated set of synchronization primitives:

• RwLock : Serializes file writes while allowing concurrent reads of the lock
• AtomicU64 : Provides lock-free tail offset tracking
• Mutex : Protects memory map updates without blocking existing readers
• RwLock : Enables highly concurrent index reads with exclusive write access

This design achieves:

• Zero-copy concurrent reads via memory mapping
• Serialized writes preventing data corruption
• Linear read scalability across CPU cores
• Consistent snapshots through atomic operations

However, these guarantees apply only within a single process. Multi-process access requires external coordination mechanisms.

Sources: README.md:170-206 src/storage_engine/data_store.rs:26-33 tests/concurrency_tests.rs:1-230

Dismiss

Refresh this wiki

Enter email to refresh

SIMD and Performance Utilities

Loading…

SIMD and Performance Utilities

Relevant source files

• extensions/Cargo.toml
• src/storage_engine/simd_copy.rs
• src/utils.rs
• src/utils/align_or_copy.rs
• tests/align_or_copy_tests.rs

The simd-r-drive engine is designed for high-throughput data operations. To achieve this, the codebase leverages hardware-accelerated memory operations, zero-copy data reinterpretation, and rigorous benchmarking suites. This page provides an overview of the utilities that underpin the system’s performance characteristics.

SIMD Copy Implementation

The engine utilizes SIMD (Single Instruction, Multiple Data) instructions to accelerate memory-to-memory copies, which are critical during data ingestion and compaction. The implementation provides architecture-specific paths for x86_64 and aarch64.

• x86_64 Path : Uses AVX2 via _mm256_loadu_si256 and _mm256_storeu_si256 to process data in 32-byte chunks [src/storage_engine/simd_copy.rs:35-62].
• aarch64 Path : Uses NEON via vld1q_u8 and vst1q_u8 to process data in 16-byte chunks [src/storage_engine/simd_copy.rs:83-108].
• Runtime Detection : The simd_copy function uses is_x86_feature_detected!("avx2") on x86 platforms to safely select the best path at runtime, falling back to standard scalar copies if features are missing [src/storage_engine/simd_copy.rs:111-138].
• Logging : A LOG_ONCE mechanism ensures that performance warnings (e.g., falling back to scalar copy) do not flood the system logs [src/storage_engine/simd_copy.rs:8-8], [src/storage_engine/simd_copy.rs:121-124].

For details, see SIMD Copy Implementation.

SIMD Copy Logic Flow

“Memory Copy Execution Path”

Sources: [src/storage_engine/simd_copy.rs:111-138], [src/storage_engine/simd_copy.rs:35-62], [src/storage_engine/simd_copy.rs:83-108]

Alignment, Checksums, and Utility Functions

Performance is further optimized through strict memory alignment and efficient integrity checks.

• Zero-Copy Alignment : The align_or_copy utility attempts to reinterpret raw byte slices into typed slices without copying using align_to. If the memory is not properly aligned for the target type or the length is not a multiple of the element size, it falls back to a Cow::Owned copy to ensure safety [src/utils/align_or_copy.rs:44-75].
• General Utilities : The engine includes helpers for format_bytes [src/utils.rs:7-8], parse_buffer_size [src/utils.rs:13-14], verify_file_existence [src/utils.rs:16-17], and NamespaceHasher for prefixed key hashing [src/utils.rs:10-11].
• Extension Support : The append_extension utility facilitates path manipulation for specialized storage files [src/utils.rs:4-5].

For details, see Alignment, Checksums, and Utility Functions.

Benchmarks

The repository includes a comprehensive benchmarking suite to validate performance across different workloads.

• Storage Benchmark : This suite tests the DataStore by writing and reading large volumes of entries. It measures:
  • Append Throughput : Performance of sequential and batched writes.
  • Sequential Reads : Throughput when iterating through the store.
  • Random Reads : Latency and throughput for single-key lookups.
  • Vectorized Reads : Efficiency of multi-key lookups.
• Contention Benchmark : Evaluates system performance under heavy concurrent load, measuring throughput across different payload sizes.

For details, see Benchmarks.

Performance Verification Entities

“Benchmarking and Testing Framework”

Sources: [src/utils/align_or_copy.rs:44-75], [src/storage_engine/simd_copy.rs:111-138]

SIMD Copy Implementation

Loading…

SIMD Copy Implementation

Relevant source files

• extensions/Cargo.toml
• src/storage_engine/simd_copy.rs
• src/utils.rs
• src/utils/align_or_copy.rs
• tests/align_or_copy_tests.rs

The simd_copy utility provides a high-performance memory copy abstraction that leverages hardware-specific vector instructions. By utilizing SIMD (Single Instruction, Multiple Data) , the system can process multiple bytes in a single CPU cycle, which is critical for the high-throughput requirements of the append-only storage engine.

Overview

The implementation targets specific architectures (x86_64 and AArch64) with specialized intrinsic functions while maintaining a safe scalar fallback for unsupported hardware src/storage_engine/simd_copy.rs:111-138

Architecture Support Matrix

Sources: src/storage_engine/simd_copy.rs:10-15 src/storage_engine/simd_copy.rs:16-108

Implementation Details

x86_64 Path: AVX2

On x86_64 systems, the implementation uses Advanced Vector Extensions 2 (AVX2) src/storage_engine/simd_copy.rs:18-19

• Intrinsics : It uses _mm256_loadu_si256 to load 256 bits (32 bytes) of unaligned data from the source and _mm256_storeu_si256 to write to the destination src/storage_engine/simd_copy.rs:47-55
• Loop Unrolling : The main loop processes data in 32-byte increments src/storage_engine/simd_copy.rs:40-58
• Tail Handling : Any remaining bytes (less than 32) are copied using the standard copy_from_slice scalar method src/storage_engine/simd_copy.rs61

AArch64 Path: NEON

For ARM64/AArch64 architectures, the implementation utilizes NEON (Advanced SIMD) src/storage_engine/simd_copy.rs:64-67

• Intrinsics : It uses vld1q_u8 for 128-bit (16-byte) loads and vst1q_u8 for 16-byte stores src/storage_engine/simd_copy.rs:94-101
• Loop Unrolling : The loop processes data in 16-byte increments src/storage_engine/simd_copy.rs:88-104
• Tail Handling : Remaining bytes are handled via copy_from_slice src/storage_engine/simd_copy.rs107

Runtime Feature Detection and Logging

The simd_copy function acts as a dispatcher. On x86_64, it uses the std::is_x86_feature_detected!("avx2") macro to check CPU capabilities at runtime src/storage_engine/simd_copy.rs114

If a compatible SIMD unit is not found (common in virtualized environments like Windows 11 Arm in UTM on Apple Silicon), the system logs a warning src/storage_engine/simd_copy.rs:119-122 To prevent log spam in high-frequency copy operations, the LOG_ONCE mechanism is employed using std::sync::Once src/storage_engine/simd_copy.rs:8-9 This ensures the “AVX2 not detected” warning is emitted exactly once per process lifetime src/storage_engine/simd_copy.rs:121-123

Logic Flow: SIMD Dispatcher

The following diagram illustrates how the simd_copy function selects the appropriate implementation path based on the target architecture and runtime features.

SIMD Dispatcher Logic

graph TD
    "Entry[simd_copy]" --> "ArchCheck{target_arch?}"
    
    "ArchCheck" -- "x86_64" --> "AVX2Check{is_x86_feature_detected!('avx2')}"
    "AVX2Check" -- "Yes" --> "simd_copy_x86"
    "AVX2Check" -- "No" --> "WarnOnce[LOG_ONCE: warn!]"
    "WarnOnce" --> "ScalarFallback[dst.copy_from_slice]"
    
    "ArchCheck" -- "aarch64" --> "simd_copy_arm"
    
    "ArchCheck" -- "other" --> "ScalarFallback"
    
    subgraph "x86_64 Implementation"
        "simd_copy_x86" --> "AVX_Loop[while i < chunks * 32]"
        "AVX_Loop" --> "AVX_Intrinsics[_mm256_loadu/storeu]"
        "AVX_Intrinsics" --> "AVX_Tail[copy_from_slice tail]"
    end
    
    subgraph "ARM Implementation"
        "simd_copy_arm" --> "NEON_Loop[while i < chunks * 16]"
        "NEON_Loop" --> "NEON_Intrinsics[vld1q/vst1q_u8]"
        "NEON_Intrinsics" --> "NEON_Tail[copy_from_slice tail]"
    end

Sources: src/storage_engine/simd_copy.rs:8-9 src/storage_engine/simd_copy.rs:35-138

Data Flow and Memory Safety

The SIMD functions are marked unsafe because they perform raw pointer arithmetic and bypass certain slice bounds checks for performance src/storage_engine/simd_copy.rs:35-83

1. Bounds Calculation : The length is determined by the minimum of the destination and source slice lengths to prevent buffer overflows src/storage_engine/simd_copy.rs36 src/storage_engine/simd_copy.rs84
2. Pointer Casting : Slices are converted to raw pointers (as_ptr() / as_mut_ptr()) and cast to the appropriate SIMD vector types (e.g., *const __m256i for AVX2) src/storage_engine/simd_copy.rs:47-55
3. Unaligned Access : The implementation specifically uses “unaligned” load/store instructions (_mm256_loadu_si256, vld1q_u8), which allows the functions to work on any byte-aligned slice without requiring strict 32-byte or 16-byte memory alignment src/storage_engine/simd_copy.rs:47-55 src/storage_engine/simd_copy.rs:94-101

Entity Mapping: Implementation to Intrinsics

Sources: src/storage_engine/simd_copy.rs:8-15 src/storage_engine/simd_copy.rs:47-55 src/storage_engine/simd_copy.rs:94-101

Related Utilities: align_or_copy

The simd_copy utility is closely related to align_or_copy, which provides zero-copy parsing of binary data into typed slices src/utils/align_or_copy.rs:3-6

align_or_copy uses slice::align_to::<T>() to attempt a zero-copy borrow src/utils/align_or_copy.rs57 If the memory is misaligned or the size is not a multiple of the element size, it falls back to an owned Vec<T> src/utils/align_or_copy.rs:60-74 This is verified in integration tests where misaligned buffers trigger the Cow::Owned fallback path tests/align_or_copy_tests.rs:23-29

Sources: src/utils/align_or_copy.rs:1-75 tests/align_or_copy_tests.rs:23-29

Alignment, Checksums, and Utility Functions

Loading…

Alignment, Checksums, and Utility Functions

Relevant source files

• simd-r-drive-entry-handle/src/debug_assert_aligned.rs
• src/utils/align_or_copy.rs
• src/utils/parse_buffer_size.rs
• src/utils/verify_file_existence.rs
• tests/alignment_tests.rs

This page details the performance-critical utilities and helper functions used across the SIMD R Drive workspace. These components ensure data integrity through CRC32C checksums, maintain hardware-friendly memory alignment for SIMD operations, and provide common parsing and formatting logic for the CLI and core engine.

Memory Alignment and Zero-Copy Reinterpretation

The storage engine enforces a strict payload alignment of 64 bytes (PAYLOAD_ALIGNMENT). This alignment matches standard CPU cache-line sizes and satisfies the requirements for most SIMD instruction sets (AVX2, NEON) tests/alignment_tests.rs12

align_or_copy

The align_or_copy function is a core utility for reinterpreting raw byte slices as typed slices (e.g., &[f32]) without allocation when possible src/utils/align_or_copy.rs:44-50

• Zero-Copy Path : Uses slice::align_to::<T>() to check if the memory address is already aligned to the target type T. If the prefix and suffix returned by align_to are empty, it returns a Cow::Borrowed slice src/utils/align_or_copy.rs:57-59
• Fallback Path : If the memory is misaligned or the length is not a multiple of size_of::<T>(), it performs a manual copy into a Cow::Owned(Vec<T>) using the provided from_le_bytes conversion function src/utils/align_or_copy.rs:60-72

Alignment Validation Logic

The codebase provides specialized assertions to ensure alignment invariants are maintained during development without impacting production performance.

These functions use a specific pattern: the function body is gated by #[cfg(any(test, debug_assertions))] simd-r-drive-entry-handle/src/debug_assert_aligned.rs27 In release builds, they compile to a no-op where arguments are marked as used to avoid compiler warnings, ensuring zero runtime cost simd-r-drive-entry-handle/src/debug_assert_aligned.rs:37-42

Tests in tests/alignment_tests.rs verify these invariants by performing unaligned writes followed by aligned overwrites, then attempting to cast the resulting slices to u32, u64, and u128 views using bytemuck::try_cast_slice tests/alignment_tests.rs:58-67

Title: Alignment Validation and SIMD Loading

Sources: src/utils/align_or_copy.rs:44-73 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:26-43 tests/alignment_tests.rs:135-200

Checksum Computation

Data integrity is maintained using the CRC32C algorithm (Castagnoli), implemented via the crc32fast crate.

• Implementation : Checksums are computed using the crc32fast::Hasher which utilizes hardware-accelerated instructions (SSE4.2 on x86_64, NEON on ARM) when available.
• Verification : The system extracts the stored CRC from the entry metadata and compares it against the computed CRC of the payload to detect corruption.

Sources: src/utils/align_or_copy.rs:1-75 tests/alignment_tests.rs:1-133

General Utility Functions

Helper functions for formatting and parsing are provided to support the CLI and initialization routines.

parse_buffer_size

Converts string representations of sizes (e.g., “1MB”, “2G”, “1024”) into usize byte counts src/utils/parse_buffer_size.rs:35-57

• Supported Units : K/KB (1024), M/MB (1024^2), G/GB (1024^3) src/utils/parse_buffer_size.rs:45-51
• Behavior : Case-insensitive and trims whitespace src/utils/parse_buffer_size.rs36

verify_file_existence

Checks if a path exists and is a regular file before initialization src/utils/verify_file_existence.rs11 It returns std::io::ErrorKind::NotFound if the path is missing or InvalidInput if the path is a directory src/utils/verify_file_existence.rs:12-24

graph LR
    subgraph "Utility Requirements"
        R1["Data Reinterpretation"]
R2["Size Parsing"]
R3["Integrity Check"]
R4["Path Validation"]
end

    subgraph "Code Implementation"
 
       R1 --> F1["align_or_copy()"]
R2 --> F2["parse_buffer_size()"]
R3 --> F3["crc32fast::Hasher"]
R4 --> F4["verify_file_existence()"]
F1 --- FILE1["src/utils/align_or_copy.rs"]
F2 --- FILE2["src/utils/parse_buffer_size.rs"]
F4 --- FILE4["src/utils/verify_file_existence.rs"]
end

System Utility Mapping

The following diagram maps high-level utility requirements to the specific implementation files and functions within the codebase.

Title: Utility Function Mapping

Sources: src/utils/parse_buffer_size.rs:1-57 src/utils/align_or_copy.rs:44-50 src/utils/verify_file_existence.rs:1-27

Benchmarks

Loading…

Benchmarks

Relevant source files

• benches/contention_benchmark.rs
• benches/storage_benchmark.rs

This page details the performance measurement suite for the SIMD R Drive. The benchmarks are designed to validate the efficiency of the append-only storage engine, particularly its SIMD-optimized paths, zero-copy reads, and concurrent access patterns.

Storage Benchmark (storage_benchmark.rs)

The storage_benchmark is a single-process micro-benchmark that evaluates the core DataStore operations under high load. It processes 1,000,000 entries to measure throughput for writes and various read patterns benches/storage_benchmark.rs:1-3

Implementation Details

The benchmark operates on a NamedTempFile to ensure a clean state and automatic cleanup benches/storage_benchmark.rs:33-46 It uses a fixed entry size of 8 bytes (storing u64 values in little-endian) to simplify validation benches/storage_benchmark.rs:20-24

Benchmark Phases:

1. Append Entries : Writes 1M entries using batch_write with a batch size of 1024. It formats keys as bench-key-{i} and flushes batches to the store benches/storage_benchmark.rs:52-83
2. Sequential Reads : Uses the DataStore iterator (via into_iter()) to traverse entries from newest to oldest, verifying data integrity via u64::from_le_bytes benches/storage_benchmark.rs:98-118
3. Random Reads : Performs 1M random lookups using read() to test KeyIndexer performance and XXH3 hashing speed benches/storage_benchmark.rs:124-149
4. Batch Reads : Uses batch_read() to verify 1M entries in vectorized chunks of 1024, testing the efficiency of bulk key lookups benches/storage_benchmark.rs:155-181

Data Flow: Storage Benchmark

The following diagram illustrates how the benchmark interacts with the DataStore and its internal components.

Storage Benchmark Execution Flow

graph TD
    subgraph "Benchmark_Process"
        [Main] --> [benchmark_append_entries]
        [Main] --> [benchmark_sequential_reads]
        [Main] --> [benchmark_random_reads]
        [Main] --> [benchmark_batch_reads]
    end

    subgraph "DataStore_API"
        [benchmark_append_entries] -- "batch_write()" --> DS_Writer["DataStoreWriter"]
        [benchmark_sequential_reads] -- "into_iter()" --> DS_Iter["EntryIterator"]
        [benchmark_random_reads] -- "read()" --> DS_Reader["DataStoreReader"]
        [benchmark_batch_reads] -- "batch_read()" --> DS_Reader
    end

    subgraph "Internal_Storage_Engine"
 
       DS_Writer --> KI["KeyIndexer"]
DS_Writer --> AL["Append-Only Log"]
DS_Reader --> KI
 
       DS_Reader --> MM["Mmap (Zero-Copy)"]
DS_Iter --> MM
    end

Sources: benches/storage_benchmark.rs:32-41 benches/storage_benchmark.rs:85-92 benches/storage_benchmark.rs:104-109 benches/storage_benchmark.rs:133-136 benches/storage_benchmark.rs:185-186

Contention Benchmark (contention_benchmark.rs)

The contention_benchmark uses the criterion framework to measure lock-contention throughput when many concurrent writers are active benches/contention_benchmark.rs:1-8

Implementation Details

The benchmark evaluates the performance of the engine under heavy concurrent write pressure using a Tokio runtime to manage asynchronous writer tasks benches/contention_benchmark.rs:33-35

Parameters:

• Payload Sizes : Tests with 128 B, 4 KiB, and 64 KiB payloads to measure how data volume affects lock hold times and write throughput benches/contention_benchmark.rs20
• Concurrency : Spawns 8 independent writer tasks using tokio::spawn benches/contention_benchmark.rs:21-49
• Workload : Each task performs 1,000 writes of random payloads (generated via rng.random()) to prevent easy compression or filesystem-level optimizations benches/contention_benchmark.rs:22-56

Code Entity Mapping: Contention

This diagram maps the benchmark entities to the synchronization primitives and runtime components.

Contention Benchmark Mapping

graph LR
    subgraph "Benchmark_Harness"
        CRIT["Criterion::BenchmarkGroup"]
RT["Tokio::Runtime"]
end

    subgraph "Task_Execution"
        RT -- "block_on" --> ITER["Iteration"]
ITER -- "spawn" --> T["Writer Tasks (x8)"]
T -- "loop" --> DS_WRITE["DataStore::write"]
end

    subgraph "Storage_Engine"
        DS_WRITE -- "Acquires" --> INDEX_LOCK["RwLock<KeyIndexer>"]
DS_WRITE -- "Acquires" --> FILE_LOCK["RwLock<File>"]
DS_WRITE -- "Writes" --> MMAP["Mmap (Atomic Tail)"]
end

Sources: benches/contention_benchmark.rs:28-35 benches/contention_benchmark.rs:45-59 benches/contention_benchmark.rs63

Execution and Interpretation

Running Benchmarks

Benchmarks are executed via Cargo. To ensure they are optimized, always run with the --release flag.

CI Integration

The GitHub Actions workflow includes a specific step to verify that benchmarks compile across all supported platforms and feature flag combinations. This is done using the cargo bench --workspace --no-run flag to prevent long-running benchmarks from stalling the CI pipeline while still ensuring API compatibility and code correctness across features like parallel or expose-internal-api.

Interpreting Results

The storage_benchmark outputs human-readable rates using the thousands crate for digit separation benches/storage_benchmark.rs14

The contention_benchmark provides detailed timing statistics per payload size via Criterion, helping identify if lock contention becomes a bottleneck as payload sizes increase benches/contention_benchmark.rs:36-40

Sources: benches/storage_benchmark.rs:77-82 benches/storage_benchmark.rs:112-117 benches/storage_benchmark.rs:143-148 benches/storage_benchmark.rs:175-180 benches/contention_benchmark.rs:36-40

Entry Handle Crate

Loading…

Entry Handle Crate

Relevant source files

• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/constants.rs
• simd-r-drive-entry-handle/src/entry_handle.rs
• simd-r-drive-entry-handle/src/entry_metadata.rs
• simd-r-drive-entry-handle/src/lib.rs
• src/storage_engine.rs
• src/storage_engine/entry_iterator.rs

The simd-r-drive-entry-handle crate provides the core data types and zero-copy access mechanisms used across the SIMD R Drive ecosystem. By decoupling these types from the main storage engine, the project allows extensions, network clients, and external tools to interact with entry data using a shared, stable ABI without depending on the full storage engine logic simd-r-drive-entry-handle/Cargo.toml:1-5

Purpose and Role

This crate serves as the “Common Type Layer.” Its primary goal is to provide a storage-agnostic way to handle entries that may be backed by memory-mapped files, anonymous memory, or network buffers simd-r-drive-entry-handle/src/entry_handle.rs:7-19 It allows different components to share data via the EntryHandle struct without re-allocating or copying the underlying payload simd-r-drive-entry-handle/src/entry_handle.rs:7-19

The following diagram illustrates how EntryHandle acts as the bridge between raw storage and high-level consumers:

Data Type Integration Flow

graph TD
    subgraph "Storage_Layer"
        [DataStore] -- "returns" --> [EntryHandle]
        [MmapMut] -- "from_owned_bytes_anon" --> [EntryHandle]
        [EntryIterator] -- "next_returns" --> [EntryHandle]
    end

    subgraph "simd-r-drive-entry-handle"
        [EntryHandle]
        [EntryMetadata]
        [EntryHandle] -- "contains" --> [EntryMetadata]
    end

    subgraph "Consumer_Layer"
        [EntryHandle] -- "as_slice" --> [Standard_Rust_Slice]
        [EntryHandle] -- "as_arrow_buffer" --> [Apache_Arrow_Buffer]
        [EntryHandle] -- "Deref" --> [Byte_Operations]
    end

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:10-19 simd-r-drive-entry-handle/src/entry_handle.rs:87-113 simd-r-drive-entry-handle/Cargo.toml:13-15 src/storage_engine/entry_iterator.rs:121-125

Core Entities

The crate is built around two primary structures that define how data is represented in memory and on disk.

EntryMetadata Layout

The EntryMetadata struct mirrors the binary layout used at the end of every entry in the storage file simd-r-drive-entry-handle/src/entry_metadata.rs:9-31 It provides serialize() and deserialize() methods to convert between the struct and the raw byte format required by the append-only log simd-r-drive-entry-handle/src/entry_metadata.rs:75-112

The layout ensures that payloads can be recovered by following the prev_offset chain, which forms a backward-linked sequence for each key simd-r-drive-entry-handle/src/entry_metadata.rs:41-43

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:44-50 simd-r-drive-entry-handle/src/entry_metadata.rs:75-83 simd-r-drive-entry-handle/src/entry_metadata.rs:101-112

Zero-Copy Architecture

The EntryHandle is designed to prevent unnecessary allocations. It achieves this by wrapping a sub-slice of an Arc<Mmap> simd-r-drive-entry-handle/src/entry_handle.rs:7-19

• File-Backed: When reading from DataStore or via an EntryIterator, the handle points directly to the memory-mapped file on disk simd-r-drive-entry-handle/src/entry_handle.rs:129-139 src/storage_engine/entry_iterator.rs:121-125
• In-Memory: Using from_owned_bytes_anon(), the crate can create handles from standard byte slices by copying them once into an anonymous memory map, thereafter providing the same zero-copy &[u8] access simd-r-drive-entry-handle/src/entry_handle.rs:87-113
• Trait Implementations: It implements Deref<Target = [u8]> simd-r-drive-entry-handle/src/entry_handle.rs:36-42 and various PartialEq variants simd-r-drive-entry-handle/src/entry_handle.rs:45-63 to allow seamless comparison with slices and vectors.

EntryHandle Internal Structure

classDiagram
    class EntryHandle {
        +Arc~Mmap~ mmap_arc
        +Range~usize~ range
        +EntryMetadata metadata
        +as_slice() &[u8]
        +clone_arc() EntryHandle
        +from_arc_mmap() EntryHandle
        +is_valid_checksum() bool
    }
    class EntryMetadata {
        +u64 key_hash
        +u64 prev_offset
        +u8_4 checksum
        +serialize() [u8; 20]
        +deserialize(data) EntryMetadata
    }
    EntryHandle *-- EntryMetadata : contains

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:10-19 simd-r-drive-entry-handle/src/entry_metadata.rs:46-50 simd-r-drive-entry-handle/src/entry_handle.rs:179-185

Sub-Pages

EntryHandle: Zero-Copy Data Access

Detailed API documentation for EntryHandle. This page covers how to instantiate handles via from_arc_mmap() or from_owned_bytes_anon(), how to verify integrity with is_valid_checksum(), and the use of Deref to treat handles as byte slices. It also explains the expose-internal-api feature gate which grants access to internal fields for specialized extensions simd-r-drive-entry-handle/Cargo.toml14 For details, see EntryHandle: Zero-Copy Data Access.

Arrow Integration

Explains the optional arrow feature in Cargo.toml simd-r-drive-entry-handle/Cargo.toml15 When enabled, EntryHandle gains the ability to export its underlying memory-mapped payload directly into an Apache Arrow Buffer. This allows SIMD-optimized analytical engines to process SIMD R Drive data without any copying or serialization overhead. For details, see Arrow Integration.

Sources: simd-r-drive-entry-handle/Cargo.toml:13-15 simd-r-drive-entry-handle/src/entry_handle.rs:151-155

EntryHandle: Zero-Copy Data Access

Loading…

EntryHandle: Zero-Copy Data Access

Relevant source files

• simd-r-drive-entry-handle/README.md
• simd-r-drive-entry-handle/src/constants.rs
• simd-r-drive-entry-handle/src/entry_handle.rs
• simd-r-drive-entry-handle/src/lib.rs

EntryHandle is the primary data structure for accessing entry payloads in rust-simd-r-drive. It acts as a zero-copy owner of a sub-slice within a memory-mapped file (Mmap), ensuring that data is never copied from the disk buffer into application memory during read operations.

Overview and Lifecycle

An EntryHandle binds an Arc<Mmap> with a specific Range<usize> and EntryMetadata simd-r-drive-entry-handle/src/entry_handle.rs:9-19 By using an Arc, the underlying memory mapping is kept alive as long as at least one handle exists, even if the DataStore that created it is closed or the file is remapped simd-r-drive-entry-handle/src/entry_handle.rs:123-125

Data Flow: From Disk to Application

The following diagram illustrates how EntryHandle bridges the gap between the raw memory map and the user’s byte-slice access.

Diagram: EntryHandle Memory Mapping Architecture

graph TD
    subgraph "Disk Storage"
        FILE["test_storage.bin"]
end

    subgraph "Process Address Space"
        MMAP["memmap2::Mmap (Memory Mapping)"]
ARC["Arc<Mmap> (Shared Ownership)"]
end

    subgraph "Code Entities (EntryHandle)"
        EH["EntryHandle"]
META["EntryMetadata"]
RANGE["Range<usize> (Payload Bounds)"]
end

 
   FILE -.->|mmap| MMAP
    MMAP --- ARC
    ARC --- EH
 
   EH --> META
 
   EH --> RANGE
    
 
   EH -.->|as_slice| SLICE["&[u8] (Zero-Copy View)"]
RANGE -.->|indexes| MMAP

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:7-19 simd-r-drive-entry-handle/src/entry_handle.rs:151-155

Core API Methods

Construction

• from_arc_mmap(mmap_arc, range, metadata) : The standard constructor used by the DataStore reader and EntryIterator. It wraps an existing shared mapping without allocations simd-r-drive-entry-handle/src/entry_handle.rs:129-139
• from_owned_bytes_anon(bytes, key_hash) : Creates an in-memory entry backed by an anonymous MmapMut. This involves exactly one copy of the input bytes into the mapping, which is then sealed as read-only simd-r-drive-entry-handle/src/entry_handle.rs:87-113 It computes a CRC32C checksum during construction to maintain consistency with file-backed entries simd-r-drive-entry-handle/src/entry_handle.rs:95-106

Data Access

• as_slice() : Returns a &[u8] pointing directly into the memory-mapped region. This is the core of the zero-copy guarantee simd-r-drive-entry-handle/src/entry_handle.rs:151-155
• Deref Implementation: EntryHandle implements std::ops::Deref, allowing it to be used interchangeably with &[u8] (e.g., *entry_handle) simd-r-drive-entry-handle/src/entry_handle.rs:36-42
• clone_arc() : Instead of a standard Clone (which might imply deep copying), this method explicitly increments the Arc reference count, creating a new handle to the same data simd-r-drive-entry-handle/src/entry_handle.rs:179-185

Metadata and Validation

• key_hash() : Returns the 64-bit XXH3 hash of the key associated with this entry simd-r-drive-entry-handle/src/entry_handle.rs:200-202
• is_valid_checksum() : Re-computes the CRC32C of the payload in the as_slice() range and compares it against the stored metadata.checksum simd-r-drive-entry-handle/src/entry_handle.rs:215-219
• size() : Returns the length of the payload in bytes simd-r-drive-entry-handle/src/entry_handle.rs:232-234
• file_size() : Returns the total size of the underlying memory-mapped file simd-r-drive-entry-handle/src/entry_handle.rs:242-244

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:21-244 simd-r-drive-entry-handle/src/entry_handle.rs:87-113

Internal API and Feature Gates

When the expose-internal-api feature is enabled (or during tests), additional methods are available to inspect the physical layout and memory addresses:

• offset_range() : Returns the Range<u64> representing the absolute byte offsets within the file where the payload is located simd-r-drive-entry-handle/src/entry_handle.rs:260-262
• address_range() : Returns the Range<usize> of virtual memory addresses (pointers) where the entry is currently mapped simd-r-drive-entry-handle/src/entry_handle.rs:271-274
• arc_ptr() : (Test-only) Returns the raw pointer to the Mmap struct to verify that multiple handles share the same underlying allocation simd-r-drive-entry-handle/src/entry_handle.rs:30-32

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:22-33 simd-r-drive-entry-handle/src/entry_handle.rs:250-275

Alignment and SIMD Compatibility

The EntryHandle is designed to support SIMD operations and zero-copy typed views. Because the DataStore enforces PAYLOAD_ALIGNMENT (64 bytes) simd-r-drive-entry-handle/src/constants.rs18 the as_slice() pointer is often suitable for direct casting to SIMD types or aligned buffers simd-r-drive-entry-handle/src/debug_assert_aligned.rs:26-35

Diagram: Zero-Copy Memory Access Path

Sources: simd-r-drive-entry-handle/src/constants.rs:13-18 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:66-81

Integration with Storage Operations

The DataStore utilizes EntryHandle in its read operations and iteration. When an entry is read, the DataStore constructs an EntryHandle using the current file mapping. This allows the system to verify integrity via is_valid_checksum() before passing the handle to the caller. The debug_assert_aligned_offset function ensures that file offsets used during handle construction adhere to the 64-byte boundary required for SIMD-optimized access simd-r-drive-entry-handle/src/debug_assert_aligned.rs:66-81

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:129-139 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:66-81

Arrow Integration

Loading…

Arrow Integration

Relevant source files

• CHANGELOG.md
• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/constants.rs
• simd-r-drive-entry-handle/src/entry_handle.rs
• simd-r-drive-entry-handle/src/entry_metadata.rs
• simd-r-drive-entry-handle/src/lib.rs
• src/storage_engine.rs
• src/storage_engine/entry_iterator.rs

The simd-r-drive-entry-handle crate provides optional integration with Apache Arrow via the arrow Cargo feature simd-r-drive-entry-handle/Cargo.toml15 This integration allows users to treat memory-mapped payloads as arrow_buffer::Buffer objects without copying data, enabling high-performance analytical processing directly on the stored bytes CHANGELOG.md:133-135

Overview of Arrow Buffers

Apache Arrow uses a specific Buffer type to represent contiguous regions of memory. By leveraging the fact that EntryHandle already manages an Arc<Mmap> simd-r-drive-entry-handle/src/entry_handle.rs12 the system can wrap this memory in an Arrow-compatible container. This is particularly effective because the storage engine defaults to a PAYLOAD_ALIGNMENT of 64 bytes simd-r-drive-entry-handle/src/constants.rs18 which satisfies Arrow’s requirement for SIMD-aligned data access CHANGELOG.md:93-97

Key Capabilities

• Zero-Copy Conversion : Transform an EntryHandle into an arrow_buffer::Buffer by sharing the underlying Arc<Mmap> simd-r-drive-entry-handle/src/entry_handle.rs:129-139
• Reference Counting : The Arrow Buffer increments the reference count of the Arc<Mmap>, ensuring the memory mapping remains valid for the lifetime of the Buffer simd-r-drive-entry-handle/src/entry_handle.rs:173-176
• Alignment Validation : Integrated checks in as_arrow_buffer and into_arrow_buffer ensure that the exported buffer meets both pointer and offset alignment requirements when compiled in debug or test mode CHANGELOG.md:106-108

Sources: simd-r-drive-entry-handle/Cargo.toml15 simd-r-drive-entry-handle/src/constants.rs18 CHANGELOG.md:93-97 CHANGELOG.md:106-108 simd-r-drive-entry-handle/src/entry_handle.rs:10-19

Data Flow: From Storage to Arrow

The following diagram illustrates how data flows from the on-disk memory-mapped file into an Arrow Buffer through the EntryHandle abstraction.

graph TD
    subgraph "Disk_Storage"
        [DataStore_File]
    end

    subgraph "simd-r-drive-entry-handle"
        [Arc_Mmap]
        [EntryHandle]
        [as_slice]
    end

    subgraph "Apache_Arrow_Ecosystem"
        [arrow_buffer_Buffer]
        [arrow_array_PrimitiveArray]
    end

    [DataStore_File] -- "mmap" --> [Arc_Mmap]
    [Arc_Mmap] -- "shared_reference" --> [EntryHandle]
    [EntryHandle] -- "into_arrow_buffer" --> [arrow_buffer_Buffer]
    [arrow_buffer_Buffer] -- "Zero-Copy_View" --> [arrow_array_PrimitiveArray]
    [as_slice] -- "Deref" --> [EntryHandle]

Logic Flow: Zero-Copy Arrow View

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:12-19 simd-r-drive-entry-handle/src/entry_handle.rs:129-139 simd-r-drive-entry-handle/src/entry_handle.rs:151-155

Implementation Details

The integration is implemented by extending EntryHandle with methods that interface with the arrow crate.

Function: as_arrow_buffer()

This method creates a new Arrow Buffer that points to the same memory range as the EntryHandle. It utilizes the shared Arc<Mmap> to ensure no additional allocations occur. Because it clones the Arc, the operation is $O(1)$ and involves no data movement simd-r-drive-entry-handle/src/entry_handle.rs:159-162

Function: into_arrow_buffer()

Similar to as_arrow_buffer(), but consumes the EntryHandle. This is the preferred method when the handle is no longer needed, as it transfers ownership of the underlying Arc<Mmap> directly to the Arrow Buffer CHANGELOG.md:133-135

Alignment and Safety

Arrow requires buffers to be aligned to specific boundaries (typically 64 bytes) for optimal SIMD performance. The storage engine enforces this via PAYLOAD_ALIGNMENT simd-r-drive-entry-handle/src/constants.rs18 When payloads are written, a pre-pad is calculated to ensure the payload start address is a multiple of the alignment simd-r-drive-entry-metadata.rs:22-24

Sources: simd-r-drive-entry-handle/src/constants.rs18 simd-r-drive-entry-handle/src/entry_metadata.rs:22-24 simd-r-drive-entry-handle/src/entry_handle.rs:10-19

Entity Mapping: Code to Concept

This diagram maps the high-level Arrow concepts to the specific structs and functions used in the rust-simd-r-drive implementation.

classDiagram
    class EntryHandle {+Arc~Mmap~ mmap_arc\n+Range~usize~ range\n+EntryMetadata metadata\n+as_slice()\n+as_arrow_buffer()\n+into_arrow_buffer()}

    class EntryMetadata {+u64 key_hash\n+u64 prev_offset\n+u8_4 checksum}

    class ArrowBufferIntegration {<<Interface>>\n+as_arrow_buffer()\n+into_arrow_buffer()}

    EntryHandle --> EntryMetadata : Contains
    EntryHandle --> ArrowBufferIntegration : Enables (via arrow feature)
    ArrowBufferIntegration ..> "arrow::buffer::Buffer" : Produces

Concept to Entity Map

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:10-19 simd-r-drive-entry-handle/Cargo.toml15 CHANGELOG.md:133-135

Usage Example

To use this feature, the arrow feature must be enabled in the simd-r-drive-entry-handle crate simd-r-drive-entry-handle/Cargo.toml15 Once enabled, the EntryHandle can be converted:

1. Retrieve Handle : Obtain an EntryHandle from the DataStore or an EntryIterator src/storage_engine/entry_iterator.rs:121-125
2. Convert : Call handle.into_arrow_buffer().
3. Wrap : Use the resulting Buffer to create an arrow_array::RecordBatch or PrimitiveArray.

This workflow is critical for applications that perform heavy computation on stored data, as it bypasses the standard overhead of serialization and deserialization CHANGELOG.md:93-97

Sources: simd-r-drive-entry-handle/Cargo.toml15 CHANGELOG.md:133-135 simd-r-drive-entry-handle/src/entry_handle.rs:129-139 src/storage_engine/entry_iterator.rs:121-125

Extensions Crate

Loading…

Extensions Crate

Relevant source files

• extensions/Cargo.toml
• extensions/README.md
• extensions/src/lib.rs
• extensions/src/storage_file_import_ext.rs
• extensions/src/utils/option_serializer.rs
• src/storage_engine/simd_copy.rs
• src/utils.rs
• tests/align_or_copy_tests.rs

The simd-r-drive-extensions crate provides higher-level storage patterns and utilities built on top of the core DataStore extensions/README.md:5-6 These extensions are implemented as Rust traits that extend the functionality of any type implementing DataStoreReader and DataStoreWriter extensions/src/storage_file_import_ext.rs:1-5 extensions/src/lib.rs:7-16

While the core engine focuses on zero-copy binary blobs, the extensions crate introduces structured data handling, automated lifecycle management (TTL), and filesystem integration extensions/README.md:15-134

Extension Architecture

The extensions use a namespacing strategy to prevent key collisions between different types of extended storage and standard raw binary storage. This is achieved using the NamespaceHasher utility extensions/src/storage_file_import_ext.rs:4-5 src/utils.rs:10-11

Extension to Core Mapping

The following diagram illustrates how extension traits map high-level concepts to the underlying DataStore entities.

Diagram: Extension Trait Mapping

graph TD
    subgraph "Natural Language Space"
        "Time-To-Live Cache"["Time-To-Live Cache"] 
        "Explicit Nulls"["Explicit Nulls"]
        "Bulk Directory Import"["Bulk Directory Import"]
end

    subgraph "Code Entity Space"
        "Time-To-Live Cache" --> "StorageCacheExt"["StorageCacheExt"]
        "Explicit Nulls" --> "StorageOptionExt"["StorageOptionExt"]
        "Bulk Directory Import" --> "StorageFileImportExt"["StorageFileImportExt"]
        
        "StorageCacheExt" -- "uses" --> "TTL_PREFIX"["TTL_PREFIX"]
        "StorageOptionExt" -- "uses" --> "OPTION_PREFIX"["OPTION_PREFIX"]
        "StorageFileImportExt" -- "uses" --> "import_dir_recursively"["import_dir_recursively()"]
        
        "TTL_PREFIX" & "OPTION_PREFIX" & "import_dir_recursively" -- "calls" --> "DataStore_write"["DataStore::write() / write_stream()"]
end

Sources: extensions/src/lib.rs:7-16 extensions/README.md:104-106 extensions/src/storage_file_import_ext.rs:90-91

Key Extensions

StorageCacheExt: TTL-Based Caching

The StorageCacheExt trait adds write_with_ttl() and read_with_ttl() methods to the DataStore extensions/README.md:68-70

• Mechanism : It prepends an 8-byte Little-Endian expiration timestamp to the payload extensions/README.md:84-85
• Eviction : Expired entries are automatically evicted upon read to prevent stale data extensions/README.md87
• Serialization : Uses bitcode for non-zero-copy data handling extensions/README.md:85-86

For details, see StorageCacheExt: TTL-Based Caching.

StorageOptionExt: Explicit None Storage

The StorageOptionExt trait allows users to distinguish between a key that is missing (NotFound) and a key that is explicitly set to None extensions/README.md:17-45

• Tombstones : Uses a specific 2-byte marker OPTION_TOMBSTONE_MARKER ([0xFF, 0xFE]) to represent None in the physical storage extensions/src/utils/option_serializer.rs:1-8 extensions/src/utils/option_serializer.rs:27-28
• Namespacing : Operates within the OPTION_PREFIX namespace to isolate these markers from regular data extensions/src/lib.rs:7-8
• Implementation : The logic is encapsulated in serialize_option and deserialize_option extensions/src/utils/option_serializer.rs:24-63

For details, see StorageOptionExt: Explicit None Storage.

StorageFileImportExt: Filesystem Import

This extension provides utilities for syncing local filesystem directories into a DataStore extensions/README.md:89-106

• Streaming : Uses import_dir_recursively() to walk directory trees via walkdir and stream file contents directly into the store using write_stream(), minimizing memory overhead extensions/src/storage_file_import_ext.rs:77-91
• Retrieval : Provides open_file_stream() to read stored files back as an EntryStream extensions/src/storage_file_import_ext.rs:54-58
• Keys : Automatically converts filesystem paths to Unix-style relative keys extensions/src/storage_file_import_ext.rs:117-124

For details, see StorageFileImportExt: Filesystem Import.

Data Flow Overview

The following diagram shows the data flow from high-level extension calls down to the serialized format on disk.

Diagram: Extension Data Flow

sequenceDiagram
    participant App as "Application Code"
    participant Ext as "Extension Trait (e.g., StorageOptionExt)"
    participant Ser as "option_serializer"
    participant DS as "DataStore"

    App->>Ext: write_option(key, None)
    Ext->>Ser: serialize_option(None)
    Ser-->>Ext: [0xFF, 0xFE] (Tombstone)
    Ext->>DS: write(namespaced_key, [0xFF, 0xFE])
    DS-->>App: Result<u64> (Offset)

Sources: extensions/src/utils/option_serializer.rs:24-29 extensions/src/lib.rs:7-8

Summary Table

Sources: extensions/README.md:5-134 extensions/src/utils/option_serializer.rs:24-63 extensions/src/storage_file_import_ext.rs:12-59

StorageCacheExt: TTL-Based Caching

Loading…

StorageCacheExt: TTL-Based Caching

Relevant source files

• extensions/src/constants.rs
• extensions/src/storage_cache_ext.rs
• extensions/src/storage_option_ext.rs
• extensions/tests/storage_cache_tests.rs
• extensions/tests/storage_option_tests.rs

The StorageCacheExt trait provides a high-level caching layer on top of the core DataStore engine. It introduces Time-To-Live (TTL) functionality, allowing entries to be stored with an expiration timestamp. This extension handles namespacing, serialization via bitcode, and automatic eviction of expired entries during read operations.

Overview of TTL Storage

Unlike the core DataStore which treats payloads as opaque byte slices, StorageCacheExt structures the payload to include metadata. Every entry written via this extension is prefixed with an 8-byte Little-Endian (LE) timestamp representing the absolute expiration time in seconds since the Unix Epoch extensions/src/storage_cache_ext.rs:19-22

Key Features

• Namespace Isolation : Uses a dedicated TTL_PREFIX (defined as a namespaced byte array 0xF7 + ttl + 0xFD) to ensure TTL-managed keys do not collide with standard keys or other extensions extensions/src/constants.rs42 extensions/src/storage_cache_ext.rs:1-7
• Automatic Eviction : If a read_with_ttl() call encounters an expired timestamp, it triggers a delete() on the underlying store and returns None extensions/src/storage_cache_ext.rs:95-98
• Bitcode Serialization : Values are serialized using the bitcode crate, making this a non-zero-copy operation compared to the core engine’s raw byte access extensions/src/storage_cache_ext.rs41
• Option Support : The implementation safely handles Option<T> types, allowing Some or None to be cached with a TTL without requiring additional logic extensions/src/storage_cache_ext.rs:21-22

Data Flow and Implementation

The following table and diagrams illustrate the transformation of data from high-level Rust types to the on-disk format.

TTL Write and Read Flow

Code Entity Relationship Diagram

This diagram shows how StorageCacheExt interacts with the core DataStore and utility classes.

Sources: extensions/src/storage_cache_ext.rs:1-54 extensions/src/storage_cache_ext.rs:73-78 extensions/src/constants.rs42

graph TD
    subgraph "Extensions_Crate [extensions/src/]"
        SCE["StorageCacheExt (Trait)"]
IMPL["DataStore Implementation"]
TNH["TTL_NAMESPACE_HASHER (OnceLock)"]
BC["bitcode (External)"]
TP["TTL_PREFIX (Constant)"]
end

    subgraph "Core_Library [src/]"
        DS["DataStore (Struct)"]
NH["NamespaceHasher"]
DSR["DataStoreReader (Trait)"]
DSW["DataStoreWriter (Trait)"]
end

 
   SCE --> IMPL
    IMPL -- "uses" --> TNH
    IMPL -- "serializes_via" --> BC
    TNH -- "initializes_with" --> TP
    TNH -- "creates" --> NH
    IMPL -- "calls" --> DS
    DS -- "implements" --> DSR
    DS -- "implements" --> DSW
    
    IMPL -- "write_with_ttl()" --> DSW
    IMPL -- "read_with_ttl()" --> DSR

Binary Layout

When using write_with_ttl(), the payload stored in the DataStore follows a specific binary structure.

TTL Payload Transformation

Sources: extensions/src/storage_cache_ext.rs:60-71

Key Functions

write_with_ttl<T: Encode>

Calculates the expiration by adding ttl_secs to the current SystemTime extensions/src/storage_cache_ext.rs:60-64 It prevents overflow using saturating_add extensions/src/storage_cache_ext.rs64 The final payload is a concatenation of the 8-byte timestamp and the bitcode-encoded value extensions/src/storage_cache_ext.rs:66-70

read_with_ttl<T: Decode>

1. Retrieval : Reads the entry from the DataStore using the namespaced key extensions/src/storage_cache_ext.rs:73-78
2. Validation : Checks if the data is at least 8 bytes long to ensure the timestamp is present extensions/src/storage_cache_ext.rs:82-87
3. Expiration Check : Extracts the u64 timestamp via u64::from_le_bytes extensions/src/storage_cache_ext.rs89 If the current time is greater than or equal to the timestamp, it calls self.delete() and returns Ok(None) extensions/src/storage_cache_ext.rs:95-98
4. Deserialization : If valid, decodes the remaining bytes starting at offset 8 into type T using bitcode::decode extensions/src/storage_cache_ext.rs:100-102

Usage Example

Sources: extensions/tests/storage_cache_tests.rs:29-69

Error Handling

• ErrorKind::NotFound : Returned if the key does not exist in the underlying store extensions/src/storage_cache_ext.rs104
• ErrorKind::InvalidData : Returned if the payload is less than 8 bytes or if bitcode deserialization fails extensions/src/storage_cache_ext.rs:83-86 extensions/src/storage_cache_ext.rs102
• Regular Read Failure : Attempting to use read_with_ttl on a key written with the standard write() method (without the 8-byte prefix) will result in a deserialization error because the first 8 bytes of the raw data will be interpreted as a timestamp extensions/tests/storage_cache_tests.rs:154-173

Sources:

• extensions/src/storage_cache_ext.rs:1-107
• extensions/src/constants.rs:20-42
• extensions/tests/storage_cache_tests.rs:29-151

StorageOptionExt: Explicit None Storage

Loading…

StorageOptionExt: Explicit None Storage

Relevant source files

• extensions/README.md
• extensions/src/storage_cache_ext.rs
• extensions/src/storage_option_ext.rs
• extensions/src/utils.rs
• extensions/src/utils/option_serializer.rs
• extensions/tests/storage_cache_tests.rs
• extensions/tests/storage_option_tests.rs

The StorageOptionExt trait provides a high-level abstraction for storing and retrieving Rust Option<T> types within a DataStore extensions/src/storage_option_ext.rs:20-23 While the core DataStore distinguishes between a key existing and not existing (returning None on a read for a missing key), StorageOptionExt allows an application to explicitly store a None value for a key that does exist in the index extensions/src/storage_option_ext.rs:25-28

Overview and Purpose

In standard DataStore operations, a None result from read() indicates the key was never written or was deleted. StorageOptionExt introduces an explicit tombstone marker to represent a logical None value extensions/src/storage_option_ext.rs:32-33 This is useful for:

• Distinguishing “Not Found” from “Explicitly Null” : Applications can tell if a field is missing vs. explicitly set to None extensions/src/storage_option_ext.rs:25-26
• Namespace Isolation : All Option operations are automatically isolated using a dedicated OPTION_PREFIX namespace extensions/src/storage_option_ext.rs:143-146
• Typed Serialization : Automatically handles serialization of T using bitcode when the value is Some(T) extensions/src/storage_option_ext.rs31

Logic Mapping: Natural Language to Code

The following diagram maps the logical concept of “Explicit None” to the specific code entities responsible for the implementation.

Explicit None Logic Mapping

graph TD
    subgraph "Natural_Language_Space"
        A["Explicit_None"]
B["Some_Value"]
C["Missing_Key"]
end

    subgraph "Code_Entity_Space"
        D["OPTION_TOMBSTONE_MARKER"]
E["bitcode::encode"]
F["std::io::ErrorKind::NotFound"]
G["StorageOptionExt::write_option"]
H["StorageOptionExt::read_option"]
I["serialize_option"]
J["deserialize_option"]
end

 
   A --> G
 
   B --> G
 
   G --> I
 
   I --> D
 
   I --> E
 
   H --> J
 
   J --> D
 
   J --> E
 
   C --> H
 
   H --> F

Sources: extensions/src/storage_option_ext.rs:1-165 extensions/src/utils/option_serializer.rs:1-63

Implementation Details

Namespace Isolation

To prevent collisions with raw binary data or other extensions (like TTL caching), StorageOptionExt uses a NamespaceHasher initialized with the OPTION_PREFIX extensions/src/storage_option_ext.rs:143-146 This ensures that a key b"user_1" written via write_option does not overwrite a key b"user_1" written via standard DataStore::write. The hasher is lazily initialized via a OnceLock named OPTION_NAMESPACE_HASHER extensions/src/storage_option_ext.rs12

Storage Format

The extension uses the option_serializer utility to transform Option<T> into a format the DataStore can persist:

Data Flow: Write and Read

The following diagram illustrates the flow of data through the extension and into the core storage engine.

StorageOptionExt Data Flow

sequenceDiagram
    participant App as "Application"
    participant Ext as "StorageOptionExt"
    participant Ser as "option_serializer"
    participant DS as "DataStore"

    Note over App, DS: write_option(key, Some(data))
    App->>Ext: write_option(key, Some(data))
    Ext->>Ser: serialize_option(Some(data))
    Ser-->>Ext: Vec<u8> (bitcode)
    Ext->>DS: write(namespaced_key, bytes)
    DS-->>App: Ok(offset)

    Note over App, DS: read_option(key)
    App->>Ext: read_option(key)
    Ext->>DS: read(namespaced_key)
    DS-->>Ext: Some(EntryHandle)
    Ext->>Ser: deserialize_option(entry.as_slice())
    alt matches [0xFF, 0xFE]
        Ser-->>Ext: Ok(None)
    else valid bitcode
        Ser-->>Ext: Ok(Some(T))
    end
    Ext-->>App: Result<Option<T>, Error>

Sources: extensions/src/storage_option_ext.rs:142-165 extensions/src/utils/option_serializer.rs:24-63

Key Functions

write_option<T: Encode>

Serializes the provided Option<&T> and writes it to the DataStore under a namespaced key extensions/src/storage_option_ext.rs:143-150

• Process : Fetches the OPTION_NAMESPACE_HASHER, hashes the key, calls serialize_option, and performs a standard DataStore::write extensions/src/storage_option_ext.rs:144-149
• Citations : extensions/src/storage_option_ext.rs:143-150 extensions/src/utils/option_serializer.rs:24-29

read_option<T: Decode>

Retrieves data from the namespaced key and attempts to deserialize it extensions/src/storage_option_ext.rs:152-164

• Returns Ok(Some(T)) if the data is a valid bitcode-encoded T extensions/src/utils/option_serializer.rs:60-61
• Returns Ok(None) if the data matches the OPTION_TOMBSTONE_MARKER extensions/src/utils/option_serializer.rs:56-58
• Returns Err(ErrorKind::NotFound) if the key does not exist in the store (i.e., DataStore::read returns None) extensions/src/storage_option_ext.rs:159-162
• Citations : extensions/src/storage_option_ext.rs:152-164 extensions/src/utils/option_serializer.rs:55-63

Technical Considerations

Non-Zero-Copy Performance

Unlike the core DataStore::read which returns an EntryHandle providing zero-copy access to memory-mapped data, read_option requires deserialization into a new instance of T extensions/src/storage_option_ext.rs93 This involves memory allocation and CPU cycles for the bitcode decoding process extensions/src/storage_option_ext.rs:137-138

Tombstone Collision

The tombstone marker [0xFF, 0xFE] is a reserved sequence within the OPTION_PREFIX namespace extensions/src/storage_option_ext.rs:15-18 If a serialized bitcode payload of T were to exactly match [0xFF, 0xFE], it would be incorrectly interpreted as None extensions/src/utils/option_serializer.rs:56-57 However, bitcode’s internal structure and the small size of the marker make this collision statistically improbable for complex types.

Persistence of None

When write_option(key, None) is called, the entry is not deleted from the underlying file. Instead, a new entry containing the tombstone marker is appended extensions/tests/storage_option_tests.rs:101-103 This preserves the append-only nature of the store and ensures that the None status is part of the versioned history of that key extensions/tests/storage_option_tests.rs:116-128

Sources:

• extensions/src/storage_option_ext.rs:1-165
• extensions/src/utils/option_serializer.rs:1-63
• extensions/README.md:17-52
• extensions/tests/storage_option_tests.rs:1-221

StorageFileImportExt: Filesystem Import

Loading…

StorageFileImportExt: Filesystem Import

Relevant source files

• extensions/src/lib.rs
• extensions/src/storage_file_import_ext.rs
• extensions/tests/storage_file_import_tests.rs

The StorageFileImportExt trait extends the DataStore with capabilities to ingest entire directory trees from the local filesystem into the append-only store. It specifically addresses the need for bulk data migration and streaming file access, using relative paths as keys to maintain directory structure within the schema-less storage engine.

Overview and Purpose

The primary goal of StorageFileImportExt is to provide a high-level API for synchronizing filesystem assets with a DataStore extensions/src/storage_file_import_ext.rs:12-59 It abstracts the complexity of recursive directory walking and ensures that file contents are written using efficient streaming I/O rather than loading entire files into memory.

Key features include:

• Recursive Ingestion : Automatically walks subdirectories using the walkdir crate extensions/src/storage_file_import_ext.rs:77-80
• Path-to-Key Mapping : Converts OS-specific paths into normalized, Unix-style binary keys (e.g., subdir/file.txt) extensions/src/storage_file_import_ext.rs:117-124
• Namespacing : Supports optional NamespaceHasher integration to prevent collisions when importing multiple directories into the same store extensions/src/storage_file_import_ext.rs:126-129
• Streaming Reads : Provides EntryStream wrappers for zero-copy access to stored file data extensions/src/storage_file_import_ext.rs:54-58

Sources : extensions/src/storage_file_import_ext.rs:1-59 extensions/src/storage_file_import_ext.rs:117-130

Implementation Detail: Recursive Import

The import_dir_recursively function serves as the entry point for filesystem ingestion. It validates the source directory using standard std::path::Path checks before initiating the walk extensions/src/storage_file_import_ext.rs:68-75

Data Flow: Filesystem to DataStore

The following diagram illustrates how a file on disk is transformed into a namespaced entry within the DataStore.

Import Pipeline: Path Normalization and Streaming Write

graph TD
    subgraph "Local Filesystem"
        A["File: ./assets/images/logo.png"]
end

    subgraph "StorageFileImportExt.import_dir_recursively()"
        B["walkdir::WalkDir"]
C["strip_prefix()"]
D["to_namespaced_key()"]
E["Unix-style Path: 'images/logo.png'"]
end

    subgraph "DataStore Logic"
        F["NamespaceHasher (Optional)"]
G["DataStore.write_stream()"]
end

 
   A --> B
 
   B --> C
 
   C --> E
 
   E --> D
 
   D --> F
 
   F --> G
 
   G --> H[("DataStore (.bin file)")]

Sources : extensions/src/storage_file_import_ext.rs:62-96 extensions/src/storage_file_import_ext.rs:117-130

Key Functions

Sources : extensions/src/storage_file_import_ext.rs:61-130

Streaming and Zero-Copy Access

A critical performance feature of this extension is its use of streaming for both ingestion and retrieval.

Streaming Write

Instead of reading a file into a buffer and passing it to DataStore.write(), import_dir_recursively opens a std::fs::File and passes it to DataStore.write_stream() extensions/src/storage_file_import_ext.rs:90-91 This allows the core engine to pipe data directly from the filesystem into the memory-mapped store, minimizing heap allocations.

Streaming Read (EntryStream)

When a file is retrieved via open_file_stream(), the system returns an EntryStream extensions/src/storage_file_import_ext.rs113 This object wraps an EntryHandle, which itself is a view into the memory-mapped storage file.

Code Entity Relationship: Read Streaming

classDiagram
    class DataStore {+read(key) EntryHandle}
    class StorageFileImportExt {+import_dir_recursively(base_dir, namespace)\n+open_file_stream(rel_path, namespace) EntryStream}
    class EntryHandle {
        +as_slice() &[u8]
    }
    class EntryStream {-handle: EntryHandle\n-pos: usize\n+read(buf) io::Result}

    DataStore <|-- StorageFileImportExt : implements
    StorageFileImportExt ..> EntryHandle : retrieves
    EntryStream o-- EntryHandle : wraps
    EntryStream ..|> "std::io::Read" : implements

Sources : extensions/src/storage_file_import_ext.rs:61-115 extensions/src/storage_file_import_ext.rs:2-3

Verification and Testing

The extension is verified through comprehensive integration tests that simulate directory imports and content validation.

• Content Integrity : Tests ensure that read_file_entry and open_file_stream return bytes identical to the original filesystem source extensions/tests/storage_file_import_tests.rs:17-61 extensions/tests/storage_file_import_tests.rs:131-168
• Path Normalization : Verification that keys are stored as Unix-style paths regardless of the host OS by converting the key back to UTF-8 and replacing separators during comparison extensions/tests/storage_file_import_tests.rs:41-42
• Error Cases : Validation that the system correctly rejects non-existent directories or file paths passed as directories extensions/tests/storage_file_import_tests.rs:171-193

Sources : extensions/tests/storage_file_import_tests.rs:1-193

Error Handling

• NotFound : Returned if the base_dir provided to import_dir_recursively does not exist or is not a directory extensions/src/storage_file_import_ext.rs:70-75
• IO Errors : Standard filesystem errors (permissions, disk full) encountered during WalkDir or File::open are propagated to the caller extensions/src/storage_file_import_ext.rs:90-91
• Key Collisions : Because DataStore is append-only, importing the same directory twice will result in new entries that mask the old ones in the KeyIndexer extensions/src/storage_file_import_ext.rs91

Sources : extensions/src/storage_file_import_ext.rs:62-96

CLI Interface

Loading…

CLI Interface

Relevant source files

• src/cli.rs
• src/cli/cli_parser.rs
• src/cli/execute_command.rs
• tests/cli_tests.rs

The simd-r-drive CLI provides a high-level interface for interacting with the underlying storage engine from the terminal. It serves as the primary entry point for manual data manipulation, administrative tasks like compaction, and piping data into or out of the append-only storage files.

The CLI is designed to be “pipe-friendly,” supporting both direct string arguments and binary streams via standard input (stdin) and standard output (stdout).

System Architecture and Delegation

The CLI layer is built using the clap library and acts as a thin wrapper around the DataStore API. When a command is issued, the CLI parses the arguments via the Cli struct src/cli/cli_parser.rs:16-26 matches the command in the Commands enum src/cli.rs:4-5 and delegates the logic to the appropriate DataStore methods via execute_command src/cli/execute_command.rs25

CLI to Code Entity Mapping

The following diagram illustrates how CLI components map to the internal code structures and how they interact with the core storage engine.

CLI Delegation Flow

graph TD
    subgraph "CLI_Space_(User_Interface)"
 
       A["Terminal Command"] --> B["Cli Struct"]
B --> C["Commands Enum"]
end

    subgraph "Code_Entity_Space_(src/cli/)"
        B["Cli Struct"] --- B_DEF["cli_parser.rs:16-26"]
C["Commands Enum"] --- C_DEF["src/cli/commands.rs"]
D["execute_command()"] --- D_DEF["execute_command.rs:25-25"]
end

    subgraph "Storage_Engine_Space_(src/storage_engine/)"
        E["DataStore"] --- E_DEF["DataStore::open()"]
F["DataStoreReader"]
G["DataStoreWriter"]
end

 
   C --> D
 
   D --> E
 
   E --> F
 
   E --> G

Sources: src/cli/cli_parser.rs:16-26 src/cli.rs:4-5 src/cli/execute_command.rs25

Command Execution Flow

The central logic for processing CLI requests resides in execute_command src/cli/execute_command.rs25 This function matches the parsed Commands enum and performs the necessary setup, such as opening the DataStore and handling I/O streams.

Sources: src/cli/execute_command.rs:25-175

Stream Handling and Terminal Detection

The CLI intelligently handles different output modes. When read is called, the system detects if the output is a TTY (terminal) or a pipe via is_terminal() src/cli/execute_command.rs48

• Terminal Mode: If outputting to a terminal, the CLI attempts to validate UTF-8 strings for readable display src/cli/execute_command.rs:58-63 It also appends a newline for cleaner terminal output src/cli/execute_command.rs:72-74
• Binary/Pipe Mode: If the output is redirected (e.g., > file.bin), it emits raw binary data without modification src/cli/execute_command.rs:64-67

For writes, the CLI supports write_stream src/cli/execute_command.rs100 allowing users to pipe large files into the database via stdin src/cli/execute_command.rs:98-103 It also supports custom buffer sizes for reading large entries, parsed via parse_buffer_size src/cli/execute_command.rs33 and defaulting to 64KB src/cli/execute_command.rs39

Integration Testing

The CLI interface is verified through integration tests in tests/cli_tests.rs. These tests spawn the binary using std::process::Command tests/cli_tests.rs:15-26 to ensure that the end-to-end flow—from argument parsing to disk persistence—works as expected. Tests cover basic read/write tests/cli_tests.rs:11-41 large file chunking via --buffer-size tests/cli_tests.rs:111-167 and multi-storage operations like copy tests/cli_tests.rs:171-219

CLI Command Integration

Sources: tests/cli_tests.rs:11-41 tests/cli_tests.rs:111-167 tests/cli_tests.rs:171-219

Sub-pages

CLI Commands Reference

Detailed documentation for every available command, including flag descriptions, binary output behavior, and buffer size configuration for large data transfers. For details, see CLI Commands Reference.

CLI Parser and Help System

Technical overview of the clap implementation, the Cli structure src/cli/cli_parser.rs:16-26 and how the custom help template src/cli/help_template.rs8 is integrated into the binary’s --help output. For details, see CLI Parser and Help System.

CLI Commands Reference

Loading…

CLI Commands Reference

Relevant source files

• src/cli/commands.rs
• src/cli/execute_command.rs
• tests/cli_tests.rs

This page provides a comprehensive technical reference for the simd-r-drive Command Line Interface (CLI). The CLI serves as the primary entry point for interacting with the append-only storage engine, providing commands for data manipulation, maintenance, and inspection.

Architecture Overview

The CLI is built using the clap crate for argument parsing and delegates all core logic to the DataStore engine. The execution flow follows a pattern of parsing arguments into a Cli struct, which contains a Commands enum, and then dispatching these to the execute_command function src/cli/execute_command.rs:25-26

CLI Data Flow and Entity Mapping

The following diagram illustrates how CLI commands map to specific functions within the DataStore and how data streams through the system.

Diagram: CLI to Storage Engine Mapping

graph TD
    subgraph "CLI Layer (src/cli/)"
 
       A["Cli Struct (src/cli/cli_parser.rs)"] --> B["Commands Enum (src/cli/commands.rs)"]
B --> C["execute_command() (src/cli/execute_command.rs)"]
end

    subgraph "Logic & Data Flow"
        C -- "Read/Stream" --> D["EntryStream (src/storage_engine/entry_stream.rs)"]
C -- "Write/Stream" --> E["DataStoreWriter::write_stream()"]
C -- "Maintenance" --> F["DataStore::compact()"]
end

    subgraph "Core Engine (src/storage_engine/)"
 
       D --> G["DataStoreReader::read()"]
E --> H["DataStore::write_stream()"]
F --> I["EntryIterator (src/storage_engine/entry_iterator.rs)"]
end

    style A stroke-dasharray: 5 5
    style G font-weight:bold
    style H font-weight:bold

Sources: src/cli/execute_command.rs:1-7 src/cli/execute_command.rs:25-174 src/cli/commands.rs:5-65

Command Reference

Read

Retrieves the value associated with a specific key and outputs it to stdout.

• Usage: <storage> read <key> [--buffer-size <size>]
• Buffer Management: Defaults to 64KB if --buffer-size is not provided src/cli/execute_command.rs:30-39 It uses parse_buffer_size to handle human-readable strings like “32K” or “1M” src/cli/execute_command.rs:33-34
• Output Modes:
  • Terminal: If stdout is a TTY (checked via is_terminal()), the CLI attempts to validate the data as UTF-8. If valid, it prints as text; otherwise, it falls back to raw bytes src/cli/execute_command.rs:48-63 A newline is appended for readability src/cli/execute_command.rs:72-74
  • Piped/Binary: If redirected (e.g., > file.bin), it outputs raw binary data without modification or trailing newlines src/cli/execute_command.rs:64-67

Sources: src/cli/commands.rs:7-14 src/cli/execute_command.rs:27-85 tests/cli_tests.rs:111-167

Write

Stores a value for a given key. Supports both direct arguments and piped input.

• Usage: <storage> write <key> [value]
• Direct Write: If value is provided as an argument, it is written directly to the store using DataStore::write() src/cli/execute_command.rs:91-95
• Piped Stdin: If value is omitted and stdin is not a terminal, the CLI uses DataStore::write_stream() to pipe data from stdin directly into the storage file src/cli/execute_command.rs:96-103
• Validation: If neither a value nor a pipe is detected (checked via is_terminal() and the FORCE_NO_TTY environment variable), the command exits with an error src/cli/execute_command.rs:96-108

Sources: src/cli/commands.rs:17-23 src/cli/execute_command.rs:87-111 tests/cli_tests.rs:45-67

Copy, Move, and Rename

These commands manage entry lifecycle across the same or different storage files.

Sources: src/cli/commands.rs:26-46 src/cli/execute_command.rs:113-160

Delete

Marks a key as deleted by appending a “tombstone” entry to the storage file.

• Usage: <storage> delete <key>
• Implementation: Calls DataStore::delete(), which ensures the key is no longer reachable via the index by writing a null-payload entry src/cli/execute_command.rs:165-167

Sources: src/cli/commands.rs:49-52 src/cli/execute_command.rs:162-169

Compact

Reclaims disk space by removing shadowed (old) versions of keys and deleted entries.

• Usage: <storage> compact
• Logic: It opens the storage in read-write mode and triggers the compaction process via DataStore::compact() src/cli/execute_command.rs:171-174

Sources: src/cli/commands.rs55 src/cli/execute_command.rs:171-174

Info and Metadata

Provides diagnostic information about the storage file or a specific entry.

• Info: Displays high-level storage stats like file size and entry count src/cli/commands.rs58
• Metadata: Retrieves the EntryMetadata for a specific key, including its hash, offset, and checksum src/cli/commands.rs:61-64

Sources: src/cli/commands.rs:58-64 src/cli/execute_command.rs26

Execution Logic and Error Handling

The CLI uses a centralized execute_command function to handle the lifecycle of a command, from opening the DataStore to handling I/O errors.

Diagram: Write Command Execution Flow

sequenceDiagram
    participant U as "User/Shell"
    participant C as "execute_command (src/cli/execute_command.rs)"
    participant S as "DataStore (src/storage_engine/mod.rs)"
    participant F as "File System"

    U->>C: write key [value]
    alt "Value Provided"
        C->>S: DataStore::write(key, value)
        S->>F: Append Entry
    else "Piped Stdin"
        U->>C: "echo 'data' | bin write key"
        C->>S: DataStore::write_stream(key, stdin)
        loop "Chunked Read"
            S->>F: Append Chunks via SIMD/Scalar
        end
    end
    C->>U: Print 'Stored key'

Key Functions

• DataStore::open_existing(&path): Used by commands that require an existing file (Read, Info, Metadata) src/cli/execute_command.rs28
• DataStore::open(&path): Used by commands that can create a file if it’s missing (Write) src/cli/execute_command.rs88
• parse_buffer_size(String): Converts inputs like “64K” into usize bytes src/cli/execute_command.rs33

Sources: src/cli/execute_command.rs:25-174 src/utils/mod.rs:1-10

Global Options

Storage Path

The first argument to the binary is always the path to the storage file. If the file does not exist, commands like write will create it automatically via DataStore::open, while read will fail via DataStore::open_existing src/cli/execute_command.rs28 src/cli/execute_command.rs88

Buffer Size Flag

Available on the read command, the --buffer-size (or -b) flag controls the internal memory allocation for streaming data from the storage engine to stdout. This is particularly useful when reading very large entries to prevent excessive memory consumption src/cli/commands.rs:11-13

Sources: src/cli/commands.rs:7-14 tests/cli_tests.rs:111-167

CLI Parser and Help System

Loading…

CLI Parser and Help System

Relevant source files

• src/cli.rs
• src/cli/cli_parser.rs
• src/cli/help_template.rs
• src/utils/parse_buffer_size.rs

The CLI Parser and Help System provide the user-facing entry point for interacting with the simd-r-drive storage engine. Built on the clap framework, this system handles command-line argument parsing, sub-command routing, and dynamic help generation. It translates user intentions into structured data that the execute_command logic uses to invoke the underlying DataStore.

CLI Structure and Data Flow

The CLI is structured around the Cli struct, which acts as the root parser. Every command requires a path to a storage file as its first positional argument, followed by a specific sub-command defined in the Commands enum.

Code Entity Mapping: CLI Definitions

The following diagram illustrates how the CLI structure in code maps to the command-line interface presented to the user.

CLI Entity Mapping

graph TD
    subgraph "Code Entity Space"
        CLI_STRUCT["Cli Struct [src/cli/cli_parser.rs]"]
CMD_ENUM["Commands Enum [src/cli/commands.rs]"]
HELP_VAR["HELP_TEMPLATE [src/cli/help_template.rs]"]
BUF_PARSER["parse_buffer_size() [src/utils/parse_buffer_size.rs]"]
end

    subgraph "Natural Language / CLI Space"
        BIN_NAME["'simd-r-drive' (Binary)"]
STORAGE_ARG["'storage' (Positional PathBuf)"]
SUB_CMD["Subcommands (read, write, etc.)"]
EXAMPLES["Extended Help Examples"]
SIZE_FLAGS["'--buffer-size' (K, MB, GB)"]
end

 
   CLI_STRUCT --> BIN_NAME
 
   CLI_STRUCT --> STORAGE_ARG
 
   CLI_STRUCT --> CMD_ENUM
 
   CMD_ENUM --> SUB_CMD
 
   HELP_VAR --> EXAMPLES
    CLI_STRUCT -. "uses" .-> HELP_VAR
    SUB_CMD -. "uses" .-> BUF_PARSER
 
   BUF_PARSER --> SIZE_FLAGS

Sources: src/cli/cli_parser.rs:5-26 src/cli/help_template.rs:4-39 src/cli.rs:1-11 src/utils/parse_buffer_size.rs:35-57

The Cli Struct

The Cli struct is the primary container for argument parsing. It uses clap derive macros to pull metadata like name, version, and description directly from the Cargo.toml environment variables at build time src/cli/cli_parser.rs:5-12

Key components of the Cli struct:

• storage : A PathBuf representing the target storage file. The help text explicitly notes that if the file does not exist, it will be created automatically src/cli/cli_parser.rs:17-22
• command : An instance of the Commands enum, representing the specific action to take src/cli/cli_parser.rs:24-25

Sources: src/cli/cli_parser.rs:5-26

Commands Enum and Sub-commands

The Commands enum defines the available operations. Each variant represents a sub-command and its specific arguments. The implementation in execute_command.rs maps these variants to DataStore methods.

Sources: src/cli/help_template.rs:4-39 src/cli/cli_parser.rs:24-25 src/cli/commands.rs:4-5

Help System and Templates

The system uses a dynamic help template to provide users with practical examples. The HELP_TEMPLATE is a raw string defined using the indoc crate to maintain formatting and readability in the terminal src/cli/help_template.rs:1-39

During parser initialization, the string %BINARY_NAME% within the template is replaced with the actual package name (obtained via env!("CARGO_PKG_NAME")) using the after_help attribute src/cli/cli_parser.rs:13-15 This ensures that the help text remains accurate regardless of the build environment or binary naming.

Sources: src/cli/help_template.rs:4-39 src/cli/cli_parser.rs:13-15

Utility: Buffer Size Parsing

For commands like read that support custom buffer sizes, the CLI utilizes parse_buffer_size in src/utils/parse_buffer_size.rs.

This utility supports case-insensitive suffixes:

• K/KB : Kilobytes ($1024^1$ bytes) src/utils/parse_buffer_size.rs10 src/utils/parse_buffer_size.rs47
• M/MB : Megabytes ($1024^2$ bytes) src/utils/parse_buffer_size.rs11 src/utils/parse_buffer_size.rs48
• G/GB : Gigabytes ($1024^3$ bytes) src/utils/parse_buffer_size.rs12 src/utils/parse_buffer_size.rs49

The function trims the input string, identifies the numeric portion, and applies the appropriate multiplier src/utils/parse_buffer_size.rs:36-57

Sources: src/utils/parse_buffer_size.rs:1-57

Execution Logic

The main function (entry point) initializes the CLI by calling Cli::parse() before delegating the parsed structure to execute_command src/cli/execute_command.rs:10-11

Command Execution Flow

Module Organization

The CLI logic is modularized under the cli module:

• cli_parser.rs: Defines the Cli struct and clap configuration src/cli/cli_parser.rs:1-26
• commands.rs: Defines the Commands enum variants src/cli.rs:4-5
• help_template.rs: Contains the static example text src/cli/help_template.rs:4-39
• execute_command.rs: Contains the bridge logic between CLI arguments and DataStore API calls src/cli.rs:10-11

Sources: src/cli.rs:1-11 src/cli/cli_parser.rs:1-26 src/cli/help_template.rs:1-39

Network Layer: WebSocket RPC (Experiments)

Loading…

Network Layer: WebSocket RPC (Experiments)

Relevant source files

• Cargo.lock
• experiments/bindings/python-ws-client/Cargo.toml
• experiments/simd-r-drive-muxio-service-definition/Cargo.toml
• experiments/simd-r-drive-ws-client/Cargo.toml
• experiments/simd-r-drive-ws-server/Cargo.toml

The network layer in simd-r-drive is an experimental, high-performance RPC system designed to expose the core storage engine over WebSockets. It leverages the muxio framework to provide multiplexed I/O, allowing multiple concurrent requests and responses over a single TCP connection without head-of-line blocking at the application level.

System Architecture

The architecture consists of a server that wraps a DataStore instance and a client that implements the standard simd-r-drive traits, making the network boundary transparent to the application logic. Communication is governed by a shared service definition that uses bitcode for efficient serialization.

Network Entity Mapping

This diagram bridges the natural language concepts of the network layer to the specific code entities and crates that implement them.

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:1-21 experiments/simd-r-drive-ws-server/Cargo.toml:1-22 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:1-16 experiments/bindings/python-ws-client/Cargo.toml:1-21

graph TD
    subgraph "Client_Space"["Client Space"]
        "WsClient"["WsClient [experiments/simd-r-drive-ws-client]"]
        "AsyncTraits"["AsyncDataStoreReader / AsyncDataStoreWriter [simd-r-drive]"]
        "PyWsClient"["simd_r_drive_ws_client [experiments/bindings/python-ws-client]"]
end

    subgraph "Service_Definition_Shared"["Service Definition (Shared)"]
        "ServiceDef"["simd-r-drive-muxio-service-definition [experiments/simd-r-drive-muxio-service-definition]"]
        "Bitcode"["bitcode [serialization]"]
end

    subgraph "Server_Space"["Server Space"]
        "WsServer"["simd-r-drive-ws-server [experiments/simd-r-drive-ws-server]"]
        "MuxioServer"["muxio-tokio-rpc-server"]
        "DataStore"["DataStore [simd-r-drive]"]
end

    "WsClient" -- "implements" --> "AsyncTraits"
    "WsClient" -- "uses" --> "ServiceDef"
    "PyWsClient" -- "wraps" --> "WsClient"
    "WsServer" -- "registers" --> "ServiceDef"
    "WsServer" -- "wraps" --> "DataStore"
    "WsClient" -- "WebSocket_muxio" --> "MuxioServer"

WebSocket Server

The simd-r-drive-ws-server crate provides a standalone binary that hosts a DataStore over a WebSocket endpoint. It uses tokio for the asynchronous runtime and integrates with muxio-tokio-rpc-server to handle incoming RPC calls.

The server registers several RPC endpoints corresponding to the DataStore API:

• Write Operations: write, batch_write, delete.
• Read Operations: read, batch_read, exists.
• Metadata Operations: len, is_empty, file_size.

For details on server implementation, transport configuration, and CLI arguments, see WebSocket Server.

Sources: experiments/simd-r-drive-ws-server/Cargo.toml:13-22

WebSocket Client and Service Definition

The client-side implementation is split between a shared service definition and a concrete client wrapper.

Service Definition

The simd-r-drive-muxio-service-definition crate defines the “contract” between the client and server. It uses the bitcode crate for high-speed serialization of keys and payloads, ensuring minimal overhead during network transit. It defines the structure of requests and responses that both the client and server must adhere to.

WsClient

The WsClient struct in simd-r-drive-ws-client implements the AsyncDataStoreReader and AsyncDataStoreWriter traits. This allows it to be used as a drop-in replacement for a local DataStore in asynchronous contexts, abstracting the network calls into standard trait methods.

For details on the multiplexed transport and serialization format, see WebSocket Client and Service Definition.

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:13-21 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:14-15

Request Flow Overview

The following diagram illustrates the flow of an RPC request from the client to the underlying storage engine on the server.

Sources: experiments/simd-r-drive-ws-client/Cargo.toml:13-21 experiments/simd-r-drive-ws-server/Cargo.toml:13-22 experiments/simd-r-drive-muxio-service-definition/Cargo.toml:14-15

sequenceDiagram
    participant C as "WsClient (simd-r-drive-ws-client)"
    participant M as "muxio-tokio-rpc-client"
    participant S as "simd-r-drive-ws-server"
    participant DS as "DataStore (simd-r-drive)"

    C->>C: Encode Request (bitcode)
    C->>M: send_request(METHOD_ID, payload)
    M->>S: WebSocket Frame (Multiplexed Stream)
    S->>S: Route to Method Handler
    S->>DS: Perform Storage Operation
    DS-->>S: Result (EntryHandle / Success)
    S->>S: Encode Response (bitcode)
    S-->>M: WebSocket Frame
    M-->>C: Response Bytes
    C->>C: Decode Response

Sub-pages

• WebSocket Server : Deep dive into how simd-r-drive-ws-server wraps DataStore in Arc<RwLock> and registers RPC endpoints (write, batch_write, read, batch_read, delete, len, is_empty, file_size, exists) via muxio-tokio-rpc-server. Covers the axum/tokio-tungstenite transport and CLI args.
• WebSocket Client and Service Definition : Details on how simd-r-drive-ws-client implements AsyncDataStoreReader/AsyncDataStoreWriter via WsClient, using muxio-tokio-rpc-client for multiplexed WebSocket transport. Covers the prebuffered service definition (bitcode serialization, METHOD_ID constants).

WebSocket Server

Loading…

WebSocket Server

Relevant source files

• experiments/simd-r-drive-ws-server/README.md
• experiments/simd-r-drive-ws-server/src/cli/cli_parser.rs
• experiments/simd-r-drive-ws-server/src/cli/help_template.rs
• experiments/simd-r-drive-ws-server/src/main.rs
• tests/compaction_tests.rs

The simd-r-drive-ws-server is an experimental RPC wrapper around the core DataStore engine. It enables remote access to the append-only storage via a multiplexed WebSocket transport, leveraging the muxio framework for high-concurrency request/response handling.

Overview and Architecture

The server acts as a bridge between the physical storage on disk and remote clients. It encapsulates a DataStore instance within an Arc<RwLock<DataStore>> to allow safe, concurrent access across multiple asynchronous WebSocket connections handled by tokio experiments/simd-r-drive-ws-server/src/main.rs:31-39

Request Lifecycle

1. Transport : The server initializes an RpcServer which manages the underlying WebSocket transport experiments/simd-r-drive-ws-server/src/main.rs:42-43
2. Multiplexing : The muxio protocol allows multiple concurrent RPC calls over a single WebSocket connection without head-of-line blocking.
3. Dispatch : The RpcServiceEndpointInterface receives a frame, identifies the METHOD_ID, and dispatches it to the registered prebuffered handler experiments/simd-r-drive-ws-server/src/main.rs:55-57
4. Execution : Handlers use task::spawn_blocking to move heavy I/O and locking operations off the async executor experiments/simd-r-drive-ws-server/src/main.rs:60-67 They acquire a read or write lock on the DataStore (e.g., blocking_write or blocking_read) and execute the corresponding operation experiments/simd-r-drive-ws-server/src/main.rs:62-63
5. Response : Results are serialized using bitcode (via the service definition) and sent back through the multiplexer experiments/simd-r-drive-ws-server/src/main.rs:64-65

Data Flow Diagram

The following diagram illustrates how the server wraps the core library and exposes it over the network.

WebSocket Server Data Flow

graph TD
    subgraph "NetworkLayer"
        "RemoteClient" -- "WebSocket (muxio)" --> "RpcServer[muxio-tokio-rpc-server]"
    end

    subgraph "ServerProcess(simd-r-drive-ws-server)"
        "RpcServer" -- "Dispatch" --> "Endpoint[RpcServiceEndpointInterface]"
        "Endpoint" -- "spawn_blocking" --> "Handlers[RPC Handlers]"
        "Handlers" -- "blocking_write()" --> "SharedStore[Arc<RwLock<DataStore>>]"
        "Handlers" -- "blocking_read()" --> "SharedStore"
        
        subgraph "CoreEngine"
            "SharedStore" -- "I/O" --> "DataStore[simd_r_drive::DataStore]"
        end
    end

    subgraph "Storage"
        "DataStore" -- "Append/Mmap" --> "StorageFile[.bin]"
    end

Sources: experiments/simd-r-drive-ws-server/src/main.rs:36-67 experiments/simd-r-drive-ws-server/src/cli/cli_parser.rs:16-42

RPC Endpoints and Service Registration

The server registers a suite of endpoints that mirror the DataStoreReader and DataStoreWriter traits experiments/simd-r-drive-ws-server/src/main.rs:55-163 These are defined in the simd-r-drive-muxio-service-definition crate to ensure type safety between the server and the client.

Code Entity Mapping

Sources: experiments/simd-r-drive-ws-server/src/main.rs:14-19 experiments/simd-r-drive-ws-server/src/main.rs:55-163 experiments/simd-r-drive-ws-server/src/main.rs:172-223

CLI Configuration

The server is started via a CLI that configures the storage path and network binding. The CLI is built using clap experiments/simd-r-drive-ws-server/src/cli/cli_parser.rs:8-16

Arguments and Flags

• storage : The positional argument specifying the path to the storage file. If the file does not exist, the server initializes a new one experiments/simd-r-drive-ws-server/src/cli/cli_parser.rs:17-22
• --host : The IP address to bind to. Defaults to 127.0.0.1 experiments/simd-r-drive-ws-server/src/cli/cli_parser.rs:25-31
• --port : The TCP port for the WebSocket server. If set to 0 (default), the OS assigns a random free port experiments/simd-r-drive-ws-server/src/cli/cli_parser.rs:33-41

Error Handling

The CLI includes a custom error handler in Cli::parse_args() that detects MissingRequiredArgument errors and displays a detailed help template instead of a generic error message experiments/simd-r-drive-ws-server/src/cli/cli_parser.rs:45-61

Sources: experiments/simd-r-drive-ws-server/src/cli/cli_parser.rs:1-62 experiments/simd-r-drive-ws-server/src/cli/help_template.rs:4-14

Implementation Details

Concurrency Model

The server uses a multi-layered concurrency approach:

1. Outer Lock : An Arc<RwLock<DataStore>> protects the DataStore instance across multiple tokio tasks experiments/simd-r-drive-ws-server/src/main.rs:36-39
2. Blocking Tasks : Because DataStore operations involve synchronous file I/O and CPU-intensive SIMD/hashing, handlers use task::spawn_blocking to prevent blocking the tokio worker threads experiments/simd-r-drive-ws-server/src/main.rs:60-67
3. Read/Write Separation : Handlers use blocking_read() for operations like read, len, and exists to allow parallel read access, while using blocking_write() for write and delete to ensure exclusive access during appends experiments/simd-r-drive-ws-server/src/main.rs62 experiments/simd-r-drive-ws-server/src/main.rs103

Serialization

Request and response parameters are serialized using bitcode. The server decodes incoming requests (e.g., Write::decode_request(&bytes)) and encodes outgoing responses (e.g., Write::encode_response(...)) using methods provided by the RpcMethodPrebuffered trait implementation experiments/simd-r-drive-ws-server/src/main.rs:61-65

Sources: experiments/simd-r-drive-ws-server/src/main.rs:31-40 experiments/simd-r-drive-ws-server/src/main.rs:55-115

WebSocket Client and Service Definition

Loading…

WebSocket Client and Service Definition

Relevant source files

• experiments/bindings/python-ws-client/src/base_ws_client_py.rs
• experiments/bindings/python-ws-client/tests/integraton/test_client.py
• experiments/simd-r-drive-muxio-service-definition/README.md
• experiments/simd-r-drive-muxio-service-definition/src/prebuffered.rs
• experiments/simd-r-drive-muxio-service-definition/src/prebuffered/batch_read.rs
• experiments/simd-r-drive-muxio-service-definition/src/prebuffered/batch_write.rs
• experiments/simd-r-drive-muxio-service-definition/src/prebuffered/write.rs
• experiments/simd-r-drive-ws-client/README.md
• experiments/simd-r-drive-ws-client/src/ws_client.rs

The WebSocket implementation in simd-r-drive provides a networked interface to the core storage engine. It leverages the muxio framework for multiplexed RPC over WebSockets, allowing multiple concurrent requests and responses over a single connection. This layer consists of a shared service definition, a high-performance server, and an asynchronous client that implements the standard data store traits.

Service Definition and Serialization

The communication between client and server is governed by a “prebuffered” service definition. This approach uses the bitcode serialization format for high-efficiency encoding of request and response parameters experiments/simd-r-drive-muxio-service-definition/src/prebuffered.rs:1-26

Each RPC method is defined as a struct implementing RpcMethodPrebuffered, which associates a unique METHOD_ID with specific input and output types via the rpc_method_id! macro experiments/simd-r-drive-muxio-service-definition/src/prebuffered/write.rs:15-19

RPC Method Mapping

Sources: experiments/simd-r-drive-ws-client/src/ws_client.rs:7-12 experiments/simd-r-drive-muxio-service-definition/src/prebuffered.rs:1-26 experiments/simd-r-drive-muxio-service-definition/src/prebuffered/write.rs:5-22 experiments/simd-r-drive-muxio-service-definition/src/prebuffered/batch_read.rs:5-22 experiments/simd-r-drive-muxio-service-definition/src/prebuffered/batch_write.rs:5-22

WsClient Implementation

The WsClient struct in the simd-r-drive-ws-client crate acts as the primary gateway for remote storage operations. It wraps an Arc<RpcClient> from the muxio-tokio-rpc-client crate experiments/simd-r-drive-ws-client/src/ws_client.rs:20-22

Async Trait Integration

WsClient implements the AsyncDataStoreReader and AsyncDataStoreWriter traits, allowing it to be used interchangeably with other asynchronous storage backends experiments/simd-r-drive-ws-client/src/ws_client.rs:42-127 experiments/simd-r-drive-ws-client/src/ws_client.rs:130-213

Key implementation details:

• Data Handling : Because WebSocket transport requires copying data into buffers for serialization, the EntryHandleType for WsClient is defined as Vec<u8> rather than a zero-copy memory map handle experiments/simd-r-drive-ws-client/src/ws_client.rs132
• Multiplexing : Requests are dispatched via METHOD::call (e.g., Write::call), which allows the muxio runtime to handle request IDs and response matching experiments/simd-r-drive-ws-client/src/ws_client.rs:56-63
• Error Handling : RPC-specific errors are converted to standard std::io::Error via rpc_error_to_io to maintain trait compatibility experiments/simd-r-drive-ws-client/src/ws_client.rs:16-18

Connection Management

The client supports monitoring the underlying WebSocket connection state through set_state_change_handler, which accepts a callback receiving RpcTransportState updates experiments/simd-r-drive-ws-client/src/ws_client.rs:33-38 This is utilized by higher-level bindings to track connection health, such as the Python BaseDataStoreWsClient which updates an AtomicBool on disconnect experiments/bindings/python-ws-client/src/base_ws_client_py.rs:48-53

Data Flow: Client to Server

The following diagram illustrates how a write call is transformed from a trait method into a serialized RPC frame.

Logic to Code Entity Map: Client Request Path

graph TD
    subgraph "ApplicationLayer"
        User["User Code"] -- "ws_client.write(key, payload)" --> WsClient["WsClient (ws_client.rs)"]
end

    subgraph "TraitImplementation"
        WsClient -- "implements" --> ADSW["AsyncDataStoreWriter (traits.rs)"]
WsClient -- "constructs" --> WRP["WriteRequestParams (prebuffered/write.rs)"]
end

    subgraph "ServiceDefinition"
        WRP -- "passed to" --> WriteCall["Write::call (RpcCallPrebuffered)"]
WriteCall -- "uses" --> Encode["Write::encode_request (bitcode)"]
end

    subgraph "TransportLayer"
        Encode -- "binary payload" --> RpcClient["RpcClient (muxio-tokio-rpc-client)"]
RpcClient -- "WebSocket Frame" --> Network["TCP/IP"]
end

Sources: experiments/simd-r-drive-ws-client/src/ws_client.rs:55-67 experiments/simd-r-drive-ws-client/src/ws_client.rs:1-12 experiments/simd-r-drive-ws-client/src/ws_client.rs:20-29 experiments/simd-r-drive-muxio-service-definition/src/prebuffered/write.rs:24-26

Server-Side Execution

The server registers handlers for each METHOD_ID defined in the service definition. When a message arrives, the server decodes the request, performs the storage operation, and encodes the response back to the client.

Request Dispatching and Threading

To support high concurrency, the server typically utilizes tokio::task::spawn_blocking for storage operations. This prevents long-running disk I/O or heavy SIMD computations from blocking the asynchronous executor’s reactor threads. The DataStore is typically wrapped in a tokio::sync::RwLock to allow multiple concurrent readers.

Logic to Code Entity Map: Server Execution Path

graph LR
    subgraph "Network"
        WS["WebSocket Stream"]
end

    subgraph "RPCDispatch"
        Endpoint["RpcServiceEndpoint (muxio-tokio-rpc-server)"] -- "matches METHOD_ID" --> Handler["Registered Closure"]
end

    subgraph "ExecutionContext"
        Handler -- "spawn_blocking" --> Task["Tokio Blocking Task"]
Task -- "decode" --> Decode["Write::decode_request"]
end

    subgraph "StorageEngine"
        Decode -- "params" --> Lock["store.blocking_write()"]
Lock -- "access" --> DS["DataStore::write (storage_engine)"]
end

 
   WS --> Endpoint
    DS -- "tail_offset" --> Task
    Task -- "encode" --> WS

Sources: experiments/simd-r-drive-muxio-service-definition/src/prebuffered/write.rs:28-32 experiments/simd-r-drive-ws-client/src/ws_client.rs:56-64

Summary of Data Flow

The system maintains a strict separation between the transport (WebSocket), the protocol (muxio RPC), and the storage logic (DataStore).

Sources: experiments/simd-r-drive-ws-client/src/ws_client.rs:1-130 experiments/bindings/python-ws-client/src/base_ws_client_py.rs:21-26 experiments/simd-r-drive-muxio-service-definition/src/prebuffered/batch_read.rs:1-44

Python Bindings

Loading…

Python Bindings

Relevant source files

• experiments/bindings/python-ws-client/Cargo.lock
• experiments/bindings/python_(old_client)/pyproject.toml/pyproject.toml)

This section provides an overview of the Python integration strategies for SIMD R Drive. The project offers two distinct approaches for interacting with the storage engine from Python: a high-performance WebSocket-based client for networked environments and a legacy direct binding for local embedded use cases. Both implementations utilize PyO3 and maturin to bridge Rust performance with Python ergonomics.

Binding Architecture Overview

The following diagram illustrates how the Python bindings interface with the core Rust components, bridging the gap between Python user-space and the underlying Rust storage logic.

Python Binding System Context

graph TD
    subgraph "Python_User_Space"
        [simd_r_drive_ws_client] --> [DataStoreWsClient_Python]
        [simd-r-drive-py] --> [DataStore_Python]
    end

    subgraph "Rust_Binding_Layer_PyO3"
        [DataStoreWsClient_Python] -- "wraps" --> [WsClient_Rust]
        [DataStore_Python] -- "wraps" --> [DataStore_Rust]
    end

    subgraph "Rust_Backend_Crates"
        [WsClient_Rust] -- "RPC_via_muxio" --> [simd-r-drive-ws-server]
        [simd-r-drive-ws-server] -- "calls" --> [DataStore_Rust]
        [DataStore_Rust] -- "io" --> [Disk_Storage]
    end

Sources: experiments/bindings/python_(old_client)/pyproject.toml1-4 experiments/bindings/python-ws-client/Cargo.lock:133-143

Python WebSocket Client Binding

The simd_r_drive_ws_client is the primary, modern binding. It acts as a thin PyO3 wrapper around the Rust WsClient, enabling Python applications to communicate with a simd-r-drive-ws-server instance over WebSockets.

• API Surface : Exposes the DataStoreWsClient class, which provides asynchronous access to the storage engine. It supports standard operations like write, read, and Pythonic metadata checks such as __contains__ and __len__.
• Tooling : The development environment uses uv for dependency management and virtual environment isolation.
• Testing : Includes an automated lifecycle script, integration_test.sh, which manages the end-to-end flow: spinning up a simd-r-drive-ws-server, building the Python wheel, and running pytest. It also features a unique extract_readme_tests.py script that converts documentation examples into executable test cases.

For details, see Python WebSocket Client Binding.

Sources: experiments/bindings/python-ws-client/Cargo.lock:133-143 experiments/bindings/python-ws-client/Cargo.lock1633

Python Direct Binding (Legacy)

The simd-r-drive-py package is a legacy implementation that binds directly to the DataStore core library. It allows Python to load the storage engine as a shared library without requiring a separate server process.

• Platform Support : Targets CPython 3.10 through 3.13 on Linux and macOS experiments/bindings/python_(old_client)/pyproject.toml20-25
• Status : This binding is considered experimental/alpha experiments/bindings/python_(old_client)/pyproject.toml3-11
• Build System : Configured via pyproject.toml using the maturin build backend experiments/bindings/python_(old_client)/pyproject.toml28-30

For details, see Python Direct Binding (Legacy)).

Sources: experiments/bindings/python_(old_client)/pyproject.toml1-34

Comparison of Approaches

Binding Logic Flow

Sources: experiments/bindings/python_(old_client)/pyproject.toml32-34 experiments/bindings/python-ws-client/Cargo.lock1633

Python WebSocket Client Binding

Loading…

Python WebSocket Client Binding

Relevant source files

• experiments/bindings/python-ws-client/README.md
• experiments/bindings/python-ws-client/extract_readme_tests.py
• experiments/bindings/python-ws-client/integration_test.sh
• experiments/bindings/python-ws-client/pyproject.toml
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/init.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi
• experiments/bindings/python-ws-client/uv.lock
• src/storage_engine/key_indexer.rs

The simd_r_drive_ws_client Python package provides a high-performance bridge to the Rust WsClient experiments/bindings/python-ws-client/README.md:7-15 it allows Python applications to interact with a simd-r-drive-ws-server using a familiar, dictionary-like API while leveraging the multiplexed I/O and SIMD-optimized serialization of the underlying Rust implementation.

Architecture and Data Flow

The binding is implemented using PyO3 and maturin experiments/bindings/python-ws-client/README.md:14-15 It wraps a Rust WsClient and manages network operations transparently for the Python caller.

Component Hierarchy

The following diagram illustrates how Python calls are dispatched through the Rust shim to the WebSocket transport.

Client Dispatch Architecture

graph TD
    subgraph "Python_Space"
        PY_APP["Python Application"]
DS_WS_CLIENT["DataStoreWsClient (Python)"]
end

    subgraph "Rust_Shim_(PyO3_Module)"
        BASE_WS_PY["BaseDataStoreWsClient (Rust Struct)"]
TOKIO_RT["Tokio Runtime"]
end

    subgraph "Rust_Core_(simd-r-drive-ws-client)"
        WS_CLIENT["WsClient"]
TRAITS["AsyncDataStoreReader / AsyncDataStoreWriter"]
end

 
   PY_APP -->|calls| DS_WS_CLIENT
 
   DS_WS_CLIENT -->|inherits| BASE_WS_PY
 
   BASE_WS_PY -->|executes_via| TOKIO_RT
 
   TOKIO_RT -->|manages| WS_CLIENT
 
   WS_CLIENT -->|implements| TRAITS

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py11 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi8 experiments/bindings/python-ws-client/README.md:14-15

Data Flow Mapping

This diagram maps Python method calls to their corresponding internal definitions and logic.

Method Binding Map

graph LR
    subgraph "Python_API_(data_store_ws_client.pyi)"
        P_WRITE["write(key, data)"]
P_READ["read(key)"]
P_LEN["__len__()"]
P_CONTAINS["__contains__(key)"]
end

    subgraph "PyO3_Rust_Shim_(BaseDataStoreWsClient)"
        R_WRITE["py_write"]
R_READ["py_read"]
R_LEN["py_len"]
R_EXISTS["py_exists"]
end

    subgraph "Python_Logic_(data_store_ws_client.py)"
        P_STRUCT["batch_read_structured()"]
end

 
   P_WRITE --> R_WRITE
 
   P_READ --> R_READ
 
   P_LEN --> R_LEN
 
   P_CONTAINS --> R_EXISTS
 
   P_STRUCT --> R_READ

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:27-150 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:11-62

Core API: DataStoreWsClient

The primary interface is the DataStoreWsClient class, which inherits from BaseDataStoreWsClient experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py11

Key Methods and Signatures

Structured Batch Reads

The Python implementation includes batch_read_structured, which accepts a dictionary or list of dictionaries where values are keys to be fetched experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:12-31

1. Decompile: Flattens all dictionary values into a single keys_to_fetch list while mapping original keys experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:35-46
2. Fetch: Executes one high-performance batch_read call experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py49
3. Reconstruct: Rebuilds the original nested structure with the fetched results experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:51-62

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:11-62

Key Utilities: NamespaceHasher

To prevent collisions across different logical domains, the NamespaceHasher utility generates 16-byte deterministic keys using XXH3 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:171-186

• Format: [8 bytes Namespace Hash] || [8 bytes Key Hash] experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:201-216
• Initialization: The prefix is hashed once upon instantiation to serve as a unique identifier experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:188-199

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:171-216

Test Environment and Lifecycle

The project uses uv for environment management and an integrated shell script for end-to-end testing.

Integration Test Lifecycle (integration_test.sh)

The integration_test.sh script automates the full stack testing experiments/bindings/python-ws-client/integration_test.sh:1-5:

1. Server Startup: Starts simd-r-drive-ws-server in the background via cargo run with temporary storage experiments/bindings/python-ws-client/integration_test.sh:50-56
2. Environment Setup: Uses uv venv and uv pip install to prepare the Python environment with pytest and maturin experiments/bindings/python-ws-client/integration_test.sh:70-78
3. Doc-Test Extraction: Runs extract_readme_tests.py to convert README examples into executable pytest functions experiments/bindings/python-ws-client/integration_test.sh:79-80
4. Execution: Runs pytest against the live server using exported TEST_SERVER_HOST and TEST_SERVER_PORT experiments/bindings/python-ws-client/integration_test.sh:82-87
5. Cleanup: Uses a trap to kill the server process group and remove temporary storage files experiments/bindings/python-ws-client/integration_test.sh:17-34

README Test Extraction

The extract_readme_tests.py script uses regular expressions to find python fenced code blocks in README.md experiments/bindings/python-ws-client/extract_readme_tests.py:21-24 It wraps these snippets into isolated test_readme_block_{i} functions and writes them to tests/test_readme_blocks.py experiments/bindings/python-ws-client/extract_readme_tests.py:30-42

Sources: experiments/bindings/python-ws-client/integration_test.sh:1-90 experiments/bindings/python-ws-client/extract_readme_tests.py:1-45

Development and Installation

The package is built with maturin and targets Python 3.10+ experiments/bindings/python-ws-client/README.md:14-21

Build Command:

experiments/bindings/python-ws-client/README.md:34-35

Sources: experiments/bindings/python-ws-client/README.md:1-59 experiments/bindings/python-ws-client/pyproject.toml:1-46 experiments/bindings/python-ws-client/uv.lock:1-120

Python Direct Binding (Legacy)

Loading…

Python Direct Binding (Legacy)

Relevant source files

• experiments/bindings/python_(old_client)/Cargo.toml/Cargo.toml)
• experiments/bindings/python_(old_client)/README.md/README.md)
• experiments/bindings/python_(old_client)/pyproject.toml/pyproject.toml)

The simd-r-drive-py package represents the legacy approach to providing Python access to the DataStore engine. Unlike the modern WebSocket-based client, this binding uses PyO3 and maturin to expose the Rust DataStore struct directly as a native Python extension module. It is currently considered experimental/alpha and has been largely superseded by the network-based client due to threading and platform compatibility constraints experiments/bindings/python_(old_client)/README.md3-5

Purpose and Scope

The primary goal of the direct binding is to provide the absolute lowest latency possible by eliminating network overhead and using memory-mapped files directly within the Python process. It allows Python users to interact with the append-only storage engine as if it were a native Python object, supporting zero-copy reads via Python’s memoryview interface experiments/bindings/python_(old_client)/README.md7

Status and Limitations

• Status: Experimental / Alpha experiments/bindings/python_(old_client)/pyproject.toml3 experiments/bindings/python_(old_client)/pyproject.toml11
• Thread Safety: This binding is not thread-safe experiments/bindings/python_(old_client)/README.md198 Concurrent streaming writes or reads from multiple threads may cause hangs or inconsistent behavior due to the interaction between the Python Global Interpreter Lock (GIL) and Rust’s internal locking experiments/bindings/python_(old_client)/README.md198-201
• Platform Support: Officially supports Linux and macOS (CPython 3.10–3.13) experiments/bindings/python_(old_client)/README.md23-32 Windows is not officially supported for the direct binding due to memory-mapping inconsistencies in the Python runtime experiments/bindings/python_(old_client)/README.md38-39

Sources: experiments/bindings/python_(old_client)/README.md3-39 experiments/bindings/python_(old_client)/pyproject.toml3-11

System Architecture

The direct binding bridges the Python interpreter to the Rust DataStore using a thin wrapper layer defined in the module entry point. The Rust simd-r-drive dependency is compiled with the expose-internal-api feature to allow the binding access to internal structures like EntryHandle experiments/bindings/python_(old_client)/Cargo.toml15-16

Data Flow: Python to Rust Engine

The following diagram illustrates how a Python call traverses the binding layer to reach the core storage engine.

Binding Execution Path

graph TD
    subgraph "Python_Space"
        ["Python_Application"] --> ["DataStore_Python_Class"]
end

    subgraph "PyO3_Native_Extension"
        ["DataStore_Python_Class"] -- "PyBytes_to_u8_slice" --> ["Rust_Wrapper_impl_DataStore"]
        ["EntryHandle_Python"] -- "as_memoryview" --> ["memoryview_Object"]
end

    subgraph "Core_Engine_simd_r_drive"
        ["Rust_Wrapper_impl_DataStore"] -- "DataStore::write" --> ["DataStore_Rust_Struct"]
        ["DataStore_Rust_Struct"] -- "Index_Lookup" --> ["KeyIndexer"]
        ["DataStore_Rust_Struct"] -- "Mmap_Access" --> ["Mmap_Shared_Memory"]
end

    ["Mmap_Shared_Memory"] -- "Direct_Slice" --> ["memoryview_Object"]

Sources: experiments/bindings/python_(old_client)/README.md164-187 experiments/bindings/python_(old_client)/Cargo.toml15-16

Implementation Details

Build System and Metadata

The project uses maturin as the build backend, targeting the pyo3 binding type experiments/bindings/python_(old_client)/pyproject.toml28-33 It specifies a minimum Python version of 3.8 but officially targets 3.10 through 3.13 for CPython experiments/bindings/python_(old_client)/pyproject.toml20-34 The crate type is set to cdylib to produce a shared library compatible with Python’s import system experiments/bindings/python_(old_client)/Cargo.toml8-10

Key Classes and Functions

Sources: experiments/bindings/python_(old_client)/README.md160-194 experiments/bindings/python_(old_client)/Cargo.toml8-16 experiments/bindings/python_(old_client)/pyproject.toml28-34

Zero-Copy Integration

One of the core features of the direct binding is the EntryHandle.as_memoryview() method. This allows Python libraries to consume data directly from the Rust-managed mmap without copying bytes into the Python heap experiments/bindings/python_(old_client)/README.md7-182

Code Entity Association: Zero-Copy Read

Sources: experiments/bindings/python_(old_client)/README.md180-182 experiments/bindings/python_(old_client)/README.md7

Usage and Lifecycle Management

Resource Cleanup

Because PyO3 does not guarantee deterministic destruction of Rust-backed objects, the underlying Rust Drop implementation handles the cleanup of file handles and memory maps. However, the Python garbage collector may delay this, potentially causing issues when trying to delete or move the underlying file on disk while the process is running, especially on Windows experiments/bindings/python_(old_client)/README.md38-39

Example: Streaming Large Payloads

The legacy binding supports streaming from Python’s io.BytesIO or other file-like objects directly into the store using the write_stream method experiments/bindings/python_(old_client)/README.md172-174

Sources: experiments/bindings/python_(old_client)/README.md136-156 experiments/bindings/python_(old_client)/README.md184-186

Supported Platforms Table

Sources: experiments/bindings/python_(old_client)/README.md23-45

Testing and CI/CD

Loading…

Testing and CI/CD

Relevant source files

• .github/workflows/rust-tests.yml
• .gitignore
• tests/README.md

The rust-simd-r-drive project employs a multi-layered testing strategy and an extensive CI/CD pipeline to ensure the correctness of its append-only storage engine across different operating systems, hardware architectures, and feature flag configurations.

The test suite is designed to validate core storage invariants—such as CRC32C checksum integrity, payload alignment, and thread safety—while the CI/CD pipelines automate linting, cross-platform builds, documentation generation, and release workflows for both Rust and Python components.

Test Suite Organization

The testing infrastructure is primarily located in the tests/ directory of the core crate, supplemented by unit tests within the source files. The suite relies heavily on conditional compilation to test various performance optimizations.

Rust Integration Tests

Integration tests are organized by functional area to isolate behaviors like concurrency, persistence, and SIMD alignment. Many tests require specific Cargo features, such as expose-internal-api to inspect the underlying file structure or parallel to exercise Rayon-based iterators.

Test Execution Logic

Sources: tests/README.md:1-26 src/storage_engine/traits/reader.rs:4-21 src/storage_engine/traits/writer.rs:4-65

For a detailed breakdown of test files and feature-gated testing, see Rust Test Suite.

Performance Benchmarking

The project includes a benchmarking suite that is checked for compilation during every CI run via cargo bench --workspace --no-run .github/workflows/rust-tests.yml:65-66 These benchmarks cover:

• Storage Benchmarks : Sequential vs. random I/O performance.
• Contention Benchmarks : Measuring lock overhead in the DataStore.

Sources: .github/workflows/rust-tests.yml:64-66 .gitignore:1-12

CI/CD Pipelines

The project uses GitHub Actions to orchestrate a comprehensive suite of workflows. These pipelines ensure that every pull request and tag follows the project’s quality standards.

Pipeline Architecture

The following diagram illustrates how the CI/CD workflows interact with the codebase and external package registries:

CI/CD Workflow Mapping

graph TD
    subgraph "GitHub_Actions"
        RT["rust-tests.yml"]
RL["rust-lint.yml"]
BD["build-docs.yml"]
RR["rust-release.yml"]
PB["python-build.yml"]
PNR["python-net-release.yml"]
end

    subgraph "Codebase_Entities"
        WS["Cargo_Workspace"]
DS["DataStore_(src/storage_engine)"]
CLI["CLI_(src/main.rs)"]
PY["Python_Bindings_(bindings/python)"]
end

    subgraph "Registries"
        CR["crates.io"]
PYPI["PyPI"]
end

 
   RT --> WS
 
   RL --> WS
 
   BD --> DS
 
   RR --> CR
 
   PB --> PY
 
   PNR --> PYPI

Sources: .github/workflows/rust-tests.yml:1-7 .github/workflows/rust-tests.yml:15-35

Cross-Platform Matrix

The rust-tests.yml workflow utilizes a build matrix to guarantee compatibility across major operating systems and feature combinations. It runs on ubuntu-latest, macos-latest, and windows-latest .github/workflows/rust-tests.yml22

Feature Flag Test Matrix

Automation and Releases

Beyond testing, the CI/CD system handles:

• Concurrency Management : Automatically cancels older jobs when new commits are pushed .github/workflows/rust-tests.yml:9-13
• Caching : Speeds up builds by caching ~/.cargo/registry and target/ .github/workflows/rust-tests.yml:45-57
• Rust Toolchain : Uses dtolnay/rust-toolchain@stable for consistent build environments .github/workflows/rust-tests.yml:41-42
• Python Integration : Validating the WebSocket-based Python client against a live Rust server instance.
• Release Orchestration : Automated publishing to Crates.io and PyPI when version tags are pushed.

For details on specific workflow configurations and release triggers, see CI/CD Pipelines.

Sources: .github/workflows/rust-tests.yml:9-66

Rust Test Suite

Loading…

Rust Test Suite

Relevant source files

• experiments/bindings/python_(old_client)/LICENSE/LICENSE)
• src/main.rs
• src/storage_engine/traits/reader.rs
• src/storage_engine/traits/writer.rs
• src/utils/format_bytes.rs
• tests/alignment_tests.rs
• tests/basic_operations_tests.rs
• tests/batch_ops_tests.rs
• tests/concurrency_tests.rs
• tests/hash_stability_tests.rs
• tests/integrity_tests.rs
• tests/mmap_and_zero_copy_tests.rs
• tests/namespace_hasher_tests.rs
• tests/parallel_iterator_tests.rs
• tests/persistence_tests.rs
• tests/storage_operation_tests.rs
• tests/streaming_tests.rs

The Rust test suite for simd-r-drive is designed to validate the core storage engine’s guarantees regarding append-only integrity, zero-copy alignment, hash stability, and thread safety. It is organized into several specialized integration test files located in the tests/ directory, leveraging Cargo feature flags like parallel and expose-internal-api to exercise different execution paths.

Test Suite Organization

The integration tests are categorized by the specific sub-system or behavior they validate:

Data Flow: From Test to Storage Engine

The following diagram illustrates how tests interact with the DataStore and its underlying traits.

Test Execution Flow

graph TD
    subgraph "Test_Space"
        T1["basic_operations_tests.rs"]
T2["concurrency_tests.rs"]
T3["batch_ops_tests.rs"]
T4["storage_operation_tests.rs"]
end

    subgraph "Code_Entity_Space_Core"
        DS["DataStore"]
DSR["trait DataStoreReader"]
DSW["trait DataStoreWriter"]
KI["KeyIndexer"]
MM["Mmap (Arc)"]
end

 
   T1 -->|calls| DSR
 
   T2 -->|spawns_tasks| DSW
 
   T3 -->|vectorized| DSW
 
   T4 -->|cross_instance| DSW
    
 
   DSR -.->|implemented_by| DS
 
   DSW -.->|implemented_by| DS
    
 
   DS -->|updates| KI
 
   DS -->|reads_remaps| MM

Sources: src/storage_engine/traits/reader.rs:4-21 src/storage_engine/traits/writer.rs:4-29 tests/basic_operations_tests.rs:16-27 tests/concurrency_tests.rs:113-137

Alignment and SIMD Validation

A critical component of the test suite is alignment_tests.rs, which ensures that every payload is written to a 64-byte aligned boundary. This allows the storage engine to provide zero-copy views that are safe for SIMD instructions (AVX2/NEON).

• Alignment Proofs : Tests use bytemuck::try_cast_slice to prove that a &[u8] from the store can be cast to &[u32], &[u64], or &[u128] without copying tests/alignment_tests.rs:58-67
• SIMD Loadability : On x86_64, tests execute _mm_load_si128 on retrieved payloads to verify they do not trigger alignment faults tests/alignment_tests.rs:69-95 On aarch64, vld1q_u8 is used for the same purpose tests/alignment_tests.rs:97-122
• Interaction Testing : The suite mixes unaligned string writes (e.g., 3-byte or 7-byte) with aligned numeric writes to ensure the pre-pad calculation correctly maintains the PAYLOAD_ALIGNMENT tests/alignment_tests.rs:135-160

Alignment Verification Logic

Sources: tests/alignment_tests.rs:12-32 tests/alignment_tests.rs:69-75 tests/alignment_tests.rs:185-192 tests/alignment_tests.rs:97-103

Concurrency and Parallelism

The test suite validates thread safety using tokio and serial_test.

Concurrent Writers

concurrency_tests.rs spawns multiple tasks to perform simultaneous write operations. It verifies that the internal locking and tail offset management prevent data corruption tests/concurrency_tests.rs:113-142

Slow Streaming

The concurrent_slow_streamed_write_test uses a custom SlowReader to simulate network latency during a write_stream operation, ensuring that the store remains consistent even when appends are delayed tests/concurrency_tests.rs:16-35

Parallel Iterators

When the parallel feature is enabled, parallel_iterator_tests.rs verifies that DataStore can be iterated using Rayon, allowing for multi-threaded processing of the entire entry log.

• Deduplication : Tests ensure that parallel iteration respects the backward-linked chain and only returns the most recent version of a key.
• Tombstones : Tests validate that entries marked for deletion are omitted from the iterator stream.

Sources: tests/concurrency_tests.rs:1-12 tests/concurrency_tests.rs:113-121 tests/concurrency_tests.rs:16-35

Persistence and Recovery

persistence_tests.rs focuses on the “on-disk” contract of the engine.

1. Re-open Validation : Ensures that data written in one process session is visible after closing and re-opening the DataStore tests/persistence_tests.rs:13-55
2. Corruption Recovery : Simulates an interrupted write by manually appending “garbage” bytes to the end of the file. The test verifies that DataStore::open detects the invalid entry (via CRC32C or metadata mismatch) and truncates the file back to the last valid state tests/persistence_tests.rs:110-176 Note that these tests are skipped on Windows due to file locking restrictions with memory maps tests/persistence_tests.rs:146-160

Sources: tests/persistence_tests.rs:13-32 tests/persistence_tests.rs:126-141 tests/persistence_tests.rs:160-176

Batch and Streaming Operations

The suite rigorously tests vectorized operations and memory-efficient streaming.

Batch Operations

batch_ops_tests.rs validates the atomicity and ordering of batch_write and batch_read.

• Atomicity : If any payload in a batch is invalid (e.g., empty or null-byte), the entire batch is rejected, and no data is persisted tests/batch_ops_tests.rs:104-127
• Hashed Reads : test_batch_read_hashed_keys_with_verification exercises the high-performance path where callers provide pre-computed hashes and original keys for collision verification tests/batch_ops_tests.rs:197-210

Streaming I/O

streaming_tests.rs validates write_stream and EntryStream.

• Chunked Reading : Tests verify that a 1MB payload can be written from a BufReader and read back in 4KB chunks using EntryStream, maintaining checksum integrity throughout tests/streaming_tests.rs:20-89

Sources: tests/batch_ops_tests.rs:104-127 tests/batch_ops_tests.rs:197-210 tests/streaming_tests.rs:20-89

Hash Stability

hash_stability_tests.rs enforces hardcoded xxh3 hash values to guard against silent regressions if the xxhash-rust dependency is updated or replaced.

• Single Key Stability : Validates hashes for empty strings, null bytes, and various sample keys like “alice” or “test_key” tests/hash_stability_tests.rs:16-48
• Batch Stability : Ensures compute_hash_batch matches individual hash results tests/hash_stability_tests.rs:58-65
• Namespace Stability : Locks down the 16-byte namespaced key derivation logic in NamespaceHasher tests/hash_stability_tests.rs:77-101

Sources: tests/hash_stability_tests.rs:1-108

Storage Operations: Copy and Transfer

storage_operation_tests.rs validates higher-level data movement between different storage instances.

• Copy Entry : Verifies that source_storage.copy(key, &target_storage) correctly replicates data and metadata (key hash, checksum) while keeping the original entry intact tests/storage_operation_tests.rs:20-74
• Transfer Entry : Validates that transfer (move) correctly writes to the target and appends a tombstone to the source tests/storage_operation_tests.rs:103-152
• Self-Copy Protection : Ensures that attempting to copy an entry to the same storage instance returns an error tests/storage_operation_tests.rs:77-100

Sources: tests/storage_operation_tests.rs:20-152

Feature Flags in Testing

The test suite is executed across a matrix of feature flags in CI to ensure compatibility:

• default : Standard configuration.
• parallel : Enables Rayon integration and parallel iterators.
• expose-internal-api : Allows tests to access low-level metadata and internal state for deeper verification.
• arrow : Enables Apache Arrow buffer integration.

CI Matrix Configuration

Sources: src/storage_engine/traits/reader.rs:1-158 src/storage_engine/traits/writer.rs:1-152

CI/CD Pipelines

Loading…

CI/CD Pipelines

Relevant source files

• .cargo/config.toml.example
• .github/dependabot.yml
• .github/workflows/build-docs.yml
• .github/workflows/python-direct-release.yml.TODO
• .github/workflows/python-net-release.yml
• .github/workflows/rust-lint.yml
• .github/workflows/rust-release.yml
• .github/workflows/rust-tests.yml
• .gitignore

This page describes the GitHub Actions workflows and automated pipelines that ensure the stability, security, and delivery of the rust-simd-r-drive ecosystem. The CI/CD infrastructure covers Rust core testing across multiple platforms, Python binding builds, security auditing, and automated releases to GitHub and PyPI.

Core Rust Workflows

The Rust CI infrastructure is split into validation (linting/security) and functional testing (matrix builds).

Rust Linting and Security

The rust-lint.yml workflow performs static analysis and security checks on every pull request [ .github/workflows/rust-lint.yml3](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-lint.yml#L3-L3) It ensures that the codebase adheres to formatting standards, passes Clippy’s suggestions, and is free from known vulnerabilities in dependencies.

Sources: [ .github/workflows/rust-lint.yml1-50](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-lint.yml#L1-L50)

Multi-Platform Testing Matrix

The rust-tests.yml workflow executes the test suite across a matrix of Operating Systems and Cargo feature flags to ensure compatibility and performance stability. It includes a specific check to ensure benchmarks compile via cargo bench --workspace --no-run [ .github/workflows/rust-tests.yml65-66](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-tests.yml#L65-L66)

Data Flow: Test Matrix Execution The diagram below illustrates how the matrix strategy expands into individual job executions.

Sources: [ .github/workflows/rust-tests.yml1-67](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-tests.yml#L1-L67)

graph TD
    subgraph "Matrix_Strategy"
        OS["OS: [ubuntu-latest, macos-latest, windows-latest]"]
FEAT["Features: [Default, No-Default, Parallel, Expose-API, Parallel+Expose, All]"]
end

 
   OS --> JOB["Job: Test (OS, Features)"]
FEAT --> JOB

 
   JOB --> CHECKOUT["actions/checkout@v4"]
CHECKOUT --> RUST["dtolnay/rust-toolchain@stable"]
RUST --> CACHE["actions/cache@v4 (Cargo deps & target/)"]
CACHE --> BUILD["cargo build --workspace --all-targets"]
BUILD --> TEST["cargo test --workspace --all-targets --verbose"]
TEST --> BENCH["cargo bench --workspace --no-run"]
style JOB stroke-width:2px

Python and Network Pipelines

The project maintains specialized pipelines for building and testing Python bindings, specifically the WebSocket-based client.

Python WebSocket Integration

The python-net-release.yml manages the lifecycle of integration tests where a Rust WebSocket server is spawned to serve a Python client.

1. Environment Setup : Uses uv to manage Python dependencies and setup-python for the interpreter [ .github/workflows/python-net-release.yml31-39](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/python-net-release.yml#L31-L39)
2. Test Execution : Runs integration_test.sh, which handles starting the simd-r-drive-ws-server, running Python tests, and tearing down the server [ .github/workflows/python-net-release.yml42-45](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/python-net-release.yml#L42-L45)

Python Release Pipeline

The python-net-release.yml workflow builds platform-specific wheels using cibuildwheel and publishes them to PyPI. A similar TODO workflow exists for the direct (legacy) bindings [ .github/workflows/python-direct-release.yml.TODO1-104](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/python-direct-release.yml.TODO#L1-L104)

• Wheel Building : Uses cibuildwheel (version 2.23.3) to compile the Rust extensions for modern Python versions [ .github/workflows/python-net-release.yml48-56](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/python-net-release.yml#L48-L56)
• Targeting : Specifically skips older Python versions and 32-bit architectures (e.g., cp36, manylinux_i686) to focus on modern 64-bit systems [ .github/workflows/python-net-release.yml54](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/python-net-release.yml#L54-L54)
• Publishing Logic : Distinguishes between test releases (tags containing -test) and production releases to target either test.pypi.org or upload.pypi.org [ .github/workflows/python-net-release.yml88-101](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/python-net-release.yml#L88-L101)

Sources: [ .github/workflows/python-net-release.yml1-102](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/python-net-release.yml#L1-L102) [ .github/workflows/python-direct-release.yml.TODO1-104](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/python-direct-release.yml.TODO#L1-L104)

Automated Release and Documentation

Rust Binary Releases

The rust-release.yml workflow automates the creation of GitHub Releases. It compiles the simd-r-drive CLI in --release mode for Linux, macOS, and Windows [ .github/workflows/rust-release.yml16-26](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-release.yml#L16-L26) It renames the binaries for clarity (e.g., adding .exe for Windows) [ .github/workflows/rust-release.yml28-37](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-release.yml#L28-L37) and uploads them as release assets when a version tag (e.g., v*) is pushed [ .github/workflows/rust-release.yml49-64](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-release.yml#L49-L64)

Sources: [ .github/workflows/rust-release.yml1-65](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-release.yml#L1-L65)

Documentation Deployment

The build-docs.yml workflow generates this wiki as a searchable mdBook and deploys it to GitHub Pages. It runs on a weekly schedule or manually via workflow_dispatch [ .github/workflows/build-docs.yml4-7](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/build-docs.yml#L4-L7)

Entity Mapping: Documentation Pipeline This diagram maps the documentation build process to the tools and outputs used in the workflow.

Sources: [ .github/workflows/build-docs.yml1-81](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/build-docs.yml#L1-L81)

graph LR
    subgraph "Input_Space"
        REPO["GitHub Repository Content"]
CRON["Schedule: Weekly (Sunday)"]
end

    subgraph "Code_Entity_Space"
        DW["jzombie/deepwiki-to-mdbook@main"]
MD["mdBook Engine"]
OUT["./output/book"]
end

 
   REPO --> DW
 
   CRON --> DW
 
   DW --> MD
 
   MD --> OUT
 
   OUT --> PAGES["GitHub Pages Deployment"]
style DW stroke-dasharray: 5 5

Dependency Management

The project uses GitHub Dependabot to maintain up-to-date and secure dependencies.

• Ecosystem : Monitored for the cargo package manager [ .github/dependabot.yml8](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/dependabot.yml#L8-L8)
• Schedule : Checks for updates on a weekly basis [ .github/dependabot.yml11](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/dependabot.yml#L11-L11)
• Configuration : Scans the root directory for Cargo.toml manifests [ .github/dependabot.yml9](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/dependabot.yml#L9-L9)

Sources: [ .github/dependabot.yml1-12](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/dependabot.yml#L1-L12)

Build Environment Configuration

The repository includes a .cargo/config.toml.example which is used to redirect crate dependencies to local paths during development, particularly for the rust-muxio suite [ .cargo/config.toml.example1-6](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .cargo/config.toml.example#L1-L6) CI environments typically rely on the standard crates.io versions unless these overrides are active.

Code Entity Mapping: CI Validation Flow The CI environment relies on specific cargo commands to validate the workspace. The following diagram shows how the CI jobs map to the actual cargo commands executed in the environment.

graph TD
    subgraph "Natural_Language_Space"
        LINT["Linting & Security"]
TESTING["Multi-Platform Tests"]
end

    subgraph "Code_Entity_Space"
        CLIPPY["cargo clippy --workspace"]
AUDIT["cargo audit"]
DENY["cargo deny check"]
T_WORK["cargo test --workspace"]
B_WORK["cargo bench --no-run"]
end

 
   LINT --> CLIPPY
 
   LINT --> AUDIT
 
   LINT --> DENY
 
   TESTING --> T_WORK
 
   TESTING --> B_WORK

    style CLIPPY stroke-width:2px
    style T_WORK stroke-width:2px

Sources: [ .github/workflows/rust-lint.yml37-49](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-lint.yml#L37-L49) [ .github/workflows/rust-tests.yml59-66](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .github/workflows/rust-tests.yml#L59-L66) [ .cargo/config.toml.example1-6](https://github.com/jzombie/rust-simd-r-drive/blob/5652ca4d/ .cargo/config.toml.example#L1-L6)

Glossary

Loading…

Glossary

Relevant source files

• CHANGELOG.md
• Cargo.lock
• README.md
• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/entry_metadata.rs
• src/lib.rs
• src/storage_engine.rs
• src/storage_engine/data_store.rs
• src/storage_engine/entry_iterator.rs

This page provides definitions for codebase-specific terms, abbreviations, and domain concepts used throughout the SIMD R Drive project. It serves as a technical reference for developers to understand the relationship between high-level concepts and their implementation in the Rust source code.

Core Concepts

Data Structures and Implementation

Entry Layout

The storage engine uses a specific binary layout for every entry to ensure alignment and integrity. Non-tombstone entries are padded to ensure the payload starts on a PAYLOAD_ALIGNMENT boundary.

Entry Layout Diagram

graph TD
    subgraph "On-Disk Entry Structure"
        [A] -- "Pre-Pad (Alignment)" --> [B]
        [B] -- "Payload (Data)" --> [C]
        [C] -- "EntryMetadata" --> [D]
    end

    subgraph "Code Entities"
        [B] -- "Accessed via" --> [EH_SLICE]
        [C] -- "Defined by" --> [EM_STRUCT]
    end

    [EH_SLICE] -- "EntryHandle::as_slice()" --> [SRC_EH]
    [EM_STRUCT] -- "EntryMetadata struct" --> [SRC_EM]

    [SRC_EH] -- "[simd-r-drive-entry-handle/src/entry_handle.rs:151-155]()" --> [FIN]
    [SRC_EM] -- "[simd-r-drive-entry-handle/src/entry_metadata.rs:46-50]()" --> [FIN]

Sources: README.md:112-120 simd-r-drive-entry-handle/src/entry_metadata.rs:11-30 src/storage_engine/entry_iterator.rs:80-125

Key Indexing and Hashing

The system utilizes XXH3 hashing for keys and a custom indexing mechanism to locate data efficiently while maintaining a small memory footprint.

• Xxh3BuildHasher : A hasher implementation using the XXH3 algorithm for high-performance key hashing. src/storage_engine/digest.rs:3-5
• NamespaceHasher : A utility to generate 16-byte namespaced keys by hashing a namespace prefix combined with a key. src/storage_engine/digest.rs8
• Tag+Offset Scheme : The KeyIndexer manages the mapping between key hashes and file locations using a packed 64-bit value containing a 16-bit tag and 48-bit offset. src/storage_engine/key_indexer.rs:15-20

Sources: src/storage_engine/key_indexer.rs:15-30 src/storage_engine/digest.rs:1-8

Extension Concepts

The simd-r-drive-extensions crate introduces higher-level abstractions built on top of the raw DataStore.

TTL (Time-To-Live)

A mechanism to automatically expire entries after a certain duration.

• Implementation : An 8-byte Little-Endian timestamp is prefixed to the serialized data. extensions/src/cache.rs:10-15
• Auto-Eviction : Expired keys are logically removed from the underlying DataStore during a read operation if the current time exceeds the stored timestamp. extensions/src/cache.rs:45-50
• Trait : StorageCacheExt adds write_with_ttl() and read_with_ttl() to DataStore. extensions/src/cache.rs:5-10

Storage Options

A way to explicitly store None values, distinguishing them from keys that simply do not exist in the index (which return NotFound).

• Tombstone Marker : Uses a specific 2-byte marker [0xFF, 0xFE] (aliased as OPTION_TOMBSTONE_MARKER) to represent None. extensions/src/utils/option_serializer.rs:1-8
• Serialization : Values are serialized using the bitcode crate, making these operations non-zero-copy as they require deserialization. CHANGELOG.md:78-80
• Trait : StorageOptionExt adds write_option() and read_option(). extensions/src/option.rs:5-10

Sources: extensions/src/utils/option_serializer.rs:24-63 CHANGELOG.md:78-80

Technical Terms Reference

System Data Flow

The following diagram bridges the natural language concept of “Reading a Key” to the specific code entities involved in the process within the DataStore implementation.

DataStore Read Flow

sequenceDiagram
    participant User as "User Code"
    participant DS as "DataStore::read()"
    participant KI as "KeyIndexer (RwLock)"
    participant MM as "Mmap (Arc<Mmap>)"
    participant EH as "EntryHandle"

    User->>DS: read(key)
    DS->>KI: lookup hash in index
    KI-->>DS: offset
    DS->>MM: Access memory at offset
    DS->>EH: EntryHandle::from_arc_mmap()
    EH-->>User: EntryHandle (Zero-Copy)

Sources: src/storage_engine/data_store.rs:27-33 src/storage_engine/traits.rs:1-5 src/storage_engine/entry_iterator.rs:121-125 simd-r-drive-entry-handle/src/entry_handle.rs:45-50