Overview

Loading…

Overview

Relevant source files

• CHANGELOG.md
• Cargo.toml
• README.md

SIMD R Drive is a high-performance, thread-safe storage engine designed for zero-copy binary access within a single-file container. It is optimized for SIMD-accelerated workloads and cache-friendly data processing, providing a schema-less, append-only architecture that bridges the gap between raw filesystem performance and structured key-value access README.md:5-9

The system is designed to handle datasets larger than available RAM by leveraging memory mapping (mmap), allowing transparent access to specific file segments while minimizing memory pressure README.md:43-50

Key Design Goals

The architecture of SIMD R Drive is centered around four primary pillars:

• Append-Only Design : High write throughput is achieved by using sequential, append-based writes, which minimizes disk seek overhead and ensures data integrity through a backward-linked validation chain README.md:98-103
• Zero-Copy Access : By memory-mapping the storage file, the engine provides direct access to data payloads via the EntryHandle. This allows applications to use data without the overhead of deserialization or intermediate buffering README.md:43-50 simd-r-drive-entry-handle/src/lib.rs:1-10
• SIMD-Optimized : Payloads are written at fixed 64-byte aligned boundaries (matching typical CPU cachelines). This ensures that SIMD/vector loads (AVX, NEON, etc.) can operate at maximum hardware speed without crossing cacheline boundaries README.md:51-60 CHANGELOG.md:92-97
• Schema-less Flexibility : The engine treats payloads as raw bytes (&[u8]). It makes no assumptions about data format, endianness, or structure, allowing it to store anything from simple strings to nested storage containers or executable binaries README.md:61-88

System Architecture and Code Entities

The following diagram illustrates how the natural language concepts of the storage engine map to specific code entities and traits within the Rust implementation.

Storage Engine Logic Mapping

graph TD
    subgraph "Natural Language Space"
        "Append-Only_Storage"["Append-Only Storage"]
        "Zero-Copy_Handles"["Zero-Copy Handles"]
        "Key_Indexing"["Key Indexing"]
        "Memory_Mapping"["Memory Mapping"]
end

    subgraph "Code Entity Space (simd-r-drive)"
        "Append-Only_Storage" --> "DataStore"["DataStore struct"]
        "DataStore" --> "DataStoreWriter"["DataStoreWriter trait"]
        "DataStore" --> "DataStoreReader"["DataStoreReader trait"]
        
        "Zero-Copy_Handles" --> "EntryHandle"["EntryHandle struct"]
        
        "Key_Indexing" --> "KeyIndexer"["KeyIndexer struct"]
        "KeyIndexer" --> "Xxh3BuildHasher"["Xxh3BuildHasher"]
        
        "Memory_Mapping" --> "Mmap"["memmap2::Mmap"]
        "Mmap" --> "EntryHandle"
    end

    subgraph "Files"
        "DataStore" --- "src_storage_engine_mod"["src/storage_engine/mod.rs"]
        "EntryHandle" --- "entry_handle_src"["simd-r-drive-entry-handle/src/lib.rs"]
        "KeyIndexer" --- "indexer_src"["src/storage_engine/index/key_indexer.rs"]
end

Sources: README.md:5-9 Cargo.toml:80-91 simd-r-drive-entry-handle/src/lib.rs:1-10

Workspace Layout

The project is organized as a Cargo workspace containing the core engine, shared data types, and experimental extensions.

Workspace Entity Relationships

Sources: Cargo.toml:14-27 Cargo.toml:60-63

Navigation

For detailed technical information, please refer to the following sub-sections:

• Getting Started : Instructions for building the project, understanding Cargo features (like parallel, arrow, and expose-internal-api), and using the CLI quick-start. For details, see Getting Started.
• Repository Layout : A comprehensive map of the directory structure, including the core library, entry-handle crate, extensions, experiments, and CI configurations. For details, see Repository Layout.

Sources: README.md:1-112 Cargo.toml:1-112 CHANGELOG.md:1-145