Core Storage Engine

Loading…

Core Storage Engine

Relevant source files

• src/lib.rs
• src/storage_engine/data_store.rs

The core storage engine of rust-simd-r-drive is an append-only, high-performance data store designed for zero-copy reads and SIMD-optimized writes. It is implemented primarily within the src/storage_engine/ directory and revolves around the DataStore struct src/storage_engine/data_store.rs:27-33

The engine utilizes memory-mapped files via the memmap2 crate to provide high-speed access to data while maintaining a simple, immutable-on-disk structure src/lib.rs:7-17 By treating the file as a continuous log, it avoids the complexities of in-place updates and fragmentation, relying on a reverse-traversal iterator and a background compaction process to manage data lifecycle and reclaim space src/storage_engine/data_store.rs:35-51

System Architecture

The storage engine is composed of several interlocking components that manage data from initial write through recovery and eventual compaction.

High-Level Component Interaction

The following diagram illustrates how the core entities interact during standard operations.

Storage Engine Component Map

graph TD
    subgraph "Public API [src/traits.rs]"
        DR["DataStoreReader"]
DW["DataStoreWriter"]
end
    
    subgraph "Core Logic [src/storage_engine/data_store.rs]"
        DS["DataStore Struct"]
DS -.-> |implements| DR
 
       DS -.-> |implements| DW
    end

    subgraph "Indexing & Search"
        KI["KeyIndexer [src/storage_engine/key_indexer.rs]"]
EH["EntryHandle [simd-r-drive-entry-handle]"]
end

    subgraph "Disk & Memory"
        MM["Mmap [memmap2]"]
BW["BufWriter [std::io]"]
FILE["On-Disk .bin File"]
end

 
   DS --> ["Arc<RwLock<BufWriter<File>>>"]
DS --> ["Arc<Mutex<Arc<Mmap>>>"]
DS --> ["AtomicU64 tail_offset"]
DS --> ["Arc<RwLock<KeyIndexer>>"]
    
    ["Arc<RwLock<BufWriter<File>>>"] -->
 FILE
    ["Arc<Mutex<Arc<Mmap>>>"] --> FILE
 
   DR --> EH
 
   EH -.-> |points into| MM

Sources: src/storage_engine/data_store.rs:27-33 src/storage_engine/data_store.rs7 src/storage_engine/key_indexer.rs:15-25

Key Components

DataStore

The DataStore struct is the primary handle for interacting with the storage engine src/storage_engine/data_store.rs:27-33 It manages the file handles, memory maps, and the in-memory key index. It implements the DataStoreReader and DataStoreWriter traits to provide a standardized API for operations like write(), read(), and batch_write() src/storage_engine/data_store.rs:66-117

• For details, see DataStore: Read and Write Operations.

On-Disk Format

Data is stored as a sequence of entries. Each entry consists of an optional pre-pad for alignment, a variable-length payload, and a fixed-size EntryMetadata footer simd-r-drive-entry-handle/src/lib.rs10 To ensure SIMD efficiency, payloads are aligned to 64-byte boundaries (PAYLOAD_ALIGNMENT) src/storage_engine/constants.rs:3-4 The file also maintains a backward-linked validation chain via prev_offset fields in the metadata simd-r-drive-entry-handle/src/lib.rs10

• For details, see On-Disk Storage Format.

Key Indexer and Hashing

To avoid scanning the entire file for every read, the engine maintains an in-memory KeyIndexer src/storage_engine/data_store.rs:107-114 This index maps 64-bit XXH3 hashes of keys to their most recent file offsets src/storage_engine/data_store.rs:2-4 It allows for O(1) lookup performance by keeping the mapping of the latest version of every key in RAM.

• For details, see Key Indexer and Hashing.

Entry Iterator and Compaction

Since the store is append-only, updates to a key result in a new entry being written at the end of the file. The EntryIterator traverses the file in reverse (newest to oldest) using the prev_offset pointers to ensure only the latest version of a key is retrieved src/storage_engine/entry_iterator.rs:24-34 Compaction reclaims space by using this iterator to identify and rewrite only “live” entries into a fresh file src/storage_engine/data_store.rs:555-562

• For details, see Entry Iterator and Compaction.

Concurrency and Thread Safety

The engine supports concurrent access through a layered locking strategy. It uses RwLock<BufWriter<File>> for serialized appends, RwLock<KeyIndexer> for index updates, and Mutex<Arc<Mmap>> to safely manage memory-map remapping during file growth src/storage_engine/data_store.rs:28-31 An AtomicU64 tracks the tail_offset to allow progress checks without acquiring heavy locks src/storage_engine/data_store.rs113

• For details, see Concurrency and Thread Safety.

sequenceDiagram
    participant User
    participant DS as "DataStore::write()"
    participant SC as "simd_copy()"
    participant KI as "KeyIndexer::insert()"
    participant MM as "remap_and_index()"

    User->>DS: key, value
    DS->>DS: compute_hash(key)
    DS->>SC: copy value to BufWriter
    DS->>DS: EntryMetadata::serialize()
    DS->>KI: update index with offset
    DS->>MM: refresh memory map and KeyIndexer
    DS-->>User: Success

Data Flow: Write vs Read

The following diagram bridges the gap between the high-level logic and the specific code entities involved in the data lifecycle.

Logic to Entity Mapping

Sources: src/storage_engine/data_store.rs:176-195 src/storage_engine/data_store.rs:2-5 src/storage_engine/data_store.rs:320-335

Child Pages

Sources: src/lib.rs:1-137 src/storage_engine/data_store.rs:1-117 src/storage_engine/constants.rs:1-7