Glossary

Loading…

Glossary

Relevant source files

• CHANGELOG.md
• Cargo.lock
• README.md
• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/entry_metadata.rs
• src/lib.rs
• src/storage_engine.rs
• src/storage_engine/data_store.rs
• src/storage_engine/entry_iterator.rs

This page provides definitions for codebase-specific terms, abbreviations, and domain concepts used throughout the SIMD R Drive project. It serves as a technical reference for developers to understand the relationship between high-level concepts and their implementation in the Rust source code.

Core Concepts

Data Structures and Implementation

Entry Layout

The storage engine uses a specific binary layout for every entry to ensure alignment and integrity. Non-tombstone entries are padded to ensure the payload starts on a PAYLOAD_ALIGNMENT boundary.

Entry Layout Diagram

graph TD
    subgraph "On-Disk Entry Structure"
        [A] -- "Pre-Pad (Alignment)" --> [B]
        [B] -- "Payload (Data)" --> [C]
        [C] -- "EntryMetadata" --> [D]
    end

    subgraph "Code Entities"
        [B] -- "Accessed via" --> [EH_SLICE]
        [C] -- "Defined by" --> [EM_STRUCT]
    end

    [EH_SLICE] -- "EntryHandle::as_slice()" --> [SRC_EH]
    [EM_STRUCT] -- "EntryMetadata struct" --> [SRC_EM]

    [SRC_EH] -- "[simd-r-drive-entry-handle/src/entry_handle.rs:151-155]()" --> [FIN]
    [SRC_EM] -- "[simd-r-drive-entry-handle/src/entry_metadata.rs:46-50]()" --> [FIN]

Sources: README.md:112-120 simd-r-drive-entry-handle/src/entry_metadata.rs:11-30 src/storage_engine/entry_iterator.rs:80-125

Key Indexing and Hashing

The system utilizes XXH3 hashing for keys and a custom indexing mechanism to locate data efficiently while maintaining a small memory footprint.

• Xxh3BuildHasher : A hasher implementation using the XXH3 algorithm for high-performance key hashing. src/storage_engine/digest.rs:3-5
• NamespaceHasher : A utility to generate 16-byte namespaced keys by hashing a namespace prefix combined with a key. src/storage_engine/digest.rs8
• Tag+Offset Scheme : The KeyIndexer manages the mapping between key hashes and file locations using a packed 64-bit value containing a 16-bit tag and 48-bit offset. src/storage_engine/key_indexer.rs:15-20

Sources: src/storage_engine/key_indexer.rs:15-30 src/storage_engine/digest.rs:1-8

Extension Concepts

The simd-r-drive-extensions crate introduces higher-level abstractions built on top of the raw DataStore.

TTL (Time-To-Live)

A mechanism to automatically expire entries after a certain duration.

• Implementation : An 8-byte Little-Endian timestamp is prefixed to the serialized data. extensions/src/cache.rs:10-15
• Auto-Eviction : Expired keys are logically removed from the underlying DataStore during a read operation if the current time exceeds the stored timestamp. extensions/src/cache.rs:45-50
• Trait : StorageCacheExt adds write_with_ttl() and read_with_ttl() to DataStore. extensions/src/cache.rs:5-10

Storage Options

A way to explicitly store None values, distinguishing them from keys that simply do not exist in the index (which return NotFound).

• Tombstone Marker : Uses a specific 2-byte marker [0xFF, 0xFE] (aliased as OPTION_TOMBSTONE_MARKER) to represent None. extensions/src/utils/option_serializer.rs:1-8
• Serialization : Values are serialized using the bitcode crate, making these operations non-zero-copy as they require deserialization. CHANGELOG.md:78-80
• Trait : StorageOptionExt adds write_option() and read_option(). extensions/src/option.rs:5-10

Sources: extensions/src/utils/option_serializer.rs:24-63 CHANGELOG.md:78-80

Technical Terms Reference

System Data Flow

The following diagram bridges the natural language concept of “Reading a Key” to the specific code entities involved in the process within the DataStore implementation.

DataStore Read Flow

sequenceDiagram
    participant User as "User Code"
    participant DS as "DataStore::read()"
    participant KI as "KeyIndexer (RwLock)"
    participant MM as "Mmap (Arc<Mmap>)"
    participant EH as "EntryHandle"

    User->>DS: read(key)
    DS->>KI: lookup hash in index
    KI-->>DS: offset
    DS->>MM: Access memory at offset
    DS->>EH: EntryHandle::from_arc_mmap()
    EH-->>User: EntryHandle (Zero-Copy)

Sources: src/storage_engine/data_store.rs:27-33 src/storage_engine/traits.rs:1-5 src/storage_engine/entry_iterator.rs:121-125 simd-r-drive-entry-handle/src/entry_handle.rs:45-50