Entry Handle Crate

Loading…

Entry Handle Crate

Relevant source files

• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/constants.rs
• simd-r-drive-entry-handle/src/entry_handle.rs
• simd-r-drive-entry-handle/src/entry_metadata.rs
• simd-r-drive-entry-handle/src/lib.rs
• src/storage_engine.rs
• src/storage_engine/entry_iterator.rs

The simd-r-drive-entry-handle crate provides the core data types and zero-copy access mechanisms used across the SIMD R Drive ecosystem. By decoupling these types from the main storage engine, the project allows extensions, network clients, and external tools to interact with entry data using a shared, stable ABI without depending on the full storage engine logic simd-r-drive-entry-handle/Cargo.toml:1-5

Purpose and Role

This crate serves as the “Common Type Layer.” Its primary goal is to provide a storage-agnostic way to handle entries that may be backed by memory-mapped files, anonymous memory, or network buffers simd-r-drive-entry-handle/src/entry_handle.rs:7-19 It allows different components to share data via the EntryHandle struct without re-allocating or copying the underlying payload simd-r-drive-entry-handle/src/entry_handle.rs:7-19

The following diagram illustrates how EntryHandle acts as the bridge between raw storage and high-level consumers:

Data Type Integration Flow

graph TD
    subgraph "Storage_Layer"
        [DataStore] -- "returns" --> [EntryHandle]
        [MmapMut] -- "from_owned_bytes_anon" --> [EntryHandle]
        [EntryIterator] -- "next_returns" --> [EntryHandle]
    end

    subgraph "simd-r-drive-entry-handle"
        [EntryHandle]
        [EntryMetadata]
        [EntryHandle] -- "contains" --> [EntryMetadata]
    end

    subgraph "Consumer_Layer"
        [EntryHandle] -- "as_slice" --> [Standard_Rust_Slice]
        [EntryHandle] -- "as_arrow_buffer" --> [Apache_Arrow_Buffer]
        [EntryHandle] -- "Deref" --> [Byte_Operations]
    end

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:10-19 simd-r-drive-entry-handle/src/entry_handle.rs:87-113 simd-r-drive-entry-handle/Cargo.toml:13-15 src/storage_engine/entry_iterator.rs:121-125

Core Entities

The crate is built around two primary structures that define how data is represented in memory and on disk.

EntryMetadata Layout

The EntryMetadata struct mirrors the binary layout used at the end of every entry in the storage file simd-r-drive-entry-handle/src/entry_metadata.rs:9-31 It provides serialize() and deserialize() methods to convert between the struct and the raw byte format required by the append-only log simd-r-drive-entry-handle/src/entry_metadata.rs:75-112

The layout ensures that payloads can be recovered by following the prev_offset chain, which forms a backward-linked sequence for each key simd-r-drive-entry-handle/src/entry_metadata.rs:41-43

Sources: simd-r-drive-entry-handle/src/entry_metadata.rs:44-50 simd-r-drive-entry-handle/src/entry_metadata.rs:75-83 simd-r-drive-entry-handle/src/entry_metadata.rs:101-112

Zero-Copy Architecture

The EntryHandle is designed to prevent unnecessary allocations. It achieves this by wrapping a sub-slice of an Arc<Mmap> simd-r-drive-entry-handle/src/entry_handle.rs:7-19

• File-Backed: When reading from DataStore or via an EntryIterator, the handle points directly to the memory-mapped file on disk simd-r-drive-entry-handle/src/entry_handle.rs:129-139 src/storage_engine/entry_iterator.rs:121-125
• In-Memory: Using from_owned_bytes_anon(), the crate can create handles from standard byte slices by copying them once into an anonymous memory map, thereafter providing the same zero-copy &[u8] access simd-r-drive-entry-handle/src/entry_handle.rs:87-113
• Trait Implementations: It implements Deref<Target = [u8]> simd-r-drive-entry-handle/src/entry_handle.rs:36-42 and various PartialEq variants simd-r-drive-entry-handle/src/entry_handle.rs:45-63 to allow seamless comparison with slices and vectors.

EntryHandle Internal Structure

classDiagram
    class EntryHandle {
        +Arc~Mmap~ mmap_arc
        +Range~usize~ range
        +EntryMetadata metadata
        +as_slice() &[u8]
        +clone_arc() EntryHandle
        +from_arc_mmap() EntryHandle
        +is_valid_checksum() bool
    }
    class EntryMetadata {
        +u64 key_hash
        +u64 prev_offset
        +u8_4 checksum
        +serialize() [u8; 20]
        +deserialize(data) EntryMetadata
    }
    EntryHandle *-- EntryMetadata : contains

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:10-19 simd-r-drive-entry-handle/src/entry_metadata.rs:46-50 simd-r-drive-entry-handle/src/entry_handle.rs:179-185

Sub-Pages

EntryHandle: Zero-Copy Data Access

Detailed API documentation for EntryHandle. This page covers how to instantiate handles via from_arc_mmap() or from_owned_bytes_anon(), how to verify integrity with is_valid_checksum(), and the use of Deref to treat handles as byte slices. It also explains the expose-internal-api feature gate which grants access to internal fields for specialized extensions simd-r-drive-entry-handle/Cargo.toml14 For details, see EntryHandle: Zero-Copy Data Access.

Arrow Integration

Explains the optional arrow feature in Cargo.toml simd-r-drive-entry-handle/Cargo.toml15 When enabled, EntryHandle gains the ability to export its underlying memory-mapped payload directly into an Apache Arrow Buffer. This allows SIMD-optimized analytical engines to process SIMD R Drive data without any copying or serialization overhead. For details, see Arrow Integration.

Sources: simd-r-drive-entry-handle/Cargo.toml:13-15 simd-r-drive-entry-handle/src/entry_handle.rs:151-155