Arrow Integration

Loading…

Arrow Integration

Relevant source files

• CHANGELOG.md
• simd-r-drive-entry-handle/Cargo.toml
• simd-r-drive-entry-handle/src/constants.rs
• simd-r-drive-entry-handle/src/entry_handle.rs
• simd-r-drive-entry-handle/src/entry_metadata.rs
• simd-r-drive-entry-handle/src/lib.rs
• src/storage_engine.rs
• src/storage_engine/entry_iterator.rs

The simd-r-drive-entry-handle crate provides optional integration with Apache Arrow via the arrow Cargo feature simd-r-drive-entry-handle/Cargo.toml15 This integration allows users to treat memory-mapped payloads as arrow_buffer::Buffer objects without copying data, enabling high-performance analytical processing directly on the stored bytes CHANGELOG.md:133-135

Overview of Arrow Buffers

Apache Arrow uses a specific Buffer type to represent contiguous regions of memory. By leveraging the fact that EntryHandle already manages an Arc<Mmap> simd-r-drive-entry-handle/src/entry_handle.rs12 the system can wrap this memory in an Arrow-compatible container. This is particularly effective because the storage engine defaults to a PAYLOAD_ALIGNMENT of 64 bytes simd-r-drive-entry-handle/src/constants.rs18 which satisfies Arrow’s requirement for SIMD-aligned data access CHANGELOG.md:93-97

Key Capabilities

• Zero-Copy Conversion : Transform an EntryHandle into an arrow_buffer::Buffer by sharing the underlying Arc<Mmap> simd-r-drive-entry-handle/src/entry_handle.rs:129-139
• Reference Counting : The Arrow Buffer increments the reference count of the Arc<Mmap>, ensuring the memory mapping remains valid for the lifetime of the Buffer simd-r-drive-entry-handle/src/entry_handle.rs:173-176
• Alignment Validation : Integrated checks in as_arrow_buffer and into_arrow_buffer ensure that the exported buffer meets both pointer and offset alignment requirements when compiled in debug or test mode CHANGELOG.md:106-108

Sources: simd-r-drive-entry-handle/Cargo.toml15 simd-r-drive-entry-handle/src/constants.rs18 CHANGELOG.md:93-97 CHANGELOG.md:106-108 simd-r-drive-entry-handle/src/entry_handle.rs:10-19

Data Flow: From Storage to Arrow

The following diagram illustrates how data flows from the on-disk memory-mapped file into an Arrow Buffer through the EntryHandle abstraction.

graph TD
    subgraph "Disk_Storage"
        [DataStore_File]
    end

    subgraph "simd-r-drive-entry-handle"
        [Arc_Mmap]
        [EntryHandle]
        [as_slice]
    end

    subgraph "Apache_Arrow_Ecosystem"
        [arrow_buffer_Buffer]
        [arrow_array_PrimitiveArray]
    end

    [DataStore_File] -- "mmap" --> [Arc_Mmap]
    [Arc_Mmap] -- "shared_reference" --> [EntryHandle]
    [EntryHandle] -- "into_arrow_buffer" --> [arrow_buffer_Buffer]
    [arrow_buffer_Buffer] -- "Zero-Copy_View" --> [arrow_array_PrimitiveArray]
    [as_slice] -- "Deref" --> [EntryHandle]

Logic Flow: Zero-Copy Arrow View

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:12-19 simd-r-drive-entry-handle/src/entry_handle.rs:129-139 simd-r-drive-entry-handle/src/entry_handle.rs:151-155

Implementation Details

The integration is implemented by extending EntryHandle with methods that interface with the arrow crate.

Function: as_arrow_buffer()

This method creates a new Arrow Buffer that points to the same memory range as the EntryHandle. It utilizes the shared Arc<Mmap> to ensure no additional allocations occur. Because it clones the Arc, the operation is $O(1)$ and involves no data movement simd-r-drive-entry-handle/src/entry_handle.rs:159-162

Function: into_arrow_buffer()

Similar to as_arrow_buffer(), but consumes the EntryHandle. This is the preferred method when the handle is no longer needed, as it transfers ownership of the underlying Arc<Mmap> directly to the Arrow Buffer CHANGELOG.md:133-135

Alignment and Safety

Arrow requires buffers to be aligned to specific boundaries (typically 64 bytes) for optimal SIMD performance. The storage engine enforces this via PAYLOAD_ALIGNMENT simd-r-drive-entry-handle/src/constants.rs18 When payloads are written, a pre-pad is calculated to ensure the payload start address is a multiple of the alignment simd-r-drive-entry-metadata.rs:22-24

Sources: simd-r-drive-entry-handle/src/constants.rs18 simd-r-drive-entry-handle/src/entry_metadata.rs:22-24 simd-r-drive-entry-handle/src/entry_handle.rs:10-19

Entity Mapping: Code to Concept

This diagram maps the high-level Arrow concepts to the specific structs and functions used in the rust-simd-r-drive implementation.

classDiagram
    class EntryHandle {+Arc~Mmap~ mmap_arc\n+Range~usize~ range\n+EntryMetadata metadata\n+as_slice()\n+as_arrow_buffer()\n+into_arrow_buffer()}

    class EntryMetadata {+u64 key_hash\n+u64 prev_offset\n+u8_4 checksum}

    class ArrowBufferIntegration {<<Interface>>\n+as_arrow_buffer()\n+into_arrow_buffer()}

    EntryHandle --> EntryMetadata : Contains
    EntryHandle --> ArrowBufferIntegration : Enables (via arrow feature)
    ArrowBufferIntegration ..> "arrow::buffer::Buffer" : Produces

Concept to Entity Map

Sources: simd-r-drive-entry-handle/src/entry_handle.rs:10-19 simd-r-drive-entry-handle/Cargo.toml15 CHANGELOG.md:133-135

Usage Example

To use this feature, the arrow feature must be enabled in the simd-r-drive-entry-handle crate simd-r-drive-entry-handle/Cargo.toml15 Once enabled, the EntryHandle can be converted:

1. Retrieve Handle : Obtain an EntryHandle from the DataStore or an EntryIterator src/storage_engine/entry_iterator.rs:121-125
2. Convert : Call handle.into_arrow_buffer().
3. Wrap : Use the resulting Buffer to create an arrow_array::RecordBatch or PrimitiveArray.

This workflow is critical for applications that perform heavy computation on stored data, as it bypasses the standard overhead of serialization and deserialization CHANGELOG.md:93-97

Sources: simd-r-drive-entry-handle/Cargo.toml15 CHANGELOG.md:133-135 simd-r-drive-entry-handle/src/entry_handle.rs:129-139 src/storage_engine/entry_iterator.rs:121-125