Extensions Crate

Loading…

Extensions Crate

Relevant source files

• extensions/Cargo.toml
• extensions/README.md
• extensions/src/lib.rs
• extensions/src/storage_file_import_ext.rs
• extensions/src/utils/option_serializer.rs
• src/storage_engine/simd_copy.rs
• src/utils.rs
• tests/align_or_copy_tests.rs

The simd-r-drive-extensions crate provides higher-level storage patterns and utilities built on top of the core DataStore extensions/README.md:5-6 These extensions are implemented as Rust traits that extend the functionality of any type implementing DataStoreReader and DataStoreWriter extensions/src/storage_file_import_ext.rs:1-5 extensions/src/lib.rs:7-16

While the core engine focuses on zero-copy binary blobs, the extensions crate introduces structured data handling, automated lifecycle management (TTL), and filesystem integration extensions/README.md:15-134

Extension Architecture

The extensions use a namespacing strategy to prevent key collisions between different types of extended storage and standard raw binary storage. This is achieved using the NamespaceHasher utility extensions/src/storage_file_import_ext.rs:4-5 src/utils.rs:10-11

Extension to Core Mapping

The following diagram illustrates how extension traits map high-level concepts to the underlying DataStore entities.

Diagram: Extension Trait Mapping

graph TD
    subgraph "Natural Language Space"
        "Time-To-Live Cache"["Time-To-Live Cache"] 
        "Explicit Nulls"["Explicit Nulls"]
        "Bulk Directory Import"["Bulk Directory Import"]
end

    subgraph "Code Entity Space"
        "Time-To-Live Cache" --> "StorageCacheExt"["StorageCacheExt"]
        "Explicit Nulls" --> "StorageOptionExt"["StorageOptionExt"]
        "Bulk Directory Import" --> "StorageFileImportExt"["StorageFileImportExt"]
        
        "StorageCacheExt" -- "uses" --> "TTL_PREFIX"["TTL_PREFIX"]
        "StorageOptionExt" -- "uses" --> "OPTION_PREFIX"["OPTION_PREFIX"]
        "StorageFileImportExt" -- "uses" --> "import_dir_recursively"["import_dir_recursively()"]
        
        "TTL_PREFIX" & "OPTION_PREFIX" & "import_dir_recursively" -- "calls" --> "DataStore_write"["DataStore::write() / write_stream()"]
end

Sources: extensions/src/lib.rs:7-16 extensions/README.md:104-106 extensions/src/storage_file_import_ext.rs:90-91

Key Extensions

StorageCacheExt: TTL-Based Caching

The StorageCacheExt trait adds write_with_ttl() and read_with_ttl() methods to the DataStore extensions/README.md:68-70

• Mechanism : It prepends an 8-byte Little-Endian expiration timestamp to the payload extensions/README.md:84-85
• Eviction : Expired entries are automatically evicted upon read to prevent stale data extensions/README.md87
• Serialization : Uses bitcode for non-zero-copy data handling extensions/README.md:85-86

For details, see StorageCacheExt: TTL-Based Caching.

StorageOptionExt: Explicit None Storage

The StorageOptionExt trait allows users to distinguish between a key that is missing (NotFound) and a key that is explicitly set to None extensions/README.md:17-45

• Tombstones : Uses a specific 2-byte marker OPTION_TOMBSTONE_MARKER ([0xFF, 0xFE]) to represent None in the physical storage extensions/src/utils/option_serializer.rs:1-8 extensions/src/utils/option_serializer.rs:27-28
• Namespacing : Operates within the OPTION_PREFIX namespace to isolate these markers from regular data extensions/src/lib.rs:7-8
• Implementation : The logic is encapsulated in serialize_option and deserialize_option extensions/src/utils/option_serializer.rs:24-63

For details, see StorageOptionExt: Explicit None Storage.

StorageFileImportExt: Filesystem Import

This extension provides utilities for syncing local filesystem directories into a DataStore extensions/README.md:89-106

• Streaming : Uses import_dir_recursively() to walk directory trees via walkdir and stream file contents directly into the store using write_stream(), minimizing memory overhead extensions/src/storage_file_import_ext.rs:77-91
• Retrieval : Provides open_file_stream() to read stored files back as an EntryStream extensions/src/storage_file_import_ext.rs:54-58
• Keys : Automatically converts filesystem paths to Unix-style relative keys extensions/src/storage_file_import_ext.rs:117-124

For details, see StorageFileImportExt: Filesystem Import.

Data Flow Overview

The following diagram shows the data flow from high-level extension calls down to the serialized format on disk.

Diagram: Extension Data Flow

sequenceDiagram
    participant App as "Application Code"
    participant Ext as "Extension Trait (e.g., StorageOptionExt)"
    participant Ser as "option_serializer"
    participant DS as "DataStore"

    App->>Ext: write_option(key, None)
    Ext->>Ser: serialize_option(None)
    Ser-->>Ext: [0xFF, 0xFE] (Tombstone)
    Ext->>DS: write(namespaced_key, [0xFF, 0xFE])
    DS-->>App: Result<u64> (Offset)

Sources: extensions/src/utils/option_serializer.rs:24-29 extensions/src/lib.rs:7-8

Summary Table

Sources: extensions/README.md:5-134 extensions/src/utils/option_serializer.rs:24-63 extensions/src/storage_file_import_ext.rs:12-59