This documentation is part of the "Projects with Books" initiative at zenOSmosis.

The source code for this project is available on GitHub.

StorageOptionExt: Explicit None Storage

Loading…

StorageOptionExt: Explicit None Storage

Relevant source files

The StorageOptionExt trait provides a high-level abstraction for storing and retrieving Rust Option<T> types within a DataStore extensions/src/storage_option_ext.rs:20-23 While the core DataStore distinguishes between a key existing and not existing (returning None on a read for a missing key), StorageOptionExt allows an application to explicitly store a None value for a key that does exist in the index extensions/src/storage_option_ext.rs:25-28

Overview and Purpose

In standard DataStore operations, a None result from read() indicates the key was never written or was deleted. StorageOptionExt introduces an explicit tombstone marker to represent a logical None value extensions/src/storage_option_ext.rs:32-33 This is useful for:

Distinguishing “Not Found” from “Explicitly Null” : Applications can tell if a field is missing vs. explicitly set to None extensions/src/storage_option_ext.rs:25-26
Namespace Isolation : All Option operations are automatically isolated using a dedicated OPTION_PREFIX namespace extensions/src/storage_option_ext.rs:143-146
Typed Serialization : Automatically handles serialization of T using bitcode when the value is Some(T) extensions/src/storage_option_ext.rs31

Logic Mapping: Natural Language to Code

The following diagram maps the logical concept of “Explicit None” to the specific code entities responsible for the implementation.

Explicit None Logic Mapping

graph TD
    subgraph "Natural_Language_Space"
        A["Explicit_None"]
B["Some_Value"]
C["Missing_Key"]
end

    subgraph "Code_Entity_Space"
        D["OPTION_TOMBSTONE_MARKER"]
E["bitcode::encode"]
F["std::io::ErrorKind::NotFound"]
G["StorageOptionExt::write_option"]
H["StorageOptionExt::read_option"]
I["serialize_option"]
J["deserialize_option"]
end

 
   A --> G
 
   B --> G
 
   G --> I
 
   I --> D
 
   I --> E
 
   H --> J
 
   J --> D
 
   J --> E
 
   C --> H
 
   H --> F

Sources: extensions/src/storage_option_ext.rs:1-165 extensions/src/utils/option_serializer.rs:1-63

Implementation Details

Namespace Isolation

To prevent collisions with raw binary data or other extensions (like TTL caching), StorageOptionExt uses a NamespaceHasher initialized with the OPTION_PREFIX extensions/src/storage_option_ext.rs:143-146 This ensures that a key b"user_1" written via write_option does not overwrite a key b"user_1" written via standard DataStore::write. The hasher is lazily initialized via a OnceLock named OPTION_NAMESPACE_HASHER extensions/src/storage_option_ext.rs12

Storage Format

The extension uses the option_serializer utility to transform Option<T> into a format the DataStore can persist:

Value	On-Disk Representation
`Some(T)`	`bitcode` serialized bytes of `T` extensions/src/utils/option_serializer.rs26
`None`	The tombstone marker: `[0xFF, 0xFE]` extensions/src/utils/option_serializer.rs27

Data Flow: Write and Read

The following diagram illustrates the flow of data through the extension and into the core storage engine.

StorageOptionExt Data Flow

sequenceDiagram
    participant App as "Application"
    participant Ext as "StorageOptionExt"
    participant Ser as "option_serializer"
    participant DS as "DataStore"

    Note over App, DS: write_option(key, Some(data))
    App->>Ext: write_option(key, Some(data))
    Ext->>Ser: serialize_option(Some(data))
    Ser-->>Ext: Vec<u8> (bitcode)
    Ext->>DS: write(namespaced_key, bytes)
    DS-->>App: Ok(offset)

    Note over App, DS: read_option(key)
    App->>Ext: read_option(key)
    Ext->>DS: read(namespaced_key)
    DS-->>Ext: Some(EntryHandle)
    Ext->>Ser: deserialize_option(entry.as_slice())
    alt matches [0xFF, 0xFE]
        Ser-->>Ext: Ok(None)
    else valid bitcode
        Ser-->>Ext: Ok(Some(T))
    end
    Ext-->>App: Result<Option<T>, Error>

Sources: extensions/src/storage_option_ext.rs:142-165 extensions/src/utils/option_serializer.rs:24-63

Key Functions

`write_option<T: Encode>`

Serializes the provided Option<&T> and writes it to the DataStore under a namespaced key extensions/src/storage_option_ext.rs:143-150

Process : Fetches the OPTION_NAMESPACE_HASHER, hashes the key, calls serialize_option, and performs a standard DataStore::write extensions/src/storage_option_ext.rs:144-149
Citations : extensions/src/storage_option_ext.rs:143-150 extensions/src/utils/option_serializer.rs:24-29

`read_option<T: Decode>`

Retrieves data from the namespaced key and attempts to deserialize it extensions/src/storage_option_ext.rs:152-164

Returns Ok(Some(T)) if the data is a valid bitcode-encoded T extensions/src/utils/option_serializer.rs:60-61
Returns Ok(None) if the data matches the OPTION_TOMBSTONE_MARKER extensions/src/utils/option_serializer.rs:56-58
Returns Err(ErrorKind::NotFound) if the key does not exist in the store (i.e., DataStore::read returns None) extensions/src/storage_option_ext.rs:159-162
Citations : extensions/src/storage_option_ext.rs:152-164 extensions/src/utils/option_serializer.rs:55-63

Technical Considerations

Non-Zero-Copy Performance

Unlike the core DataStore::read which returns an EntryHandle providing zero-copy access to memory-mapped data, read_option requires deserialization into a new instance of T extensions/src/storage_option_ext.rs93 This involves memory allocation and CPU cycles for the bitcode decoding process extensions/src/storage_option_ext.rs:137-138

Tombstone Collision

The tombstone marker [0xFF, 0xFE] is a reserved sequence within the OPTION_PREFIX namespace extensions/src/storage_option_ext.rs:15-18 If a serialized bitcode payload of T were to exactly match [0xFF, 0xFE], it would be incorrectly interpreted as None extensions/src/utils/option_serializer.rs:56-57 However, bitcode’s internal structure and the small size of the marker make this collision statistically improbable for complex types.

Persistence of None

When write_option(key, None) is called, the entry is not deleted from the underlying file. Instead, a new entry containing the tombstone marker is appended extensions/tests/storage_option_tests.rs:101-103 This preserves the append-only nature of the store and ensures that the None status is part of the versioned history of that key extensions/tests/storage_option_tests.rs:116-128

Sources:

Keyboard shortcuts

rust-simd-r-drive Documentation