Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

DeepWiki GitHub

DataStore API

Relevant source files

This document describes the public API of the DataStore struct, which provides the primary interface for interacting with the SIMD R Drive storage engine. It covers the methods for opening storage files, reading and writing data, batch operations, streaming, iteration, and maintenance operations.

For details on the underlying storage architecture and design principles, see Storage Architecture. For information about entry structure and metadata format, see Entry Structure and Metadata. For details on concurrency mechanisms, see Concurrency and Thread Safety.

API Structure

The DataStore API is organized around two primary traits that separate read and write capabilities, along with additional utility methods provided directly on the DataStore struct.

Sources: src/storage_engine/data_store.rs:26-750 src/storage_engine/traits.rs src/storage_engine.rs:1-24

graph TB
    subgraph "Trait Definitions"
        DSReader["DataStoreReader\ntrait"]
DSWriter["DataStoreWriter\ntrait"]
end
    
    subgraph "Core Implementation"
        DS["DataStore\nstruct"]
end
    
    subgraph "Associated Types"
        EH["EntryHandle"]
EM["EntryMetadata"]
EI["EntryIterator"]
ES["EntryStream"]
end
    
 
   DSReader -->|implemented by| DS
 
   DSWriter -->|implemented by| DS
    
 
   DS -->|returns| EH
 
   DS -->|returns| EM
 
   DS -->|provides| EI
 
   EH -->|converts to| ES
    
 
   DSReader -->|defines read ops| ReadMethods["read()\nbatch_read()\nexists()\nread_metadata()\nread_last_entry()"]
DSWriter -->|defines write ops| WriteMethods["write()\nbatch_write()\nwrite_stream()\ndelete()\nrename()\ncopy()"]
DS -->|direct methods| DirectMethods["open()\niter_entries()\ncompact()\nget_path()"]

Core Types

TypePurposeDefined In
DataStoreMain storage engine implementationsrc/storage_engine/data_store.rs:27-33
DataStoreReaderTrait defining read operationsTrait definition
DataStoreWriterTrait defining write operationsTrait definition
EntryHandleZero-copy reference to stored datasimd_r_drive_entry_handle crate
EntryMetadataEntry metadata structure (key hash, prev offset, checksum)simd_r_drive_entry_handle crate
EntryIteratorSequential iterator over entriessrc/storage_engine/entry_iterator.rs:21-127
EntryStreamStreaming reader for entry datasrc/storage_engine/entry_stream.rs

Sources: src/storage_engine/data_store.rs:27-33 src/storage_engine/entry_iterator.rs:8-25 src/storage_engine.rs:1-24

Opening Storage Files

The DataStore provides two methods for opening storage files, both returning Result<DataStore>.

Sources: src/storage_engine/data_store.rs:84-117 src/storage_engine/data_store.rs:141-144 src/storage_engine/data_store.rs:161-170

graph LR
    subgraph "Open Methods"
        Open["open(path)"]
OpenExisting["open_existing(path)"]
end
    
    subgraph "Process Flow"
        OpenFile["open_file_in_append_mode()"]
InitMmap["init_mmap()"]
RecoverChain["recover_valid_chain()"]
BuildIndex["KeyIndexer::build()"]
end
    
    subgraph "Result"
        DSInstance["DataStore instance"]
end
    
 
   Open -->|creates if missing| OpenFile
 
   OpenExisting -->|requires existing file| VerifyFile["verify_file_existence()"]
VerifyFile --> OpenFile
    
 
   OpenFile --> InitMmap
 
   InitMmap --> RecoverChain
 
   RecoverChain -->|may truncate| BuildIndex
 
   BuildIndex --> DSInstance

open

Opens or creates a storage file at the specified path. This method performs the following operations:

  1. Opens the file in read/write mode (creates if necessary)
  2. Memory-maps the file using mmap
  3. Runs recover_valid_chain() to validate data integrity
  4. Truncates the file if corruption is detected
  5. Builds the in-memory key index using KeyIndexer::build()

Parameters:

  • path: Path to the storage file

Returns:

  • Ok(DataStore): Successfully opened storage instance
  • Err(std::io::Error): File operation failure

Sources: src/storage_engine/data_store.rs:84-117

open_existing

Opens an existing storage file. Unlike open(), this method returns an error if the file does not exist. It calls verify_file_existence() before delegating to open().

Parameters:

  • path: Path to the existing storage file

Returns:

  • Ok(DataStore): Successfully opened storage instance
  • Err(std::io::Error): File does not exist or cannot be opened

Sources: src/storage_engine/data_store.rs:141-144

Type Conversion Constructors

Allows creating a DataStore directly from a PathBuf:

This conversion calls DataStore::open() internally and panics if the file cannot be opened.

Sources: src/storage_engine/data_store.rs:53-64

Read Operations

Read operations are defined by the DataStoreReader trait and return EntryHandle instances that provide zero-copy access to stored data.

Sources: src/storage_engine/data_store.rs:1027-1182

graph TB
    subgraph "Read API Methods"
        Read["read(key)"]
ReadHash["read_with_key_hash(hash)"]
BatchRead["batch_read(keys)"]
BatchReadHash["batch_read_hashed_keys(hashes)"]
ReadLast["read_last_entry()"]
ReadMeta["read_metadata(key)"]
Exists["exists(key)"]
ExistsHash["exists_with_key_hash(hash)"]
end
    
    subgraph "Internal Flow"
        ComputeHash["compute_hash()"]
GetIndexLock["key_indexer.read()"]
GetMmap["get_mmap_arc()"]
ReadContext["read_entry_with_context()"]
end
    
    subgraph "Return Types"
        OptHandle["Option<EntryHandle>"]
VecOptHandle["Vec<Option<EntryHandle>>"]
OptMeta["Option<EntryMetadata>"]
Bool["bool"]
end
    
 
   Read --> ComputeHash
 
   ComputeHash --> ReadHash
 
   ReadHash --> GetIndexLock
 
   GetIndexLock --> GetMmap
 
   GetMmap --> ReadContext
 
   ReadContext --> OptHandle
    
 
   BatchRead --> BatchReadHash
 
   BatchReadHash --> ReadContext
 
   ReadContext --> VecOptHandle
    
 
   ReadMeta --> Read
 
   Read --> OptMeta
    
 
   Exists --> Read
 
   Read --> Bool

read

Reads a single entry by key. Computes the key hash using compute_hash(), acquires a read lock on the key index, and calls read_entry_with_context() to retrieve the entry.

Parameters:

  • key: Raw key bytes

Returns:

  • Ok(Some(EntryHandle)): Entry found and valid
  • Ok(None): Entry not found or is a tombstone
  • Err(std::io::Error): Lock acquisition failure

Tag Verification: When the original key is provided, the method verifies the stored tag matches KeyIndexer::tag_from_key(key) to detect hash collisions.

Sources: src/storage_engine/data_store.rs:1040-1049

read_with_key_hash

Reads an entry using a pre-computed hash. This method skips the hashing step and tag verification, useful when the hash is already known.

Parameters:

  • prehashed_key: Pre-computed XXH3 hash of the key

Returns:

  • Ok(Some(EntryHandle)): Entry found
  • Ok(None): Entry not found
  • Err(std::io::Error): Lock acquisition failure

Note: Tag verification is not performed when using pre-computed hashes.

Sources: src/storage_engine/data_store.rs:1051-1059

batch_read

Reads multiple entries in a single operation. Computes all hashes using compute_hash_batch(), then delegates to batch_read_hashed_keys().

Parameters:

  • keys: Slice of key references

Returns:

  • Ok(Vec<Option<EntryHandle>>): Vector of results, same length as input
  • Err(std::io::Error): Lock acquisition failure

Efficiency: Acquires the index read lock once for all lookups, reducing lock contention compared to multiple read() calls.

Sources: src/storage_engine/data_store.rs:1105-1109

batch_read_hashed_keys

Reads multiple entries using pre-computed hashes. Optionally accepts original keys for tag verification.

Parameters:

  • prehashed_keys: Slice of pre-computed key hashes
  • non_hashed_keys: Optional slice of original keys for tag verification (must match length of prehashed_keys)

Returns:

  • Ok(Vec<Option<EntryHandle>>): Vector of results
  • Err(std::io::Error): Lock failure or length mismatch

Sources: src/storage_engine/data_store.rs:1111-1158

read_last_entry

Reads the most recently written entry (at tail_offset - METADATA_SIZE). Does not verify the key or check the index; directly reads from the tail position.

Returns:

  • Ok(Some(EntryHandle)): Last entry retrieved
  • Ok(None): Storage is empty or tail is invalid
  • Err(std::io::Error): Not applicable (infallible after initial checks)

Sources: src/storage_engine/data_store.rs:1061-1103

read_metadata

Reads only the metadata for a key without accessing the payload data. Calls read() and extracts the metadata field.

Returns:

  • Ok(Some(EntryMetadata)): Metadata retrieved
  • Ok(None): Key not found
  • Err(std::io::Error): Read failure

Sources: src/storage_engine/data_store.rs:1160-1162

exists

Checks if a key exists in the storage.

Returns:

  • Ok(true): Key exists and is not a tombstone
  • Ok(false): Key not found or is deleted
  • Err(std::io::Error): Read failure

Sources: src/storage_engine/data_store.rs:1030-1032

exists_with_key_hash

Checks key existence using a pre-computed hash.

Sources: src/storage_engine/data_store.rs:1034-1038

graph TB
    subgraph "Write API Methods"
        Write["write(key, payload)"]
WriteHash["write_with_key_hash(hash, payload)"]
BatchWrite["batch_write(entries)"]
BatchWriteHash["batch_write_with_key_hashes(entries)"]
WriteStream["write_stream(key, reader)"]
WriteStreamHash["write_stream_with_key_hash(hash, reader)"]
end
    
    subgraph "Core Write Flow"
        AcquireLock["file.write()"]
GetTail["tail_offset.load()"]
CalcPrepad["prepad_len(tail)"]
WritePrepad["write zeros for alignment"]
WritePayload["write payload data"]
WriteMeta["write EntryMetadata"]
Flush["file.flush()"]
Reindex["reindex()"]
end
    
    subgraph "Post-Write"
        RemapMmap["init_mmap()"]
UpdateIndex["key_indexer.insert()"]
UpdateTail["tail_offset.store()"]
end
    
 
   Write --> WriteHash
 
   WriteHash --> BatchWriteHash
 
   WriteStream --> WriteStreamHash
    
 
   BatchWriteHash --> AcquireLock
 
   WriteStreamHash --> AcquireLock
    
 
   AcquireLock --> GetTail
 
   GetTail --> CalcPrepad
 
   CalcPrepad --> WritePrepad
 
   WritePrepad --> WritePayload
 
   WritePayload --> WriteMeta
 
   WriteMeta --> Flush
 
   Flush --> Reindex
    
 
   Reindex --> RemapMmap
 
   RemapMmap --> UpdateIndex
 
   UpdateIndex --> UpdateTail

Write Operations

Write operations are defined by the DataStoreWriter trait. All writes are append-only and atomic with respect to the file lock.

Sources: src/storage_engine/data_store.rs:752-1025

write

Writes a single key-value pair. Computes the key hash and delegates to write_with_key_hash().

Parameters:

  • key: Key bytes
  • payload: Payload bytes (cannot be empty or all NULL bytes)

Returns:

  • Ok(tail_offset): New tail offset after write
  • Err(std::io::Error): Write failure

Sources: src/storage_engine/data_store.rs:827-830

write_with_key_hash

Writes a payload using a pre-computed key hash. Delegates to batch_write_with_key_hashes() with a single entry.

Sources: src/storage_engine/data_store.rs:832-834

batch_write

Writes multiple key-value pairs in a single atomic operation. All entries are buffered and written together, then flushed once.

Parameters:

  • entries: Slice of (key, payload) tuples

Returns:

  • Ok(tail_offset): New tail offset after batch write
  • Err(std::io::Error): Write failure (entire batch is rejected)

Efficiency: Reduces I/O overhead and lock contention compared to multiple write() calls.

Sources: src/storage_engine/data_store.rs:838-843

batch_write_with_key_hashes

Core batch write implementation. Processes each entry, calculating alignment padding and writing pre-pad, payload, and metadata. All data is buffered before flushing to disk.

Parameters:

  • prehashed_keys: Vector of (hash, payload) tuples
  • allow_null_bytes: Whether to allow single NULL byte payloads (used for tombstones)

Returns:

  • Ok(tail_offset): New tail offset
  • Err(std::io::Error): Write failure

Write Process:

  1. Acquire exclusive write lock on file
  2. Load current tail_offset
  3. For each entry:
    • Calculate prepad_len(tail_offset) for 64-byte alignment
    • Buffer pre-pad zeros (if needed)
    • Buffer payload (using simd_copy())
    • Buffer metadata (key_hash, prev_offset, checksum)
    • Update tail_offset
  4. Write entire buffer to file
  5. Flush to disk
  6. Call reindex() to update mmap and index

Sources: src/storage_engine/data_store.rs:847-939

write_stream

Writes a payload from a streaming source without loading it entirely into memory. Useful for large entries.

Parameters:

  • key: Key bytes
  • reader: Any type implementing Read trait

Returns:

  • Ok(tail_offset): New tail offset after write
  • Err(std::io::Error): Write or read failure

Sources: src/storage_engine/data_store.rs:753-756

write_stream_with_key_hash

Core streaming write implementation. Reads from the source in chunks of WRITE_STREAM_BUFFER_SIZE bytes, computing the checksum incrementally.

Write Process:

  1. Acquire write lock
  2. Write pre-pad bytes for alignment
  3. Read chunks from reader using WRITE_STREAM_BUFFER_SIZE buffer
  4. Write each chunk immediately to file
  5. Update checksum incrementally
  6. Write metadata after all payload bytes
  7. Flush and reindex

Validation:

  • Rejects empty payloads
  • Rejects payloads containing only NULL bytes

Sources: src/storage_engine/data_store.rs:758-825

Delete Operations

Delete operations write tombstones (single NULL byte entries) to the storage. They are implemented using the write infrastructure with allow_null_bytes = true.

delete

Deletes a single key by writing a tombstone. Delegates to batch_delete().

Returns:

  • Ok(tail_offset): New tail offset
  • Err(std::io::Error): Write failure

Sources: src/storage_engine/data_store.rs:986-988

batch_delete

Deletes multiple keys in a single operation. Computes hashes and delegates to batch_delete_key_hashes().

Sources: src/storage_engine/data_store.rs:990-993

batch_delete_key_hashes

Core batch delete implementation. First checks which keys actually exist to avoid writing unnecessary tombstones, then writes tombstones for existing keys only.

Process:

  1. Acquire read lock on index
  2. Filter keys to only those that exist
  3. Release read lock
  4. If no keys exist, return immediately without I/O
  5. Create delete operations: (hash, &NULL_BYTE)
  6. Call batch_write_with_key_hashes() with allow_null_bytes = true

Optimization: Tombstones are written without pre-pad bytes since they are always exactly 1 byte.

Sources: src/storage_engine/data_store.rs:995-1024

Key Management Operations

These operations manipulate entries by key, providing higher-level functionality built on read and write operations.

rename

Renames a key by copying its data to a new key and deleting the old key.

Parameters:

  • old_key: Current key
  • new_key: New key (must be different from old key)

Returns:

  • Ok(tail_offset): New tail offset after rename
  • Err(std::io::Error): Key not found, or keys are identical

Process:

  1. Read entry at old_key
  2. Convert to EntryStream
  3. Write stream to new_key
  4. Delete old_key

Sources: src/storage_engine/data_store.rs:941-958

copy

Copies an entry from this storage to a different storage instance.

Parameters:

  • key: Key to copy
  • target: Target DataStore (must have a different path)

Returns:

  • Ok(target_offset): Offset in target storage where entry was written
  • Err(std::io::Error): Key not found, same storage path, or write failure

Process:

  1. Read entry handle for key
  2. Call copy_handle() to write to target
  3. Uses EntryStream for streaming copy

Sources: src/storage_engine/data_store.rs:960-979

transfer

Transfers an entry to another storage by copying then deleting.

Parameters:

  • key: Key to transfer
  • target: Target DataStore

Returns:

  • Ok(tail_offset): New tail offset after deletion
  • Err(std::io::Error): Copy or delete failure

Process:

graph TB
    subgraph "Iteration Methods"
        IntoIter["into_iter()"]
IterEntries["iter_entries()"]
ParIterEntries["par_iter_entries()"]
end
    
    subgraph "Iterator Types"
        EI["EntryIterator"]
PI["ParallelIterator<EntryHandle>"]
end
    
    subgraph "Implementation Details"
        GetMmap["get_mmap_arc()"]
GetTail["tail_offset.load()"]
CollectOffsets["collect packed offsets"]
FilterMap["filter_map entries"]
end
    
 
   IntoIter -->|consumes DataStore| IterEntries
 
   IterEntries --> GetMmap
 
   GetMmap --> GetTail
 
   GetTail --> EI
    
 
   ParIterEntries -->|requires parallel feature| CollectOffsets
 
   CollectOffsets --> FilterMap
 
   FilterMap --> PI
  1. Copy entry to target
  2. Delete entry from source

Sources: src/storage_engine/data_store.rs:981-984

Iteration

The DataStore provides methods for iterating over all valid entries. Iteration yields EntryHandle instances for each unique key, providing zero-copy access.

Sources: src/storage_engine/data_store.rs:269-361 src/storage_engine/entry_iterator.rs:8-127

iter_entries

Creates a sequential iterator over all valid entries. The iterator:

  • Starts at tail_offset and moves backward
  • Skips tombstones (deleted entries)
  • Ensures unique keys (only returns latest version)
  • Uses zero-copy access via EntryHandle

Returns:

  • EntryIterator: Sequential iterator over EntryHandle

Example:

Sources: src/storage_engine/data_store.rs:276-280

IntoIterator Implementation

Allows consuming a DataStore to produce an iterator:

This delegates to iter_entries() and consumes the DataStore instance.

Sources: src/storage_engine/data_store.rs:44-51

par_iter_entries (Parallel Feature)

Creates a parallel iterator using Rayon for multi-threaded processing. This method:

  1. Acquires a read lock on the index
  2. Collects all packed offset values (fast operation)
  3. Releases the lock immediately
  4. Returns a ParallelIterator that processes entries across threads

Returns:

  • ParallelIterator<Item = EntryHandle>: Parallel iterator over entries

Performance: Suitable for bulk operations on multi-core systems where each entry can be processed independently.

Sources: src/storage_engine/data_store.rs:296-361

Maintenance Operations

These methods provide utilities for monitoring and optimizing storage.

compact

Compacts the storage by creating a new file containing only the latest version of each key. This operation:

  1. Creates temporary file at {path}.bk
  2. Iterates through current entries
  3. Copies latest version of each key to temporary file
  4. Swaps temporary file with original file

Requirements:

  • Requires &mut self (exclusive mutable reference)
  • Should only be called when no other threads access the storage

Warning: While &mut self prevents concurrent mutations, it does not prevent other threads from holding shared references (&DataStore) if wrapped in Arc<DataStore>. External synchronization may be required.

Returns:

  • Ok(()): Compaction successful
  • Err(std::io::Error): I/O failure

Sources: src/storage_engine/data_store.rs:706-749

estimate_compaction_savings

Calculates potential space savings from compaction by comparing total file size to the size needed for unique entries only.

Returns:

  • Number of bytes that would be saved by compaction

Implementation: Iterates through entries, tracking seen keys, and sums the file size of unique entries.

Sources: src/storage_engine/data_store.rs:605-616

get_path

Returns the file path of the storage.

Sources: src/storage_engine/data_store.rs:265-267

len

Returns the number of unique keys currently stored (excluding tombstones).

Returns:

  • Ok(count): Number of keys in index
  • Err(std::io::Error): Lock acquisition failure

Sources: src/storage_engine/data_store.rs:1164-1171

is_empty

Checks if the storage contains any entries.

Sources: src/storage_engine/data_store.rs:1173-1177

file_size

Returns the total size of the storage file in bytes.

Sources: src/storage_engine/data_store.rs:1179-1181

Internal Methods

The following methods are internal implementation details but are important for understanding the API's behavior.

read_entry_with_context

Core read logic used by all read methods. Performs:

  1. Index lookup via key_indexer_guard.get_packed(&key_hash)
  2. Unpacking of tag and offset via KeyIndexer::unpack()
  3. Tag verification (if non_hashed_key is provided)
  4. Metadata deserialization
  5. Entry bounds calculation (handling pre-pad and tombstones)
  6. Construction of EntryHandle

Tag Verification: If non_hashed_key is provided and tag does not match KeyIndexer::tag_from_key(), returns None and logs a warning about potential hash collision.

Sources: src/storage_engine/data_store.rs:502-565

reindex

Updates the memory map and key index after write operations. This method:

  1. Creates new mmap via init_mmap()
  2. Acquires locks on mmap and key_indexer
  3. Inserts or removes key mappings
  4. Updates atomic tail_offset

Hash Collision Handling: If key_indexer_guard.insert() returns an error (collision detected), the entire operation fails to prevent inconsistent state.

Sources: src/storage_engine/data_store.rs:224-259

prepad_len

Calculates the number of padding bytes needed to align offset to PAYLOAD_ALIGNMENT (64 bytes). Formula:

prepad = (PAYLOAD_ALIGNMENT - (offset % PAYLOAD_ALIGNMENT)) & (PAYLOAD_ALIGNMENT - 1)

This ensures all non-tombstone payloads start on 64-byte boundaries.

Sources: src/storage_engine/data_store.rs:670-673

API Usage Patterns

Basic Read-Write Pattern

Batch Operations Pattern

Streaming Pattern

Iteration Pattern

Sources: src/lib.rs:20-115 README.md:208-246

Dismiss

Refresh this wiki

This wiki was recently refreshed. Please wait 4 days to refresh again.

On this page

  • DataStore API
  • API Structure
  • Core Types
  • Opening Storage Files
  • open
  • open_existing
  • Type Conversion Constructors
  • Read Operations
  • read
  • read_with_key_hash
  • batch_read
  • batch_read_hashed_keys
  • read_last_entry
  • read_metadata
  • exists
  • exists_with_key_hash
  • Write Operations
  • write
  • write_with_key_hash
  • batch_write
  • batch_write_with_key_hashes
  • write_stream
  • write_stream_with_key_hash
  • Delete Operations
  • delete
  • batch_delete
  • batch_delete_key_hashes
  • Key Management Operations
  • rename
  • copy
  • transfer
  • Iteration
  • iter_entries
  • IntoIterator Implementation
  • par_iter_entries (Parallel Feature)
  • Maintenance Operations
  • compact
  • estimate_compaction_savings
  • get_path
  • len
  • is_empty
  • file_size
  • Internal Methods
  • read_entry_with_context
  • reindex
  • prepad_len
  • API Usage Patterns
  • Basic Read-Write Pattern
  • Batch Operations Pattern
  • Streaming Pattern
  • Iteration Pattern