DataStore: Read and Write Operations

Loading…

DataStore: Read and Write Operations

Relevant source files

• experiments/bindings/python_(old_client)/LICENSE/LICENSE)
• src/storage_engine/data_store.rs
• src/storage_engine/entry_stream.rs
• src/storage_engine/traits/reader.rs
• src/storage_engine/traits/writer.rs
• tests/basic_operations_tests.rs
• tests/batch_ops_tests.rs

The DataStore is the central entity of the rust-simd-r-drive storage engine. It provides a thread-safe, append-only interface for managing data with a focus on zero-copy reads and SIMD-accelerated writes. Operations are primarily defined through the DataStoreReader and DataStoreWriter traits, allowing for high-level abstractions over the underlying memory-mapped file.

Lifecycle and Initialization

A DataStore instance manages a buffered file writer, a memory-mapped view (Mmap), an atomic tail offset, and an in-memory KeyIndexer src/storage_engine/data_store.rs:27-33

Opening a Store

The DataStore::open() function is the primary entry point src/storage_engine/data_store.rs:84-117 It performs the following sequence:

1. File Opening : Opens the file in read/write mode, creating it if it doesn’t exist via open_file_in_append_mode src/storage_engine/data_store.rs:161-170
2. Mmap Initialization : Maps the file into memory via init_mmap src/storage_engine/data_store.rs:172-174
3. Chain Recovery : Validates the backward-linked integrity chain to find the last valid entry via recover_valid_chain src/storage_engine/data_store.rs89
4. Truncation : If corruption is detected (i.e., the valid chain is shorter than the file length), the file is truncated to the last known good state src/storage_engine/data_store.rs:91-104
5. Index Building : The KeyIndexer is populated by scanning the validated file content src/storage_engine/data_store.rs108

Entity Relationship: Initialization

The following diagram illustrates the transition from a file system path to an active DataStore instance.

graph TD
    subgraph "Natural Language Space"
        Path["File System Path"]
Recovery["Integrity Recovery"]
end

    subgraph "Code Entity Space"
        DS_Open["DataStore::open(path)"]
OpenAppend["open_file_in_append_mode()"]
InitMmap["init_mmap()"]
Recover["recover_valid_chain()"]
BuildIdx["KeyIndexer::build()"]
DS_Struct["struct DataStore"]
end

 
   Path --> DS_Open
 
   DS_Open --> OpenAppend
 
   OpenAppend --> InitMmap
 
   InitMmap --> Recover
 
   Recovery -.-> Recover
 
   Recover --> BuildIdx
 
   BuildIdx --> DS_Struct

Title: DataStore Initialization Flow

Sources: src/storage_engine/data_store.rs:66-117 src/storage_engine/data_store.rs:161-174

Write Operations

Write operations are defined in the DataStoreWriter trait src/storage_engine/traits/writer.rs:4-152 All writes are append-only.

Standard and Batch Writes

• write(key, payload) : Computes the key hash and appends the payload followed by EntryMetadata src/storage_engine/traits/writer.rs65
• batch_write(entries) : Writes multiple key-value pairs in a single operation src/storage_engine/traits/writer.rs106 This is more efficient as it acquires the write lock only once src/storage_engine/traits/writer.rs131
• batch_write_with_key_hashes : The low-level implementation for batching. It uses simd_copy for high-performance data movement and performs a single mmap remapping after the batch is complete src/storage_engine/traits/writer.rs:108-138

Streaming Writes

For payloads larger than available RAM, write_stream allows writing data from any source implementing std::io::Read src/storage_engine/traits/writer.rs29

• Buffer Size : Data is read in 64KB chunks (WRITE_STREAM_BUFFER_SIZE) src/storage_engine/traits/writer.rs25
• Incremental Checksum : The CRC32C checksum is updated as chunks are streamed to disk src/storage_engine/traits/writer.rs27

Write Data Flow

Title: Write Operation Data Flow

Sources: src/storage_engine/traits/writer.rs:5-138 src/storage_engine/data_store.rs:176-210

Read Operations

Read operations are defined in the DataStoreReader trait src/storage_engine/traits/reader.rs:4-160

Zero-Copy Access

The read(key) method returns an Option<EntryHandle> src/storage_engine/traits/reader.rs54

1. The key is hashed, and the KeyIndexer provides the file offset src/storage_engine/traits/reader.rs:41-42
2. The EntryHandle contains an Arc<Mmap>, allowing the application to access the payload as a slice (&[u8]) without copying data from the kernel buffer to user space src/storage_engine/traits/reader.rs:89-90

Batch and Hashed Reads

• batch_read(keys) : Vectorized lookup for multiple keys. It takes a read lock on the index once for the entire batch src/storage_engine/traits/reader.rs101
• batch_read_hashed_keys : Optimized version for callers who already have precomputed hashes. It includes an optional verification step to handle hash collisions by comparing the original keys src/storage_engine/traits/reader.rs:134-138

Streaming Reads

The EntryStream struct wraps an EntryHandle to provide a std::io::Read interface src/storage_engine/entry_stream.rs:44-47 Note that while the EntryHandle is zero-copy, the EntryStream::read() method does perform copies into the provided buffer src/storage_engine/entry_stream.rs:76-91

Sources: src/storage_engine/traits/reader.rs:4-138 src/storage_engine/entry_stream.rs:1-92

Management Operations

Deletion and Compaction

• delete(key) : Appends a “tombstone” entry to the file src/storage_engine/traits/writer.rs145 The key is removed from the KeyIndexer, making it immediately invisible to readers src/storage_engine/traits/reader.rs:11-12
• compact() : Reclaims space by creating a new storage file and copying only “live” (non-deleted, latest version) entries into it. This uses the EntryIterator to traverse the store src/storage_engine/data_store.rs:420-475

Copy, Move, and Rename

• copy(key, target_store) : Reads an entry from the current store and writes it to a different DataStore instance src/storage_engine/traits/writer.rs152
• transfer(key, target_store) : Performs a copy to the target store and then deletes the key from the source store (equivalent to a “move”) src/storage_engine/traits/writer.rs152
• rename(old_key, new_key) : Copies an entry to a new key within the same store and deletes the old key src/storage_engine/traits/writer.rs:140-152

Summary of Traits

Sources: src/storage_engine/traits/reader.rs:4-160 src/storage_engine/traits/writer.rs:4-152 src/storage_engine/data_store.rs:420-475