Concurrency and Thread Safety

Loading…

Concurrency and Thread Safety

Relevant source files

• README.md
• benches/storage_benchmark.rs
• src/main.rs
• src/storage_engine/data_store.rs
• src/utils/format_bytes.rs
• tests/concurrency_tests.rs

Purpose and Scope

This document describes the concurrency model and thread safety guarantees of the SIMD R Drive storage engine. It covers the synchronization primitives used to enable safe multi-threaded access within a single process, including lock strategies for reads and writes, atomic operations, and memory map management.

For information about the core storage architecture and data structures, see Storage Architecture. For details on memory-mapped file usage, see Memory Management and Zero-Copy Access.

Key Limitation : The concurrency mechanisms described here apply only to single-process, multi-threaded environments. Multiple processes accessing the same storage file simultaneously are not supported and require external file locking mechanisms.

Concurrency Model Overview

The DataStore structure uses a combination of read-write locks, atomic operations, and mutexes to enable safe concurrent access across multiple threads while maintaining data consistency.

Diagram: DataStore Synchronization Architecture

Sources: src/storage_engine/data_store.rs:27-33 README.md:172-183

Synchronization Primitives

DataStore Field Overview

The DataStore struct contains four primary fields that implement concurrency control:

Sources: src/storage_engine/data_store.rs:27-33

RwLock for File Writes

All write operations acquire an exclusive write lock on the file handle to prevent concurrent modifications.

Diagram: Write Lock Serialization

Write Lock Acquisition

Write operations acquire the lock at the start of the write process:

This pattern appears in:

• write_stream_with_key_hash: src/storage_engine/data_store.rs:759-762
• batch_write_with_key_hashes: src/storage_engine/data_store.rs:852-855

The write lock ensures that only one thread can append data to the file at any given time, preventing:

• Race conditions on file position
• Interleaved writes corrupting the append-only chain
• Inconsistent metadata ordering

Sources: src/storage_engine/data_store.rs:752-825 src/storage_engine/data_store.rs:847-945 README.md176

AtomicU64 for Tail Offset

The tail_offset field tracks the current end of the valid data in the storage file using atomic operations, enabling lock-free reads of the current file position.

Atomic Operations Used

Load Operation

Reads use Acquire ordering to ensure they see all previous writes:

Examples:

• iter_entries: src/storage_engine/data_store.rs278
• Write operations reading previous tail: src/storage_engine/data_store.rs763 src/storage_engine/data_store.rs858

Store Operation

Writes use Release ordering to ensure all previous writes are visible:

Location: src/storage_engine/data_store.rs256

This atomic coordination ensures that:

• Readers always see a consistent tail offset
• Writers update the tail only after data is flushed
• No locks are needed for reading the tail position

Sources: src/storage_engine/data_store.rs30 src/storage_engine/data_store.rs256 src/storage_engine/data_store.rs278 README.md182

Mutex for Memory Map

The memory-mapped file reference is protected by a Mutex<Arc<Mmap>> to prevent concurrent remapping during reads.

Diagram: Memory Map Arc Cloning Pattern

Accessing the Memory Map

Read operations clone the Arc<Mmap> to obtain a stable reference:

Source: src/storage_engine/data_store.rs:658-663

This pattern ensures:

• Readers hold a reference to a specific memory map version
• Writers can create a new memory map without invalidating existing readers
• The Arc reference counting prevents premature deallocation
• The mutex is held only briefly during the clone operation

Remapping After Writes

After writing and flushing data, the reindex function creates a new memory map:

Source: src/storage_engine/data_store.rs:231-255

Sources: src/storage_engine/data_store.rs29 src/storage_engine/data_store.rs:224-259 src/storage_engine/data_store.rs:658-663 README.md180

RwLock for Key Index

The KeyIndexer is protected by a read-write lock, allowing multiple concurrent readers but exclusive writers.

Read Access Pattern

Multiple threads can acquire read locks simultaneously for lookups:

Example: src/storage_engine/data_store.rs509

Write Access Pattern

Index updates require exclusive write access:

Source: src/storage_engine/data_store.rs:233-253

Parallel Iterator Lock Strategy

The parallel iterator minimizes lock holding time by collecting offsets first:

Source: src/storage_engine/data_store.rs:300-302

Sources: src/storage_engine/data_store.rs31 src/storage_engine/data_store.rs:233-253 src/storage_engine/data_store.rs:300-302 README.md178

Lock-Free Read Operations

Read operations achieve lock-free access through memory-mapped files and atomic operations.

Diagram: Concurrent Lock-Free Read Pattern

Zero-Copy Read Implementation

Once the offset is obtained from the index, data access is lock-free:

Source: src/storage_engine/data_store.rs:502-565

Benefits of Lock-Free Reads

1. No Read Contention : Multiple readers access different memory regions simultaneously
2. Zero-Copy : Data is accessed directly from the memory map without copying
3. Scalability : Read throughput scales linearly with CPU cores
4. Low Latency : No lock acquisition overhead after index lookup

Sources: README.md174 src/storage_engine/data_store.rs:502-565 tests/concurrency_tests.rs:163-229

Write Synchronization

Write operations are fully serialized through the file lock, ensuring consistency.

Diagram: Write Operation Synchronization Flow

Single Write Flow

Source: src/storage_engine/data_store.rs:758-825

Batch Write Optimization

Batch writes hold the lock once for multiple entries:

Source: src/storage_engine/data_store.rs:847-945

Sources: src/storage_engine/data_store.rs:752-825 src/storage_engine/data_store.rs:847-945 README.md176

Thread Safety Guarantees

Thread Safety Matrix

The following table summarizes thread safety guarantees for different environments:

Source: README.md:196-200

Safe Concurrency Properties

The design ensures the following properties in single-process, multi-threaded environments:

Diagram: Thread Safety Property Dependencies

1. Atomicity : All operations on shared state are atomic or properly locked
2. Visibility : Changes made by one thread are visible to others through Release/Acquire semantics
3. Ordering : The append-only design ensures writes happen in a strict sequence
4. Isolation : Readers see a consistent snapshot via Arc<Mmap> cloning

Sources: README.md:172-206 src/storage_engine/data_store.rs:27-33

Single-Process vs Multi-Process

Single-Process Multi-Threaded (Supported)

All synchronization primitives work correctly within a single process:

Diagram: Single-Process Shared State

Example from concurrency tests:

Source: tests/concurrency_tests.rs:117-137

Multi-Process (Not Supported)

Multiple processes have separate address spaces and cannot share the in-memory synchronization primitives:

Diagram: Multi-Process Unsafe Access

Why Multi-Process is Unsafe

1. Separate Index State : Each process has its own KeyIndexer in memory
2. Independent Mmap Views : Memory maps are not synchronized across processes
3. No Lock Coordination : RwLock and Mutex are process-local, not system-wide
4. Race Conditions : Concurrent writes can corrupt the file structure

Recommendation : Use external file locking (e.g., flock, advisory locks) if multi-process access is required.

Sources: README.md:186-206 README.md:189-191

Testing Concurrency

The test suite validates concurrent access patterns to ensure thread safety guarantees.

Concurrent Write Test

Tests multiple threads writing simultaneously:

Source: tests/concurrency_tests.rs:111-161

Interleaved Read-Write Test

Tests read-after-write consistency with coordinated threads:

Source: tests/concurrency_tests.rs:163-229

Concurrent Streamed Write Test

Tests slow, streaming writes that hold the lock for extended periods:

Source: tests/concurrency_tests.rs:14-109

Sources: tests/concurrency_tests.rs:1-230

Summary

The SIMD R Drive concurrency model provides thread-safe access through a carefully coordinated set of synchronization primitives:

• RwLock : Serializes file writes while allowing concurrent reads of the lock
• AtomicU64 : Provides lock-free tail offset tracking
• Mutex : Protects memory map updates without blocking existing readers
• RwLock : Enables highly concurrent index reads with exclusive write access

This design achieves:

• Zero-copy concurrent reads via memory mapping
• Serialized writes preventing data corruption
• Linear read scalability across CPU cores
• Consistent snapshots through atomic operations

However, these guarantees apply only within a single process. Multi-process access requires external coordination mechanisms.

Sources: README.md:170-206 src/storage_engine/data_store.rs:26-33 tests/concurrency_tests.rs:1-230

Dismiss

Refresh this wiki

Enter email to refresh