Rust Test Suite

Loading…

Rust Test Suite

Relevant source files

• experiments/bindings/python_(old_client)/LICENSE/LICENSE)
• src/main.rs
• src/storage_engine/traits/reader.rs
• src/storage_engine/traits/writer.rs
• src/utils/format_bytes.rs
• tests/alignment_tests.rs
• tests/basic_operations_tests.rs
• tests/batch_ops_tests.rs
• tests/concurrency_tests.rs
• tests/hash_stability_tests.rs
• tests/integrity_tests.rs
• tests/mmap_and_zero_copy_tests.rs
• tests/namespace_hasher_tests.rs
• tests/parallel_iterator_tests.rs
• tests/persistence_tests.rs
• tests/storage_operation_tests.rs
• tests/streaming_tests.rs

The Rust test suite for simd-r-drive is designed to validate the core storage engine’s guarantees regarding append-only integrity, zero-copy alignment, hash stability, and thread safety. It is organized into several specialized integration test files located in the tests/ directory, leveraging Cargo feature flags like parallel and expose-internal-api to exercise different execution paths.

Test Suite Organization

The integration tests are categorized by the specific sub-system or behavior they validate:

Data Flow: From Test to Storage Engine

The following diagram illustrates how tests interact with the DataStore and its underlying traits.

Test Execution Flow

graph TD
    subgraph "Test_Space"
        T1["basic_operations_tests.rs"]
T2["concurrency_tests.rs"]
T3["batch_ops_tests.rs"]
T4["storage_operation_tests.rs"]
end

    subgraph "Code_Entity_Space_Core"
        DS["DataStore"]
DSR["trait DataStoreReader"]
DSW["trait DataStoreWriter"]
KI["KeyIndexer"]
MM["Mmap (Arc)"]
end

 
   T1 -->|calls| DSR
 
   T2 -->|spawns_tasks| DSW
 
   T3 -->|vectorized| DSW
 
   T4 -->|cross_instance| DSW
    
 
   DSR -.->|implemented_by| DS
 
   DSW -.->|implemented_by| DS
    
 
   DS -->|updates| KI
 
   DS -->|reads_remaps| MM

Sources: src/storage_engine/traits/reader.rs:4-21 src/storage_engine/traits/writer.rs:4-29 tests/basic_operations_tests.rs:16-27 tests/concurrency_tests.rs:113-137

Alignment and SIMD Validation

A critical component of the test suite is alignment_tests.rs, which ensures that every payload is written to a 64-byte aligned boundary. This allows the storage engine to provide zero-copy views that are safe for SIMD instructions (AVX2/NEON).

• Alignment Proofs : Tests use bytemuck::try_cast_slice to prove that a &[u8] from the store can be cast to &[u32], &[u64], or &[u128] without copying tests/alignment_tests.rs:58-67
• SIMD Loadability : On x86_64, tests execute _mm_load_si128 on retrieved payloads to verify they do not trigger alignment faults tests/alignment_tests.rs:69-95 On aarch64, vld1q_u8 is used for the same purpose tests/alignment_tests.rs:97-122
• Interaction Testing : The suite mixes unaligned string writes (e.g., 3-byte or 7-byte) with aligned numeric writes to ensure the pre-pad calculation correctly maintains the PAYLOAD_ALIGNMENT tests/alignment_tests.rs:135-160

Alignment Verification Logic

Sources: tests/alignment_tests.rs:12-32 tests/alignment_tests.rs:69-75 tests/alignment_tests.rs:185-192 tests/alignment_tests.rs:97-103

Concurrency and Parallelism

The test suite validates thread safety using tokio and serial_test.

Concurrent Writers

concurrency_tests.rs spawns multiple tasks to perform simultaneous write operations. It verifies that the internal locking and tail offset management prevent data corruption tests/concurrency_tests.rs:113-142

Slow Streaming

The concurrent_slow_streamed_write_test uses a custom SlowReader to simulate network latency during a write_stream operation, ensuring that the store remains consistent even when appends are delayed tests/concurrency_tests.rs:16-35

Parallel Iterators

When the parallel feature is enabled, parallel_iterator_tests.rs verifies that DataStore can be iterated using Rayon, allowing for multi-threaded processing of the entire entry log.

• Deduplication : Tests ensure that parallel iteration respects the backward-linked chain and only returns the most recent version of a key.
• Tombstones : Tests validate that entries marked for deletion are omitted from the iterator stream.

Sources: tests/concurrency_tests.rs:1-12 tests/concurrency_tests.rs:113-121 tests/concurrency_tests.rs:16-35

Persistence and Recovery

persistence_tests.rs focuses on the “on-disk” contract of the engine.

1. Re-open Validation : Ensures that data written in one process session is visible after closing and re-opening the DataStore tests/persistence_tests.rs:13-55
2. Corruption Recovery : Simulates an interrupted write by manually appending “garbage” bytes to the end of the file. The test verifies that DataStore::open detects the invalid entry (via CRC32C or metadata mismatch) and truncates the file back to the last valid state tests/persistence_tests.rs:110-176 Note that these tests are skipped on Windows due to file locking restrictions with memory maps tests/persistence_tests.rs:146-160

Sources: tests/persistence_tests.rs:13-32 tests/persistence_tests.rs:126-141 tests/persistence_tests.rs:160-176

Batch and Streaming Operations

The suite rigorously tests vectorized operations and memory-efficient streaming.

Batch Operations

batch_ops_tests.rs validates the atomicity and ordering of batch_write and batch_read.

• Atomicity : If any payload in a batch is invalid (e.g., empty or null-byte), the entire batch is rejected, and no data is persisted tests/batch_ops_tests.rs:104-127
• Hashed Reads : test_batch_read_hashed_keys_with_verification exercises the high-performance path where callers provide pre-computed hashes and original keys for collision verification tests/batch_ops_tests.rs:197-210

Streaming I/O

streaming_tests.rs validates write_stream and EntryStream.

• Chunked Reading : Tests verify that a 1MB payload can be written from a BufReader and read back in 4KB chunks using EntryStream, maintaining checksum integrity throughout tests/streaming_tests.rs:20-89

Sources: tests/batch_ops_tests.rs:104-127 tests/batch_ops_tests.rs:197-210 tests/streaming_tests.rs:20-89

Hash Stability

hash_stability_tests.rs enforces hardcoded xxh3 hash values to guard against silent regressions if the xxhash-rust dependency is updated or replaced.

• Single Key Stability : Validates hashes for empty strings, null bytes, and various sample keys like “alice” or “test_key” tests/hash_stability_tests.rs:16-48
• Batch Stability : Ensures compute_hash_batch matches individual hash results tests/hash_stability_tests.rs:58-65
• Namespace Stability : Locks down the 16-byte namespaced key derivation logic in NamespaceHasher tests/hash_stability_tests.rs:77-101

Sources: tests/hash_stability_tests.rs:1-108

Storage Operations: Copy and Transfer

storage_operation_tests.rs validates higher-level data movement between different storage instances.

• Copy Entry : Verifies that source_storage.copy(key, &target_storage) correctly replicates data and metadata (key hash, checksum) while keeping the original entry intact tests/storage_operation_tests.rs:20-74
• Transfer Entry : Validates that transfer (move) correctly writes to the target and appends a tombstone to the source tests/storage_operation_tests.rs:103-152
• Self-Copy Protection : Ensures that attempting to copy an entry to the same storage instance returns an error tests/storage_operation_tests.rs:77-100

Sources: tests/storage_operation_tests.rs:20-152

Feature Flags in Testing

The test suite is executed across a matrix of feature flags in CI to ensure compatibility:

• default : Standard configuration.
• parallel : Enables Rayon integration and parallel iterators.
• expose-internal-api : Allows tests to access low-level metadata and internal state for deeper verification.
• arrow : Enables Apache Arrow buffer integration.

CI Matrix Configuration

Sources: src/storage_engine/traits/reader.rs:1-158 src/storage_engine/traits/writer.rs:1-152