Alignment, Checksums, and Utility Functions

Loading…

Alignment, Checksums, and Utility Functions

Relevant source files

• simd-r-drive-entry-handle/src/debug_assert_aligned.rs
• src/utils/align_or_copy.rs
• src/utils/parse_buffer_size.rs
• src/utils/verify_file_existence.rs
• tests/alignment_tests.rs

This page details the performance-critical utilities and helper functions used across the SIMD R Drive workspace. These components ensure data integrity through CRC32C checksums, maintain hardware-friendly memory alignment for SIMD operations, and provide common parsing and formatting logic for the CLI and core engine.

Memory Alignment and Zero-Copy Reinterpretation

The storage engine enforces a strict payload alignment of 64 bytes (PAYLOAD_ALIGNMENT). This alignment matches standard CPU cache-line sizes and satisfies the requirements for most SIMD instruction sets (AVX2, NEON) tests/alignment_tests.rs12

align_or_copy

The align_or_copy function is a core utility for reinterpreting raw byte slices as typed slices (e.g., &[f32]) without allocation when possible src/utils/align_or_copy.rs:44-50

• Zero-Copy Path : Uses slice::align_to::<T>() to check if the memory address is already aligned to the target type T. If the prefix and suffix returned by align_to are empty, it returns a Cow::Borrowed slice src/utils/align_or_copy.rs:57-59
• Fallback Path : If the memory is misaligned or the length is not a multiple of size_of::<T>(), it performs a manual copy into a Cow::Owned(Vec<T>) using the provided from_le_bytes conversion function src/utils/align_or_copy.rs:60-72

Alignment Validation Logic

The codebase provides specialized assertions to ensure alignment invariants are maintained during development without impacting production performance.

These functions use a specific pattern: the function body is gated by #[cfg(any(test, debug_assertions))] simd-r-drive-entry-handle/src/debug_assert_aligned.rs27 In release builds, they compile to a no-op where arguments are marked as used to avoid compiler warnings, ensuring zero runtime cost simd-r-drive-entry-handle/src/debug_assert_aligned.rs:37-42

Tests in tests/alignment_tests.rs verify these invariants by performing unaligned writes followed by aligned overwrites, then attempting to cast the resulting slices to u32, u64, and u128 views using bytemuck::try_cast_slice tests/alignment_tests.rs:58-67

Title: Alignment Validation and SIMD Loading

Sources: src/utils/align_or_copy.rs:44-73 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:26-43 tests/alignment_tests.rs:135-200

Checksum Computation

Data integrity is maintained using the CRC32C algorithm (Castagnoli), implemented via the crc32fast crate.

• Implementation : Checksums are computed using the crc32fast::Hasher which utilizes hardware-accelerated instructions (SSE4.2 on x86_64, NEON on ARM) when available.
• Verification : The system extracts the stored CRC from the entry metadata and compares it against the computed CRC of the payload to detect corruption.

Sources: src/utils/align_or_copy.rs:1-75 tests/alignment_tests.rs:1-133

General Utility Functions

Helper functions for formatting and parsing are provided to support the CLI and initialization routines.

parse_buffer_size

Converts string representations of sizes (e.g., “1MB”, “2G”, “1024”) into usize byte counts src/utils/parse_buffer_size.rs:35-57

• Supported Units : K/KB (1024), M/MB (1024^2), G/GB (1024^3) src/utils/parse_buffer_size.rs:45-51
• Behavior : Case-insensitive and trims whitespace src/utils/parse_buffer_size.rs36

verify_file_existence

Checks if a path exists and is a regular file before initialization src/utils/verify_file_existence.rs11 It returns std::io::ErrorKind::NotFound if the path is missing or InvalidInput if the path is a directory src/utils/verify_file_existence.rs:12-24

graph LR
    subgraph "Utility Requirements"
        R1["Data Reinterpretation"]
R2["Size Parsing"]
R3["Integrity Check"]
R4["Path Validation"]
end

    subgraph "Code Implementation"
 
       R1 --> F1["align_or_copy()"]
R2 --> F2["parse_buffer_size()"]
R3 --> F3["crc32fast::Hasher"]
R4 --> F4["verify_file_existence()"]
F1 --- FILE1["src/utils/align_or_copy.rs"]
F2 --- FILE2["src/utils/parse_buffer_size.rs"]
F4 --- FILE4["src/utils/verify_file_existence.rs"]
end

System Utility Mapping

The following diagram maps high-level utility requirements to the specific implementation files and functions within the codebase.

Title: Utility Function Mapping

Sources: src/utils/parse_buffer_size.rs:1-57 src/utils/align_or_copy.rs:44-50 src/utils/verify_file_existence.rs:1-27