Repository Structure

Relevant source files

• Cargo.lock
• Cargo.toml

Purpose and Scope

This document provides a comprehensive overview of the SIMD R Drive repository structure, including its Cargo workspace organization, crate hierarchy, and inter-crate dependencies. It covers the core storage engine crates, experimental network components, and build configuration. For details about the storage architecture and API, see Core Storage Engine. For network communication details, see Network Layer and RPC. For Python integration specifics, see Python Integration.

Workspace Organization

The SIMD R Drive repository is organized as a Cargo workspace containing six published crates and two excluded Python binding crates. The workspace uses Cargo resolver version 2 and maintains unified versioning (0.15.5-alpha) across all workspace members.

Workspace Configuration:

The workspace is defined in Cargo.toml:65-78 with members spanning core functionality, utilities, and experimental features.

Workspace Structure Diagram:

graph TB
    subgraph "Core Crates"
        Core["simd-r-drive\n(Root Package)\nsrc/"]
EntryHandle["simd-r-drive-entry-handle\nsimd-r-drive-entry-handle/"]
Extensions["simd-r-drive-extensions\nextensions/"]
end
    
    subgraph "Experimental Network Crates"
        MuxioDef["simd-r-drive-muxio-service-definition\nexperiments/simd-r-drive-muxio-service-definition/"]
WSClient["simd-r-drive-ws-client\nexperiments/simd-r-drive-ws-client/"]
WSServer["simd-r-drive-ws-server\nexperiments/simd-r-drive-ws-server/"]
end
    
    subgraph "Excluded Python Bindings"
        PyBinding["Python Direct Binding\nexperiments/bindings/python/\n(Excluded from workspace)"]
PyWSClient["Python WebSocket Client\nexperiments/bindings/python-ws-client/\n(Excluded from workspace)"]
end
    
 
   Core --> EntryHandle
 
   Extensions --> Core
 
   WSServer --> Core
 
   WSServer --> MuxioDef
 
   WSClient --> Core
 
   WSClient --> MuxioDef
 
   PyBinding -.-> Core
 
   PyWSClient -.-> WSClient
    
    style Core fill:#f9f9f9,stroke:#333,stroke-width:3px
    style EntryHandle fill:#f9f9f9,stroke:#333,stroke-width:2px
    style Extensions fill:#f9f9f9,stroke:#333,stroke-width:2px

Sources: Cargo.toml:65-78

Core Crates

simd-r-drive (Root Package)

The main storage engine crate providing append-only, schema-less binary storage with SIMD optimizations. Located at the repository root with source code in src/.

Key Components:

• DataStore - Main storage interface implementing DataStoreReader and DataStoreWriter traits
• KeyIndexer - Hash-based index for O(1) key lookups using XXH3
• Storage format implementation with 64-byte payload alignment
• SIMD-optimized memory operations (simd_copy)
• Recovery and validation logic

Features:

• default - Standard functionality
• expose-internal-api - Exposes internal APIs for testing
• parallel - Enables Rayon-based parallel iteration
• arrow - Proxy feature enabling Apache Arrow support in entry-handle

CLI Binary: The crate includes a CLI application for direct storage operations without network overhead.

Sources: Cargo.toml:11-56 Cargo.lock:1795-1820

simd-r-drive-entry-handle

Zero-copy data access abstraction located in simd-r-drive-entry-handle/. Provides safe wrappers around memory-mapped file regions without copying payload data.

Key Components:

• EntryHandle - Zero-copy view into memory-mapped payload data
• EntryMetadata - Fixed 20-byte metadata structure
• Arrow integration (optional) for columnar data access
• CRC32C checksum validation

Dependencies:

• memmap2 - Memory-mapped file support
• crc32fast - Checksum validation
• arrow (optional) - Apache Arrow integration

The crate is dependency-minimal to enable use as a standalone zero-copy accessor.

Sources: Cargo.toml83 Cargo.lock:1823-1829

simd-r-drive-extensions

Utility functions and helper modules located in extensions/. Provides commonly-needed functionality for working with the storage engine.

Key Components:

• align_or_copy - Utility for handling alignment requirements
• Serialization helpers (bincode integration)
• Testing utilities
• File system utilities (walkdir integration)

Dependencies:

• simd-r-drive - Core storage engine
• bincode - Serialization support
• serde - Serialization framework
• tempfile - Temporary file handling
• walkdir - Directory traversal

Sources: Cargo.toml69 Cargo.lock:1832-1841

Experimental Network Crates

These crates enable remote access to the storage engine via WebSocket-based RPC. Located in experiments/.

simd-r-drive-muxio-service-definition

RPC service contract definition using the Muxio framework. Located in experiments/simd-r-drive-muxio-service-definition/.

Purpose:

• Defines RPC service interface shared between client and server
• Uses bitcode for efficient binary serialization
• Provides type-safe method definitions

Dependencies:

• bitcode - Binary serialization
• muxio-rpc-service - RPC service framework

Sources: Cargo.toml72 Cargo.lock:1844-1849

simd-r-drive-ws-client

Native Rust WebSocket client for remote storage access. Located in experiments/simd-r-drive-ws-client/.

Key Components:

• BaseDataStoreWsClient - WebSocket client implementation
• RPC method callers
• Connection management
• Async/await integration with Tokio

Dependencies:

• simd-r-drive - Storage engine traits
• simd-r-drive-muxio-service-definition - Shared RPC contract
• muxio-tokio-rpc-client - WebSocket RPC client runtime
• muxio-rpc-service-caller - RPC method invocation
• tokio - Async runtime

Sources: Cargo.toml71 Cargo.lock:1852-1863

simd-r-drive-ws-server

WebSocket server exposing the storage engine over RPC. Located in experiments/simd-r-drive-ws-server/.

Key Components:

• Axum-based HTTP/WebSocket server
• RPC endpoint routing
• DataStore backend integration
• Configuration via CLI arguments

Dependencies:

• simd-r-drive - Storage engine backend
• simd-r-drive-muxio-service-definition - Shared RPC contract
• muxio-tokio-rpc-server - WebSocket RPC server runtime
• clap - CLI argument parsing
• tokio - Async runtime

Sources: Cargo.toml70 Cargo.lock:1866-1878

Python Bindings (Excluded)

Two Python binding crates exist but are excluded from the main workspace due to their PyO3 and Maturin build requirements.

experiments/bindings/python

Direct Python bindings to the core storage engine using PyO3. Excluded at Cargo.toml75

Purpose:

• FFI bridge between Python and Rust
• Direct local access to DataStore from Python
• PyO3-based class and method exports

experiments/bindings/python-ws-client

Python WebSocket client wrapping the Rust client. Excluded at Cargo.toml76

Purpose:

• Remote storage access from Python
• PyO3 wrapper around simd-r-drive-ws-client
• Async/await bridge between Python and Tokio

For detailed documentation on these bindings, see Python Integration.

Sources: Cargo.toml:74-77

Dependency Graph

Inter-Crate Dependencies:

Sources: Cargo.lock:1795-1878 Cargo.toml:80-90

External Dependencies

Workspace-Level Shared Dependencies

The workspace defines common dependencies to ensure version consistency across all crates:

Storage & Performance:

Async & Network:

Serialization:

CLI & Utilities:

Sources: Cargo.toml:80-112

Build Configuration

Workspace Settings

All crates share common metadata defined at the workspace level:

Sources: Cargo.toml:1-9

Benchmark Configuration

The root crate includes two benchmark harnesses:

storage_benchmark:

• Harness: Criterion
• Purpose: Measure write/read throughput and latency
• Path: benches/storage_benchmark.rs

contention_benchmark:

• Harness: Criterion
• Purpose: Measure concurrent access performance
• Path: benches/contention_benchmark.rs

Sources: Cargo.toml:57-63

Directory Layout

rust-simd-r-drive/
├── Cargo.toml                          # Workspace root
├── Cargo.lock                          # Dependency lock file
├── src/                                # simd-r-drive core source
│   ├── lib.rs
│   ├── data_store.rs
│   ├── key_indexer.rs
│   └── ...
├── simd-r-drive-entry-handle/          # Zero-copy handle crate
│   ├── Cargo.toml
│   └── src/
├── extensions/                         # Utilities crate
│   ├── Cargo.toml
│   └── src/
└── experiments/
    ├── simd-r-drive-ws-server/         # WebSocket server
    │   ├── Cargo.toml
    │   └── src/
    ├── simd-r-drive-ws-client/         # Rust WebSocket client
    │   ├── Cargo.toml
    │   └── src/
    ├── simd-r-drive-muxio-service-definition/  # RPC contract
    │   ├── Cargo.toml
    │   └── src/
    └── bindings/
        ├── python/                     # PyO3 direct binding (excluded)
        │   ├── Cargo.toml
        │   ├── pyproject.toml
        │   └── src/
        └── python-ws-client/           # PyO3 WS client (excluded)
            ├── Cargo.toml
            ├── pyproject.toml
            └── src/

Sources: Cargo.toml:65-78

Publishing Strategy

All workspace crates (except Python bindings) are configured for publishing to crates.io with publish = true. The Python bindings use Maturin for building Python wheels and are distributed separately via PyPI.

Crates.io Packages:

• simd-r-drive - Core storage engine
• simd-r-drive-entry-handle - Zero-copy handle
• simd-r-drive-extensions - Utilities
• simd-r-drive-ws-server - WebSocket server
• simd-r-drive-ws-client - Rust WebSocket client
• simd-r-drive-muxio-service-definition - RPC contract

PyPI Packages:

• simd-r-drive (Python binding via Maturin)
• simd-r-drive-ws-client-py (Python WebSocket client via Maturin)

For Python package build instructions, see Building Python Bindings.

Sources: Cargo.toml:1-9 Cargo.toml21