Repository Layout

Loading…

Repository Layout

Relevant source files

• Cargo.lock
• Cargo.toml
• LICENSE
• experiments/bindings/python-ws-client/LICENSE

The SIMD R Drive repository is organized as a Cargo workspace containing a core storage engine, specialized utility crates, high-level extensions, and experimental network-based components. The project emphasizes a clean separation between the low-level binary storage logic and the high-level interfaces (CLI, RPC, and Python bindings) that consume it.

Workspace Structure Overview

The repository follows a modular architecture where the core logic is kept lean, and additional functionality is opted into via sub-crates or feature flags.

Sources: Cargo.toml:14-22 Cargo.toml:60-63

Component Relationship Diagram

The following diagram illustrates how the various crates and directories interact within the workspace.

Workspace Dependency and Data Flow

graph TD
    subgraph "Core_Workspace"
        CORE["simd-r-drive_(src/)"]
HANDLE["simd-r-drive-entry-handle"]
EXT["simd-r-drive-extensions"]
end

    subgraph "Experiments"
        WS_SERV["simd-r-drive-ws-server"]
WS_CLI["simd-r-drive-ws-client"]
DEF["simd-r-drive-muxio-service-definition"]
end

    subgraph "Bindings"
        PY_WS["python-ws-client_(PyO3)"]
end

    EXT -- "uses" --> CORE
    CORE -- "depends_on" --> HANDLE
    WS_SERV -- "wraps" --> CORE
    WS_SERV -- "implements" --> DEF
    WS_CLI -- "calls" --> DEF
    PY_WS -- "wraps" --> WS_CLI

Sources: Cargo.toml:14-22 Cargo.toml:60-63 Cargo.toml88

1. Core Library (src/)

The root directory contains the primary simd-r-drive crate. This is the heart of the project, responsible for the append-only storage format, memory-mapped I/O, and SIMD-accelerated operations.

• src/storage_engine/ : Implementation of DataStore, which manages file handles and the KeyIndexer.
• src/utils/ : Utility functions and performance-critical helpers, including align_or_copy and NamespaceHasher.
• src/lib.rs : The main entry point for the library, exporting core traits like DataStoreReader and DataStoreWriter.
• SIMD Copy : Specialized implementations for x86_64 (AVX2) and aarch64 (NEON) are housed in the storage engine.

Sources: Cargo.toml:2-12 Cargo.toml:80-91

2. Entry Handle Crate (simd-r-drive-entry-handle/)

This is a standalone crate designed to be as lightweight as possible. It defines the EntryHandle type, which provides a zero-copy view into the memory-mapped storage.

• Role : Allows third-party crates to read SIMD R Drive data without pulling in the full storage engine or its heavy dependencies.
• Key Entities : EntryHandle provides the primary interface for accessing payloads and validating checksums.
• Features : Includes an optional arrow feature to provide Apache Arrow buffer compatibility via simd-r-drive-entry-handle/arrow. Cargo.toml78

Sources: Cargo.toml21 Cargo.toml61 Cargo.toml78

3. Extensions Crate (extensions/)

The simd-r-drive-extensions crate provides high-level storage patterns built on top of the base DataStore.

• TTL Support : Adds Time-To-Live metadata to entries via StorageCacheExt.
• Option Storage : Differentiates between a missing key and a key explicitly set to None via StorageOptionExt.
• Filesystem Import : Utilities to recursively import local directories into a storage file via StorageFileImportExt using the walkdir dependency.

Sources: Cargo.toml20 Cargo.toml69

4. Experiments Directory (experiments/)

This directory houses work-in-progress or network-related components that are not part of the core “local-first” storage philosophy.

WebSocket RPC Stack

The system uses muxio for multiplexed asynchronous RPC over WebSockets.

• simd-r-drive-muxio-service-definition : Shared trait definitions and bitcode serialization schemas for RPC calls. Cargo.toml17 Cargo.toml62
• simd-r-drive-ws-server : A tokio-based server that exposes a DataStore instance over a network port. Cargo.toml19 Cargo.toml66
• simd-r-drive-ws-client : An async client implementing RPC-based access to a remote DataStore. Cargo.toml18 Cargo.toml63

Sources: Cargo.toml:17-19 Cargo.toml:62-63

5. Bindings and CI

The repository includes infrastructure for cross-language support and automated quality assurance.

• Python Bindings : Located in experiments/bindings/. These sub-projects (e.g., python-ws-client) are excluded from the main Cargo workspace to avoid mandatory local dependency on Python development headers during standard Rust builds. Cargo.toml:23-26
• CI Configuration : Found in .github/workflows/. It covers Rust linting, multi-platform testing (Ubuntu, macOS, Windows), and various feature flag combinations including parallel (which enables rayon) and expose-internal-api. Cargo.toml:74-75

Sources: Cargo.toml:23-26 Cargo.toml:74-75

Code Entity Mapping

The following diagrams map high-level repository concepts to specific code structures and file locations.

Entity Mapping: Storage and Metadata

classDiagram
    class DataStore {<<src/storage_engine/mod.rs>>\n+write(key, payload)\n+read(key)\n+compact()\n+write_stream(key, reader)}
    class DataStoreReader {<<src/storage_engine/traits.rs>>\n+read(key)\n+exists(key)}
    class EntryHandle {<<simd-r-drive-entry-handle/src/lib.rs>>\n+as_slice()\n+is_valid_checksum()}
    class EntryStream {<<src/storage_engine/entry_stream.rs>>\n+read(buf)\n+from(EntryHandle)}

    DataStore ..|> DataStoreReader : implements
    DataStore ..> EntryHandle : returns_on_read
    EntryStream ..> EntryHandle : wraps_for_io

Sources: Cargo.toml:60-61

Entity Mapping: Utility and Performance

Sources: Cargo.toml70 Cargo.toml91