Development Guide

Relevant source files

• .github/workflows/rust-tests.yml
• .gitignore
• Cargo.toml

This document provides an overview of the development process for SIMD R Drive, covering workspace structure, feature flags, build commands, and development workflows. This page serves as an entry point for contributors and developers working on the codebase.

For detailed instructions on building and running tests, see Building and Testing. For information about the CI/CD pipeline and automated testing, see CI/CD Pipeline. For version history and migration guides, see Version History and Migration.

Prerequisites

SIMD R Drive requires:

• Rust : Edition 2024 (as specified in Cargo.toml4)
• Cargo : Workspace-aware build system
• Platform Support : Linux, macOS, Windows (tested via CI matrix)

Optional tools for specific workflows:

• Python 3.10-3.13 : For Python bindings development (see Python Integration)
• Maturin : For building Python wheels (see Building Python Bindings)
• Criterion : For benchmarking (included as dev-dependency)

Workspace Structure

SIMD R Drive uses a Cargo workspace architecture with multiple crates organized by function. The workspace is defined in Cargo.toml:65-78 with specific members included and Python bindings excluded from the main workspace.

graph TB
    Root["simd-r-drive\n(root crate)"]
EntryHandle["simd-r-drive-entry-handle\n(zero-copy data access)"]
Extensions["extensions\n(utilities and helpers)"]
subgraph "Network Experiments"
        WSServer["experiments/simd-r-drive-ws-server\n(WebSocket server)"]
WSClient["experiments/simd-r-drive-ws-client\n(Rust client)"]
ServiceDef["experiments/simd-r-drive-muxio-service-definition\n(RPC contract)"]
end
    
    subgraph "Python Bindings (Excluded)"
        PyBindings["experiments/bindings/python\n(PyO3 bindings)"]
PyWSClient["experiments/bindings/python-ws-client\n(Python WebSocket client)"]
end
    
 
   Root --> EntryHandle
 
   Root --> Extensions
 
   WSServer --> Root
 
   WSServer --> ServiceDef
 
   WSClient --> ServiceDef
 
   PyWSClient --> WSClient
 
   PyBindings --> Root
    
    style Root stroke-width:3px
    style PyBindings stroke-dasharray: 5 5
    style PyWSClient stroke-dasharray: 5 5

Workspace Members Diagram

Sources: Cargo.toml:65-78

Workspace Member Descriptions

Python bindings are excluded from the main workspace because they use Maturin's build system, which conflicts with Cargo's workspace resolver. They must be built separately using maturin build or maturin develop.

Sources: Cargo.toml:74-77

Feature Flags

The core simd-r-drive crate exposes multiple feature flags that enable optional functionality. Features are defined in Cargo.toml:49-55

Feature Dependencies Diagram

Sources: Cargo.toml:49-56

Feature Flag Reference

The arrow feature is a proxy feature: enabling simd-r-drive/arrow automatically enables simd-r-drive-entry-handle/arrow, allowing users to enable Arrow support without knowing the internal crate structure.

Sources: Cargo.toml:49-55 Cargo.toml30 Cargo.toml:54-55

Common Development Commands

Building

Testing

Benchmarking

The crate includes two Criterion-based benchmarks defined in Cargo.toml:57-63:

To compile benchmarks without running them (useful for CI): cargo bench --no-run

Sources: Cargo.toml:57-63 .github/workflows/rust-tests.yml:60-61

Development Workflow

Standard Development Cycle

Sources: .github/workflows/rust-tests.yml:1-62

Multi-Platform Testing

The CI pipeline tests on three operating systems with multiple feature flag combinations. The test matrix is defined in .github/workflows/rust-tests.yml:14-30:

This results in 18 test jobs (3 OS × 6 feature combinations) per CI run.

Sources: .github/workflows/rust-tests.yml:14-30

Ignored Files and Directories

The .gitignore:1-10 file specifies build artifacts and local files to exclude from version control:

The /data directory is explicitly ignored for local development and experimentation without polluting the repository.

Sources: .gitignore:1-10

Dependency Management

Workspace-Level Dependencies

Shared dependencies are defined in Cargo.toml:80-112 to ensure version consistency across workspace members:

Core Dependencies:

• xxhash-rust = "0.8.15" with xxh3 and const_xxh3 features for hashing
• memmap2 = "0.9.5" for memory-mapped file access
• crc32fast = "1.4.2" for checksum validation
• dashmap = "6.1.0" for concurrent hash maps

Optional Dependencies:

• rayon = "1.10.0" (gated by parallel feature)
• arrow = "57.0.0" (gated by arrow feature, default-features disabled)

Development Dependencies:

• criterion = "0.6.0" for benchmarking
• tempfile = "3.19.0" for test isolation
• serial_test = "3.2.0" for sequential test execution
• tokio = "1.45.1" for async testing (not used in core crate)

Sources: Cargo.toml:23-47 Cargo.toml:80-112

Intra-Workspace Dependencies

Workspace crates reference each other using path dependencies with explicit versions:

This ensures workspace members can be published independently to crates.io while maintaining version consistency during development.

Sources: Cargo.toml:81-85

Caching Strategy

The CI pipeline uses GitHub Actions cache to accelerate builds. The caching configuration is defined in .github/workflows/rust-tests.yml:40-51:

Cached Directories:

• ~/.cargo/bin/ - Compiled binaries (cargo tools)
• ~/.cargo/registry/index/ - Crate registry index
• ~/.cargo/registry/cache/ - Downloaded crate archives
• ~/.cargo/git/db/ - Git dependencies
• target/ - Build artifacts

Cache Key Strategy:

• Primary key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}-${{ matrix.flags }}
• Fallback key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}-

This strategy provides OS-specific, feature-flag-specific caching with fallback to shared dependencies when feature flags change.

Sources: .github/workflows/rust-tests.yml:39-51

Publishing Configuration

All workspace crates share publishing configuration through Cargo.toml:1-9:

Individual crates inherit these values using workspace inheritance: edition.workspace = true.

Sources: Cargo.toml:1-21

Next Steps

• Building and Testing : See Building and Testing for detailed instructions on running unit tests, integration tests, concurrency tests, and benchmarks.
• CI/CD Pipeline : See CI/CD Pipeline for information about GitHub Actions workflows, matrix testing strategy, and quality checks.
• Version History and Migration : See Version History and Migration for breaking changes, migration paths, and storage format evolution.

For information about specific subsystems:

• Storage engine internals: Core Storage Engine
• Network layer: Network Layer and RPC
• Python integration: Python Integration
• Performance features: Performance Optimizations