Development Guide

Loading…

Development Guide

Relevant source files

• .github/workflows/rust-tests.yml
• .gitignore
• Cargo.toml

This document provides an overview of the development process for SIMD R Drive, covering workspace structure, feature flags, build commands, and development workflows. This page serves as an entry point for contributors and developers working on the codebase.

For detailed instructions on building and running tests, see Building and Testing. For information about the CI/CD pipeline and automated testing, see CI/CD Pipeline. For version history and migration guides, see Version History and Migration.

Prerequisites

SIMD R Drive requires:

• Rust : Edition 2024 (as specified in Cargo.toml4)
• Cargo : Workspace-aware build system
• Platform Support : Linux, macOS, Windows (tested via CI matrix)

Optional tools for specific workflows:

• Python 3.10-3.13 : For Python bindings development (see Python Integration)
• Maturin : For building Python wheels (see Building Python Bindings)
• Criterion : For benchmarking (included as dev-dependency)

Workspace Structure

SIMD R Drive uses a Cargo workspace architecture with multiple crates organized by function. The workspace is defined in Cargo.toml:65-78 with specific members included and Python bindings excluded from the main workspace due to Maturin’s incompatible build system.

graph TB
    Root["simd-r-drive\nname: 'simd-r-drive'\npath: '.'"]
EntryHandle["simd-r-drive-entry-handle\npath: './simd-r-drive-entry-handle'"]
Extensions["simd-r-drive-extensions\npath: './extensions'"]
subgraph experiments["experiments/ (workspace members)"]
WSServer["simd-r-drive-ws-server\npath: 'experiments/simd-r-drive-ws-server'"]
WSClient["simd-r-drive-ws-client\npath: 'experiments/simd-r-drive-ws-client'"]
ServiceDef["simd-r-drive-muxio-service-definition\npath: 'experiments/simd-r-drive-muxio-service-definition'"]
end
    
    subgraph excluded["excluded (maturin-managed)"]
PyBindings["simd-r-drive-py\npath: 'experiments/bindings/python'"]
PyWSClient["simd-r-drive-ws-client-py\npath: 'experiments/bindings/python-ws-client'"]
end
    
 
   Root --> EntryHandle
 
   WSServer --> Root
 
   WSServer --> ServiceDef
 
   WSClient --> Root
 
   WSClient --> ServiceDef
 
   PyWSClient --> WSClient
 
   PyBindings --> Root
    
 
   Extensions --> Root

Workspace Dependency Graph

Workspace Configuration:

• [workspace] block: Cargo.toml:65-78
• members: Lines 66-73
• exclude: Lines 74-77
• resolver = "2": Line 78 (uses v2 feature resolver)

Sources: Cargo.toml:65-78 Cargo.toml:11-21

Workspace Member Descriptions

Exclusion Rationale: Python binding crates use Maturin’s PyO3 build system, which conflicts with Cargo’s workspace resolver. These crates must be built separately using maturin build or maturin develop. The exclusion is specified in Cargo.toml:74-77

Sources: Cargo.toml:11-21 Cargo.toml:65-78 Cargo.toml:81-85

Feature Flags

The core simd-r-drive crate exposes multiple feature flags that enable optional functionality. Features are defined in Cargo.toml:49-55

Feature Configuration and Dependencies

Sources: Cargo.toml:49-55 Cargo.toml:23-34

Feature Flag Reference

The arrow feature is a proxy feature: enabling simd-r-drive/arrow automatically enables simd-r-drive-entry-handle/arrow, allowing users to enable Arrow support without knowing the internal crate structure.

Sources: Cargo.toml:49-55 Cargo.toml30 Cargo.toml:54-55

Common Development Commands

Building

Build Target Types:

• lib - Core library (src/lib.rs)
• bin - CLI executable (src/main.rs)
• test - Unit and integration tests (tests/)
• bench - Criterion benchmarks (benches/)

Testing

Dev Dependencies for Testing:

• tempfile = "3.19.0" - Temporary file creation (Cargo.toml45)
• serial_test = "3.2.0" - Sequential test execution (Cargo.toml44)
• rand = "0.9.0" - Random data generation (Cargo.toml41)

Benchmarking

The crate includes two Criterion-based benchmarks defined in Cargo.toml:57-63:

Benchmark Configuration:

• harness = false (Cargo.toml59 Cargo.toml63) - Use Criterion’s harness instead of libtest
• Compile without running: cargo bench --no-run (.github/workflows/rust-tests.yml:60-61)

Sources: Cargo.toml:36-47 Cargo.toml:57-63 .github/workflows/rust-tests.yml:53-61

Development Workflow

Standard Development Cycle

Sources: .github/workflows/rust-tests.yml:1-62

Multi-Platform Testing

The CI pipeline tests on three operating systems with multiple feature flag combinations. The test matrix is defined in .github/workflows/rust-tests.yml:14-30:

This results in 18 test jobs (3 OS × 6 feature combinations) per CI run.

Build and Test Targets Flow

Sources: .github/workflows/rust-tests.yml:14-30 .github/workflows/rust-tests.yml:53-61

Ignored Files and Directories

The .gitignore:1-10 file specifies build artifacts and local files to exclude from version control:

The /data directory is explicitly ignored for local development and experimentation without polluting the repository.

Sources: .gitignore:1-10

Dependency Management

Workspace-Level Dependencies

Shared dependencies are defined in [workspace.dependencies] block (Cargo.toml:80-112) to ensure version consistency across workspace members:

Core Storage Dependencies:

Optional Feature Dependencies:

Development-Only Dependencies:

Network Stack Dependencies (Muxio Framework):

Sources: Cargo.toml:23-34 Cargo.toml:36-47 Cargo.toml:80-112

Intra-Workspace Dependencies

Workspace crates reference each other using path dependencies with explicit versions:

This ensures workspace members can be published independently to crates.io while maintaining version consistency during development.

Sources: Cargo.toml:81-85

CI/CD Caching Strategy

The CI pipeline uses GitHub Actions cache to accelerate builds. The caching configuration is defined in .github/workflows/rust-tests.yml:40-51

Cached Directory Structure

Cache Key Components:

1. ${{ runner.os }} - Platform-specific cache (ubuntu/macos/windows)
2. ${{ hashFiles('**/Cargo.lock') }} - Dependency version fingerprint
3. ${{ matrix.flags }} - Feature flag combination fingerprint

Fallback Strategy: If exact match fails, falls back to same OS + same Cargo.lock, recompiling only feature-specific code.

Sources: .github/workflows/rust-tests.yml:40-51

Publishing Configuration

All workspace crates share publishing configuration through Cargo.toml:1-9:

Individual crates inherit these values using workspace inheritance: edition.workspace = true.

Sources: Cargo.toml:1-21

Next Steps

• Building and Testing : See Building and Testing for detailed instructions on running unit tests, integration tests, concurrency tests, and benchmarks.
• CI/CD Pipeline : See CI/CD Pipeline for information about GitHub Actions workflows, matrix testing strategy, and quality checks.
• Version History and Migration : See Version History and Migration for breaking changes, migration paths, and storage format evolution.

For information about specific subsystems:

• Storage engine internals: Core Storage Engine
• Network layer: Network Layer and RPC
• Python integration: Python Integration
• Performance features: Performance Optimizations

Dismiss

Refresh this wiki

Enter email to refresh