Development Guide

Loading…

Development Guide

Relevant source files

• .github/workflows/rust-tests.yml
• .gitignore
• Cargo.toml

This document provides an overview of development practices for the SIMD R Drive repository. It covers workspace organization, build processes, feature configuration, testing strategies, and CI/CD integration. This guide is intended for contributors and maintainers working on the codebase.

For detailed instructions on building and testing specific features, see Building and Testing. For CI/CD pipeline details, see CI/CD Pipeline. For version history and migration information, see Version History and Migration.

Workspace Organization

The repository uses a Cargo workspace structure with multiple interdependent packages. The workspace is defined in the root Cargo.toml and includes both core components and experimental features.

graph TB
    subgraph "Root Workspace"
        ROOT["Cargo.toml\nworkspace root"]
end
    
    subgraph "Core Packages"
        CORE["simd-r-drive\nmain storage engine"]
HANDLE["simd-r-drive-entry-handle\nentry abstraction"]
EXT["extensions\nutility functions"]
end
    
    subgraph "Experimental Packages"
        WS_SERVER["experiments/simd-r-drive-ws-server\nWebSocket RPC server"]
WS_CLIENT["experiments/simd-r-drive-ws-client\nWebSocket RPC client"]
SERVICE_DEF["experiments/simd-r-drive-muxio-service-definition\nRPC contract"]
end
    
    subgraph "Excluded from Workspace"
        PY_DIRECT["experiments/bindings/python\nPyO3 direct bindings"]
PY_WS["experiments/bindings/python-ws-client\nPython WebSocket client"]
end
    
 
   ROOT --> CORE
 
   ROOT --> HANDLE
 
   ROOT --> EXT
 
   ROOT --> WS_SERVER
 
   ROOT --> WS_CLIENT
 
   ROOT --> SERVICE_DEF
    
 
   CORE --> HANDLE
 
   WS_SERVER --> CORE
 
   WS_SERVER --> SERVICE_DEF
 
   WS_CLIENT --> SERVICE_DEF
    
    PY_DIRECT -.excluded.-> ROOT
    PY_WS -.excluded.-> ROOT

Workspace Structure

Sources: Cargo.toml:65-77

The workspace includes six member packages Cargo.toml:66-73 and excludes two Python binding packages Cargo.toml:74-77 which use their own build systems via maturin.

Package Dependencies

Sources: Cargo.toml:23-34 Cargo.toml:80-112

Feature Flags and Configuration

The core simd-r-drive package provides three optional feature flags that enable additional capabilities or internal API access.

graph LR
    subgraph "Available Features"
        DEFAULT["default = []\nbaseline features"]
PARALLEL["parallel\nenables rayon"]
EXPOSE["expose-internal-api\nexposes internals"]
ARROW["arrow\nproxy to entry-handle"]
end
    
    subgraph "Dependencies Enabled"
        RAYON["rayon = '1.10.0'"]
ARROW_DEP["arrow = '57.0.0'"]
end
    
    subgraph "Use Cases"
        UC_DEFAULT["Standard storage operations"]
UC_PARALLEL["Parallel batch operations"]
UC_EXPOSE["Testing/benchmarking access"]
UC_ARROW["Zero-copy Arrow buffers"]
end
    
 
   DEFAULT --> UC_DEFAULT
 
   PARALLEL --> RAYON
 
   RAYON --> UC_PARALLEL
 
   EXPOSE --> UC_EXPOSE
 
   ARROW --> ARROW_DEP
 
   ARROW_DEP --> UC_ARROW

Feature Flag Overview

Sources: Cargo.toml:49-55

Feature Flag Definitions

The feature flags are defined in Cargo.toml:49-55:

The arrow feature is a proxy feature Cargo.toml:54-55 that forwards to the simd-r-drive-entry-handle package, enabling zero-copy Apache Arrow buffer integration.

Sources: Cargo.toml:49-55

Development Workflow

The typical development workflow involves building, testing, and validating changes across multiple feature combinations before committing.

graph TB
    subgraph "Local Development"
        BUILD["cargo build --workspace"]
TEST["cargo test --workspace"]
BENCH_CHECK["cargo bench --workspace --no-run"]
CHECK["cargo check --workspace"]
end
    
    subgraph "Feature Testing"
        TEST_DEFAULT["cargo test"]
TEST_PARALLEL["cargo test --features parallel"]
TEST_EXPOSE["cargo test --features expose-internal-api"]
TEST_ALL["cargo test --all-features"]
end
    
    subgraph "Code Quality"
        FMT["cargo fmt --all -- --check"]
CLIPPY["cargo clippy --workspace"]
DENY["cargo deny check"]
AUDIT["cargo audit"]
end
    
 
   BUILD --> TEST
 
   TEST --> BENCH_CHECK
    
 
   TEST --> TEST_DEFAULT
 
   TEST --> TEST_PARALLEL
 
   TEST --> TEST_EXPOSE
 
   TEST --> TEST_ALL
    
 
   TEST --> FMT
 
   TEST --> CLIPPY
 
   CLIPPY --> DENY
 
   DENY --> AUDIT

Standard Development Commands

Sources: .github/workflows/rust-tests.yml:54-61

Workspace Commands

All workspace members can be built, tested, and checked simultaneously using workspace flags:

Individual packages can be built by navigating to their directory or using the -p flag:

Sources: .github/workflows/rust-tests.yml:54-57

Testing Strategy

The repository employs multiple testing approaches to ensure correctness and performance across different configurations and platforms.

graph TB
    subgraph "Unit Tests"
        UNIT_CORE["Core storage tests\nsrc/ inline #[test]"]
UNIT_HANDLE["EntryHandle tests\nsimd-r-drive-entry-handle"]
UNIT_EXT["Extension utility tests"]
end
    
    subgraph "Integration Tests"
        INT_STORAGE["Storage operations\ntests/ directory"]
INT_CONCURRENCY["Concurrency tests\nserial_test"]
INT_RPC["RPC integration\nexperiments/"]
end
    
    subgraph "Benchmarks"
        BENCH_STORAGE["storage_benchmark\nCriterion harness:false"]
BENCH_CONTENTION["contention_benchmark\nCriterion harness:false"]
end
    
    subgraph "Python Tests"
        PY_UNIT["pytest unit tests"]
PY_README["README example tests"]
PY_INT["Integration tests"]
end
    
 
   UNIT_CORE --> INT_STORAGE
 
   UNIT_HANDLE --> INT_STORAGE
 
   UNIT_EXT --> INT_STORAGE
    
 
   INT_STORAGE --> BENCH_STORAGE
 
   INT_CONCURRENCY --> BENCH_CONTENTION
    
 
   INT_RPC --> PY_UNIT
 
   PY_UNIT --> PY_README
 
   PY_README --> PY_INT

Test Hierarchy

Sources: Cargo.toml:36-63

Test Types

The benchmark suite uses Criterion.rs with custom harness configuration Cargo.toml:57-63 to disable the default test harness and enable statistical analysis.

Sources: Cargo.toml:36-63

CI/CD Integration

The repository uses GitHub Actions for continuous integration, running tests across multiple operating systems and feature combinations.

CI Matrix Strategy

The test pipeline .github/workflows/rust-tests.yml:1-62 executes a matrix build strategy:

Sources: .github/workflows/rust-tests.yml:10-61

Matrix Configuration

The CI pipeline tests 18 total combinations (3 OS × 6 feature sets) .github/workflows/rust-tests.yml:14-30:

The fail-fast: false configuration .github/workflows/rust-tests.yml15 ensures all matrix jobs complete even if one fails, providing comprehensive test coverage feedback.

Sources: .github/workflows/rust-tests.yml:14-30

Caching Strategy

The CI pipeline caches Cargo dependencies .github/workflows/rust-tests.yml:40-51 to reduce build times:

Sources: .github/workflows/rust-tests.yml:40-51

The cache key includes the OS, Cargo.lock hash, and feature flags, ensuring separate caches for different configurations while enabling reuse across builds with identical dependencies.

Version Management

The workspace uses unified versioning across all packages Cargo.toml:1-6:

All workspace members inherit these values via workspace inheritance Cargo.toml:14-21:

This ensures consistent versioning across the entire project. For detailed version history and migration guides, see Version History and Migration.

Sources: Cargo.toml:1-21

Ignored Files and Directories

The repository excludes certain files and directories from version control .gitignore:1-11:

The /data directory .gitignore5 is specifically noted for debugging and experimentation purposes, allowing developers to maintain local test data without committing it.

Sources: .gitignore:1-11

Summary

The SIMD R Drive development environment is organized as a Cargo workspace with multiple packages, optional feature flags, comprehensive testing across platforms and configurations, and automated CI/CD validation. The workspace structure separates core functionality from experimental features, while unified versioning ensures consistency across all packages.

Key development practices include:

• Building and testing all workspace members simultaneously
• Testing multiple feature flag combinations locally before committing
• Leveraging CI/CD matrix builds for comprehensive platform coverage
• Using benchmarks with statistical analysis via Criterion.rs
• Maintaining separate build systems for Python bindings

For specific build instructions and test execution details, refer to Building and Testing. For CI/CD pipeline configuration details, refer to CI/CD Pipeline. For version history and migration guidance, refer to Version History and Migration.

Dismiss

Refresh this wiki

Enter email to refresh