CI/CD Pipeline

Loading…

CI/CD Pipeline

Relevant source files

• .github/workflows/rust-lint.yml
• .github/workflows/rust-tests.yml
• .gitignore
• CHANGELOG.md
• simd-r-drive-entry-handle/src/debug_assert_aligned.rs

Purpose and Scope

This document describes the continuous integration and continuous delivery (CI/CD) infrastructure for the SIMD R Drive project. The CI/CD pipeline is implemented using GitHub Actions workflows that automatically validate code quality, run tests across multiple platforms and feature combinations, and perform security audits on every commit and pull request.

For information about building and testing the project locally, see Building and Testing. For details about version management and breaking changes that may affect CI/CD configuration, see Version History and Migration.

Sources: .github/workflows/rust-tests.yml:1-62 .github/workflows/rust-lint.yml:1-44

Workflow Overview

The CI/CD pipeline consists of two primary GitHub Actions workflows:

Both workflows run in parallel to provide comprehensive validation before code is merged.

Sources: .github/workflows/rust-tests.yml:1-9 .github/workflows/rust-lint.yml:1-3

CI/CD Architecture

CI/CD Architecture Overview

This diagram shows the complete CI/CD pipeline structure. The test workflow creates a matrix of 18 job combinations (3 operating systems × 6 feature configurations), while the lint workflow runs a sequential series of quality checks.

Sources: .github/workflows/rust-tests.yml:10-31 .github/workflows/rust-lint.yml:5-43

Test Workflow (rust-tests.yml)

Test Matrix Configuration

The test workflow uses a GitHub Actions matrix strategy to validate the codebase across multiple dimensions:

Test Matrix Execution Flow

graph LR
    subgraph "Operating Systems"
        OS_U["ubuntu-latest"]
OS_M["macos-latest"]
OS_W["windows-latest"]
end
    
    subgraph "Feature Configurations"
        F1["flags: empty\nDefault"]
F2["flags: --no-default-features\nNo Default Features"]
F3["flags: --features parallel\nParallel"]
F4["flags: --features expose-internal-api\nExpose Internal API"]
F5["flags: --features=parallel,expose-internal-api\nParallel + Expose API"]
F6["flags: --all-features\nAll Features"]
end
    
    subgraph "Test Steps"
        CHECKOUT["actions/checkout@v4"]
RUST_INSTALL["dtolnay/rust-toolchain@stable"]
CACHE["actions/cache@v4\nCargo dependencies"]
BUILD["cargo build --workspace --all-targets"]
TEST["cargo test --workspace --all-targets"]
BENCH["cargo bench --workspace --no-run"]
end
    
 
   OS_U --> F1
 
   OS_U --> F2
 
   OS_U --> F3
 
   OS_U --> F4
 
   OS_U --> F5
 
   OS_U --> F6
    
 
   F1 --> CHECKOUT
 
   CHECKOUT --> RUST_INSTALL
 
   RUST_INSTALL --> CACHE
 
   CACHE --> BUILD
 
   BUILD --> TEST
 
   TEST --> BENCH

Each of the 18 matrix combinations follows the same execution flow: checkout code, install Rust toolchain, restore cached dependencies, build all workspace targets, run all tests, and verify benchmarks compile.

Sources: .github/workflows/rust-tests.yml:14-31 .github/workflows/rust-tests.yml:33-61

Matrix Strategy Details

Feature Combinations:

1. Default (flags: ""): Standard feature set with default features enabled
2. No Default Features (flags: "--no-default-features"): Minimal build without optional features
3. Parallel (flags: "--features parallel"): Enables parallel processing capabilities
4. Expose Internal API (flags: "--features expose-internal-api"): Exposes internal APIs for testing/experimentation
5. Parallel + Expose API (flags: "--features=parallel,expose-internal-api"): Combination of parallel and internal API features
6. All Features (flags: "--all-features"): Enables all available features including arrow integration

Sources: .github/workflows/rust-tests.yml:18-30

Test Execution Steps

The test workflow executes the following steps for each matrix combination:

1. Repository Checkout

Uses GitHub’s official checkout action to clone the repository.

Sources: .github/workflows/rust-tests.yml:33-34

2. Rust Toolchain Installation

Installs the stable Rust toolchain using the dtolnay/rust-toolchain action.

Sources: .github/workflows/rust-tests.yml:36-37

3. Dependency Caching

Caches Cargo dependencies and build artifacts to speed up subsequent runs. The cache key includes:

• Operating system (runner.os)
• Cargo.lock file hash for dependency versioning
• Matrix flags to separate caches for different feature combinations

Sources: .github/workflows/rust-tests.yml:39-51

4. Build

Builds all workspace members and all target types (lib, bin, tests, benches, examples) with the specified feature flags.

Sources: .github/workflows/rust-tests.yml:53-54

5. Test Execution

Runs all tests across the entire workspace with verbose output enabled.

Sources: .github/workflows/rust-tests.yml:56-57

6. Benchmark Compilation Check

Verifies that all benchmarks compile successfully without actually executing them. The --no-run flag ensures benchmarks are only compiled, not executed, which would be time-consuming in CI.

Sources: .github/workflows/rust-tests.yml:59-61

graph TB
    TRIGGER["Push or Pull Request"]
subgraph "Setup Steps"
        CHECKOUT["actions/checkout@v3"]
RUST_INSTALL["dtolnay/rust-toolchain@stable"]
COMPONENTS["rustup component add\nrustfmt clippy"]
TOOLS["cargo install\ncargo-deny cargo-audit"]
end
    
    subgraph "Quality Checks"
        FMT["cargo fmt --all -- --check\nVerify formatting"]
CLIPPY["cargo clippy --workspace\n--all-targets --all-features\nLint warnings"]
DOC["RUSTDOCFLAGS=-D warnings\ncargo doc --workspace\nDocumentation quality"]
end
    
    subgraph "Security Checks"
        DENY["cargo deny check\nLicense/dependency policy"]
AUDIT["cargo audit\nKnown vulnerabilities"]
end
    
 
   TRIGGER --> CHECKOUT
 
   CHECKOUT --> RUST_INSTALL
 
   RUST_INSTALL --> COMPONENTS
 
   COMPONENTS --> TOOLS
 
   TOOLS --> FMT
 
   FMT --> CLIPPY
 
   CLIPPY --> DOC
 
   DOC --> DENY
 
   DENY --> AUDIT

Lint Workflow (rust-lint.yml)

The lint workflow performs comprehensive code quality and security checks on a single platform (Ubuntu):

Lint Workflow Execution Graph

The lint workflow runs sequentially through setup, quality checks, and security audits. All checks must pass for the workflow to succeed.

Sources: .github/workflows/rust-lint.yml:1-44

Lint Steps Breakdown

1. Component Installation Workaround

This step addresses a GitHub Actions environment issue where rustfmt and clippy may not be automatically available. The workflow explicitly installs these components to ensure consistent behavior.

Sources: .github/workflows/rust-lint.yml:13-18

2. Tool Installation

Installs third-party Cargo subcommands:

• cargo-deny : Validates dependency licenses, sources, and advisories against policy rules
• cargo-audit : Checks dependencies against the RustSec Advisory Database for known security vulnerabilities

Sources: .github/workflows/rust-lint.yml:20-23

3. Format Verification

Verifies that all code follows Rust’s standard formatting conventions using rustfmt. The --check flag ensures the command fails if any files need reformatting without modifying them.

Sources: .github/workflows/rust-lint.yml:25-27

4. Clippy Linting

Runs Clippy, Rust’s official linter, with the following configuration:

• --workspace: Lint all workspace members
• --all-targets: Lint library, binaries, tests, benchmarks, and examples
• --all-features: Enable all features when linting
• -D warnings: Treat all warnings as errors, failing the build if any issues are found

Sources: .github/workflows/rust-lint.yml:29-31

5. Documentation Verification

Generates and validates documentation with strict checks:

• RUSTDOCFLAGS="-D warnings": Treats documentation warnings as errors
• --workspace: Document all workspace members
• --no-deps: Only document workspace crates, not dependencies
• --document-private-items: Include documentation for private items to ensure comprehensive coverage

Sources: .github/workflows/rust-lint.yml:33-35

6. Dependency Policy Enforcement

Validates dependencies against policy rules defined in a deny.toml configuration file (if present). This checks:

• License compatibility
• Banned/allowed crates
• Advisory database for security issues
• Source verification (crates.io, git repositories)

Sources: .github/workflows/rust-lint.yml:37-39

7. Security Audit

Scans Cargo.lock against the RustSec Advisory Database to identify dependencies with known security vulnerabilities. This provides early warning of security issues in the dependency tree.

Sources: .github/workflows/rust-lint.yml:41-43

Caching Strategy

The test workflow implements an intelligent caching strategy to reduce build times:

Cache Key Structure

${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}-${{ matrix.flags }}

Components:

1. runner.os : Operating system (Linux, macOS, Windows)
2. hashFiles('**/Cargo.lock') : Hash of all Cargo.lock files in the repository
3. matrix.flags : Feature flag combination being tested

This multi-dimensional key ensures:

• Different operating systems maintain separate caches
• Cache invalidation occurs when dependencies change
• Different feature combinations don’t share potentially incompatible build artifacts

Cached Directories

Cache Restore Fallback

If an exact cache match is not found, the workflow attempts to restore a cache with a partial key match (same OS and Cargo.lock hash but different flags). This provides some benefit even when testing different feature combinations.

Sources: .github/workflows/rust-tests.yml:39-51

Workflow Triggers and Conditions

Test Workflow Triggers

The test workflow (rust-tests.yml) activates on:

This configuration ensures:

• All changes to the main branch are tested
• Release tags trigger comprehensive validation
• Pull requests are validated before merge

Sources: .github/workflows/rust-tests.yml:3-8

Lint Workflow Triggers

The lint workflow (rust-lint.yml) activates on:

This broader trigger ensures code quality checks run on all development branches, not just main.

Sources: .github/workflows/rust-lint.yml3

Integration with Repository Configuration

Ignored Files and Directories

The CI/CD workflows respect the repository’s .gitignore configuration:

Key exclusions:

• **/target : Build artifacts (handled by caching)
• *.bin : Binary data files created by storage engine tests
• /data : Debugging and experimentation directory
• .cargo/config.toml : Local Cargo configuration overrides

Sources: .gitignore:1-11

Alignment Changes and CI Impact

The CI/CD pipeline automatically validates alignment-sensitive code across all platforms. Version 0.15.0 introduced a breaking change increasing PAYLOAD_ALIGNMENT from 16 to 64 bytes, which the CI validates through:

1. Debug assertions in simd-r-drive-entry-handle/src/debug_assert_aligned.rs:

   • debug_assert_aligned(): Validates pointer alignment
   • debug_assert_aligned_offset(): Validates file offset alignment

2. Cross-platform testing ensures alignment works correctly on:

   • x86_64 (AVX2 256-bit)
   • ARM (NEON 128-bit)
   • Both 32-bit and 64-bit architectures

The debug assertions compile to no-ops in release builds but provide comprehensive validation in CI test runs:

Sources: simd-r-drive-entry-handle/src/debug_assert_aligned.rs:26-43 CHANGELOG.md:25-51

Failure Modes and Debugging

Matrix Job Independence

The test workflow sets fail-fast: false, which means:

• If one OS/feature combination fails, others continue to completion
• Developers can see all failure patterns at once
• Useful for identifying platform-specific or feature-specific issues

Sources: .github/workflows/rust-tests.yml15

Common Failure Scenarios

Debugging Failed Matrix Jobs

To reproduce a specific matrix job failure locally:

1. Identify the failing OS and feature combination from the GitHub Actions log

2. Use the exact command shown in the workflow:

3. For cross-platform issues, use Docker or a VM matching the CI environment

Sources: .github/workflows/rust-tests.yml:53-57

CI/CD Pipeline Maintenance

Adding New Feature Flags

To add a new feature flag to the test matrix:

1. Add a new entry to the matrix.include section in rust-tests.yml:

2. Consider whether the feature should be included in the “All Features” test

3. Update documentation if the feature has platform-specific behavior

Sources: .github/workflows/rust-tests.yml:18-30

Updating Toolchain Versions

Both workflows use GitHub Actions to manage Rust toolchain versions:

• Stable toolchain : dtolnay/rust-toolchain@stable automatically tracks the latest stable release
• Pinning a specific version : Replace @stable with @1.XX.X if needed
• Nightly features : Change @stable to @nightly (may require additional stability considerations)

Sources: .github/workflows/rust-tests.yml:36-37 .github/workflows/rust-lint.yml11

Monitoring CI Performance

Key metrics to monitor:

• Cache hit rate : Check if Cargo caches are being restored effectively
• Build time trends : Monitor for increases that might indicate dependency bloat
• Test execution time : Identify slow tests that could benefit from optimization
• Matrix job duration : Ensure no single OS/feature combination becomes a bottleneck

GitHub Actions provides timing information for each step and job in the workflow run logs.

Dismiss

Refresh this wiki

Enter email to refresh