Building and Testing

Loading…

Building and Testing

Relevant source files

• .github/workflows/rust-tests.yml
• .gitignore
• Cargo.toml
• benches/storage_benchmark.rs
• src/main.rs
• src/utils/format_bytes.rs
• tests/concurrency_tests.rs

This document covers building the Rust workspace, running tests, and using feature flags for the simd-r-drive storage engine. It provides instructions for building individual crates, executing test suites, and running benchmarks.

For information about CI/CD automation, see CI/CD Pipeline. For building Python bindings specifically, see Building Python Bindings.

Prerequisites

The project requires:

• Rust toolchain (edition 2024, as specified in Cargo.toml4)
• Cargo (workspace resolver version 2)
• Platform support : Linux, macOS, Windows (all tested in CI)

Optional dependencies for specific features:

• Rayon (for parallel feature)
• Apache Arrow libraries (for arrow feature)
• Tokio runtime (for async tests and network experiments)

Sources: Cargo.toml:1-113

Workspace Structure

The project is organized as a Cargo workspace with multiple member crates:

Workspace Members (built together with --workspace):

• simd-r-drive - Core storage engine
• simd-r-drive-entry-handle - Entry data structures
• extensions - Helper utilities
• Network experiment crates

Excluded Members (must be built separately):

• Python binding directories use separate build systems (maturin/PyO3)

Sources: Cargo.toml:65-78

Building the Core Library

Basic Build

Build the core library with default features:

Build in release mode for production use:

Building the Entire Workspace

Build all workspace members:

Build all targets (lib, bins, tests, benches):

Sources: Cargo.toml:11-21 .github/workflows/rust-tests.yml:53-54

Feature Flags

The project uses Cargo feature flags to enable optional functionality:

Building with Features

Build with specific features:

Sources: Cargo.toml:49-55 Cargo.toml30

Running Tests

Test Suite Organization

The project includes multiple test types:

Basic Test Execution

Run all tests in the workspace:

Run tests with verbose output:

Run tests for a specific crate:

Sources: .github/workflows/rust-tests.yml:56-57

Testing with Different Feature Combinations

The CI matrix tests 6 feature combinations across 3 operating systems:

Run tests matching CI configurations:

Sources: .github/workflows/rust-tests.yml:14-31

Concurrency Tests

The concurrency test suite validates thread-safety under high contention:

Concurrency Test Cases

The test suite includes three primary concurrency tests:

Running Concurrency Tests

The tests use #[serial] annotation to prevent parallel execution (since they test shared state):

Test Requirements:

• Uses serial_test::serial to serialize test execution tests/concurrency_tests.rs1
• Requires Tokio multi-threaded runtime tests/concurrency_tests.rs14
• Uses tempfile for isolated storage instances tests/concurrency_tests.rs9

Sources: tests/concurrency_tests.rs:1-230

Benchmarking

Benchmark Suite

The project includes two benchmark suites:

Benchmark Configuration:

Running Benchmarks

Compile benchmarks without running:

Run all benchmarks:

Run specific benchmark:

Storage Benchmark Operations

The storage_benchmark suite measures four operation types:

Key Constants:

• ENTRY_SIZE = 8 bytes benches/storage_benchmark.rs20
• WRITE_BATCH_SIZE = 1024 benches/storage_benchmark.rs21
• READ_BATCH_SIZE = 1024 benches/storage_benchmark.rs22
• NUM_ENTRIES = 1_000_000 benches/storage_benchmark.rs24

Sources: Cargo.toml:57-63 benches/storage_benchmark.rs:1-234

Building the CLI Binary

The project includes a command-line interface:

Building the CLI

Build the CLI binary:

Run the CLI directly:

The CLI binary will be located at:

• Debug: target/debug/simd-r-drive (or .exe on Windows)
• Release: target/release/simd-r-drive

Sources: src/main.rs:1-12 Cargo.toml25

Cross-Platform Considerations

Platform-Specific Testing

The CI pipeline tests on three operating systems:

Known Platform Differences

1. File Locking : The BufWriter<File> uses platform-specific file locking Cargo.toml:23-24
2. Memory Mapping : memmap2 handles platform differences in mmap APIs Cargo.toml29
3. Path Separators : Tests use tempfile for cross-platform temporary paths Cargo.toml45

Running Platform-Specific Tests

Test on your local platform:

Sources: .github/workflows/rust-tests.yml:14-18

Development Workflow

Quick Test Commands

Common development commands:

Caching Dependencies

The CI uses cargo caching to speed up builds:

Local development automatically uses Cargo’s built-in caching in these directories.

Sources: .github/workflows/rust-tests.yml:40-51

Test Data Management

Test isolation strategies:

1. Temporary Files : Use tempfile::tempdir() for isolated test storage tests/concurrency_tests.rs37
2. Serial Execution : Use #[serial] for tests sharing state tests/concurrency_tests.rs15
3. Cleanup : Temporary files are automatically removed on drop tests/concurrency_tests.rs37

Sources: tests/concurrency_tests.rs:37-40 .gitignore:1-11

Test Debugging

Enabling Trace Logs

The CLI and tests use tracing for structured logging:

The main CLI initializes tracing with info level by default src/main.rs7

Test Output Verbosity

Capture test output:

Sources: src/main.rs7 Cargo.toml:32-33

Verifying Build Artifacts

Checking Benchmark Compilation

The CI ensures benchmarks compile even though they’re not run in CI:

This is important because benchmark compilation uses different code paths (criterion harness disabled) Cargo.toml59

Build All Targets

Verify all code compiles:

This builds:

• Library crates (--lib)
• Binary targets (--bins)
• Test targets (--tests)
• Benchmark targets (--benches)
• Example code (--examples)

Sources: .github/workflows/rust-tests.yml:53-61

Document Sources:

• Cargo.toml:1-113
• .github/workflows/rust-tests.yml:1-62
• tests/concurrency_tests.rs:1-230
• benches/storage_benchmark.rs:1-234
• src/main.rs:1-12
• .gitignore:1-11
• src/utils/format_bytes.rs:1-31

Dismiss

Refresh this wiki

Enter email to refresh