Version History and Migration

Loading…

Version History and Migration

Relevant source files

• .github/workflows/rust-lint.yml
• CHANGELOG.md
• simd-r-drive-entry-handle/src/debug_assert_aligned.rs

This page documents the version history of simd-r-drive, tracking breaking changes, file format evolution, and providing actionable migration guides for upgrading between versions. For information about building and testing procedures, see Building and Testing. For CI/CD pipeline details, see CI/CD Pipeline.

Versioning Strategy

The project follows a modified Semantic Versioning approach while in alpha status:

Breaking Change Policy : Any change to the on-disk storage format or core API that prevents backward compatibility results in a MINOR version bump. The project maintains a strict policy of documenting all breaking changes in the changelog with migration instructions.

Sources : CHANGELOG.md:1-5

Version Timeline

The following diagram shows the evolution of major versions and their breaking changes:

Sources : CHANGELOG.md:19-82

timeline
    title "simd-r-drive Version History"
    section 0.14.x Series
        0.14.0-alpha : Introduced payload alignment
                     : Added pre-padding mechanism
                     : 16-byte default alignment
                     : BREAKING: Format incompatible with 0.13.x\nsection 0.15.x Series\n0.15.0-alpha : Increased alignment to 64 bytes\n: Added debug alignment assertions\n: BREAKING: Format incompatible with 0.14.x\n0.15.5-alpha : Arrow dependency bump to 57.0.0\n: No format changes

Breaking Changes History

Version 0.15.5-alpha (2025-10-27)

Type : Non-breaking maintenance release

Changes :

• Apache Arrow dependency updated to version 57.0.0
• Affects arrow feature flag only
• No changes to DataStore, EntryHandle, or storage format
• No migration required

Sources : CHANGELOG.md:19-22

Version 0.15.0-alpha (2025-09-25)

Type : BREAKING - On-disk format incompatible with 0.14.x

Critical Changes :

The PAYLOAD_ALIGNMENT constant in src/storage_engine/constants.rs increased from 16 bytes (log₂ = 4) to 64 bytes (log₂ = 6). This ensures safe zero-copy access for:

• SSE: 16-byte operations
• AVX2: 32-byte operations
• AVX-512: 64-byte operations
• CPU cache lines: 64 bytes on modern x86_64/ARM

Title : On-Disk Format Comparison: 0.14.x vs 0.15.x

Added Features :

• debug_assert_aligned(ptr: *const u8, align: usize) in simd-r-drive-entry-handle/src/debug_assert_aligned.rs:26-43
  • Validates pointer alignment at runtime in debug/test builds
  • No-op in release builds (zero cost abstraction)
• debug_assert_aligned_offset(off: u64) in simd-r-drive-entry-handle/src/debug_assert_aligned.rs:66-88
  • Validates file offset alignment using PAYLOAD_ALIGNMENT constant
  • Checks off.is_multiple_of(PAYLOAD_ALIGNMENT)
• Enhanced EntryHandle::as_arrow_buffer() with alignment assertions
• Enhanced EntryHandle::into_arrow_buffer() with alignment assertions

Affected Code Entities :

Technical Incompatibility Details :

When a 0.14.x reader opens a 0.15.x file:

1. Incorrect offset calculation : Reader calculates payload_start = metadata_end + (16 - offset % 16) % 16
2. Actual offset : File contains payload_start = metadata_end + (64 - offset % 64) % 64
3. Result : Reader may access pre-pad bytes or miss payload start, causing:
   • CRC32 checksum failures
   • Deserialization errors
   • Silent data corruption if payload happens to parse

Title : Read Operation Failure in Mixed Versions

Sources : CHANGELOG.md:25-52 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-88

Version 0.14.0-alpha (2025-09-08)

Type : BREAKING - On-disk format incompatible with 0.13.x

Critical Changes :

First version to introduce configurable payload alignment via pre-padding mechanism. Payloads now guaranteed to start at addresses that are multiples of PAYLOAD_ALIGNMENT.

Title : Introduction of Pre-Padding in 0.14.0-alpha

New Constants Introduced :

New Feature Flags :

• arrow: Enables Apache Arrow integration via EntryHandle::as_arrow_buffer() and into_arrow_buffer()

Added Methods (requires arrow feature):

• EntryHandle::as_arrow_buffer(&self) -> arrow_buffer::Buffer
  • Creates zero-copy Arrow buffer view over payload
  • Requires payload alignment to satisfy Arrow’s requirements
• EntryHandle::into_arrow_buffer(self) -> arrow_buffer::Buffer
  • Converts EntryHandle into Arrow buffer (consumes handle)

Pre-Padding Calculation Algorithm :

Given:
  - metadata_end: u64  // offset after EntryMetadata
  - PAYLOAD_ALIGNMENT: u64 = 16

Calculate:
  padding = (PAYLOAD_ALIGNMENT - (metadata_end % PAYLOAD_ALIGNMENT)) % PAYLOAD_ALIGNMENT
  payload_start = metadata_end + padding
  
Assert:
  payload_start % PAYLOAD_ALIGNMENT == 0

Technical Incompatibility Details :

0.13.x readers do not skip pre-padding bytes. When reading 0.14.x files:

1. Reader expects payload immediately after metadata
2. Reads pre-pad zero bytes as payload start
3. Payload deserialization fails or produces corrupt data
4. CRC32 checksum computed over wrong byte range

Sources : CHANGELOG.md:55-82

Migration Procedures

Migrating from 0.14.x to 0.15.x

Title : Migration Process: 0.14.x to 0.15.x

Detailed Migration Steps :

Step 1: Environment Setup

• Compile 0.14.x binary: git checkout v0.14.0-alpha && cargo build --release
• Compile 0.15.x binary: git checkout v0.15.0-alpha && cargo build --release
• Verify disk space: new file size ≈ old file size × 1.1 (due to increased padding)
• Backup: cp data.store data.store.backup

Step 2: Extract Data with 0.14.x Binary

Using DataStore API:

Step 3: Create New Store with 0.15.x Binary

Using DataStore::create():

Step 4: Verification Pass

Check all entries are readable and valid:

Step 5: Deployment Strategy

For single-process deployments:

1. Stop process
2. Run migration script
3. Atomic file replacement: mv new.store data.store
4. Restart process with 0.15.x binary

For multi-service deployments:

1. Deploy all reader services to 0.15.x first (backward compatible reads)
2. Stop all writer services
3. Run migration on primary data store
4. Verify migrated store
5. Deploy writer services to 0.15.x
6. Resume writes

Sources : CHANGELOG.md:43-51

Migrating from 0.13.x to 0.14.x

Step-by-Step Migration :

Technical Changes :

• 0.13.x: No pre-padding, payload immediately follows EntryMetadata
• 0.14.x: Pre-padding inserted, PAYLOAD_ALIGNMENT = 16 guaranteed

Migration Process :

1. Data extraction (using 0.13.x binary):

2. Store creation (using 0.14.x binary):

3. Verification (using 0.14.x binary):

   • Read each key and validate CRC32
   • Compare payloads with original data

Service Deployment Order :

• Stage 1 : Deploy readers with 0.14.x (can read old format)
• Stage 2 : Migrate data store to new format
• Stage 3 : Deploy writers with 0.14.x
• Rationale : Prevents writers from creating 0.14.x files before readers can handle them

Sources : CHANGELOG.md:75-81

Compatibility Matrix

The following table shows version compatibility for readers and writers:

Legend :

• ✅ Compatible: Reader can correctly parse writer’s format
• ❌ Breaks: Reader cannot correctly parse writer’s format (data corruption risk)

Key Observations :

• Newer readers are backward-compatible (can read older formats)
• Older readers cannot read newer formats (forward compatibility not guaranteed)
• Each MINOR version bump introduces a new on-disk format

Sources : CHANGELOG.md:25-82

File Format Version Detection

The storage engine does not embed a file format version marker in the data file. Version compatibility must be managed externally through:

1. Deployment tracking : Maintain records of which binary version wrote each store
2. File naming conventions : Include version in filename (e.g., data-v0.15.store)
3. Metadata sidecars : Store version information in separate metadata files
4. Service configuration : Configure services with expected format version

Limitations :

• No automatic format detection at runtime
• Mixed-version deployment requires careful orchestration
• Checksum validation alone cannot detect version mismatches

Sources : CHANGELOG.md:1-82

Upgrade Strategies

Strategy 1: Blue-Green Deployment

Advantages :

• Clean separation of old and new versions
• Easy rollback if issues discovered
• No mixed-version complexity

Disadvantages :

• Requires duplicate infrastructure during migration
• Data must be fully copied
• Higher resource cost

Strategy 2: Rolling Upgrade (Reader-First)

Advantages :

• Minimal infrastructure duplication
• Gradual rollout reduces risk
• Reader compatibility maintained throughout

Disadvantages :

• Requires maintenance window for data migration
• More complex orchestration
• Must coordinate across multiple services

Sources : CHANGELOG.md:43-51 CHANGELOG.md:75-81

Alignment Configuration Reference

Title : Code Entity Mapping: Alignment System

Alignment Constants :

Debug Assertion Functions :

Function Implementation Details :

Both assertion functions use conditional compilation to ensure zero runtime cost in release builds:

Pre-Padding Calculation (used in write path):

Given:
  metadata_end: u64
  PAYLOAD_ALIGNMENT: u64 (16 or 64)

Compute:
  padding = (PAYLOAD_ALIGNMENT - (metadata_end % PAYLOAD_ALIGNMENT)) % PAYLOAD_ALIGNMENT
  payload_start = metadata_end + padding

Invariant:
  payload_start % PAYLOAD_ALIGNMENT == 0

Version-Specific Alignment Values :

Sources : CHANGELOG.md:25-42 CHANGELOG.md:55-74 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-88

CI/CD Integration for Version Management

Title : CI/CD Pipeline Steps for Version Validation

Linting Steps (from .github/workflows/rust-lint.yml:1-44):

Pre-Release Checklist :

Before incrementing version number in Cargo.toml:

1. Run full lint suite: cargo fmt && cargo clippy --all-features
2. Test all feature combinations (see CI/CD Pipeline)
3. Update CHANGELOG.md:1-82 with changes:
   • List breaking changes under ### Breaking
   • Provide migration steps under ### Migration
   • Document affected code entities
4. Verify backward compatibility claims
5. Test migration procedure on sample data store

Testing Matrix Coverage : See CI/CD Pipeline for full matrix of OS and feature combinations tested.

Sources : .github/workflows/rust-lint.yml:1-44 CHANGELOG.md:1-16

Future Considerations

Toward 1.0 Release :

• Embed format version marker in file header
• Implement automatic format detection
• Support multi-version reader capability
• Define stable API surface with backward compatibility guarantees

Deprecation Policy (Post-1.0):

• Major version bumps for breaking changes
• Deprecated features maintained for one major version
• Clear migration paths documented before removals

Sources : CHANGELOG.md:1-5

Dismiss

Refresh this wiki

Enter email to refresh