Version History and Migration

Relevant source files

• .github/workflows/rust-lint.yml
• CHANGELOG.md
• README.md
• simd-r-drive-entry-handle/src/debug_assert_aligned.rs

This page documents the version history of SIMD R Drive, detailing breaking changes, format evolutions, and migration procedures required when upgrading between versions. The primary focus is on storage format compatibility and the alignment strategy evolution that occurred between versions 0.13.x and 0.15.x.

For information about the current storage architecture and entry structure, see Entry Structure and Metadata. For details on the current alignment strategy and performance characteristics, see Payload Alignment and Cache Efficiency.

Version Overview

SIMD R Drive follows a loosely semantic versioning scheme and is currently in alpha (0.x.x-alpha). Breaking changes in the storage format have occurred as the alignment strategy evolved to optimize zero-copy access and SIMD performance.

Sources: CHANGELOG.md:1-82

Storage Format Evolution

The most significant changes in SIMD R Drive's history involve the evolution of payload alignment strategy. These changes were driven by the need to optimize zero-copy access, SIMD operations, and CPU cache efficiency.

timeline
    title "Storage Format Alignment Evolution"
    section Pre-0.14
        No Alignment : Payloads written sequentially\n: No pre-padding\n: Variable starting addresses\nsection 0.14.0-alpha\n16-byte Alignment : Introduced PAYLOAD_ALIGNMENT constant
                          : Added pre-pad bytes (0-15)
                          : SSE-friendly boundaries
    section 0.15.0-alpha
        64-byte Alignment : Increased to cache-line size
                          : Pre-pad bytes (0-63)
                          : AVX-512 and cacheline optimized

Timeline: Alignment Strategy Evolution

Format Comparison: Pre-0.14 vs 0.14+ vs 0.15+

Sources: CHANGELOG.md:55-82 CHANGELOG.md:25-52 README.md:51-59

Version Compatibility Matrix

Understanding compatibility between readers and writers is critical when managing SIMD R Drive deployments across multiple services or upgrading existing storage files.

Reader/Writer Compatibility

Legend:

• ✅ Compatible: Reader can correctly parse the storage format
• ❌ Incompatible: Reader will misinterpret pre-pad bytes as payload data
• • Newer readers may attempt to parse old formats but will fail validation

Incompatibility Root Cause

The incompatibility arises because:

1. Pre-0.14 readers expect prev_offset to point directly to payload start
2. 0.14+ formats insert 0-15 or 0-63 pre-pad bytes before the payload
3. Old readers interpret these zero bytes as part of the payload, corrupting data

Sources: CHANGELOG.md:55-60 CHANGELOG.md:27-31

Migration Procedures

Migrating from 0.13.x or earlier to 0.14.x+

When upgrading from pre-alignment versions to any aligned version (0.14.x or 0.15.x), you must regenerate storage files.

flowchart TD
    Start["Start: Have 0.13.x store"]
ReadOld["Read all entries\nwith 0.13.x binary"]
CreateNew["Create new store\nwith 0.14.x+ binary"]
WriteNew["Write entries to\nnew store"]
Verify["Verify new store\nintegrity"]
Replace["Replace old file\nwith new file"]
Start --> ReadOld
 
   ReadOld --> CreateNew
 
   CreateNew --> WriteNew
 
   WriteNew --> Verify
 
   Verify -->|Success| Replace
 
   Verify -->|Failure| CreateNew
    
    MultiService{"Multi-service\ndeployment?"}
UpgradeReaders["Deploy reader upgrades\nto all services first"]
UpgradeWriters["Deploy writer upgrades\nafter readers"]
Replace --> MultiService
 
   MultiService -->|Yes| UpgradeReaders
 
   MultiService -->|No| Done["Migration Complete"]
UpgradeReaders --> UpgradeWriters
 
   UpgradeWriters --> Done

Migration Workflow

Step-by-Step Migration: 0.13.x → 0.14.x/0.15.x

1. Prepare the old binary:

2. Build the new binary:

3. Extract data from old store:

4. Create new store with new binary:

5. Verify integrity:

6. Replace old file:

Sources: CHANGELOG.md:75-82 CHANGELOG.md:43-52

Migrating from 0.14.x to 0.15.x

The migration from 0.14.x (16-byte alignment) to 0.15.x (64-byte alignment) follows the same regeneration process, as the alignment constants are incompatible.

Alignment Constant Changes

The key difference between versions:

Migration Steps: 0.14.x → 0.15.x

Follow the same procedure as 0.13.x → 0.14.x migration:

1. Read entries with 0.14.x binary
2. Write to new store with 0.15.x binary
3. Verify new store integrity
4. Deploy reader upgrades before writer upgrades in multi-service environments

Rationale for 64-byte alignment:

• Matches typical CPU cache line size
• Enables full-speed AVX-512 operations
• Provides safety margin for future SIMD extensions
• Optimizes for zero-copy typed slices (see Payload Alignment and Cache Efficiency)

Sources: CHANGELOG.md:25-52 README.md:51-59 CHANGELOG.md39

Version-Specific Details

Version 0.15.5-alpha (2025-10-27)

Type: Non-breaking maintenance release

Changes:

• Updated Apache Arrow dependency from version 56.x to 57.0.0
• No storage format changes
• No API changes
• No migration required

Compatibility: Fully compatible with 0.15.0-alpha through 0.15.4-alpha

Sources: CHANGELOG.md:19-22

Version 0.15.0-alpha (2025-09-25)

Type: Breaking storage format change

Primary Change: Increased default PAYLOAD_ALIGNMENT from 16 bytes to 64 bytes

Motivation:

• Cache-line alignment for optimal memory access
• Support for wider SIMD instructions (AVX-512)
• Improved zero-copy performance for typed slices
• Future-proofing for emerging CPU architectures

Breaking Changes:

1. Storage files created with 0.15.x cannot be read by 0.14.x or earlier
2. Pre-pad calculation changed: pad = (64 - (prev_tail % 64)) & 63
3. Pointer and offset alignment assertions added in debug/test builds

New Features:

• Debug-only alignment assertions via debug_assert_aligned() function
• Offset-level alignment validation via debug_assert_aligned_offset()
• Enhanced documentation for alignment guarantees

Code Changes:

• simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-89 - New alignment assertion functions
• Configuration via PAYLOAD_ALIGNMENT constant (64 bytes)

Migration Path: See [Migrating from 0.14.x to 0.15.x](https://github.com/jzombie/rust-simd-r-drive/blob/487b7b98/Migrating from 0.14.x to 0.15.x) above.

Sources: CHANGELOG.md:25-52 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-89

Version 0.14.0-alpha (2025-09-08)

Type: Breaking storage format change

Primary Change: Introduced fixed payload alignment with pre-padding

Motivation:

• Enable zero-copy typed views (e.g., &[u16], &[u32], &[u64])
• Guarantee SIMD-safe alignment for AVX2/NEON operations
• Improve CPU cache utilization

Breaking Changes:

1. New on-disk layout with optional pre-pad bytes (0-15 bytes)
2. Files written by 0.14.0-alpha are incompatible with ≤0.13.x readers
3. Pre-pad calculation: pad = (16 - (prev_tail % 16)) & 15

New Features:

• Experimental arrow feature flag
• EntryHandle::as_arrow_buffer() method for Apache Arrow integration
• EntryHandle::into_arrow_buffer() method for zero-copy Arrow buffers
• Configurable alignment via PAYLOAD_ALIGN_LOG2 and PAYLOAD_ALIGNMENT constants

Storage Layout:

[Pre-Pad: 0-15 bytes] [Payload: variable] [Metadata: 20 bytes]

Configuration:

• Alignment controlled in src/storage_engine/constants.rs
• PAYLOAD_ALIGN_LOG2 = 4 (2^4 = 16 bytes)
• PAYLOAD_ALIGNMENT = 1 << PAYLOAD_ALIGN_LOG2

Migration Path: See [Migrating from 0.13.x or earlier to 0.14.x+](https://github.com/jzombie/rust-simd-r-drive/blob/487b7b98/Migrating from 0.13.x or earlier to 0.14.x+) above.

Sources: CHANGELOG.md:55-82 README.md:112-138

graph TB
    subgraph Phase1["Phase 1: Reader Upgrades"]
R1["Service A\n(Reader)"]
R2["Service B\n(Reader)"]
R3["Service C\n(Reader)"]
end
    
    subgraph Phase2["Phase 2: Writer Upgrades"]
W1["Service D\n(Writer)"]
W2["Service E\n(Writer)"]
end
    
    subgraph Phase3["Phase 3: Storage Migration"]
M1["Migrate shared\nstorage files"]
M2["Switch to new\nformat"]
end
    
    Start["Start: All services\non old version"]
Start --> Phase1
 
   Phase1 -->|Verify all readers operational| Phase2
 
   Phase2 -->|Verify all writers operational| Phase3
 
   Phase3 --> Complete["Complete: All services\non new version"]
Note1["Readers can handle\nold format during\ntransition"]
Note2["Writers continue\nproducing old format\nuntil migration"]
Phase1 -.-> Note1
 
   Phase2 -.-> Note2

Multi-Service Deployment Strategy

When upgrading SIMD R Drive across multiple services that share storage files or communicate via the WebSocket RPC interface, a staged deployment approach prevents compatibility issues.

Recommended Upgrade Order

Deployment Steps

1. Phase 1 - Upgrade all readers:

   • Deploy new binaries to services that only read from storage
   • Deploy new WebSocket clients (Python or Rust)
   • Verify that services can still read existing (old format) data
   • Monitor for errors or compatibility issues

2. Phase 2 - Upgrade writers (optional hold):

   • Deploy new binaries to services that write to storage
   • Important: Writers should continue using the old format during this phase
   • Verify writer functionality with old format

3. Phase 3 - Migrate storage and switch format:

   • Use migration procedure to regenerate storage files
   • Update writer configuration to use new format
   • Monitor write operations for alignment correctness

4. Rollback strategy:

   • Keep backups of old format files until migration is verified
   • Keep old binaries available for emergency rollback
   • Test rollback procedure before starting migration

Sources: CHANGELOG.md:43-52 CHANGELOG.md:75-82

graph TB
    subgraph DebugMode["Debug/Test Mode"]
Call1["debug_assert_aligned(ptr, align)"]
Call2["debug_assert_aligned_offset(offset)"]
Check1["Verify ptr % align == 0"]
Check2["Verify offset % PAYLOAD_ALIGNMENT == 0"]
Pass1["Assertion passes"]
Fail1["Panic with alignment error"]
Call1 --> Check1
 
       Call2 --> Check2
 
       Check1 -->|Aligned| Pass1
 
       Check1 -->|Misaligned| Fail1
 
       Check2 -->|Aligned| Pass1
 
       Check2 -->|Misaligned| Fail1
    end
    
    subgraph ReleaseMode["Release/Bench Mode"]
Call3["debug_assert_aligned(ptr, align)"]
Call4["debug_assert_aligned_offset(offset)"]
NoOp["No-op (zero cost)"]
Call3 --> NoOp
 
       Call4 --> NoOp
    end

Alignment Validation and Debugging

Starting with version 0.15.0, SIMD R Drive includes debug-time alignment assertions to catch alignment violations early in development.

Debug Assertion Functions

Implementation Details

The assertion functions are exported from simd-r-drive-entry-handle and can be called from any crate:

• debug_assert_aligned(ptr: *const u8, align: usize)

  • Validates pointer alignment in debug/test builds
  • Used for verifying Arrow buffer compatibility
  • Zero cost in release builds

• debug_assert_aligned_offset(offset: u64)

  • Validates file offset alignment in debug/test builds
  • Checks against PAYLOAD_ALIGNMENT constant
  • Zero cost in release builds

Implementation approach:

• Functions always exist (stable symbol for cross-crate calls)
• Body is gated by #[cfg(any(test, debug_assertions))]
• Release/bench builds compile to no-ops with no runtime cost

Sources: simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-89 CHANGELOG.md:33-42

Future Format Stability

Current Status

SIMD R Drive is in alpha (0.x.x-alpha) and storage format stability is not guaranteed. Each alignment change requires storage regeneration.

Path to Stability

Future work may include:

1. Format version headers - Store format version in file header for automatic detection
2. Multi-format readers - Support reading multiple alignment versions in a single binary
3. In-place migration - Tools to migrate storage without full regeneration
4. Stable 1.0 format - Commitment to format stability after 1.0 release

Current Recommendations

Until format stability is achieved:

• Plan for regeneration: Assume storage files will need regeneration on major updates
• Version pinning: Pin specific SIMD R Drive versions in production deployments
• Test thoroughly: Validate migration procedures in staging environments
• Keep backups: Maintain backups of storage files before migrations
• Monitor changelogs: Review CHANGELOG.md for breaking changes before upgrading

Sources: CHANGELOG.md:1-82 README.md:1-10

Summary: Key Takeaways

For specific migration procedures, refer to the [Migration Procedures](https://github.com/jzombie/rust-simd-r-drive/blob/487b7b98/Migration Procedures) section above. For understanding the current storage format, see Entry Structure and Metadata.

Sources: CHANGELOG.md:1-82 README.md:1-282