Version History and Migration

Loading…

Version History and Migration

Relevant source files

• .github/workflows/rust-lint.yml
• CHANGELOG.md
• simd-r-drive-entry-handle/src/debug_assert_aligned.rs

This page documents the version history of simd-r-drive, tracking breaking changes, file format evolution, and providing actionable migration guides for upgrading between versions. For information about building and testing procedures, see Building and Testing. For CI/CD pipeline details, see CI/CD Pipeline.

Versioning Strategy

The project follows a modified Semantic Versioning approach while in alpha status:

Breaking Change Policy : Any change to the on-disk storage format or core API that prevents backward compatibility results in a MINOR version bump. The project maintains a strict policy of documenting all breaking changes in the changelog with migration instructions.

Sources : CHANGELOG.md:1-5

Version Timeline

The following diagram shows the evolution of major versions and their breaking changes:

Sources : CHANGELOG.md:19-82

timeline
    title "simd-r-drive Version History"
    section 0.14.x Series
        0.14.0-alpha : Introduced payload alignment
                     : Added pre-padding mechanism
                     : 16-byte default alignment
                     : BREAKING: Format incompatible with 0.13.x\nsection 0.15.x Series\n0.15.0-alpha : Increased alignment to 64 bytes\n: Added debug alignment assertions\n: BREAKING: Format incompatible with 0.14.x\n0.15.5-alpha : Arrow dependency bump to 57.0.0\n: No format changes

Breaking Changes History

Version 0.15.5-alpha (2025-10-27)

Type : Non-breaking maintenance release

Changes :

• Apache Arrow dependency updated from previous version to 57.0.0
• No functional changes to storage engine
• No migration required

Sources : CHANGELOG.md:19-22

Version 0.15.0-alpha (2025-09-25)

Type : BREAKING - On-disk format incompatible with 0.14.x

Critical Changes :

graph LR
    subgraph "0.14.x Format"
 
       A["Metadata"] --> B["Pre-pad\n(0-15 bytes)"]
B --> C["Payload\n(16-byte aligned)"]
end
    
    subgraph "0.15.x Format"
 
       D["Metadata"] --> E["Pre-pad\n(0-63 bytes)"]
E --> F["Payload\n(64-byte aligned)"]
end
    
 
   G["Incompatibility:\nDifferent alignment\nassumptions"] -.-> A
 
   G -.-> D

The default PAYLOAD_ALIGNMENT constant increased from 16 bytes to 64 bytes to ensure SIMD- and cacheline-safe zero-copy access across SSE/AVX/AVX-512 code paths.

Added Features :

• debug_assert_aligned() function in simd-r-drive-entry-handle/src/debug_assert_aligned.rs:26-43
• debug_assert_aligned_offset() function in simd-r-drive-entry-handle/src/debug_assert_aligned.rs:66-88
• Alignment validation in EntryHandle::as_arrow_buffer and into_arrow_buffer when compiled in test/debug mode

Affected Code Entities :

• PAYLOAD_ALIGNMENT constant in src/storage_engine/constants.rs
• PAYLOAD_ALIGN_LOG2 constant in src/storage_engine/constants.rs
• All write operations that calculate pre-padding
• All read operations that derive payload start offsets

Why This Change Breaks Compatibility :

Readers compiled with ≤0.14.x expect payloads to be 16-byte aligned. When they read a 0.15.x file with 64-byte alignment, they calculate incorrect payload boundaries, reading pre-pad bytes as payload data or skipping actual payload bytes.

Sources : CHANGELOG.md:25-52 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-88

Version 0.14.0-alpha (2025-09-08)

Type : BREAKING - On-disk format incompatible with 0.13.x

Critical Changes :

Introduction of fixed payload alignment with configurable pre-padding. This was the first version to guarantee that payload starts align to a power-of-two boundary for zero-copy typed views.

On-Disk Layout Changes :

Added Features :

• PAYLOAD_ALIGN_LOG2 configuration constant
• PAYLOAD_ALIGNMENT = 1 << PAYLOAD_ALIGN_LOG2 derived constant
• Experimental arrow feature flag
• EntryHandle::as_arrow_buffer() method (requires arrow feature)
• EntryHandle::into_arrow_buffer() method (requires arrow feature)

Why This Change Breaks Compatibility :

Older readers (≤0.13.x) do not account for pre-padding bytes. When reading 0.14.x files, they interpret pre-pad bytes as part of the payload, causing data corruption or deserialization errors.

Sources : CHANGELOG.md:55-82

Migration Procedures

Migrating from 0.14.x to 0.15.x

The following sequence diagram illustrates the migration process:

Step-by-Step Migration :

1. Prepare the environment :

   • Build both 0.14.x and 0.15.x binaries
   • Ensure sufficient disk space (temporary duplicate of store)
   • Backup original store file

2. Read legacy data :

   # Using 0.14.x binary
   for each entry in old_store:
       data[entry.key] = entry.read_payload()
       verify entry.is_valid_checksum()
   

3. Create new store :

   # Using 0.15.x binary
   new_store = DataStore::create("new.store")
   for key, payload in data:
       new_store.write(key, payload)
   

4. Verification :

   # Using 0.15.x binary
   for key in data.keys():
       entry = new_store.read(key)
       assert entry.is_valid_checksum()
       assert entry.payload() == data[key]
   

5. Deployment in multi-service environments :

   • Deploy reader services with 0.15.x before writer services
   • Ensures readers can handle both formats during transition
   • Cut over writers only after all readers upgraded

Sources : CHANGELOG.md:43-51

Migrating from 0.13.x to 0.14.x

Step-by-Step Migration :

1. Regenerate stores with alignment:

   • Open old store with 0.13.x binary
   • Read each entry
   • Write to new 0.14.x store (automatically adds pre-padding)
   • Replace old file after verification

2. Service deployment order (multi-service environments):

   • Deploy reader upgrades first
   • Deploy writer upgrades second
   • Prevents mixed-version read failures

Sources : CHANGELOG.md:75-81

Compatibility Matrix

The following table shows version compatibility for readers and writers:

Legend :

• ✅ Compatible: Reader can correctly parse writer’s format
• ❌ Breaks: Reader cannot correctly parse writer’s format (data corruption risk)

Key Observations :

• Newer readers are backward-compatible (can read older formats)
• Older readers cannot read newer formats (forward compatibility not guaranteed)
• Each MINOR version bump introduces a new on-disk format

Sources : CHANGELOG.md:25-82

File Format Version Detection

The storage engine does not embed a file format version marker in the data file. Version compatibility must be managed externally through:

1. Deployment tracking : Maintain records of which binary version wrote each store
2. File naming conventions : Include version in filename (e.g., data-v0.15.store)
3. Metadata sidecars : Store version information in separate metadata files
4. Service configuration : Configure services with expected format version

Limitations :

• No automatic format detection at runtime
• Mixed-version deployment requires careful orchestration
• Checksum validation alone cannot detect version mismatches

Sources : CHANGELOG.md:1-82

Upgrade Strategies

Strategy 1: Blue-Green Deployment

Advantages :

• Clean separation of old and new versions
• Easy rollback if issues discovered
• No mixed-version complexity

Disadvantages :

• Requires duplicate infrastructure during migration
• Data must be fully copied
• Higher resource cost

Strategy 2: Rolling Upgrade (Reader-First)

Advantages :

• Minimal infrastructure duplication
• Gradual rollout reduces risk
• Reader compatibility maintained throughout

Disadvantages :

• Requires maintenance window for data migration
• More complex orchestration
• Must coordinate across multiple services

Sources : CHANGELOG.md:43-51 CHANGELOG.md:75-81

Alignment Configuration Reference

The alignment constants are defined in the storage engine’s constants module:

Version History :

• 0.13.x and earlier : No guaranteed alignment
• 0.14.x : Configurable alignment (default 16 bytes)
• 0.15.x and later : Configurable alignment (default 64 bytes)

Sources : CHANGELOG.md:25-42 CHANGELOG.md:55-74 simd-r-drive-entry-handle/src/debug_assert_aligned.rs:1-88

CI/CD Integration for Version Management

The continuous integration pipeline helps prevent unintentional breaking changes:

Linting and Validation :

• cargo fmt --all -- --check: Enforces consistent formatting across versions
• cargo clippy --workspace --all-targets --all-features -- -D warnings: Catches potential issues
• RUSTDOCFLAGS="-D warnings" cargo doc: Ensures documentation stays current
• cargo deny check: Validates dependency licenses and security
• cargo audit: Checks for known vulnerabilities

Testing Matrix : The CI/CD pipeline tests across multiple feature combinations to ensure compatibility. See CI/CD Pipeline for details on the testing infrastructure.

Sources : .github/workflows/rust-lint.yml:1-44

Future Considerations

Toward 1.0 Release :

• Embed format version marker in file header
• Implement automatic format detection
• Support multi-version reader capability
• Define stable API surface with backward compatibility guarantees

Deprecation Policy (Post-1.0):

• Major version bumps for breaking changes
• Deprecated features maintained for one major version
• Clear migration paths documented before removals

Sources : CHANGELOG.md:1-5

Dismiss

Refresh this wiki

Enter email to refresh