Python Direct Binding (Legacy)

Loading…

Python Direct Binding (Legacy)

Relevant source files

• experiments/bindings/python_(old_client)/Cargo.toml/Cargo.toml)
• experiments/bindings/python_(old_client)/README.md/README.md)
• experiments/bindings/python_(old_client)/pyproject.toml/pyproject.toml)

The simd-r-drive-py package represents the legacy approach to providing Python access to the DataStore engine. Unlike the modern WebSocket-based client, this binding uses PyO3 and maturin to expose the Rust DataStore struct directly as a native Python extension module. It is currently considered experimental/alpha and has been largely superseded by the network-based client due to threading and platform compatibility constraints experiments/bindings/python_(old_client)/README.md3-5

Purpose and Scope

The primary goal of the direct binding is to provide the absolute lowest latency possible by eliminating network overhead and using memory-mapped files directly within the Python process. It allows Python users to interact with the append-only storage engine as if it were a native Python object, supporting zero-copy reads via Python’s memoryview interface experiments/bindings/python_(old_client)/README.md7

Status and Limitations

• Status: Experimental / Alpha experiments/bindings/python_(old_client)/pyproject.toml3 experiments/bindings/python_(old_client)/pyproject.toml11
• Thread Safety: This binding is not thread-safe experiments/bindings/python_(old_client)/README.md198 Concurrent streaming writes or reads from multiple threads may cause hangs or inconsistent behavior due to the interaction between the Python Global Interpreter Lock (GIL) and Rust’s internal locking experiments/bindings/python_(old_client)/README.md198-201
• Platform Support: Officially supports Linux and macOS (CPython 3.10–3.13) experiments/bindings/python_(old_client)/README.md23-32 Windows is not officially supported for the direct binding due to memory-mapping inconsistencies in the Python runtime experiments/bindings/python_(old_client)/README.md38-39

Sources: experiments/bindings/python_(old_client)/README.md3-39 experiments/bindings/python_(old_client)/pyproject.toml3-11

System Architecture

The direct binding bridges the Python interpreter to the Rust DataStore using a thin wrapper layer defined in the module entry point. The Rust simd-r-drive dependency is compiled with the expose-internal-api feature to allow the binding access to internal structures like EntryHandle experiments/bindings/python_(old_client)/Cargo.toml15-16

Data Flow: Python to Rust Engine

The following diagram illustrates how a Python call traverses the binding layer to reach the core storage engine.

Binding Execution Path

graph TD
    subgraph "Python_Space"
        ["Python_Application"] --> ["DataStore_Python_Class"]
end

    subgraph "PyO3_Native_Extension"
        ["DataStore_Python_Class"] -- "PyBytes_to_u8_slice" --> ["Rust_Wrapper_impl_DataStore"]
        ["EntryHandle_Python"] -- "as_memoryview" --> ["memoryview_Object"]
end

    subgraph "Core_Engine_simd_r_drive"
        ["Rust_Wrapper_impl_DataStore"] -- "DataStore::write" --> ["DataStore_Rust_Struct"]
        ["DataStore_Rust_Struct"] -- "Index_Lookup" --> ["KeyIndexer"]
        ["DataStore_Rust_Struct"] -- "Mmap_Access" --> ["Mmap_Shared_Memory"]
end

    ["Mmap_Shared_Memory"] -- "Direct_Slice" --> ["memoryview_Object"]

Sources: experiments/bindings/python_(old_client)/README.md164-187 experiments/bindings/python_(old_client)/Cargo.toml15-16

Implementation Details

Build System and Metadata

The project uses maturin as the build backend, targeting the pyo3 binding type experiments/bindings/python_(old_client)/pyproject.toml28-33 It specifies a minimum Python version of 3.8 but officially targets 3.10 through 3.13 for CPython experiments/bindings/python_(old_client)/pyproject.toml20-34 The crate type is set to cdylib to produce a shared library compatible with Python’s import system experiments/bindings/python_(old_client)/Cargo.toml8-10

Key Classes and Functions

Sources: experiments/bindings/python_(old_client)/README.md160-194 experiments/bindings/python_(old_client)/Cargo.toml8-16 experiments/bindings/python_(old_client)/pyproject.toml28-34

Zero-Copy Integration

One of the core features of the direct binding is the EntryHandle.as_memoryview() method. This allows Python libraries to consume data directly from the Rust-managed mmap without copying bytes into the Python heap experiments/bindings/python_(old_client)/README.md7-182

Code Entity Association: Zero-Copy Read

Sources: experiments/bindings/python_(old_client)/README.md180-182 experiments/bindings/python_(old_client)/README.md7

Usage and Lifecycle Management

Resource Cleanup

Because PyO3 does not guarantee deterministic destruction of Rust-backed objects, the underlying Rust Drop implementation handles the cleanup of file handles and memory maps. However, the Python garbage collector may delay this, potentially causing issues when trying to delete or move the underlying file on disk while the process is running, especially on Windows experiments/bindings/python_(old_client)/README.md38-39

Example: Streaming Large Payloads

The legacy binding supports streaming from Python’s io.BytesIO or other file-like objects directly into the store using the write_stream method experiments/bindings/python_(old_client)/README.md172-174

Sources: experiments/bindings/python_(old_client)/README.md136-156 experiments/bindings/python_(old_client)/README.md184-186

Supported Platforms Table

Sources: experiments/bindings/python_(old_client)/README.md23-45