Building Python Bindings

Loading…

Building Python Bindings

Relevant source files

• experiments/bindings/python-ws-client/Cargo.lock
• experiments/bindings/python-ws-client/README.md
• experiments/bindings/python-ws-client/extract_readme_tests.py
• experiments/bindings/python-ws-client/integration_test.sh
• experiments/bindings/python-ws-client/pyproject.toml
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/init.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi
• experiments/bindings/python-ws-client/uv.lock
• src/storage_engine/key_indexer.rs

Purpose and Scope

This page describes the build system, tooling, and workflow for generating Python bindings for the SIMD R Drive storage engine. It covers the PyO3/Maturin build pipeline, dependency management with uv, local development workflows, and wheel distribution. For the Python API surface and usage patterns, see Python WebSocket Client API. For CI/CD workflows and automated release processes, see CI/CD Pipeline.

Build System Architecture

The Python bindings are built using PyO3 (Rust-Python FFI) and Maturin (build backend and wheel generator). The uv tool replaces traditional pip and venv for faster, more reliable dependency management.

Build Pipeline Overview

graph TB
    subgraph "Source Code"
        RUST_LIB["experiments/bindings/python-ws-client/src/lib.rs\nPyO3 FFI Layer"]
PY_WRAPPER["simd_r_drive_ws_client/data_store_ws_client.py\nPython Wrapper"]
TYPE_STUBS["simd_r_drive_ws_client/data_store_ws_client.pyi\nType Hints"]
end
    
    subgraph "Build Configuration"
        PYPROJECT["pyproject.toml\nProject Metadata + Build Backend"]
CARGO_TOML["Cargo.toml\nRust Dependencies"]
UV_LOCK["uv.lock\nPinned Python Deps"]
end
    
    subgraph "Build Tools"
        PYO3["PyO3 0.25.1\nRust-Python Bridge"]
MATURIN["Maturin 1.8.7\nBuild System"]
UV["uv\nDependency Manager"]
end
    
    subgraph "Build Outputs"
        NATIVE_LIB["simd_r_drive_ws_client.so/.pyd\nNative Extension Module"]
WHEEL["simd_r_drive_ws_client-*.whl\nDistributable Package"]
end
    
 
   RUST_LIB --> PYO3
 
   PY_WRAPPER --> MATURIN
 
   TYPE_STUBS --> MATURIN
 
   PYPROJECT --> MATURIN
 
   CARGO_TOML --> PYO3
 
   UV_LOCK --> UV
    
 
   PYO3 --> MATURIN
 
   MATURIN --> NATIVE_LIB
 
   MATURIN --> WHEEL
 
   UV --> MATURIN
    
    NATIVE_LIB -.packaged into.-> WHEEL

Sources: experiments/bindings/python-ws-client/pyproject.toml:29-36 experiments/bindings/python-ws-client/Cargo.lock:832-905

Project Configuration Files

pyproject.toml Structure

The pyproject.toml file defines project metadata, build system requirements, and development dependencies.

Key Configuration Entries:

The bindings = "pyo3" directive tells Maturin to compile Rust code using PyO3’s FFI macros and generate a native Python extension module.

Sources: experiments/bindings/python-ws-client/pyproject.toml:1-46

Cargo Dependencies

The Rust side declares dependencies for PyO3, async runtime bridging, and core storage functionality:

Sources: experiments/bindings/python-ws-client/Cargo.lock:832-905 experiments/bindings/python-ws-client/Cargo.lock:849-860

Dependency Management with uv

The project uses uv instead of traditional pip for significantly faster dependency resolution and installation. uv is an all-in-one replacement for pip, pip-tools, and virtualenv.

uv Workflow Diagram

graph LR
    subgraph "Traditional pip"
        PIP_VENV["python -m venv"]
PIP_INSTALL["pip install"]
PIP_LOCK["pip freeze > requirements.txt"]
end
    
    subgraph "uv Workflow"
        UV_VENV["uv venv"]
UV_SYNC["uv pip install -e . --group dev"]
UV_LOCK["uv.lock\nAuto-generated"]
UV_RUN["uv run pytest"]
end
    
 
   UV_VENV --> UV_SYNC
 
   UV_SYNC --> UV_LOCK
 
   UV_LOCK --> UV_RUN

Development Dependencies

Development dependencies are specified using the [dependency-groups] table in pyproject.toml:

This group can be installed with:

Sources: experiments/bindings/python-ws-client/pyproject.toml:37-46 experiments/bindings/python-ws-client/integration_test.sh:70-77

Building from Source

Local Development Build

The fastest way to build and install bindings for local development is using maturin develop:

Build Process Details:

The develop command creates an editable installation , meaning changes to Python wrapper code (experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py) take effect immediately without reinstalling. Rust code changes require re-running maturin develop.

Sources: experiments/bindings/python-ws-client/README.md:32-36 experiments/bindings/python-ws-client/integration_test.sh:74-77

Release Wheel Build

To build distributable wheels for PyPI:

Maturin automatically:

1. Compiles Rust code with --release optimizations
2. Generates platform-specific wheel filename (e.g., cp310-cp310-linux_x86_64)
3. Bundles native extension, Python wrappers, and type stubs
4. Creates wheel in target/wheels/

Platform-Specific Wheels:

Sources: experiments/bindings/python-ws-client/uv.lock:117-130

Integration Testing Infrastructure

The project includes an automated integration test system that:

1. Extracts code examples from README.md
2. Starts the WebSocket server
3. Runs pytest against live server
4. Cleans up resources

graph TB
    subgraph "integration_test.sh"
        START["Start script"]
REGISTER_CLEANUP["Register cleanup trap"]
BUILD_SERVER["cargo run --package simd-r-drive-ws-server"]
START_SERVER["Start server in background\nPID captured"]
SETUP_UV["uv venv\nuv pip install"]
EXTRACT_TESTS["uv run extract_readme_tests.py"]
RUN_PYTEST["uv run pytest -v -s"]
CLEANUP["Kill server PID\nRemove storage file"]
end
    
    subgraph "extract_readme_tests.py"
        READ_README["Read README.md"]
EXTRACT_BLOCKS["Regex: ```python...```"]
WRAP_TEST_FN["Wrap as test_readme_block_N()"]
WRITE_TEST_FILE["Write tests/test_readme_blocks.py"]
end
    
 
   START --> REGISTER_CLEANUP
 
   REGISTER_CLEANUP --> BUILD_SERVER
 
   BUILD_SERVER --> START_SERVER
 
   START_SERVER --> SETUP_UV
 
   SETUP_UV --> EXTRACT_TESTS
    
 
   EXTRACT_TESTS --> READ_README
 
   READ_README --> EXTRACT_BLOCKS
 
   EXTRACT_BLOCKS --> WRAP_TEST_FN
 
   WRAP_TEST_FN --> WRITE_TEST_FILE
    
 
   WRITE_TEST_FILE --> RUN_PYTEST
 
   RUN_PYTEST --> CLEANUP

Test Orchestration Flow

Sources: experiments/bindings/python-ws-client/integration_test.sh:1-91

README Test Extraction

The extract_readme_tests.py script converts documentation examples into executable tests:

Extraction Logic:

1. Pattern Matching: Uses regex r"```python\n(.*?)```" to extract fenced code blocks (experiments/bindings/python-ws-client/extract_readme_tests.py:22-24)
2. Sanitization: Strips non-ASCII characters to avoid encoding issues (experiments/bindings/python-ws-client/extract_readme_tests.py:26-28)
3. Test Wrapping: Wraps each block in a test_readme_block_{i}() function (experiments/bindings/python-ws-client/extract_readme_tests.py:30-34)
4. File Generation: Writes to tests/test_readme_blocks.py (experiments/bindings/python-ws-client/extract_readme_tests.py:36-42)

Example Transformation:

Input (README.md):

Output (tests/test_readme_blocks.py):

Sources: experiments/bindings/python-ws-client/extract_readme_tests.py:14-45

Integration Test Server Management

The test script manages the WebSocket server lifecycle:

Server Startup Sequence:

Cleanup Trap:

The script registers a cleanup() function that executes on exit (success or failure):

This ensures the server never remains running after tests complete, even if pytest crashes.

Sources: experiments/bindings/python-ws-client/integration_test.sh:17-33 experiments/bindings/python-ws-client/integration_test.sh:47-56

Wheel Distribution and CI Integration

Maturin Wheel Building

Maturin generates platform-specific binary wheels that include the compiled Rust extension. Each wheel is tagged with Python version, ABI, and platform identifiers.

Wheel Naming Convention:

{distribution}-{version}-{python tag}-{abi tag}-{platform tag}.whl

Example: simd_r_drive_ws_client-0.11.1-cp310-cp310-manylinux_2_17_x86_64.whl

Sources: experiments/bindings/python-ws-client/uv.lock:117-130

Build Matrix Configuration

The CI system (see CI/CD Pipeline) builds wheels for multiple platforms using a matrix strategy:

This produces wheels for:

• Linux: manylinux2014 x86_64, aarch64
• macOS: x86_64, arm64 (universal2)
• Windows: win_amd64, win32, win_arm64

CI Workflow Reference:

The GitHub Actions workflow at .github/workflows/python-net-release.yml orchestrates multi-platform builds using Maturin’s maturin build --release command across matrix configurations.

Sources: experiments/bindings/python-ws-client/README.md38

PyO3 Feature Configuration

Async Runtime Bridge

The bindings use pyo3-async-runtimes to bridge Python’s asyncio event loop with Rust’s tokio runtime:

This allows Python code to use await with methods that internally execute Rust async code:

The wrapper methods handle the await internally, presenting a synchronous interface to Python users.

Sources: experiments/bindings/python-ws-client/Cargo.lock:849-860

Type Stub Generation

Type stubs (.pyi files) provide IDE autocomplete and mypy type checking. They are manually maintained to match the PyO3 API:

These stubs are packaged into the wheel alongside the native extension, enabling static type checking without runtime overhead.

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:7-169

Development Workflow Summary

Recommended Development Cycle:

Quick Command Reference:

Sources: experiments/bindings/python-ws-client/integration_test.sh:70-87 experiments/bindings/python-ws-client/README.md:32-36

Dismiss

Refresh this wiki

Enter email to refresh