Python WebSocket Client API

Relevant source files

Purpose and Scope

This document describes the Python WebSocket client API for remote access to SIMD R Drive storage. The API provides idiomatic Python interfaces backed by high-performance Rust implementations via PyO3 bindings. This page covers the DataStoreWsClient class, NamespaceHasher utility, and their usage patterns.

For information about building and installing the Python bindings, see Building Python Bindings. For details about the underlying native Rust WebSocket client, see Native Rust Client. For server-side configuration and deployment, see WebSocket Server.

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/init.py:1-14 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:1-219

Architecture Overview

The Python WebSocket client uses a multi-layer architecture that bridges Python's async/await with Rust's Tokio runtime while maintaining idiomatic Python APIs.

graph TB
    UserCode["Python User Code\nimport simd_r_drive_ws_client"]
DataStoreWsClient["DataStoreWsClient\nPython wrapper class"]
BaseDataStoreWsClient["BaseDataStoreWsClient\nPyO3 #[pyclass]"]
PyO3FFI["PyO3 FFI Layer\npyo3-async-runtimes"]
RustClient["simd-r-drive-ws-client\nNative Rust implementation"]
MuxioRPC["muxio-tokio-rpc-client\nWebSocket + RPC"]
Server["simd-r-drive-ws-server\nRemote DataStore"]
UserCode --> DataStoreWsClient
 
   DataStoreWsClient --> BaseDataStoreWsClient
 
   BaseDataStoreWsClient --> PyO3FFI
 
   PyO3FFI --> RustClient
 
   RustClient --> MuxioRPC
 
   MuxioRPC --> Server

Python Integration Stack

Sources: experiments/bindings/python-ws-client/Cargo.lock:1096-1108 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:1-10

Class Hierarchy

The BaseDataStoreWsClient class is implemented in Rust and exposes core storage operations through PyO3. The DataStoreWsClient Python class extends it with additional convenience methods implemented in pure Python.

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:1-10 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:11-62

DataStoreWsClient Class

The DataStoreWsClient class provides the primary interface for interacting with a remote SIMD R Drive storage engine over WebSocket connections.

Connection Initialization

Constructor	Description
`__init__(host: str, port: int)`	Establishes WebSocket connection to the specified server

The constructor creates a WebSocket connection to the remote storage server. Connection establishment is synchronous and will raise an exception if the server is unreachable.

Example:

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:17-25

Write Operations

Method	Parameters	Description
`write(key, data)`	`key: bytes`, `data: bytes`	Appends single key/value pair
`batch_write(items)`	`items: list[tuple[bytes, bytes]]`	Writes multiple pairs in one operation

Write operations are append-only and atomic. If a key already exists, writing to it creates a new version while the old data remains on disk (marked as superseded via the index).

Example:

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:27-51

Read Operations

Method	Parameters	Return Type	Copy Behavior
`read(key)`	`key: bytes`	`Optional[bytes]`	Performs memory copy
`batch_read(keys)`	`keys: list[bytes]`	`list[Optional[bytes]]`	Performs memory copy
`batch_read_structured(data)`	`data: dict or list[dict]`	Same structure with values	Python-side wrapper

The read and batch_read methods perform memory copies when returning data. For zero-copy access patterns, the native Rust client provides read_entry methods that return memory-mapped views.

The batch_read_structured method is a Python convenience wrapper that:

Accepts dictionaries or lists of dictionaries where values are datastore keys
Flattens the structure into a single key list
Calls batch_read for efficient parallel fetching
Reconstructs the original structure with fetched values

Example:

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:79-129 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py:12-62

Deletion and Existence Checks

Method	Parameters	Return Type	Description
`delete(key)`	`key: bytes`	`None`	Marks key as deleted (tombstone)
`exists(key)`	`key: bytes`	`bool`	Checks if key is active
`__contains__(key)`	`key: bytes`	`bool`	Python `in` operator support

Deletion is logical, not physical. The delete method appends a tombstone entry to the storage file. The physical data remains on disk but is no longer accessible through reads.

Example:

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:53-77 experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:131-141

Utility Methods

Method	Return Type	Description
`__len__()`	`int`	Returns count of active entries
`is_empty()`	`bool`	Checks if store has any active keys
`file_size()`	`int`	Returns physical file size on disk

Example:

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:143-168

NamespaceHasher Utility

The NamespaceHasher class provides deterministic key namespacing using XXH3 hashing to prevent key collisions across logical domains.

graph LR
    Input1["Namespace prefix\ne.g., b'users'"]
Input2["Key\ne.g., b'user123'"]
Hash1["XXH3 hash\n8 bytes"]
Hash2["XXH3 hash\n8 bytes"]
Output["Namespaced key\n16 bytes total"]
Input1 -->|hash once at init| Hash1
 
   Input2 -->|hash per call| Hash2
 
   Hash1 --> Output
 
   Hash2 --> Output

Architecture

Usage Pattern

Method	Parameters	Return Type	Description
`__init__(prefix)`	`prefix: bytes`	N/A	Initializes hasher with namespace
`namespace(key)`	`key: bytes`	`bytes`	Returns 16-byte namespaced key

The output key structure is:

Bytes 0-7: XXH3 hash of namespace prefix
Bytes 8-15: XXH3 hash of input key

This design ensures:

Deterministic key generation (same input → same output)
Collision isolation between namespaces
Fixed-length keys regardless of input size

Example:

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:170-219

graph TB
    Stub["data_store_ws_client.pyi\nType definitions"]
Impl["data_store_ws_client.py\nImplementation"]
Base["simd_r_drive_ws_client\nCompiled Rust module"]
Stub -.->|describes| Impl
 
   Impl -->|imports from| Base
 
   Stub -.->|describes| Base

Type Stubs and IDE Support

The package provides complete type stubs for IDE integration and static type checking.

Type Stub Structure

The type stubs (experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:1-219) provide:

Full method signatures with type annotations
Return type information (Optional[bytes], list[Optional[bytes]], etc.)
Docstrings for IDE hover documentation
@final decorators indicating classes cannot be subclassed

Type Checking Example

Python Version Support:

The package targets Python 3.10-3.13 as specified in experiments/bindings/python-ws-client/pyproject.toml7 and experiments/bindings/python-ws-client/pyproject.toml:21-24

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:1-219 experiments/bindings/python-ws-client/pyproject.toml7 experiments/bindings/python-ws-client/pyproject.toml:19-27

graph TB
    PythonMain["Python Main Thread\nSynchronous API calls"]
PyO3["PyO3 Bridge\npyo3-async-runtimes"]
TokioRT["Tokio Runtime\nAsync event loop"]
WSClient["WebSocket Client\ntokio-tungstenite"]
PythonMain -->|sync call| PyO3
 
   PyO3 -->|spawn + block_on| TokioRT
 
   TokioRT --> WSClient
 
   WSClient -.->|result| TokioRT
 
   TokioRT -.->|return| PyO3
 
   PyO3 -.->|return| PythonMain

Async Runtime Bridging

The client uses pyo3-async-runtimes to bridge Python's async/await with Rust's Tokio runtime. This allows the underlying Rust WebSocket client to use native async I/O while exposing synchronous APIs to Python.

Runtime Architecture

The pyo3-async-runtimes crate (experiments/bindings/python-ws-client/Cargo.lock:849-860) provides:

Runtime spawning: Manages Tokio runtime lifecycle
Future blocking: Converts Rust async operations to Python-blocking calls
Thread safety: Ensures proper synchronization between Python GIL and Rust runtime

This design allows Python code to use simple synchronous APIs while benefiting from Rust's high-performance async networking under the hood.

Sources: experiments/bindings/python-ws-client/Cargo.lock:849-860 experiments/bindings/python-ws-client/Cargo.lock:1096-1108

API Summary

Complete Method Reference

Category	Method	Parameters	Return	Description
Connection	`__init__`	`host: str, port: int`	N/A	Establish WebSocket connection
Write	`write`	`key: bytes, data: bytes`	`None`	Write single entry
Write	`batch_write`	`items: list[tuple[bytes, bytes]]`	`None`	Write multiple entries
Read	`read`	`key: bytes`	`Optional[bytes]`	Read single entry (copies)
Read	`batch_read`	`keys: list[bytes]`	`list[Optional[bytes]]`	Read multiple entries
Read	`batch_read_structured`	`data: dict or list[dict]`	Same structure	Read with structure preservation
Delete	`delete`	`key: bytes`	`None`	Mark key as deleted
Query	`exists`	`key: bytes`	`bool`	Check key existence
Query	`__contains__`	`key: bytes`	`bool`	Python `in` operator
Info	`__len__`	N/A	`int`	Active entry count
Info	`is_empty`	N/A	`bool`	Check if empty
Info	`file_size`	N/A	`int`	Physical file size

NamespaceHasher Reference

Method	Parameters	Return	Description
`__init__`	`prefix: bytes`	N/A	Initialize namespace
`namespace`	`key: bytes`	`bytes`	Generate 16-byte namespaced key

Sources: experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi:1-219

Keyboard shortcuts

rust-simd-r-drive Documentation