Integration Testing

Relevant source files

• experiments/bindings/python-ws-client/README.md
• experiments/bindings/python-ws-client/extract_readme_tests.py
• experiments/bindings/python-ws-client/integration_test.sh
• experiments/bindings/python-ws-client/pyproject.toml
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/init.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.py
• experiments/bindings/python-ws-client/simd_r_drive_ws_client/data_store_ws_client.pyi
• experiments/bindings/python-ws-client/uv.lock

Purpose and Scope

This document describes the integration testing infrastructure for the Python WebSocket client bindings. The testing system validates end-to-end functionality by orchestrating a complete test environment: starting a SIMD R Drive WebSocket server, extracting test cases from documentation, and executing them against the live server.

For information about the Python client API itself, see Python WebSocket Client API. For build system details, see Building Python Bindings.

Integration Test Architecture

The integration testing system consists of three primary components: a shell orchestrator that manages the test lifecycle, a Python script that extracts executable examples from documentation, and the pytest framework that executes the tests.

Sources: experiments/bindings/python-ws-client/integration_test.sh:1-90 experiments/bindings/python-ws-client/extract_readme_tests.py:1-45

graph TB
    subgraph "Test Orchestration"
        Script["integration_test.sh\nBash Orchestrator"]
Cleanup["cleanup()\nTrap Handler"]
end
    
    subgraph "Server Management"
        CargoRun["cargo run\n--package simd-r-drive-ws-server"]
Server["simd-r-drive-ws-server\nProcess (PID tracked)"]
Storage["/tmp/simd-r-drive-pytest-storage.bin"]
end
    
    subgraph "Test Generation"
        ExtractScript["extract_readme_tests.py"]
ReadmeMd["README.md\nPython code blocks"]
TestFile["tests/test_readme_blocks.py\nGenerated pytest functions"]
end
    
    subgraph "Test Execution"
        UvRun["uv run pytest"]
Pytest["pytest framework"]
TestCases["test_readme_block_0()\ntest_readme_block_1()\n..."]
end
    
    subgraph "Client Under Test"
        Client["DataStoreWsClient"]
WsConnection["WebSocket Connection\n127.0.0.1:34129"]
end
    
 
   Script --> CargoRun
 
   CargoRun --> Server
 
   Server --> Storage
 
   Script --> ExtractScript
 
   ExtractScript --> ReadmeMd
 
   ReadmeMd --> TestFile
 
   Script --> UvRun
 
   UvRun --> Pytest
 
   Pytest --> TestCases
 
   TestCases --> Client
 
   Client --> WsConnection
 
   WsConnection --> Server
 
   Script --> Cleanup
 
   Cleanup -.->|kill -9 -$SERVER_PID| Server
 
   Cleanup -.->|rm -f| Storage

Test Workflow Orchestration

The integration_test.sh script manages the complete test lifecycle through a series of coordinated steps. The script implements robust cleanup handling using shell traps to ensure resources are released even if tests fail.

sequenceDiagram
    participant Script as integration_test.sh
    participant Trap as cleanup() trap
    participant Cargo as cargo run
    participant Server as simd-r-drive-ws-server
    participant UV as uv (Python)
    participant Extract as extract_readme_tests.py
    participant Pytest as pytest
    
    Script->>Script: set -e (fail on error)
    Script->>Trap: trap cleanup EXIT
    Script->>Script: cd experiments/
    Script->>Cargo: cargo run --package simd-r-drive-ws-server
    Cargo->>Server: start background process
    Server-->>Script: return PID
    Script->>Script: SERVER_PID=$!
    Script->>UV: uv venv
    Script->>UV: uv pip install pytest maturin
    Script->>UV: uv pip install -e . --group dev
    Script->>Extract: uv run extract_readme_tests.py
    Extract-->>Script: generate test_readme_blocks.py
    Script->>Script: export TEST_SERVER_HOST/PORT
    Script->>Pytest: uv run pytest -v -s
    Pytest->>Server: test connections
    Server-->>Pytest: responses
    Pytest-->>Script: test results
    Script->>Trap: EXIT signal
    Trap->>Server: kill -9 -$SERVER_PID
    Trap->>Script: rm -f /tmp/simd-r-drive-pytest-storage.bin

Workflow Sequence

Sources: experiments/bindings/python-ws-client/integration_test.sh:1-90

Configuration Variables

The script defines several configuration variables at the top for easy modification:

Sources: experiments/bindings/python-ws-client/integration_test.sh:8-15

Cleanup Mechanism

The cleanup function ensures proper resource release regardless of test outcome. It uses process group termination to kill the server and all child processes:

graph LR
    Exit["Script Exit\n(success or failure)"]
Trap["trap cleanup EXIT"]
Cleanup["cleanup()
function"]
KillServer["kill -9 -$SERVER_PID"]
RemoveFile["rm -f $STORAGE_FILE"]
Exit --> Trap
 
   Trap --> Cleanup
 
   Cleanup --> KillServer
 
   Cleanup --> RemoveFile

The kill -9 "-$SERVER_PID" syntax terminates the entire process group (note the minus sign prefix), ensuring the server and any spawned threads are fully stopped. The 2>/dev/null || true pattern prevents errors if the process already exited.

Sources: experiments/bindings/python-ws-client/integration_test.sh:17-30

graph TB
    ReadmeMd["README.md"]
ExtractScript["extract_readme_tests.py"]
subgraph "Extraction Functions"
        ExtractBlocks["extract_python_blocks()\nregex: ```python...```"]
StripNonAscii["strip_non_ascii()\nRemove non-ASCII chars"]
WrapTest["wrap_as_test_fn()\nGenerate pytest function"]
end
    
    subgraph "Generated Output"
        TestFile["tests/test_readme_blocks.py"]
TestFn0["def test_readme_block_0():\n # extracted code"]
TestFn1["def test_readme_block_1():\n # extracted code"]
TestFnN["def test_readme_block_N():\n # extracted code"]
end
    
 
   ReadmeMd --> ExtractScript
 
   ExtractScript --> ExtractBlocks
 
   ExtractBlocks --> StripNonAscii
 
   StripNonAscii --> WrapTest
 
   WrapTest --> TestFile
 
   TestFile --> TestFn0
 
   TestFile --> TestFn1
 
   TestFile --> TestFnN

README Test Extraction

The extract_readme_tests.py script automates the conversion of documentation examples into executable test cases. This approach ensures that code examples in the README remain accurate and functional.

Extraction Process

Sources: experiments/bindings/python-ws-client/extract_readme_tests.py:1-45

Extraction Implementation

The extraction process follows three steps:

1. Pattern Matching : Uses regex to find all ````python` fenced code blocks

   • Pattern: r"```python\n(.*?)```" with re.DOTALL flag
   • Returns list of code block strings

2. Text Sanitization : Removes non-ASCII characters to prevent encoding issues

   • Uses encode("ascii", errors="ignore").decode("ascii")
   • Ensures clean Python code for pytest execution

3. Test Function Generation : Wraps each code block in a pytest-compatible function

   • Function naming: test_readme_block_{idx}()
   • Indentation: 4 spaces for function body
   • Empty blocks generate pass statement

Sources: experiments/bindings/python-ws-client/extract_readme_tests.py:21-34

Example Transformation

Given a README code block:

The script generates:

Sources: experiments/bindings/python-ws-client/README.md:42-51 experiments/bindings/python-ws-client/extract_readme_tests.py:30-34

Test Environment Setup

The integration test uses uv as the Python package manager for fast, reliable dependency management. The environment setup occurs after server startup but before test execution.

Environment Setup Sequence

Sources: experiments/bindings/python-ws-client/integration_test.sh:62-87

Development Dependencies

The project defines development dependencies in pyproject.toml under the [dependency-groups] section:

Sources: experiments/bindings/python-ws-client/pyproject.toml:37-46

graph LR
    Script["integration_test.sh"]
SetM["set -m\n(enable job control)"]
CargoRun["cargo run --package simd-r-drive-ws-server\n-- /tmp/storage.bin --host 127.0.0.1 --port 34129 &"]
CapturePid["SERVER_PID=$!"]
UnsetM["set +m\n(disable job control)"]
Script --> SetM
 
   SetM --> CargoRun
 
   CargoRun --> CapturePid
 
   CapturePid --> UnsetM

Server Lifecycle Management

The integration test script manages the server as a background process with careful PID tracking and cleanup handling.

Server Startup

The server starts with the following configuration:

• Package : simd-r-drive-ws-server (built via cargo)
• Storage File : /tmp/simd-r-drive-pytest-storage.bin
• Host : 127.0.0.1 (localhost only)
• Port : 34129 (test-specific port)
• Mode : Background process (& suffix)

The set -m and set +m wrapper enables job control temporarily, ensuring the shell correctly captures the background process PID with $!.

Sources: experiments/bindings/python-ws-client/integration_test.sh:47-56

Server Shutdown

The cleanup trap ensures the server terminates cleanly:

1. Check PID Exists : [[ ! -z "$SERVER_PID" ]]
2. Kill Process Group : kill -9 "-$SERVER_PID"
   • The - prefix kills the entire process group
   • Signal -9 (SIGKILL) ensures immediate termination
3. Ignore Errors : 2>/dev/null || true
   • Prevents script failure if process already exited
4. Remove Storage : rm -f "$STORAGE_FILE"
   • Cleans up temporary test data

Sources: experiments/bindings/python-ws-client/integration_test.sh:19-29

Test Execution Environment

The test suite runs with specific environment variables that configure the client connection parameters. These variables bridge the shell orchestration layer with the Python test code.

Environment Variable Injection

The script exports server configuration to the test environment:

Python tests can access these via os.environ:

Sources: experiments/bindings/python-ws-client/integration_test.sh:83-85

Pytest Execution Flags

The test suite runs with specific pytest flags:

• -v: Verbose output (shows individual test names)
• -s: No output capture (displays print statements immediately)

This configuration aids debugging by providing real-time test output.

Sources: experiments/bindings/python-ws-client/integration_test.sh87

graph TB
    Main["main()
entry point"]
ReadFile["README.read_text()"]
Extract["extract_python_blocks()"]
Generate["List comprehension:\n[wrap_as_test_fn(code, i)\nfor i, code in enumerate(blocks)]"]
WriteFile["TEST_FILE.write_text()"]
Main --> ReadFile
 
   ReadFile --> Extract
 
   Extract --> Generate
 
   Generate --> WriteFile
    
    subgraph "File Structure"
        Header["# Auto-generated from README.md\nimport pytest"]
TestFunctions["test_readme_block_0()\ntest_readme_block_1()\n..."]
end
    
 
   WriteFile --> Header
 
   WriteFile --> TestFunctions

Test Case Structure

Generated test cases follow a consistent structure based on the README extraction pattern. Each test becomes an isolated function within the generated test file.

Test File Generation

Sources: experiments/bindings/python-ws-client/extract_readme_tests.py:36-42

Test Isolation

Each extracted code block becomes an independent test function:

• Naming Convention : test_readme_block_{idx}() where idx is zero-based
• Isolation : Each function has its own scope
• Independence : Tests can run in any order (no shared state)
• Clarity : Test number maps directly to README code block order

This pattern enables pytest to discover and execute each documentation example as a separate test case, providing clear pass/fail feedback for each code snippet.

Sources: experiments/bindings/python-ws-client/extract_readme_tests.py:30-34

Dependencies and Platform Requirements

The integration testing system requires specific tools and versions to function correctly.

Required Tools

Sources: experiments/bindings/python-ws-client/integration_test.sh:62-68 experiments/bindings/python-ws-client/integration_test.sh1

Python Version Support

The client supports Python 3.10 through 3.13 (CPython only):

Note : NumPy version constraints differ by Python version. The project uses numpy>=2.2.6 for Python 3.10 compatibility, as NumPy 2.3.0+ dropped Python 3.10 support.

Sources: experiments/bindings/python-ws-client/pyproject.toml7 experiments/bindings/python-ws-client/pyproject.toml:21-24 experiments/bindings/python-ws-client/pyproject.toml41

Operating System Support

The integration tests target POSIX-compliant systems:

• Linux : Primary development platform (Ubuntu, Debian, etc.)
• macOS : Full support (Darwin)
• Windows : Supported but uses different process management

The shell script uses POSIX-compatible syntax ([[ ]], $!, trap) that works across these platforms with appropriate shells.

Sources: experiments/bindings/python-ws-client/integration_test.sh1 experiments/bindings/python-ws-client/pyproject.toml:25-26

Running the Integration Tests

Execution Command

From the repository root:

Or from the Python client directory:

Expected Output Sequence

1. Script initialization and configuration
2. Server compilation (if needed) and startup
3. Python environment creation
4. Dependency installation
5. README test extraction
6. Pytest test execution with detailed output
7. Automatic cleanup on exit

Troubleshooting

Common failure scenarios:

Sources: experiments/bindings/python-ws-client/integration_test.sh:62-68