Integration Testing

Loading…

Integration Testing

Relevant source files

• experiments/bindings/python-ws-client/README.md
• experiments/bindings/python-ws-client/extract_readme_tests.py
• experiments/bindings/python-ws-client/integration_test.sh
• experiments/bindings/python-ws-client/uv.lock

Purpose and Scope

This document describes the Python integration test infrastructure for the simd-r-drive-ws-client-py bindings. The testing system comprises two complementary strategies: (1) automatic extraction and validation of README code examples, and (2) full-stack integration testing that orchestrates a complete WebSocket server lifecycle. For information about building the Python bindings themselves, see Building Python Bindings. For details on the Python WebSocket Client API, see Python WebSocket Client API.

Sources : experiments/bindings/python-ws-client/README.md:1-60 experiments/bindings/python-ws-client/integration_test.sh:1-91

Overview of Integration Testing Strategy

The integration testing system ensures that both documentation examples and end-to-end functionality remain correct as the codebase evolves. The architecture consists of two primary components:

1. Documentation Test Extraction : The extract_readme_tests.py script automatically converts Python code blocks from README.md into executable pytest test functions, ensuring that documentation examples remain accurate.

2. Full-Stack Integration Testing : The integration_test.sh script orchestrates a complete test environment, including launching a WebSocket server (simd-r-drive-ws-server), setting up Python dependencies, running the extracted tests, and performing cleanup.

This dual approach provides confidence that both user-facing examples and the complete client-server interaction path function correctly across different environments.

Sources : experiments/bindings/python-ws-client/extract_readme_tests.py:1-46 experiments/bindings/python-ws-client/integration_test.sh:1-91

Integration Test Architecture

Architecture Overview : The integration test system uses integration_test.sh as the central orchestrator that manages three concurrent workflows: (1) server lifecycle management using cargo run and background process control, (2) Python environment setup using uv package manager, and (3) test extraction and execution using extract_readme_tests.py and pytest. The cleanup() function, registered with a trap EXIT, ensures proper resource cleanup even if tests fail.

Sources : experiments/bindings/python-ws-client/integration_test.sh:17-33 experiments/bindings/python-ws-client/integration_test.sh:47-88

README Test Extraction Process

The extract_readme_tests.py script implements a documentation-as-test pattern by converting code blocks from README files into executable pytest functions. This ensures that user-facing examples remain functional.

graph LR
    README["README.md"]
SCRIPT["extract_readme_tests.py"]
BLOCKS["extract_python_blocks()"]
STRIP["strip_non_ascii()"]
WRAP["wrap_as_test_fn()"]
OUTPUT["tests/test_readme_blocks.py"]
README --> SCRIPT
 
   SCRIPT --> BLOCKS
 
   BLOCKS --> STRIP
 
   STRIP --> WRAP
 
   WRAP --> OUTPUT
    
    style SCRIPT fill:#f9f9f9
    style OUTPUT fill:#f9f9f9

Extraction Workflow

Extraction Pipeline : The script reads README.md, applies a regex pattern to find Python code blocks, strips non-ASCII characters to prevent encoding issues, wraps each block in a pytest function, and writes the result to tests/test_readme_blocks.py.

Sources : experiments/bindings/python-ws-client/extract_readme_tests.py:15-46

Key Functions

Sources : experiments/bindings/python-ws-client/extract_readme_tests.py:21-45

Generated Test Structure

The extraction process transforms README examples into isolated test functions. For example, a README block:

Becomes:

Each test function is isolated, meaning they can be run independently or in parallel by pytest. The header includes import pytest to ensure pytest-specific features are available if needed.

Sources : experiments/bindings/python-ws-client/extract_readme_tests.py:30-41

Full-Stack Integration Test Script

The integration_test.sh script provides end-to-end testing by managing the complete lifecycle of a WebSocket server and executing tests against it.

Script Configuration

The script defines configuration constants at the top:

Sources : experiments/bindings/python-ws-client/integration_test.sh:7-14

Server Lifecycle Management

Server Lifecycle : The script uses Bash job control (set -m) to start the server in the background and capture its PID. The server is launched via cargo run --package simd-r-drive-ws-server -- $STORAGE_FILE --host $SERVER_HOST --port $SERVER_PORT. The trap cleanup EXIT ensures that the server process group is killed and storage is removed, even if tests fail or the script is interrupted.

Sources : experiments/bindings/python-ws-client/integration_test.sh:47-56 experiments/bindings/python-ws-client/integration_test.sh:19-33

Cleanup Function

The cleanup() function handles resource deallocation:

Key features:

• Process Group Kill : Uses -$SERVER_PID to kill the entire process group, ensuring child processes are also terminated
• Idempotent : The || true suffix prevents errors if the process already exited
• Trap Registration : Registered with trap cleanup EXIT to execute on any exit (success, failure, or interrupt)

Sources : experiments/bindings/python-ws-client/integration_test.sh:19-33

graph TB
    CHECK["Check uv availability\ncommand -v uv"]
VENV["uv venv\nCreate virtual environment"]
BASE["uv pip install pytest maturin"]
DEV["uv pip install -e . --group dev"]
EXTRACT["uv run extract_readme_tests.py"]
PYTEST["uv run pytest -v -s"]
CHECK -->|exists| VENV
 
   CHECK -->|not found| EXIT["Exit with error message"]
VENV --> BASE
 
   BASE --> DEV
 
   DEV --> EXTRACT
 
   EXTRACT --> PYTEST
    
    style CHECK fill:#f9f9f9
    style PYTEST fill:#f9f9f9

Python Environment and Dependency Management

The integration test script uses uv as the Python environment and dependency manager.

Environment Setup Workflow

Environment Setup : The script first verifies uv availability, then creates a virtual environment, installs base dependencies (pytest, maturin), installs the client package in editable mode with development dependencies, and finally runs the test extraction and pytest.

Sources : experiments/bindings/python-ws-client/integration_test.sh:62-87

Dependency Specification

The uv.lock file specifies exact versions of all dependencies:

Sources : experiments/bindings/python-ws-client/uv.lock:1-243

Environment Variable Export

Before running pytest, the script exports server connection details as environment variables:

This allows test code to access server configuration without hardcoding values. Tests can retrieve these via os.environ.get("TEST_SERVER_HOST").

Sources : experiments/bindings/python-ws-client/integration_test.sh:84-87

Integration Test Execution Flow

Complete Execution Sequence

Execution Flow : The integration test follows a strict sequence: (1) register cleanup trap, (2) start server in background, (3) setup Python environment, (4) extract and generate tests, (5) export configuration, (6) run pytest, (7) cleanup resources. Each step logs progress for debugging.

Sources : experiments/bindings/python-ws-client/integration_test.sh:36-89

Error Handling

The script uses set -e at the beginning to exit immediately on any error. This ensures that:

1. If the server fails to start, tests don’t run
2. If environment setup fails, tests don’t run
3. If tests fail, the script exits with non-zero code
4. In all cases, the cleanup() function executes via the trap

Sources : experiments/bindings/python-ws-client/integration_test.sh:4-5

Verbose Output

The script uses extensive echo statements to provide visibility into each stage:

Each major step is prefixed with either --- (major section) or --> (sub-step) for clarity.

Sources : experiments/bindings/python-ws-client/integration_test.sh:36-88

Test Infrastructure File Structure

The integration testing system consists of the following key files:

experiments/bindings/python-ws-client/
├── integration_test.sh              # Main orchestration script
├── extract_readme_tests.py          # README extraction utility
├── README.md                         # Source of test examples
├── uv.lock                          # Dependency lock file
├── pyproject.toml                   # Python project configuration
├── Cargo.toml                       # Rust build configuration
└── tests/
    └── test_readme_blocks.py        # Generated test file (created by extract_readme_tests.py)

graph TB
    README["README.md\nDocumentation with examples"]
EXTRACT["extract_readme_tests.py\nExtraction script"]
TESTFILE["tests/test_readme_blocks.py\nGenerated pytest file"]
INTEGRATION["integration_test.sh\nOrchestration script"]
UVLOCK["uv.lock\nDependency specifications"]
PYPROJECT["pyproject.toml\nPackage metadata"]
CARGO["Cargo.toml\nRust build config"]
SERVER["../../../simd-r-drive-ws-server\nServer binary"]
README --> EXTRACT
 
   EXTRACT --> TESTFILE
 
   INTEGRATION --> EXTRACT
 
   INTEGRATION --> TESTFILE
 
   INTEGRATION --> SERVER
 
   UVLOCK --> INTEGRATION
 
   PYPROJECT --> INTEGRATION
 
   CARGO --> INTEGRATION
    
    style INTEGRATION fill:#f9f9f9
    style TESTFILE fill:#f9f9f9

File Dependencies

File Dependencies : The integration_test.sh script depends on the WebSocket server binary (built from simd-r-drive-ws-server), the extract_readme_tests.py script, and package configuration files. The extraction script depends only on README.md and generates tests/test_readme_blocks.py, which is consumed by pytest.

Sources : experiments/bindings/python-ws-client/integration_test.sh:1-91 experiments/bindings/python-ws-client/extract_readme_tests.py:1-46

Running Integration Tests

Manual Execution

To run the integration tests manually:

The script is self-contained and handles all setup and cleanup automatically.

Sources : experiments/bindings/python-ws-client/integration_test.sh:1-5

Prerequisites

Before running integration tests, ensure:

1. Rust Toolchain : Required to build simd-r-drive-ws-server
2. uv Package Manager : Install from https://github.com/astral-sh/uv
3. Python 3.10+ : Required by the client package
4. Port Availability : Ensure port 34129 is available

Sources : experiments/bindings/python-ws-client/integration_test.sh:62-68 experiments/bindings/python-ws-client/README.md:17-23

CI/CD Integration

The integration test script is designed for use in CI/CD pipelines. The GitHub Actions workflow python-net-release.yml invokes it to validate builds before publishing packages. The script’s exit code (0 for success, non-zero for failure) integrates naturally with CI systems.

Sources : experiments/bindings/python-ws-client/README.md38

Test Isolation and Repeatability

Storage File Isolation

Each test run uses a unique temporary storage file (/tmp/simd-r-drive-pytest-storage.bin) that is deleted during cleanup. This ensures:

• No state persists between test runs
• Tests are repeatable
• Parallel test runs on the same machine don’t conflict (as long as they use different ports)

Sources : experiments/bindings/python-ws-client/integration_test.sh12 experiments/bindings/python-ws-client/integration_test.sh28

Test Function Isolation

Generated test functions are isolated at the pytest level. Each test_readme_block_{i}() function runs independently, allowing pytest to:

• Run tests in parallel with pytest -n auto (if pytest-xdist is installed)
• Report individual test failures
• Skip or mark tests individually

Sources : experiments/bindings/python-ws-client/extract_readme_tests.py:30-34

Port Configuration

The default port (34129) is configurable via script variables. For concurrent testing, users can modify SERVER_PORT to avoid conflicts. The port is exported as TEST_SERVER_PORT for test code to access.

Sources : experiments/bindings/python-ws-client/integration_test.sh14 experiments/bindings/python-ws-client/integration_test.sh85

Debugging Integration Tests

Server Output

The server runs in the foreground of its process group, so its stdout/stderr are visible in the terminal. To capture server logs separately, modify the cargo run command:

Sources : experiments/bindings/python-ws-client/integration_test.sh53

Pytest Verbose Mode

The script runs pytest with -v -s flags:

• -v: Verbose output showing individual test names
• -s: Don’t capture stdout/stderr, allowing print statements to display

To increase verbosity further, modify the pytest invocation to use -vv or -vvv.

Sources : experiments/bindings/python-ws-client/integration_test.sh87

Manual Cleanup

If the script is interrupted before cleanup completes, manually remove resources:

Sources : experiments/bindings/python-ws-client/integration_test.sh:19-29

Summary

The Python integration testing system provides two complementary validation mechanisms:

1. Documentation Validation : extract_readme_tests.py ensures that README examples remain executable and correct by converting them to pytest functions.

2. Full-Stack Validation : integration_test.sh orchestrates a complete test environment with WebSocket server lifecycle management, Python environment setup, and pytest execution.

This architecture ensures that both user-facing documentation and end-to-end functionality remain correct across platforms and versions. The use of uv for Python dependency management and Bash scripting for orchestration makes the system portable and reliable.

Sources : experiments/bindings/python-ws-client/integration_test.sh:1-91 experiments/bindings/python-ws-client/extract_readme_tests.py:1-46

Dismiss

Refresh this wiki

Enter email to refresh