langwatch-scenario

What it is and what it does

Langwatch-scenario is an agent testing framework designed to validate LLM-based agent behavior through automated simulation. It runs multi-turn conversations between your agent implementation and simulated users (powered by LLMs), optionally with judge agents that evaluate outcomes against predefined criteria. You integrate your agent by implementing a single `call()` method, then define scenarios as pytest tests or standalone scripts that describe the context and expected behavior.

The framework handles the conversation loop, message passing, and evaluation. It supports both scripted control (where you define exact turn sequences) and autopilot mode (where the user simulator drives the conversation until success or max turns). Runtime dependencies include pytest, litellm, openai, pydantic, fastapi, and opentelemetry-sdk, reflecting its design for LLM integration, async execution, and observability. The package is actively maintained and available in Python, TypeScript, and Go.

Use it for:

• Test that a customer support agent asks follow-up questions and provides accurate information before resolving tickets.
• Validate a recipe recommendation agent generates vegetarian options and includes ingredient lists and cooking instructions.
• Verify a weather agent calls the correct tool and handles edge cases like missing location data.
• Evaluate a coding assistant's ability to explain code changes and respond to clarification requests.
• Benchmark agent behavior across different LLM models to compare response quality and tool usage.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Langwatch-scenario is an agent testing framework that simulates multi-turn conversations between your agent and LLM-powered user simulators, with optional judge agents to evaluate behavior against criteria.

Yes. The package is actively maintained, has no known vulnerabilities, uses a permissive Apache-2.0 license, and solves a real problem—testing agent behavior in realistic multi-turn scenarios. Low install friction and a large dependency set (28 runtime packages) are typical for LLM testing frameworks. Install if you need to validate agent behavior beyond unit tests or if you're building agents that must handle complex, multi-turn interactions reliably.

Install

langwatch-scenario on PyPI

pip install langwatch-scenario

uv add langwatch-scenario

poetry add langwatch-scenario

Installing langwatch-scenario

Before you install

Low install friction with a pure-Python wheel distribution. Active maintenance with a recent release (1 day old) and 951 GitHub stars. Requires Python 3.10 or later.

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions; you must include a copy of the license and state significant changes.

Quickstart

pip install langwatch-scenario pytest

import scenario
import pytest

@pytest.mark.asyncio
async def test_agent():
    class MyAgent(scenario.AgentAdapter):
        async def call(self, input: scenario.AgentInput):
            return {"role": "assistant", "content": "response"}
    
    result = await scenario.run(
        name="test",
        description="Test scenario",
        agents=[MyAgent(), scenario.UserSimulatorAgent()]
    )
    assert result.success

Requires Python 3.10 or later. Depends on 28 runtime packages including litellm, openai, fastapi, and opentelemetry-sdk; ensure your environment can resolve all transitive dependencies.

Verify before relying

• Whether the framework supports custom evaluation metrics beyond the built-in judge agent criteria.
• Performance characteristics when running many concurrent simulations or long multi-turn conversations.
• Compatibility with non-OpenAI LLM providers beyond what litellm abstracts.

Package facts

Evidence: langwatch_scenario-1.1.1-py3-none-any.whl

Tags

More Testing packages

Further reading