testing-llm
testing-llm equips you with patterns for mocking LLM API calls, evaluating output quality across multiple dimensions, and automating test workflows through planner, generator, and healer agents. Use it to build deterministic CI tests, validate RAG pipelines, and apply structured output validation with Pydantic schemas.
testing-llm provides patterns for mocking LLM responses and validating output quality with DeepEval and RAGAS metrics.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-07-28
testing-llm provides patterns for mocking LLM responses and validating output quality with DeepEval and RAGAS metrics. testing-llm equips you with patterns for mocking LLM API calls, evaluating output quality across multiple dimensions, and automating test workflows through planner, generator, and healer agents. Use it to build deterministic CI tests, validate RAG pipelines, and apply structured output validation with Pydantic schemas.
Use it when
- testing-llm integrates DeepEval and RAGAS to validate AI output quality across multiple dimensions.
- Yes.
Verify before relying
Read SKILL.md below before installing (11 files). Open directory: indexed for reading, not audited.
Install
yonatangross/orchestkit/testing-llm · repository language: TypeScript
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I test LLM API calls with testing-llm?
testing-llm provides patterns for mocking LLM API calls in your unit tests. You can record and replay LLM responses deterministically using VCR.py integration, allowing you to test code that calls OpenAI, Anthropic, and other APIs without making live requests. This enables fast, repeatable CI/CD test runs while validating your integration logic.
What evaluation metrics does testing-llm support?
testing-llm integrates DeepEval and RAGAS to validate AI output quality across multiple dimensions. You can measure faithfulness, detect hallucinations, evaluate RAG pipeline output, and set quality gate thresholds. These frameworks help you catch issues like prompt injection, structured output violations, and semantic drift in your LLM responses.
Can testing-llm validate structured output from AI models?
Yes. testing-llm supports Pydantic schema validation for AI responses, ensuring your LLM outputs conform to expected types and constraints. Combined with DeepEval and RAGAS evaluation metrics, you can both validate structure and measure quality, catching malformed or hallucinated content before it reaches production.
How does testing-llm set up deterministic LLM testing in CI/CD?
testing-llm enables deterministic AI testing by mocking LLM responses and recording them for replay. This removes non-determinism from your CI/CD pipelines, allowing tests to run consistently without external API calls. You can combine mocked responses with quality metrics to build reliable, fast test suites that validate both integration and output quality.
What are testing-llm's agentic test automation capabilities?
testing-llm automates test generation and fixing through planner, generator, and healer agents. The planner designs test strategies, the generator creates test cases, and the healer fixes failing tests. This end-to-end agentic workflow reduces manual test maintenance and scales your LLM evaluation pipelines across golden datasets and quality thresholds.
How does testing-llm help evaluate RAG pipeline quality?
testing-llm evaluates RAG pipeline output using RAGAS and DeepEval metrics to measure retrieval accuracy, generation quality, and hallucination rates. You can set up quality gates, validate faithfulness against source documents, and detect when your pipeline returns incorrect or fabricated information, ensuring reliable retrieval-augmented generation in production.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
LLM & AI Testing Patterns
Patterns and tools for testing LLM integrations, evaluating AI output quality, mocking responses for deterministic CI, and applying agentic test workflows (planner, generator, healer).
Quick Reference
| Area | File | Purpose |
|---|---|---|
| Rules | rules/llm-evaluation.md |
DeepEval quality metrics, Pydantic schema validation, timeout testing |
(truncated - see the full file via the links below)
File tree — 11 files
src/skills/testing-llm/SKILL.md
src/skills/testing-llm/checklists/llm-test-checklist.md
src/skills/testing-llm/examples/llm-test-patterns.md
src/skills/testing-llm/references/deepeval-ragas-api.md
src/skills/testing-llm/references/generator-agent.md
src/skills/testing-llm/references/healer-agent.md
src/skills/testing-llm/references/planner-agent.md
src/skills/testing-llm/rules/_sections.md
src/skills/testing-llm/rules/llm-evaluation.md
src/skills/testing-llm/rules/llm-mocking.md
src/skills/testing-llm/test-cases.json
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Test LLM API integrations with mocks and quality metrics”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
This skill has been split into five specialized sub-skills, each addressing a distinct testing need. Choose testing-unit for isolated logic, testing-e2e for browser automation, testing-integration for cross-boundary APIs and databases, testing-llm for AI evaluation, or testing-perf for load and performance work.
testing-e2e enables you to craft comprehensive end-to-end tests leveraging Playwright's semantic locators and page object model architecture. This skill simplifies test automation by providing structured patterns for reliable, maintainable test suites that scale across complex applications.
Write unit tests for PydanticAI agents without calling real APIs using TestModel for deterministic outputs, FunctionModel for custom logic, or VCR cassettes to replay recorded interactions. Mock dependencies, force tool calls, and validate agent behavior with inline snapshots.
This skill delivers operational patterns for tuning LLM inference performance, cost, and reliability in production. It covers latency budgeting, continuous batching, KV-cache optimization, speculative decoding, quantization strategies, and parallelism choices—with decision trees and checklists to guide infrastructure and serving stack tuning.
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
This skill provides patterns for k6 load testing, Locust stress testing, and pytest execution optimization. It covers staged ramp-up configuration, threshold setup, test type classification (load, stress, spike, soak), and parallel execution with pytest-xdist for faster test suites.