testing-llm
testing-llm equips you with patterns for mocking LLM API calls, evaluating output quality across multiple dimensions, and automating test workflows through planner, generator, and healer agents. Use it to build deterministic CI tests, validate RAG pipelines, and apply structured output validation with Pydantic schemas.
testing-llm provides patterns for mocking LLM responses and validating output quality with DeepEval and RAGAS metrics.
AI-generated summary based on this skill's SKILL.md
Install
yonatangross/orchestkit/testing-llm · repository language: TypeScript
git clone https://github.com/yonatangross/orchestkit
cp -r orchestkit/src/skills/testing-llm ~/.claude/skills/testing-llmnpx skillfed install yonatangross/orchestkit/testing-llmFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I test LLM API calls with testing-llm?
testing-llm provides patterns for mocking LLM API calls in your unit tests. You can record and replay LLM responses deterministically using VCR.py integration, allowing you to test code that calls OpenAI, Anthropic, and other APIs without making live requests. This enables fast, repeatable CI/CD test runs while validating your integration logic.
What evaluation metrics does testing-llm support?
testing-llm integrates DeepEval and RAGAS to validate AI output quality across multiple dimensions. You can measure faithfulness, detect hallucinations, evaluate RAG pipeline output, and set quality gate thresholds. These frameworks help you catch issues like prompt injection, structured output violations, and semantic drift in your LLM responses.
Can testing-llm validate structured output from AI models?
Yes. testing-llm supports Pydantic schema validation for AI responses, ensuring your LLM outputs conform to expected types and constraints. Combined with DeepEval and RAGAS evaluation metrics, you can both validate structure and measure quality, catching malformed or hallucinated content before it reaches production.
How does testing-llm set up deterministic LLM testing in CI/CD?
testing-llm enables deterministic AI testing by mocking LLM responses and recording them for replay. This removes non-determinism from your CI/CD pipelines, allowing tests to run consistently without external API calls. You can combine mocked responses with quality metrics to build reliable, fast test suites that validate both integration and output quality.
What are testing-llm's agentic test automation capabilities?
testing-llm automates test generation and fixing through planner, generator, and healer agents. The planner designs test strategies, the generator creates test cases, and the healer fixes failing tests. This end-to-end agentic workflow reduces manual test maintenance and scales your LLM evaluation pipelines across golden datasets and quality thresholds.
How does testing-llm help evaluate RAG pipeline quality?
testing-llm evaluates RAG pipeline output using RAGAS and DeepEval metrics to measure retrieval accuracy, generation quality, and hallucination rates. You can set up quality gates, validate faithfulness against source documents, and detect when your pipeline returns incorrect or fabricated information, ensuring reliable retrieval-augmented generation in production.
SKILL.md
rendered from the published skill — quoted content, verbatim
LLM & AI Testing Patterns
Patterns and tools for testing LLM integrations, evaluating AI output quality, mocking responses for deterministic CI, and applying agentic test workflows (planner, generator, healer).
Quick Reference
| Area | File | Purpose |
|---|---|---|
| Rules | rules/llm-evaluation.md |
DeepEval quality metrics, Pydantic schema validation, timeout testing |
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 11 files
src/skills/testing-llm/SKILL.md
src/skills/testing-llm/checklists/llm-test-checklist.md
src/skills/testing-llm/examples/llm-test-patterns.md
src/skills/testing-llm/references/deepeval-ragas-api.md
src/skills/testing-llm/references/generator-agent.md
src/skills/testing-llm/references/healer-agent.md
src/skills/testing-llm/references/planner-agent.md
src/skills/testing-llm/rules/_sections.md
src/skills/testing-llm/rules/llm-evaluation.md
src/skills/testing-llm/rules/llm-mocking.md
src/skills/testing-llm/test-cases.json