{"enrichment":{"faq":[{"a":"testing-llm provides patterns for mocking LLM API calls in your unit tests. You can record and replay LLM responses deterministically using VCR.py integration, allowing you to test code that calls OpenAI, Anthropic, and other APIs without making live requests. This enables fast, repeatable CI/CD test runs while validating your integration logic.","q":"How do I test LLM API calls with testing-llm?"},{"a":"testing-llm integrates DeepEval and RAGAS to validate AI output quality across multiple dimensions. You can measure faithfulness, detect hallucinations, evaluate RAG pipeline output, and set quality gate thresholds. These frameworks help you catch issues like prompt injection, structured output violations, and semantic drift in your LLM responses.","q":"What evaluation metrics does testing-llm support?"},{"a":"Yes. testing-llm supports Pydantic schema validation for AI responses, ensuring your LLM outputs conform to expected types and constraints. Combined with DeepEval and RAGAS evaluation metrics, you can both validate structure and measure quality, catching malformed or hallucinated content before it reaches production.","q":"Can testing-llm validate structured output from AI models?"},{"a":"testing-llm enables deterministic AI testing by mocking LLM responses and recording them for replay. This removes non-determinism from your CI/CD pipelines, allowing tests to run consistently without external API calls. You can combine mocked responses with quality metrics to build reliable, fast test suites that validate both integration and output quality.","q":"How does testing-llm set up deterministic LLM testing in CI/CD?"},{"a":"testing-llm automates test generation and fixing through planner, generator, and healer agents. The planner designs test strategies, the generator creates test cases, and the healer fixes failing tests. This end-to-end agentic workflow reduces manual test maintenance and scales your LLM evaluation pipelines across golden datasets and quality thresholds.","q":"What are testing-llm's agentic test automation capabilities?"},{"a":"testing-llm evaluates RAG pipeline output using RAGAS and DeepEval metrics to measure retrieval accuracy, generation quality, and hallucination rates. You can set up quality gates, validate faithfulness against source documents, and detect when your pipeline returns incorrect or fabricated information, ensuring reliable retrieval-augmented generation in production.","q":"How does testing-llm help evaluate RAG pipeline quality?"}],"shadow_tags":["ai-quality-assurance","response-validation","deterministic-testing","evaluation-framework","agentic-automation","mock-responses","metric-thresholds","rag-testing","prompt-safety","ci-integration"],"summary_rewrite":"testing-llm equips you with patterns for mocking LLM API calls, evaluating output quality across multiple dimensions, and automating test workflows through planner, generator, and healer agents. Use it to build deterministic CI tests, validate RAG pipelines, and apply structured output validation with Pydantic schemas."},"files":[{"bytes":7509,"path":"src/skills/testing-llm/SKILL.md","sha256":"7cc4430fafdc7b8878d71327f70d42a9a6357b21cb73c9673b8ea5a350422c5e","url":"https://skillfed.io/files/yonatangross/orchestkit/testing-llm/cdb92895/SKILL.md"}],"id":"yonatangross/orchestkit/testing-llm","links":{"html":"https://skillfed.io/yonatangross/orchestkit/testing-llm","md":"https://skillfed.io/yonatangross/orchestkit/testing-llm.md","repo":"https://github.com/yonatangross/orchestkit"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":20,"language":"TypeScript","last_updated":"2026-07-28","license":"MIT","name":"testing-llm","publisher":"yonatangross","stars":208},"relations":{"categories":["testing","ai-engineering","quality-assurance"],"similar":[{"id":"sammcj/agentic-coding/deepeval"},{"id":"yonatangross/orchestkit/testing-patterns"},{"id":"yonatangross/orchestkit/testing-unit"},{"id":"yonatangross/orchestkit/cover"},{"id":"yonatangross/orchestkit/testing-e2e"},{"id":"ancoleman/ai-design-components/evaluating-llms"},{"id":"yonatangross/orchestkit/llm-integration"},{"id":"confident-ai/deepeval/deepeval-tracing"},{"id":"existential-birds/beagle/pydantic-ai-testing"},{"id":"confident-ai/deepeval/deepeval"}]},"slug":{"owner":"yonatangross","repo":"orchestkit","skill":"testing-llm"},"version":"cdb92895"}