Skip to main content
bash TV

Promptfoo Explained: Test LLM Prompts in CI (+ AI Agents)

TestMu AI (Formerly LambdaTest)

63 views21 Nov 2026

YouTube

Promptfoo is an open-source tool that turns LLM prompts into repeatable tests. Learn the config, run evals, grade with llm-rubric and gate every PR in CI. ⏱ Chapters 0:00 What Promptfoo does + promptfooconfig.yaml 0:12 Running promptfoo eval 0:22 Assertions, llm-rubric and model comparison 0:34 CI before/after, red teaming, where prompts stop 0:47 Long chats, voice IVR, real users 0:59 Keep Promptfoo + add Agent Testing In this Promptfoo tutorial: • Assertions: icontains, regex and llm-rubric (LLM as a judge) • Compare models side by side • GitHub Action: before/after pass rates on every pull request • Red team scans for prompt injection, jailbreaks and PII leaks Promptfoo (now part of OpenAI, still open source) is built for prompts. Testing a whole AI agent end to end, across long chats, voice and IVR calls, and real users, needs more. Keep Promptfoo for prompts, and use TestMu AI Agent Testing for chatbots, voice agents and IVR. 🔗 Agent Testing (chatbots, voice, IVR): https://www.testmuai.com/agent-testing/ 🔗 Promptfoo CI/CD docs: https://www.promptfoo.dev/docs/integrations/ci-cd/ TestMu AI (formerly LambdaTest) is an agentic quality engineering platform for testing web apps, mobile apps and AI agents. #Promptfoo #LLMEvaluation #AIAgents

Join the discussion

Have something to add?

Sign in to reply, follow this thread, and earn XP when your answer helps someone.