The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

Benchmarks won't tell you how a model behaves in your app. Swapping it in and testing will. NEXT UP: AI ENGINEER CODE SUMMIT Join us in SF for the second annual AIE CODE Summit, Nov 10–12, our last flagship summit of 2026. Applications live now: https://aie.codes Pamela Fox, Principal Cloud Advocate at Microsoft, runs the same scenarios against GPT-5.5, Kimi K2.6, DeepSeek V4 Flash and Mistral Large 3 on Microsoft Foundry so you can see where models differ and what you can do about it. She works up from a letter-counting prompt to RAG groundedness, image input, multilingual pitfalls, tool calling, tool calling in a loop and a LangChain v1 travel agent, then moves to evaluation: string-check evals, LLM-as-a-judge, Microsoft's new ASSERT tool for generating test sets from a spec, and DSPy prompt optimization that finds a prompt the cheapest model can satisfy. In this video: - Where top models differ: counting, groundedness, vision, tool calls and parallel tool calling - How prompt engineering closes some gaps but not image understanding - Bulk evals with string checks and LLM-as-a-judge - Generating test sets with ASSERT and optimizing prompts with DSPy SPEAKER Pamela Fox, Principal Cloud Advocate, Microsoft X: https://x.com/pamelafox LinkedIn: https://www.linkedin.com/in/pamela-s-fox GitHub: https://github.com/pamelafox LINKS ASSERT (GitHub): https://github.com/responsibleai/ASSERT CHAPTERS 0:00 Workshop setup 2:23 Why you need to be able to swap models 3:08 The four models 5:18 Benchmarks vs your scenario 5:38 Test 1: counting letters 7:48 Fixing it with prompt engineering 8:38 Test 2: RAG and groundedness 13:03 Test 3: image input 16:18 Switching languages 17:13 Test 4: tool calling 21:03 Test 5: tool calling in a loop 23:58 Agents with LangChain v1 26:17 Parallel tool calls 30:27 Evals with string checks 33:42 Q&A: eval frameworks 35:12 LLM as a judge 40:07 ASSERT: evals from a spec 43:32 ASSERT results across models 45:52 Prompt optimization with DSPy 47:52 A DSPy prompt for Mistral 49:22 Anthropic and citations 50:32 Wrap-up Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLM #Evals #AIEngineer