Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog

Latency, errors, traffic and saturation tell you your app is up. They don't tell you if it's right. Marina Petzel, GenAI advocate at Datadog, explains why the classic golden signals are necessary but no longer enough for generative AI applications, and which metrics to add on top. She covers cost monitoring (token creep, model drift, uncached calls and tagging for cost attribution), safety monitoring (prompt injection, PII leakage, toxicity and jailbreaks) and quality monitoring (hallucination rate, relevance, user satisfaction, completeness and RAG retrieval quality). In this talk: • Four ways GenAI apps break traditional monitoring assumptions • Three hidden cost drivers and a four-level tagging strategy • Safety metrics and the thresholds to aim for • Five quality metrics, from hallucination rate to RAG retrieval quality SPEAKER Marina Petzel, GenAI Advocate, Datadog LinkedIn: https://www.linkedin.com/in/marina-petzel/ LINKS Datadog Agent Observability: https://www.datadoghq.com/products/ai/agent-observability/ LLM Observability docs: https://docs.datadoghq.com/llm_observability/ Datadog: https://www.datadoghq.com CHAPTERS 0:00 Intro 0:12 Golden signals aren't enough anymore 1:12 How GenAI apps are different 2:22 New attack vectors 2:47 Quality is subjective 3:51 Cost monitoring 4:56 Token creep, model drift, uncached calls 6:55 Cost attribution: tag everything 9:15 Safety metrics 9:55 Prompt injection, PII, toxicity, jailbreaks 12:09 Quality monitoring 13:19 Five quality metrics 15:43 Putting it all together 16:38 Datadog Agent Observability Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMObservability #GenAI #AIEngineer