Databricks Zerobus + OpenTelemetry: Observability in Unity Catalog

Databricks Zerobus now exposes OpenTelemetry-compatible endpoints that forward logs, metrics, and traces straight into Unity Catalog Delta tables no streaming pipeline, no compute, no proprietary observability tool required. In this NextGenLakehouse episode, I sit down with Bobby, a Senior Solutions Architect at Databricks, to trace observability from fragmented 1990s telemetry to today's open standard. We cover why "build a pipeline or pay Splunk/Datadog/New Relic" breaks at enterprise scale, how the 2019 merge of OpenCensus and OpenTracing created OpenTelemetry, and how Zerobus (part of Lakeflow Connect) now lands that telemetry directly in Unity Catalog. Key takeaways: • Zerobus OTLP endpoints wmetrics, and traces directly to Unity Catalog no Kafka, no multi-hop pipeline, no compute. • Three telemetry pillars ture/network/app telemetry, agent observability via MLks Apps telemetry — all in one Unity Catalog view. • Putting telemetry next tow, Workday, ERP) turnsmonitoring into explainabity and cheap long-termlog retention. Chapters 00:00 Why 30 years of telemetry stayed fragmented 00:56 Meet Bobby: an observability architect at Databricks 01:25 Why "build a pipeline or pay Splunk/Datadog" breaks at scale 02:26 How OpenTelemetry won: the 2019 OpenCensus + OpenTracing merge 03:03 Databricks goes all-in on OpenTelemetry with Zerobus 03:24 Why agent observability is now existential 04:34 Demo: streaming data-center telemetry with Fluent Bit & collectors 05:14 Zerobus OTLP: land telemetry straight into Unity Catalog 06:09 Under the hood: the Fluent Bit config that forwards to Zerobus 07:05 Ask your logs in plain English with a Genie agent 07:43 No compute required: how Zerobus ingestion works 08:25 Root-cause intelligence: linking incidents to revenue impact 09:33 What happens to Splunk, Datadog & proprietary tools 10:21 Cost & governance: telemetry next to ServiceNow and ERP data 11:46 Three pillars: infra, agent (MLflow) & Databricks Apps telemetry 13:06 How app, agent & enterprise teams should get started 15:01 Cheap log storage and going from reactive to proactive 16:17 Try it free on Databricks Free Edition 17:03 Get the solution accelerator on GitHub 🔗 Let's Connect: • Github: https://github.com/bobertx3/dbrks-otel-sim-sidecar • 🌐 Master Databricks at our official hub: https://nextgenlakehouse.com/ • 📩 Subscribe to our Substack for monthly recaps: https://nextgenlakehouse.com • 💼 Connect with Youssef: https://www.linkedin.com/in/youssefmrini