Secrets in Unity Catalog

Secrets in Unity Catalog store credentials like API keys as governed objects named catalog.schema.secret: created, granted, rotated, and audited with standard Unity Catalog privileges, and redacted from notebook and job logs. This is a full end-to-end demo: create a secret with the REST API and in the Catalog UI, read it in a notebook with dbutils.secrets.get, use it to call the OpenAI API, grant read/reference/write access to a user or group, rotate it programmatically, and audit every access from a system table. Instead of hardcoding credentials in notebooks or job definitions, you reference a secret by its catalog.schema.secret name. Because it lives in Unity Catalog, one grant governs it across every workspace attached to the same metastore, and values are encrypted at rest and automatically redacted from logs. Compute requirement: a serverless environment on version 4 or higher, or classic compute in standard access mode running Databricks Runtime 17.3 LTS or above. Chapters: 00:00 Why credential management is hard (API keys, passwords, tokens) 00:30 What a Unity Catalog secret is: the catalog.schema.secret object 01:02 Cross-workspace access, encryption at rest, and log redaction 01:34 Compute requirements: serverless env 4+ or DBR 17.3 LTS+ 02:07 Create a secret with the REST API 02:37 Create a secret in the Catalog UI 03:10 Read a secret with dbutils.secrets.get 03:41 Use the secret to call the OpenAI API 04:14 Grant permissions: read, reference, and write secret 04:45 Rotate a secret programmatically 05:47 Audit secret access with system tables 06:17 Drop a secret and wrap-up Resources & Documentation: 📝 Documentation: https://docs.databricks.com/aws/en/security/secrets/unity-catalog-secrets 📝Notebook: https://github.com/youssefmrini/nextgenlakehouse/blob/main/Unity%20Catalog%20Secrets%20%E2%80%94%20Demo%20%26%20Script.ipynb Connect with the community: 🚀 Subscribe to the NextGenLakehouse Newsletter: https://www.nextgenlakehouse.com/ 👥 Join our Reddit Community: https://www.reddit.com/r/databricks/ 🐦 Follow us on X: https://x.com/YoussefMrini 📸 Follow us on Instagram: https://www.instagram.com/nextgenlakehouse 🎥 Follow us on TikTok: https://www.tiktok.com/@nextgenlakehouse