Production Data Applications for AI-Driven Observability

The Data Intelligence Platform is the first of its kind with a massive product surface area. The primary challenge that comes with that is the difficulty in building observability and management frameworks into the platform for non technical users. This talk will show how to combine system tables, audit logging, and other Databricks APIs to create scalable Production Data Apps in Plotly Dash to give non technical users a unified end-to-end workflow to manage their Data Intelligence Platform. We will then showcase how to embed LLMs into the app to enable non technical users to ask questions of their specific data models to set up alerts, produce new charts, and create custom actions with no coding necessary. These apps allow python developers to create, share, and even commercialize re-usable operational user workflows and put non technical users in the drivers seat. We will cover use cases from financial management to access audit analysis and compute/storage resource utilization.