Democratize Data Discovery and Data Insight at Databricks

In the rapidly advancing morden data world, the power of data is undeniable. However, unlocking this power requires data engineers, data producers, and data consumers must collaborate together to build the data trust to understand the entire picture of the data lifecycle. In this talk, we will discuss how democratizing data insights leads to enhanced decision-making, fosters a data-driven culture, and accelerates innovation by allowing more members of the organization to leverage data in their roles. This will include sharing a few updates in the following areas:

• Catalog Explorer: how we build the smart and intuitive modernized data portal for Databricks Unity Catalog.
• Data Lineage: How do we build the end-to-end lakehouse lineage, including ML lineage for Unity Catalog?
• Data insight: how we build the data insight, including leveraging AI / LLM to generate auto-documentation, surfacing data popularity, and other trust signals.