Tianru Zhou is currently working at Databricks on data discovery related projects, including integrating Amundsen with existing infrastructure to do data discovery. Previously, he worked at AWS Elasticsearch on the storage layer development for UltraWarm.
May 27, 2021 03:15 PM PT
Databricks used to use a static manually maintained wiki page for internal data exploration. We will discuss how we leverage Amundsen, an open source data discovery tool from Linux Foundation AI & Data, to improve productivity with trust by surfacing the most relevant dataset and SQL analytics dashboard with its important information programmatically at Databricks internally.
We will also talk about how we integrate Amundsen with Databricks world class infrastructure to surface metadata including:
Last but not least, we will discuss how we incorporate internal user feedback and provide the same discovery productivity improvements for Databricks customers in the future.