Yuliana Havryshchuk

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale.

At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including:

• Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal 
• Performing data quality validations using libraries built to work with spark
• Dynamically generating pipelines that can be abstracted away from users
• Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers
• Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time