Qiang Meng is a data engineer at Levis Strauss & Co. leading a team involved in building future data platforms across the entire fashion value chain from design and production to customer experience.
Qiang has over 8 years of experience in creating enterprise analytics products applying Apache Spark, Hadoop, YARN, and Cloud Services. In the past, Qiang worked on the Big Data Platform at H&M Group and Softbank Robotics Europe.
Qiang gained his MSc. in CS from Telecom Paristech and MSc. in Applied Maths from the University of Paris XI. In addition, Qiang is a fashion designer and a proud member of the LGBTQI community.
November 17, 2020 04:00 PM PT
In rapidly changing conditions, many companies build ETL pipelines using ad-hoc strategy. Such an approach makes automated testing for data reliability almost impossible and leads to ineffective and time-consuming manual ETL monitoring.
Software engineering decouples code dependency, enables automated testing, and powers engineers to design, deploy, and serve reliable data in a module manner. As a consequence, the organization is able to easily reuse and maintain its ETL code base.
In this presentation, we discuss the challenges data engineers face when it comes to data reliability. Furthermore, we demonstrate how software engineering best practices help to build code modularity and automated testings for modern data engineering pipelines.
What you'll learn:
- Software engineering best practices in ETL projects: design patterns and modularity
- Automate ETL testings for data reliability: unit test, functional test, and end2end test.
Speaker: Qiang Meng