Vini Jaiswal

Best Practices for Building Robust Data Platform with Apache Spark and DeltaSummit 2020

This talk will focus on Journey of technical challenges, trade offs and ground-breaking achievements for building performant and scalable pipelines from the experience working with our customers. The problems encountered are shared by many organizations and so the lessons learned and best practices are widely applicable.

These include:

• Operational tips and best practices with Apache Spark in production
• How your choice of overall data pipeline design influences performance and outcomes.
• Common misconfigurations that prevent users from getting the most out of their Apache Spark deployment.

Attendees will come out of the session with Best Practices and Strategies that can be applied to their Big Data architecture, such as:

• Optimizing Cost to Drive Business Value
• Achieving Performance at Scale with Apache Spark and Delta Lake
• Ensuring security guarantees including, recommendations on handling GDPR and CCPA requests

Audience: The attendees should have some knowledge of setting up the Big Data Pipelines and Apache Spark.