In this talk, we will highlight major efforts happening in the Spark ecosystem. In particular, we will dive into the details of adaptive and static query optimizations in Spark 3.0 to make Spark easier to use and faster to run. We will also demonstrate how new features in Koalas, an open source library that provides Pandas-like API on top of Spark, helps data scientists gain insights from their data quicker.
Reynold is an Apache Spark PMC member and the top contributor to the project. He initiated and led efforts such as DataFrames and Project Tungsten. He is also a cofounder and Chief Architect at Databricks.