In-Memory Storage Evolution in Apache Spark

This talk will summarize recent activities in Apache Spark developer’s community to enhance columnar storage in Spark 2.3. Columnar storage is known as an efficient format for keeping consecutive fields of a column. On the other hand, previous versions of Spark used columnar storage in a few places. Columnar storage was an internal data structure. Spark 2.3 published an abstract class ColumnVector as a public API. Then, Spark 2.3 uses ColumnVector to effectively support several columnar storages with huge performance improvements. Pre-Spark 2.3 uses columnar storages for reading Apache Parquet and creating table cache in a program written in SQL, DataFrame, or Dataset e.g. df.cache.

These columnar storages are accessed using different internal APIs. This difference led to performance inefficiency of table cache. Spark 2.3 defined ColumnVector as a public API. Then, Spark 2.3 can read data in Apache Arrow and Apache ORC thru ColumnVector without extra data conversion and data copy. While PySpark in pre-Spark 2.3 had huge overhead regarding serialization and desterilization, Spark 2.3 eliminated this overhead by using to use pandas with Apache Arrow. Thus, Spark 2.3 improves performance of PySpark. Spark 2.3 accesses columnar storage for table cache thru ColumnVector without data copy. Spark 2.3 also improves performance of table cache.

Here are takeaways of this talk:
(1) ColumnVector in Spark 2.3 is a public API of columnar storage to exchange data with other columnar storages.
(2) Spark 2.3 uses ColumnVector to exchange famous columnar storages Apache Arrow and Apache ORC with low overhead, and improves performance.
(3) Spark 2.3 and later versions improve performance of PySpark by using Pandas.
(4) Spark 2.3 and later versions use ColumnVector for table cache and improved performance.


Try Databricks
See More Spark + AI Summit in San Francisco 2019 Videos

« back
About Dr. Kazuaki Ishizaki

Dr. Kazuaki Ishizaki is a senior technical staff member at IBM Research - Tokyo. He has over 20 years of experience conducting research and development of dynamic compilers for Java and other languages. He is an expert in compiler optimizations, runtime systems, and parallel processing. He has been working for IBM Java just-in-time compiler and virtual machine from JDK 1.0 to Java 8. His research has focused on how system software can enable programmers to automatically exploit hardware accelerators, such as SIMD, GPU, and deep learning accelator, in high-level languages and frameworks. He is an Apache Spark committer, working for SQL component. He is an ACM distinguished member.