Andreas Weininger

Leading Technical Professional, IBM

Andreas Weininger is part of the Information Architecture team of IBM in Munich. He has more than 20 year of experience with scalable solutions for data analytics focusing especially on the performance aspects. He is a regular speaker at database and data analytics events, and has also frequently published articles on these topics.

Past sessions

Summit Europe 2020 Polymorphic Table Functions: The Best Way to Integrate SQL and Apache Spark

November 18, 2020 04:00 PM PT

Databases store not only the data used for computations in Spark, but they often also want to consume the output of Spark computations directly in queries using the output like a relational table. So-called polymorphic table functions provide an mechanism for achieving this. This presentation explains what polymorphic table functions are, how they are used, and why they are a very efficient way of communicating between Spark and a SQL engine minimizing the amount of network traffic and maximizing the used parallelism by co-locating the workers of the SQL engine and the Spark executors. In addition use cases are presented like performing complex transformations on a table in the SQL engine by passing the table as argument to a polymorphic table function and then using the result of the transformation again as a table.

Speaker: Andreas Weininger