David Palaitis - Databricks

David Palaitis

Engineer, Two Sigma Investments

David Palaitis is an engineering manager with Two Sigma Investments. In this role, he is responsible for the compute and analysis platform that powers the firms research and investment strategies across the world’s financial markets.


New Directions in pySpark for Time Series Analysis

Whether it's Internet of Things (IoT), analysis of Financial Data, or Adtech, the arrival of events in time order requires tools and techniques that are noticeably missing from the Pandas and pySpark software stack. In this talk, we'll cover Two Sigma's contribution to time series analysis for Spark, our work with Pandas, and propose a roadmap for to future-proof pySpark and establish Python as a first class language in the Spark Ecosystem.