Large-Scale Text Processing Pipeline with Spark ML and GraphFrames

Download Slides

In this talk we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas.
We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark ML and GraphFrames.

Histogrammar package—a cross-platform suite of data aggregation primitives for making histograms, calculating descriptive statistics and plotting in Scala—is introduced to enable interactive data analysis in Spark REPL.

We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.

About Alexey Svyatkovskiy

Alexey Svyatkovskiy is a staff research scientist in Princeton Institute for Computational Science and Engineering (PICSciE) at Princeton University and a Co-Investigator at the Princeton Plasma Physics Lab (PPPL). He is focusing on Big Data, machine learning and high-performance computing for scientific applications.

He works on several projects including disruption forecasting in tokamak fusion plasmas using deep recurrent neural networks, large-scale text processing and NLP with Spark ML in application to modern American politics, and contributing to the development of statistical analysis and plotting package in Scala.

In his free time, Alexey organizes workshops on Apache Spark and Python at Princeton University, contributes to local data science meetup group and PrincetonPy community. Alexey holds a Ph.D. in particle physics with a specialization in computational engineering from Purdue University. His research focused on Standard Model pf particle interactions and charged particle reconstruction with the CMS detector at the CERN LHC and he has been published in the leading journals in the field.