Alexey Svyatkovskiy is a staff research scientist at Princeton University focusing on Big Data, machine learning and high performance computing. He works on several projects including disruption forecasting in tokamak fusion plasmas using deep recurrent neural networks, natural language processing applications to modern American politics, and contributing to the development of data analysis package in Scala. In his free time, Alexey organizes workshops on Apache Spark and Python, contributes to local data science meetup group and PrincetonPy community.
In this talk, we evaluate training of deep recurrent neural networks with half-precision floats on Pascal and Volta GPUs. We implement a distributed, data-parallel, synchronous training algorithm by integrating TensorFlow and CUDA-aware MPI to enable execution across multiple GPU nodes and making use of high-speed interconnects. We introduce a learning rate schedule facilitating neural network convergence at up to O(100) workers. Strong scaling tests performed on GPU clusters show linear runtime scaling and logarithmic communication time scaling for both single and mixed precision training modes. Performance is evaluated on a scientific dataset taken from the Joint European Torus (JET) tokamak, containing multi-modal time series of sensory measurements leading up to deleterious events called plasma disruptions. Half-precision significantly reduces memory and network bandwidth, allowing training of state-of-the-art models with over 70 million trainable parameters while achieving a comparable test set performance as single precision.
In this talk we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas. We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark ML and GraphFrames. Histogrammar package—a cross-platform suite of data aggregation primitives for making histograms, calculating descriptive statistics and plotting in Scala—is introduced to enable interactive data analysis in Spark REPL. We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.