High Performance Data Analytics (HPDA) is an emerging market being driven by the convergence of big data with HPC. Catalyzing this emergence are the spectacular recent successes of AI (machine learning and especially deep learning) in extracting knowledge from today’s big data, and across a huge range of domains. Just as GPU-computing has dramatically improved performance and lowered cost in highly-scaled HPC applications, GPU-computing now promises to play a key role in driving breakthrough performance and dramatic cost reduction in HPDA. To achieve its full potential, GPU integration into the Spark environment will be a critical step. We report on progress and plans for enhanced GPU-integration into Spark, and discuss key issues around the reliability, scalability and the ability to use deep learning frameworks in the Spark environment, including recent proof points of this integration.
Andy Steinbach has over 20 years experience developing revolutionary new technologies and products in the semiconductor & imaging technology domains. Andy holds a PhD in device physics from the University of Colorado, Boulder and was a National Science Foundation postdoctoral Fellow at CEA Saclay in France. He has developed novel optoelectronic and consumer electronic devices at JDS Uniphase and Intel. More recently, Andy led a team developing data science and machine learning techniques for the microscopy domain at Carl Zeiss. Andy is now Senior Director at NVIDIA developing the deep learning market for industrial applications.