Andy Steinbach has over 20 years experience developing revolutionary new technologies and products in the semiconductor & imaging technology domains. Andy holds a PhD in device physics from the University of Colorado, Boulder and was a National Science Foundation postdoctoral Fellow at CEA Saclay in France. He has developed novel optoelectronic and consumer electronic devices at JDS Uniphase and Intel. More recently, Andy led a team developing data science and machine learning techniques for the microscopy domain at Carl Zeiss. Andy is now Senior Director at NVIDIA developing the deep learning market for industrial applications.
Spark’s large success stems from providing a comprehensive environment for both data analytics & predictive analytics (machine learning) scaled out across multi-node clusters holding big data in memory. Successfully enabling the power of deep learning, with its superior performance on unstructured data, will be a key requirement for Spark’s continued ascendancy. Deep learning however relies on training with very large datasets to achieve superior performance, and this leads to a need for GPU-enablement to train models in business-like timeframes. We review different methods for achieving this GPU-enabled deep learning within Spark and the broader Spark ecosystem, and discuss potential strategies for future progress.
High Performance Data Analytics (HPDA) is an emerging market being driven by the convergence of big data with HPC. Catalyzing this emergence are the spectacular recent successes of AI (machine learning and especially deep learning) in extracting knowledge from today's big data, and across a huge range of domains. Just as GPU-computing has dramatically improved performance and lowered cost in highly-scaled HPC applications, GPU-computing now promises to play a key role in driving breakthrough performance and dramatic cost reduction in HPDA. To achieve its full potential, GPU integration into the Spark environment will be a critical step. We report on progress and plans for enhanced GPU-integration into Spark, and discuss key issues around the reliability, scalability and the ability to use deep learning frameworks in the Spark environment, including recent proof points of this integration.