Siu Kwan Lam is a software developer at Continuum Analytics and the lead developer of the Numba open source compiler project. He has a B.S. and M.S. degree in Computer Engineering from San Jose State University. He taught CUDA at San Jose State University during his senior year and has researched TCP covert channel detection for NSF, STC, and TRUST.
We'll demonstrate how Python and the Numba JIT compiler can be used for GPU programming that easily scales from your workstation to an Apache Spark cluster. Using an example application, we show how to write CUDA kernels in Python, compile and call them using the open source Numba JIT compiler, and execute them both locally and remotely with Spark. We also describe techniques for managing Python dependencies in a Spark cluster with the tools in the Anaconda Platform. Finally, we conclude with some tips and tricks for getting best performance when doing GPU computing with Spark and Python.