Joy is a Distributed Application Architect, 15+ yrs of Application Software development experience, 10+ yrs of Java/Scala development experience, 5+ yrs of work experience in Big-Data and Hadoop application, 2+ yrs of Apache Spark experience with a special interest in distributed/parallel computing, currently working on Cloud and Big Data technologies. Also, he is actively part of various Software architectural organization. Joy is a frequent speaker in various user groups and code camp.
This presentation will provide technical design and development insights in order to set up a Kerberosied (secured) JupyterHub notebook using Spark. Joy will show how Bloomberg set up the Kerberos-based Spark-notebook-integrating JupyterHub, Sparkmagic, and Levy. Sparkmagic provides the Spark kernel for Scala and Python. Livy is one of the most promising open source software to allow to submit Spark jobs over http-based REST interfaces. In this presentation, Joy will highlight the capabilities of Sparkmagic and Livy, along with the gap or development required in order to integrate the software seamlessly to work with your secured Spark cluster. The Kerberos integration techniques that he'll discuss can be applied to other types of authenticators, such as OAuth. No prior knowledge of any of these technologies is requied in order to understand this presentation.