Mastering Spark Unit Testing

Download Slides

Traveling to different companies and building out a number of Spark solutions, I have found that there is a lack of knowledge around how to unit test Spark applications. In this talk we will address that by walking through examples for unit testing, Spark Core, Spark MlLib, Spark GraphX, Spark SQL, and Spark Streaming. We will build and run the unit tests in real time and show additional how to debug Spark as easier as any other Java process. The end goal is to encourage more developers to build unit tests along side their Spark applications to increase velocity of development, increase stability and production quality.



« back
About Ted Malaska

Ted has seen the world of data from helping out hundreds of different companies while serving as a Printable Solutions Architect at Cloudera to multiple years at the leading game company Blizzard building out data pipelines, and managing data engineering efforts. Now Ted, servers as a Directory of Enterprise Architecture at Capital One, solving data problems at every level of the company.