Moving eBay’s Data Warehouse Over to Apache Spark – Spark as Core ETL Platform at eBay – Databricks

Moving eBay’s Data Warehouse Over to Apache Spark – Spark as Core ETL Platform at eBay

Download Slides

How did eBay move their ETL computation from conventional RDBMS environment over to Spark? What did it take to go from a strategic vision to a viable solution? This paper will take you through a journey which lead to an implementation of a 1000+ node Spark Cluster running 10,000+ ETL jobs daily, all done in a span of less than 6 months, by a team with limited Spark experience. We will share the vision, technical architecture, critical Management decisions, Challenges and Road ahead. This will be a unique opportunity to look into this awesome Spark success story at eBay!

Session  hashtag: #EntSAIS13



« back
About Kimberly Curtis

Director of software development in eBay's Data Services and Solutions (DSS) group.

About Brian Knauss

Leading Architecture and Engineering for eBay Marketplaces' Analytics/Big Data platforms, including Relational (EDW), Open-Processing (Hadoop), Semi-Structured, Business Intelligence, Data Movement, and Data Collaboration.