How did eBay move their ETL computation from conventional RDBMS environment over to Spark? What did it take to go from a strategic vision to a viable solution? This paper will take you through a journey which lead to an implementation of a 1000+ node Spark Cluster running 10,000+ ETL jobs daily, all done in a span of less than 6 months, by a team with limited Spark experience. We will share the vision, technical architecture, critical Management decisions, Challenges and Road ahead. This will be a unique opportunity to look into this awesome Spark success story at eBay!
Session hashtag: #EntSAIS13
Director of software development in eBay's Data Services and Solutions (DSS) group.
Leading Architecture and Engineering for eBay Marketplaces' Analytics/Big Data platforms, including Relational (EDW), Open-Processing (Hadoop), Semi-Structured, Business Intelligence, Data Movement, and Data Collaboration.