Hive to Spark—Journey and Lessons Learned - Databricks

Hive to Spark—Journey and Lessons Learned

Download Slides

Unity is one of the largest game development platforms. We analyze billions of events daily. Our legacy data infrastructure is based on Hive and MapReduce. Having reached a scalability barrier we decided to re-architect our infrastructure to use Spark as the primary processing engine. It supports our realtime and on-demand processing needs. In this talk we will present how we approached migrating from Hive to Spark and the issues that we faced along the way.

About William Lau

William Lau, is a senior member of Unity's Analytics Team, helping build and re-architect the big data infrastructure. Prior he worked on building large-scale distributed services at AppDynamics, Microsoft, and Amazon.

About Kent Buenaventura

Kent Buenaventura, is a senior member of Unity's Analytics Team, helping build and re-architect the big data infrastructure. Previously, he worked on building scalable game servers at DeNA and developed games at Sunstorm Interactive.