William Lau, is a senior member of Unity’s Analytics Team, helping build and re-architect the big data infrastructure. Prior he worked on building large-scale distributed services at AppDynamics, Microsoft, and Amazon.
October 25, 2016 05:00 PM PT
Unity is one of the largest game development platforms. We analyze billions of events daily. Our legacy data infrastructure is based on Hive and MapReduce. Having reached a scalability barrier we decided to re-architect our infrastructure to use Spark as the primary processing engine. It supports our realtime and on-demand processing needs. In this talk we will present how we approached migrating from Hive to Spark and the issues that we faced along the way.