Weiwei Yang is a Staff Software Engineer from Cloudera, an Apache Hadoop committer and PMC member. He is focused on evolving large scale, hybrid computation systems, he has lots of experience with building mission-critical infrastructure for storage and computes. Before Cloudera, he worked in Alibaba’s realtime computation infrastructure team that serves large scale big data workloads. Currently, Weiwei is leading the efforts for resource scheduling and management on K8s in Cloudera. Weiwei holds a master’s degree from Peking University.
Kubernetes is the most popular container orchestration system that is natively designed for Cloud. At Lyft and Cloudera, we have both emerged the next-generation, cloud-native infrastructure based on Kubernetes, which supports various distributed workloads. We embrace Apache Spark for data engineering and machine learning, and by running Spark on Kubernetes, we are able to exploit compute power promisingly under such highly elastic, scalable and decoupled architecture. We made a lot of effort on enhancing the core resource scheduling, in order to bring high performance, efficient-sharing and multi-tenancy oriented capabilities to Spark jobs. In this talk, we will focus on revealing the architecture of the cloud-native infrastructure; How we leverage YuniKorn - an open-source resource scheduler to redefine the resource scheduling on Cloud. We will introduce how YuniKorn manages quotas, resource sharing, and auto-scaling, and ultimately how to schedule large scale Spark jobs efficiently on Kubernetes in the cloud.