Last year we presented the current state of the art for Spark on Mesos, including their origins together at UC Berkeley AMPlab, features such as dynamic allocations, major users of Mesos, such as Apple Siri, and the details of job invocation and resource allocation. This year we’ll recap features in the Spark and Mesos integration, emphasizing new features and development in Mesos that can improve Spark deployment and scheduling, such as Quotas and GPU isolation. We’ll finish with a few stories from real-world deployments and show a demo of the Spark on Mesos integration in the context of the maturing SMACK/Infinity stack, integrating Spark, Mesos, Akka, Cassandra, and Kafka (plus other tools) for streaming applications.
Timothy Chen is the CTO of Hyperpilot, and also a PMC/committer on Apache Drill and Apache Mesos. Before joining Hyperpilot, Timothy was the lead engineer at Mesosphere working on container runtime and Spark on Mesos.