Jörg Schad is a Developer Evangelist at Mesosphere who works on DC/OS and Apache Mesos. Prior to this he worked on SAP Hana and in the Information Systems Group at Saarland University. His passions are distributed (database) systems, data analytics, and distributed algorithms and his speaking experience include various Meetups, international conferences, and lecture halls.
The current craze of Docker has everyone sticking their processes inside a container... but do you really understand cgroups and how they work? Do you understand the difference between CPU Sets and CPU Shares? Spark is a Scala application that lives inside a Java Runtime, do you understand the consequence of what impact the cgroup constraints have on the JRE? This talk starts with a deep understand of Java's memory management and GC characteristics and how JRE characteristics change based on core count. We will continue the talk looking at containers and how resource isolation works. The session will detail specifically the difference between CPU sets and CPU shares and memory management. The session will close with a deep understanding of the consequences of running the JRE in a CPU share environment and the potential for pseudo-random behavior of running in a heterogeneous datacenter.
There are an ever increasing number of use cases, like online fraud detection, for which the response times of traditional batch processing are too slow. In order to be able to react to such events in close to real-time, you need to go beyond classical batch processing and utilize stream processing systems such as Apache Spark Streaming, Apache Flink, or Apache Storm. These systems, however, are not sufficient on their own. For an efficient and fault-tolerant setup, you also need a message queue and storage system. One common example for setting up a fast data pipeline is the SMACK stack. SMACK stands for Spark (Streaming) - the stream processing system Mesos - the cluster orchestrator Akka - the system for providing custom actors for reacting upon the analyses Cassandra - the storage system Kafka - the message queue Setting up this kind of pipeline in a scalable, efficient and fault-tolerant manner is not trivial. First, this workshop will discuss the different components in the SMACK stack. Then, participants will get hands-on experience in setting up and maintaining data pipelines. Session hashtag: #EUeco1