Mike Olson from Cloudera has called Spark the killer App for Scala.Not only is it written in Scala but it is also deeply integrated in
its toolchain, making intensive use of the Scala REPL and with it the
Scala compiler. One good way to see Spark is as a domain specific
language embedded in Scala as a host language. The domain in this
setting is big data, or, more specifically, collections distributed
over a cluster. This makes Spark the most popular and in some sense
the most powerful implementation of Scala-like collections.
In this talk I will highlight similarities and differences of the
collections used in Scala and Spark. I will outline some areas where
Scala’s collections could benefit from the experience gained from
Spark, and, conversely, where improved Scala tooling can help Spark.