Continuous Applications: Evolving Streaming in Apache Spark 2.0
Since its release, Spark Streaming has become one of the most widely used distributed streaming engines, thanks to its high-level API and exactly-once semantics. Nonetheless, as these types of engines became common, we’ve noticed that developers often need more than just a streaming programming model to build real-time applications. At Databricks, we’ve worked with thousands of...
Introducing Apache Spark 2.0
Today, we're excited to announce the general availability of Apache Spark 2.0 on Databricks. This release builds on what the community has learned in the past two years, doubling down on what users love and fixing the pain points. This post summarizes the three major themes—easier, faster, and smarter—that comprise Spark 2.0. We also explore many of...
Introducing Databricks Community Edition: Apache Spark for All
As developers at heart, we at Databricks are committed to the development of Apache Spark and the continued growth of the community. Today we took another step towards delivering on that goal with the beta release of Databricks Community Edition, a free version of our cloud-based Spark platform. You can read the press release here,...
Databricks 2015 Year In Review: Democratizing Access to Data
To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016. 2015 has been a phenomenal year of growth for both Databricks and the Apache Spark project. In June, we launched general availability (GA) of our cloud platform, the first end-to-end enterprise data platform based on Spark. At the same time,...
Apache Spark 2015 Year In Review
To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016. 2015 has been a year of tremendous growth for Apache Spark. The pace of development is the fastest ever. We went through 4 releases (Spark 1.3 to 1.6) in a single year, and each of them added hundreds of...
Introducing Apache Spark Datasets
To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016. Developers have always loved Apache Spark for providing APIs that are simple yet powerful, a combination of traits that makes complex analysis possible with minimal programmer effort. At Databricks, we have continued to push Spark’s usability and performance envelope...
Spark Survey 2015 Results are now available
We ran the Spark Survey 2015 this summer to gain insights on how organizations are using Apache Spark. Press Release: Apache Spark Outgrowing Hadoop as Users Increasingly Move to the Cloud The results of this year’s Spark Survey - reflecting the answers and opinions of over 1,417 respondents representing 842 organizations - strongly indicate the...
Diving into Apache Spark Streaming’s Execution Model
With so many distributed stream processing engines available, people often ask us about the unique benefits of Apache Spark Streaming. From early on, Apache Spark has provided an unified engine that natively supports both batch and streaming workloads. This is different from other systems that either have a processing engine designed only for streaming, or...
Databricks is now Generally Available
We are excited to announce today, at Spark Summit 2015, the general availability of the Databricks – a hosted data platform from the team that created Apache Spark. With Databricks, you can effortlessly launch Spark clusters, explore data interactively, run production jobs, and connect third-party applications. We believe Databricks is the easiest way to use...
Deep Dive into Spark SQL’s Catalyst Optimizer
Spark SQL is one of the newest and most technically involved components of Spark. It powers both SQL queries and the new DataFrame API. At the core of Spark SQL is the Catalyst optimizer, which leverages advanced programming language features (e.g. Scala's pattern matching and quasiquotes) in a novel way to build an extensible query...
