Matei Zaharia’s 5 predictions about big data and AI in 2018
Over the past few years, the demand for artificial intelligence (AI) and machine learning capabilities has surged with innovations in natural language processing, task automation, and predictions. From autonomous cars to a more personalized shopping experience, big data and artificial intelligence is at the forefront of new solutions that are delighting customers, improving business operations...
Spark Summit is Becoming the Spark + AI Summit
We’re excited to announce that Spark Summit is expanding its coverage in 2018 to include in-depth content on artificial intelligence. We are also renaming the conference Spark + AI Summit. AI has always been one of the most exciting applications of big data and Apache Spark, so with this change, we are planning to bring...
A Technical Overview of Azure Databricks
This is a joint blog post from Matei Zaharia, Chief Technologist at Databricks and Peter Carlin, Distinguished Engineer at Microsoft. Today at Microsoft Connect(); we introduced Azure Databricks, an exciting new service in preview that brings together the best of the Apache Spark analytics platform and Azure cloud. As a close partnership between Databricks and...
Databricks Delta: A Unified Data Management System for Real-time Big Data
Combining the best of data warehouses, data lakes and streaming For an in-depth look and demo, join the webinar. Today we are proud to introduce Databricks Delta, a unified data management system to simplify large-scale data management. Currently, organizations build their big data architectures using a mix of systems, including data warehouses, data lakes and...
Sharing Knowledge with the Community in a Preview of Apache Spark: The Definitive Guide
Apache Spark has seen immense growth over the past several years. The size and scale of this Spark Summit is a true reflection of innovation after innovation that has made itself into the Apache Spark project. Hundreds of contributors working collectively have made Spark an amazing piece of the technology powering thousands of organizations, and...
Databricks and Apache Spark 2016 Year in Review
In 2016, Apache Spark released its second major version 2.0 and outgrew our wildest expectations: 4X growth in meetup members reaching 240,000 globally, and 2X growth in code contributors reaching 1000. In addition to contributing to the success of Spark, Databricks also had a phenomenal year. We have rolled out a large number of features...
Structured Streaming In Apache Spark
Apache Spark 2.0 adds the first version of a new higher-level API, Structured Streaming, for building continuous applications. The main goal is to make it easier to build end-to-end streaming applications, which integrate with storage, serving systems, and batch jobs in a consistent and fault-tolerant way. In this post, we explain why this is hard...
Continuous Applications: Evolving Streaming in Apache Spark 2.0
Since its release, Spark Streaming has become one of the most widely used distributed streaming engines, thanks to its high-level API and exactly-once semantics. Nonetheless, as these types of engines became common, we’ve noticed that developers often need more than just a streaming programming model to build real-time applications. At Databricks, we’ve worked with thousands of...
Introducing Apache Spark 2.0
Today, we're excited to announce the general availability of Apache Spark 2.0 on Databricks. This release builds on what the community has learned in the past two years, doubling down on what users love and fixing the pain points. This post summarizes the three major themes—easier, faster, and smarter—that comprise Spark 2.0. We also explore many of...
Introducing Databricks Community Edition: Apache Spark for All
As developers at heart, we at Databricks are committed to the development of Apache Spark and the continued growth of the community. Today we took another step towards delivering on that goal with the beta release of Databricks Community Edition, a free version of our cloud-based Spark platform. You can read the press release here,...
