Apache Avro as a Built-in Data Source in Apache Spark 2.4
Try this notebook in Databricks Apache Avro is a popular data serialization format. It is widely used in the Apache Spark and Apache...
How to Avoid Drowning in GDPR Data Subject Requests in a Data Lake
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. With GDPR enforcement rapidly...
Introducing Low-latency Continuous Processing Mode in Structured Streaming in Apache Spark 2.3
Import this notebook on Databricks Structured Streaming in Apache Spark 2.0 decoupled micro-batch processing from its high-level APIs for a couple of reasons...
Databricks Delta: A Unified Data Management System for Real-time Big Data
Combining the best of data warehouses, data lakes and streaming For an in-depth look and demo, join the webinar . Today we are...
Introducing Apache Spark 2.2
Today we are happy to announce the availability of Apache Spark 2.2.0 on Databricks as part of the Databricks Runtime 3.0. This release...
Processing Data in Apache Kafka with Structured Streaming in Apache Spark 2.2
This is the third post in a multi-part series about how you can perform complex streaming analytics using Apache Spark. In this blog...
Working with Complex Data Formats with Structured Streaming in Apache Spark 2.1
In part 1 of this series on Structured Streaming blog posts, we demonstrated how easy it is to write an end-to-end streaming ETL...
Real-time Streaming ETL with Structured Streaming in Apache Spark 2.1
Explore why lakehouses are the data architecture of the future with the father of the data warehouse, Bill Inmon. Try this notebook in...
Introducing Apache Spark Datasets
Developers have always loved Apache Spark for providing APIs that are simple yet powerful, a combination of traits that makes complex analysis possible...
Apache Spark 1.5 DataFrame API Highlights: Date/Time/String Handling, Time Intervals, and UDAFs
To try new features highlighted in this blog post, download Spark 1.5 or sign up Databricks for a 14-day free trial today...