Easily Clone your Delta Lake for Testing, Sharing, and ML Reproducibility
Introducing Clones An efficient way to make copies of large datasets for testing, sharing and reproducing ML experiments We are excited to introduce…
Enabling Spark SQL DDL and DML in Delta Lake on Apache Spark 3.0
Last week, we had a fun Delta Lake 0.7.0 + Apache Spark 3.0 AMA where Burak Yavuz, Tathagata Das, and Denny Lee provided…
Time Traveling with Delta Lake: A Retrospective of the Last Year
Try out Delta Lake 0.7.0 with Spark 3.0 today! It has been a little more than a year since Delta Lake became an…
Diving Into Delta Lake: Schema Enforcement & Evolution
Data, like our experiences, is always evolving and accumulating. To keep up, our mental models of the world must adapt to new data,…
Diving Into Delta Lake: Unpacking The Transaction Log
The transaction log is key to understanding Delta Lake because it is the common thread that runs through many of its most important…
Introducing Delta Time Travel for Large Scale Data Lakes
Data versioning for reproducing experiments, rolling back, and auditing data We are thrilled to introduce time travel capabilities in Databricks Delta Lake, the…
Benchmarking Structured Streaming on Databricks Runtime Against State-of-the-Art Streaming Systems
Update Dec 14, 2017: As a result of a fix in the toolkit’s data generator, Apache Flink’s performance on a cluster of 10…
Running Streaming Jobs Once a Day For 10x Cost Savings
This is the sixth post in a multi-part series about how you can perform complex streaming analytics using Apache Spark. Traditionally, when people…
Working with Complex Data Formats with Structured Streaming in Apache Spark 2.1
In part 1 of this series on Structured Streaming blog posts, we demonstrated how easy it is to write an end-to-end streaming ETL…
New Features in Machine Learning Pipelines in Apache Spark 1.4
Apache Spark 1.2 introduced Machine Learning (ML) Pipelines to facilitate the creation, tuning, and inspection of practical ML workflows. Spark’s latest release, Spark…
