Data Streaming | Databricks Blog

Page 1

Training Highly Scalable Deep Recommender Systems on Databricks (Part 1)

September 4, 2024 by Rithwik Ediga Lakhamsani, Asfandyar Qureshi, Karan Jariwala, Lin Yuan, Lu Wang (Mosaic AI), Saaketh Narayan and Ning Wang in Generative AI

Recommender systems (RecSys) have become an integral part of modern digital experiences, powering personalized content suggestions across various platforms. These sophisticated systems and...

Adaptive Query Execution in Structured Streaming

June 2, 2023 by Steven Chen, MaryAnn Xue and Jungtaek Lim in Engineering Blog

In Databricks Runtime, Adaptive Query Execution (AQE) is a performance feature that continuously re-optimizes batch queries using runtime statistics during query execution. Starting...

Latency goes subsecond in Apache Spark Structured Streaming

May 15, 2023 by Jerry Peng, Pranav Anand, Sourav Gulati, Karthik Ramasamy, Michael Armbrust and Matei Zaharia in Engineering Blog

Apache Spark Structured Streaming is the leading open source stream processing platform. It is also the core technology that powers streaming on the...

How Collective Health uses Delta Live Tables and Structured Streaming for Data Integration

April 13, 2023 by Mragesh Khandelwal and Mahmoud Saleh in Customers

Collective Health is not an insurance company. We're a technology company that's fundamentally making health insurance work better for everyone— starting with the...

Scalable Spark Structured Streaming for REST API Destinations

March 2, 2023 by Art Rask and Jay Palaniappan in Engineering Blog

Spark Structured Streaming is the widely-used open source engine at the foundation of data streaming on the Databricks Lakehouse Platform . It can...

Build Reliable and Cost Effective Streaming Data Pipelines With Delta Live Tables’ Enhanced Autoscaling

December 8, 2022 by Paul Lappas, Li Zhang, Alex Ott, Kiavash Kianfar, Yuhong Chen and Prashanth Babu Velanati Venkata in Platform Blog

This year we announced the general availability of Delta Live Tables (DLT) , the first ETL framework to use a simple, declarative approach...

Python Arbitrary Stateful Processing in Structured Streaming

October 18, 2022 by Hyukjin Kwon and Jungtaek Lim in Engineering Blog

More and more customers are using Databricks for their real-time analytics and machine learning workloads to meet the ever increasing demand of their...

State Rebalancing in Structured Streaming

October 4, 2022 by Tristen Wentling and Alex Balikov in Engineering Blog

In light of the accelerated growth and adoption of Apache Spark Structured Streaming, Databricks announced Project Lightspeed at Data + AI Summit 2022...

Using Streaming Delta Live Tables and AWS DMS for Change Data Capture From MySQL

September 29, 2022 by Neil Patel in Platform Blog

In this article we will walk you through the steps to create an end-to-end CDC pipeline with Terraform using Delta Live Tables, AWS...

Databricks at Current 2022

September 28, 2022 by Matt Jones, Frank Munz, Emma Liu, Karthik Ramasamy and Riley Maris in Company Blog

Current 2022 , organized by Confluent, is the first-ever data streaming industry event – and it's coming up soon! No matter where you...