Previewing Updates to the Databricks Notebook
At Databricks, we are committed to delivering a world-class, data-driven development experience in the Notebook, and we are very excited to preview the...
Cybersecurity in the Era of Multiple Clouds and Regions
In 2021, more than three quarters of all enterprises have infrastructure in multiple clouds . This trend shows no signs of slowdown with...
Databricks Workspace Administration – Best Practices for Account, Workspace and Metastore Admins
This blog is part of our Admin Essentials series, where we discuss topics relevant to Databricks administrators. Other blogs include our Workspace Management...
Feature Deep Dive: Watermarking in Apache Spark Structured Streaming
Key Takeaways Watermarks help Spark understand the processing progress based on event time, when to produce windowed aggregates and when to trim the...
Orchestrating Data and ML Workloads at Scale: Create and Manage Up to 10k Jobs Per Workspace
Databricks Workflows is the fully-managed orchestrator for data, analytics, and AI. Today, we are happy to announce several enhancements that make it easier...
Low-latency Streaming Data Pipelines with Delta Live Tables and Apache Kafka
Delta Live Tables (DLT) is the first ETL framework that uses a simple declarative approach for creating reliable data pipelines and fully manages...
Databricks and Jupyter: Announcing ipywidgets in the Databricks Notebook
Today, we are excited to announce a deeper integration between the Databricks Notebook and the ecosystem established by Project Jupyter, a leader in...
Identity Columns to Generate Surrogate Keys Are Now Available in a Lakehouse Near You!
What is an identity column? An identity column is a column in a database that automatically generates a unique ID number for each...
Announcing Photon Engine General Availability on the Databricks Lakehouse Platform
We are pleased to announce that Photon , the record-setting next-generation query engine for lakehouse systems, is now generally available on Databricks across...
How to Build a Marketing Analytics Solution Using Fivetran and dbt on the Databricks Lakehouse
Marketing teams use many different platforms to drive marketing and sales campaigns, which can generate a significant volume of valuable but disconnected data...