Restricting Libraries in JVM Compute Platforms
Security challenges with Scala and Java libraries Open source communities have built incredibly useful libraries. They simplify many common development scenarios. Through our...
Feature Deep Dive: Watermarking in Apache Spark Structured Streaming
Key Takeaways Watermarks help Spark understand the processing progress based on event time, when to produce windowed aggregates and when to trim the...
How to Migrate Your Data and AI Workloads to Databricks With the AWS Migration Acceleration Program
In this blog we define the process for earning AWS customer credits when migrating Data and AI workloads to Databricks on Amazon Web...
Low-Code Exploratory Data Analysis with Bamboolib in Databricks
We are very excited to announce that the public preview of bamboolib in the Databricks Notebook begins today! It is available with the...
MLOps on Databricks with Vertex AI on Google Cloud
Since the launch of Databricks on Google Cloud in early 2021, Databricks and Google Cloud have been partnering together to further integrate the...
Orchestrating Data and ML Workloads at Scale: Create and Manage Up to 10k Jobs Per Workspace
Databricks Workflows is the fully-managed orchestrator for data, analytics, and AI. Today, we are happy to announce several enhancements that make it easier...
Low-latency Streaming Data Pipelines with Delta Live Tables and Apache Kafka
Delta Live Tables (DLT) is the first ETL framework that uses a simple declarative approach for creating reliable data pipelines and fully manages...
Databricks and Jupyter: Announcing ipywidgets in the Databricks Notebook
Today, we are excited to announce a deeper integration between the Databricks Notebook and the ecosystem established by Project Jupyter, a leader in...
Near Real-Time Anomaly Detection with Delta Live Tables and Databricks Machine Learning
Why is Anomaly Detection Important? Whether in retail, finance, cyber security, or any other industry, spotting anomalous behavior as soon as it happens...
Identity Columns to Generate Surrogate Keys Are Now Available in a Lakehouse Near You!
What is an identity column? An identity column is a column in a database that automatically generates a unique ID number for each...