Introducing Databricks Library Utilities for Notebooks
Databricks has introduced a new feature, Library Utilities for Notebooks, as part of Databricks Runtime version 5.1. It allows you to install and…
Transparent Autoscaling of Instance Storage
Big data workloads require access to disk space for a variety of operations, generally when intermediate results will not fit in memory. When…
Databricks Serverless: Next Generation Resource Management for Apache Spark
As the amount of data in an organization grows, more and more engineers, analysts and data scientists need to analyze this data using…
Transactional Writes to Cloud Storage on Databricks
In another blog post published today, we showed the top five reasons for choosing S3 over HDFS. With the dominance of simple and…
Query Watchdog: Handling Disruptive Queries in Spark SQL
At Databricks, our users range from SQL Analysts who explore data through JDBC connections and SQL Notebooks to Data Engineers orchestrating large scale…
