Apparate: Managing Libraries in Databricks with CI/CD

by

This is a guest blog from Hanna Torrence, Data Scientist at ShopRunner. Introduction As leveraging data becomes a more vital component of organizations' tech stacks, it becomes increasingly important for data teams to make use of software engineering best-practices. The Databricks platform provides excellent tools for exploratory Apache Spark workflows in notebooks as well as...

Announcing Databricks Runtime 5.1

by

We are excited to announce the release of Databricks Runtime 5.1. This release includes several new features, performance and quality improvements. We highly recommend upgrading to Databricks Runtime 5.1 to benefit from all of these. Release highlights: Azure Active Directory Credential Passthrough to ADLS Gen1 (Preview) 7% performance gain overall for Databricks Azure New in...