All posts

Azure Databricks Unified Analytics Platform is the result of a joint product/engineering effort between Databricks and Microsoft. It’s available as a managed first-party service on Azure Public Cloud. Along with one-click setup (manual/automated), managed clusters (including Delta), and collaborative workspaces, the platform has native integration with other Azure first-party services, such as Azure Blob Storage,...

With a tsunami of data, scale of computing resources available, and rapid development of easy-to-learn open source Machine Learning frameworks, data science and machine learning concepts are much easier to learn and implement today than they were a decade ago. As a result, across all industries, practitioners are using cutting-edge ML algorithms to solve tough...

Simplify building big data pipelines for change data capture (CDC) and GDPR use cases. Databricks Delta, the next-generation unified analytics engine built on top of Apache Spark™, now supports the MERGE command, which allows you to efficiently upsert and delete records in your data lakes. MERGE dramatically simplifies how a number of common data pipelines...

With 2019 in full swing, the excitement for data and AI driven innovation continues. Over the past few years, we’ve seen leading innovators - like Riot Games, Regeneron and Shell - become early adopters of the latest machine learning and AI technologies, building and deploying AI applications into production. But with great promise comes great...

Building production machine learning applications is challenging because there is no standard way to record experiments, ensure reproducible runs, and manage and deploy models. To address these challenges, last June we introduced MLflow, an open source platform to manage the ML lifecycle that works with any machine learning library and environment. The project has grown...

Artificial Intelligence (AI) adoption is picking up steam. Enterprises are looking for new ways to deploy AI to transform their business and gain a competitive edge. With the power of machine learning (ML) fueled by the insights mined from massive volumes of data, AI is no longer only for the 1%, but quickly becoming an...

Databricks migrated over from the standard Scala Build Tool (SBT) to using Bazel to build, test and deploy our Scala code. Through improvements in our build infrastructure, Scala compilation workflows that previously took minutes to tens of minutes now complete in seconds. This post will walk you through the improvements we made to achieve that,...