Deep Learning Tutorial Demonstrates How to Simplify Distributed Deep Learning Model Inference Using Delta Lake and Apache Spark™
On October 10th, our team hosted a live webinar—Simple Distributed Deep Learning Model Inference—with Xiangrui Meng, Software Engineer at Databricks. Model inference, unlike model training, is usually embarrassingly parallel and hence simple to distribute. However, in practice, complex data scenarios and compute infrastructure often make this "simple" task hard to do from data source to...
Delta Lake Now Hosted by the Linux Foundation to Become the Open Standard for Data Lakes
At today’s Spark + AI Summit Europe in Amsterdam, we announced that Delta Lake is becoming a Linux Foundation project. Together with the community, the project aims to establish an open standard for managing large amounts of data in data lakes. The Apache 2.0 software license remains unchanged. Delta Lake focuses on improving the reliability...
Productionizing Machine Learning: From Deployment to Drift Detection
Try this notebook to reproduce the steps outlined below and watch our on-demand webinar to learn more. In many articles and blogs the machine learning workflow starts with data prep and ends with deploying a model to production. But in reality, that's just the beginning of the lifecycle of a machine learning model. As they say,...
Guest Blog: Using Databricks, MLflow, and Amazon SageMaker at Brandless to Bring Recommendation Systems to Production
This is a guest blog from Adam Barnhard, Head of Data at Brandless, Inc., and Bing Liang, Data Scientist at Brandless, Inc. Launched in July 2017, Brandless makes hundreds of high-quality items, curated for every member of your family and room of your home, and all sold at more accessible price points than similar products on the market. We...
Diving Into Delta Lake: Unpacking The Transaction Log
The transaction log is key to understanding Delta Lake because it is the common thread that runs through many of its most important features, including ACID transactions, scalable metadata handling, time travel, and more. In this article, we’ll explore what the Delta Lake transaction log is, how it works at the file level, and how...
AutoML on Databricks: Augmenting Data Science from Data Prep to Operationalization
Thousands of data science jobs are going unfilled today as global demand for the talent greatly outstrips supply. Every day, businesses pay the price of the data scientist shortage in missed opportunities and slow innovation. For organizations to realize the full potential of machine learning, data teams have to build hundreds of predictive models a...
Network performance regressions from TCP SACK vulnerability fixes
Update on Aug 2, 2019: Added to the end explaining our kernel patch and additional details we found. On June 17, three vulnerabilities in Linux’s networking stack were published. The most severe one could allow remote attackers to impact the system’s availability. We believe in offering the most secure image available to our customers, so...
Announcing Databricks Runtime 5.5 with Conda (Beta)
Databricks is pleased to announce the release of Databricks Runtime 5.5 with Conda (Beta). We introduced Databricks Runtime 5.4 with Conda (Beta), with the goal of making Python library and environment management very easy. This release includes several important improvements and bug fixes as noted in the latest release notes [Azure|AWS]. We recommend all users...
Announcing the MLflow 1.1 Release
We’re excited to announce today the release of MLflow 1.1. In this release, we’ve focused on fleshing out the tracking component of MLflow and improving visualization components in the UI. Some of the major features include: Automatic logging from TensorFlow and Keras Parallel coordinate plots in the tracking UI Pandas DataFrame based search API Java...
Announcing Databricks Runtime 5.4
Databricks is pleased to announce the release of Databricks Runtime 5.4. This release includes Apache Spark 2.4.3 along with several important improvements and bug fixes . We recommend all users upgrade to take advantage of this new runtime release. This blog post gives a brief overview of some of the new high value features that...