How (Not) To Scale Deep Learning in 6 Easy Steps
Introduction: The Problem Deep learning sometimes seems like sorcery. Its state-of-the-art applications are at times delightful and at times disturbing. The tools that achieve these results are, amazingly, mostly open source, and can work their magic on powerful hardware available to rent by the hour in the cloud. It's no wonder that companies are eager...
Productionizing Machine Learning with Delta Lake
Try out this notebook series in Databricks - part 1 (Delta Lake), part 2 (Delta Lake + ML) For many data scientists, the process of building and tuning machine learning models is only a small portion of the work they do every day. The vast majority of their time is spent doing the less-than-glamorous (but...
Deep Learning on Medical Images at Population Scale: On-Demand Webinar and FAQ Now Available!
On June 26th, we hosted a live webinar — Deep Learning on Medical Images at Population-scale— with members of the data science and engineering teams from Human Longevity Inc (HLI), a leader in medical imaging and genomics. During the webinar, HLI shared how they use MRI images, whole-genome sequencing data, and other clinical data sets...
Announcing Databricks Runtime 5.5 with Conda (Beta)
Databricks is pleased to announce the release of Databricks Runtime 5.5 with Conda (Beta). We introduced Databricks Runtime 5.4 with Conda (Beta), with the goal of making Python library and environment management very easy. This release includes several important improvements and bug fixes as noted in the latest release notes [Azure|AWS]. We recommend all users...
Announcing the MLflow 1.1 Release
We’re excited to announce today the release of MLflow 1.1. In this release, we’ve focused on fleshing out the tracking component of MLflow and improving visualization components in the UI. Some of the major features include: Automatic logging from TensorFlow and Keras Parallel coordinate plots in the tracking UI Pandas DataFrame based search API Java...
Automated Hyperparameter Tuning, Scaling and Tracking: On-Demand Webinar and FAQs now available!
On June 20th, our team hosted a live webinar—Automated Hyperparameter Tuning, Scaling and Tracking on Databricks—with Joseph Bradley, Software Engineer, and Yifan Cao, Senior Product Manager at Databricks. Automated Machine Learning (AutoML) has received significant interest recently because of its ability to shorten time-to-value for data science teams and maximize the predictive performance of models....
Announcing Databricks Runtime 5.5 and Runtime 5.5 for Machine Learning
Databricks is pleased to announce the release of Databricks Runtime 5.5. This release includes Apache Spark 2.4.3 along with several important improvements and bug fixes as noted in the latest release notes [Azure|AWS]. We recommend all users upgrade to take advantage of this new runtime release. This blog post gives a brief overview of some...
What’s new with MLflow? On-Demand Webinar and FAQs now available!
On June 6th, our team hosted a live webinar—Managing the Complete Machine Learning Lifecycle: What's new with MLflow—with Clemens Mewald, Director of Product Management at Databricks. Machine learning development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools and parameters to...
Detecting Bias with SHAP
StackOverflow's annual developer survey concluded earlier this year, and they have graciously published the (anonymized) 2019 results for analysis. They're a rich view into the experience of software developers around the world -- what's their favorite editor? how many years of experience? tabs or spaces? and crucially, salary. Software engineers' salaries are good, and sometimes...
Hyperparameter Tuning with MLflow, Apache Spark MLlib and Hyperopt
Hyperparameter tuning is a common technique to optimize machine learning models based on hyperparameters, or configurations that are not learned during model training. Tuning these configurations can dramatically improve model performance. However, hyperparameter tuning can be computationally expensive, slow, and unintuitive even for experts. Databricks Runtime 5.4 and 5.4 ML (Azure | AWS) introduce new...
