Scaling Hyperopt to Tune Machine Learning Models in Python
Try the Hyperopt notebook to reproduce the steps outlined below and watch our on-demand webinar to learn more. Hyperopt is one of the most popular open-source libraries for tuning Machine Learning models in Python. We’re excited to announce that Hyperopt 0.2.1 supports distributed tuning via Apache Spark. The new SparkTrials class allows you to scale out...
Automated Hyperparameter Tuning, Scaling and Tracking: On-Demand Webinar and FAQs now available!
On June 20th, our team hosted a live webinar—Automated Hyperparameter Tuning, Scaling and Tracking on Databricks—with Joseph Bradley, Software Engineer, and Yifan Cao, Senior Product Manager at Databricks. Automated Machine Learning (AutoML) has received significant interest recently because of its ability to shorten time-to-value for data science teams and maximize the predictive performance of models....
Hyperparameter Tuning with MLflow, Apache Spark MLlib and Hyperopt
Hyperparameter tuning is a common technique to optimize machine learning models based on hyperparameters, or configurations that are not learned during model training. Tuning these configurations can dramatically improve model performance. However, hyperparameter tuning can be computationally expensive, slow, and unintuitive even for experts. Databricks Runtime 5.4 and 5.4 ML (Azure | AWS) introduce new...
Enhanced Hyperparameter Tuning and Optimized AWS Storage with Databricks Runtime 5.4 ML
We are excited to announce the release of Databricks Runtime 5.4 ML (Azure | AWS). This release includes two Public Preview features to improve data science productivity, optimized storage in AWS for developing distributed applications, and a number of Python library upgrades. To get started, you simply select the Databricks Runtime 5.4 ML from the...
Databricks Runtime 5.2 ML Features Multi-GPU Workflow, Pregel API, and Performant GraphFrames
We are excited to announce the release of Databricks Runtime 5.2 for Machine Learning. This release includes several new features and performance improvements to help developers easily use machine learning on the Databricks Unified Analytics Platform. Continuing our efforts to make developers’ lives easy to build deep learning applications, this release includes the following features...
Introducing HorovodRunner for Distributed Deep Learning Training
Today, we are excited to introduce HorovodRunner in our Databricks Runtime 5.0 ML! HorovodRunner provides a simple way to scale up your deep learning training workloads from a single machine to large clusters, reducing overall training time. Motivated by the needs of many of our users who want to train deep learning models on datasets...
Databricks Engineering Interns & Impact in Summer 2018
Thanks to our awesome interns! This summer, our Engineering interns at Databricks did amazing work. Our interns, working on teams from Developer Tools to Machine Learning, built features and improvements which are already impacting our customers and the Apache Spark and AI communities. Spending a summer at Databricks Databricks Engineering internships are a mix of...
Developing Custom Machine Learning Algorithms in PySpark
Developing custom Machine Learning (ML) algorithms in PySpark—the Python API for Apache Spark—can be challenging and laborious. In this blog post, we describe our work to improve PySpark APIs to simplify the development of custom algorithms. Our key improvement reduces hundreds of lines of boilerplate code for persistence (saving and loading models) to a single...
Detecting Abuse at Scale: Locality Sensitive Hashing at Uber Engineering
This is a cross blog post effort between Databricks and Uber Engineering. Yun Ni is a software engineer on Uber’s Machine Learning Platform team, Kelvin Chu is technical lead engineer on Uber’s Complex Data Processing/Speak team, and Joseph Bradley is a software engineer on Databricks’ Machine Learning team. With 5 million Uber trips taken daily...
Intel’s BigDL on Databricks
Intel recently released its BigDL project for distributed deep learning on Apache Spark. BigDL has native Spark integration, allowing it to leverage Spark during model training, prediction, and tuning. This blog post gives highlights of BigDL and a tutorial showing how to get started with BigDL on Databricks. Intel’s BigDL project BigDL is an open...
