Managed MLflow Now Available on Databricks Community Edition
In February 2016, we introduced Databricks Community Edition, a free edition for big data developers to learn and get started quickly with Apache Spark. Since then our commitment to foster a community of developers remains steadfast: to date, we have over 150K registered Community Edition users; we have trained thousands of people at meetups and...
Democratizing Financial Time Series Analysis with Databricks
Introduction The role of data scientists, data engineers, and analysts at financial institutions includes (but is not limited to) protecting hundreds of billions of dollars worth of assets and protecting investors from trillion-dollar impacts, say from a flash crash. One of the biggest technical challenges underlying these problems is scaling time series manipulation. Tick data,...
Analyzing Your MLflow Data with DataFrames
Max Allen interned with Databricks Engineering in the Summer of 2019. This blog post, written by Max, highlights the great work he did while on the team. Introduction to MLflow and the Machine Learning Development Lifecycle MLflow is an open source platform for the machine learning lifecycle, and many Databricks customers have been using it...
A Guide to AI, Data Science, Machine Learning and Deep Learning Talks at Spark+AI Summit Europe 2019
The Spark + AI Summit Europe is just around the corner, and it's a great opportunity for data scientists and Machine Learning (ML) practitioners to get up to speed on the latest tools and innovations in the field! Below is a selection of talks on ML best practices for productionizing ML at scale, real-life use...
Using AutoML Toolkit to Automate Loan Default Predictions
Download the following notebooks and try the AutoML Toolkit today: Evaluating Risk for Loan Approvals using XGBoost (0.90) | Using AutoML Toolkit to Simplify Loan Risk Analysis XGBoost Model Optimization This blog was originally published on September 10th, 2019; it has been updated on October 2nd, 2019. In a previous blog and notebook, Loan Risk Analysis...
Guest Blog: How Virgin Hyperloop One reduced processing time from hours to minutes with Koalas
At Virgin Hyperloop One, we work on making Hyperloop a reality, so we can move passengers and cargo at airline speeds but at a fraction of the cost of air travel. In order to build a commercially viable system, we collect and analyze a large, diverse quantity of data, including Devloop Test Track runs, numerous...
AutoML on Databricks: Augmenting Data Science from Data Prep to Operationalization
Thousands of data science jobs are going unfilled today as global demand for the talent greatly outstrips supply. Every day, businesses pay the price of the data scientist shortage in missed opportunities and slow innovation. For organizations to realize the full potential of machine learning, data teams have to build hundreds of predictive models a...
How (Not) To Scale Deep Learning in 6 Easy Steps
Introduction: The Problem Deep learning sometimes seems like sorcery. Its state-of-the-art applications are at times delightful and at times disturbing. The tools that achieve these results are, amazingly, mostly open source, and can work their magic on powerful hardware available to rent by the hour in the cloud. It's no wonder that companies are eager...
Productionizing Machine Learning with Delta Lake
Try out this notebook series in Databricks - part 1 (Delta Lake), part 2 (Delta Lake + ML) For many data scientists, the process of building and tuning machine learning models is only a small portion of the work they do every day. The vast majority of their time is spent doing the less-than-glamorous (but...
Deep Learning on Medical Images at Population Scale: On-Demand Webinar and FAQ Now Available!
On June 26th, we hosted a live webinar — Deep Learning on Medical Images at Population-scale— with members of the data science and engineering teams from Human Longevity Inc (HLI), a leader in medical imaging and genomics. During the webinar, HLI shared how they use MRI images, whole-genome sequencing data, and other clinical data sets...
