Detecting Bias with SHAP
StackOverflow's annual developer survey concluded earlier this year, and they have graciously published the (anonymized) 2019 results for analysis. They're a rich view into the experience of software developers around the world -- what's their favorite editor? how many years of experience? tabs or spaces? and crucially, salary. Software engineers' salaries are good, and sometimes...
Announcing the MLflow 1.0 Release
MLflow is an open source platform to help manage the complete machine learning lifecycle. With MLflow, data scientists can track and share experiments locally (on a laptop) or remotely (in the cloud), package and share models across frameworks, and deploy models virtually anywhere. Today we are excited to announce the release of MLflow 1.0. Since...
Hyperparameter Tuning with MLflow, Apache Spark MLlib and Hyperopt
Hyperparameter tuning is a common technique to optimize machine learning models based on hyperparameters, or configurations that are not learned during model training. Tuning these configurations can dramatically improve model performance. However, hyperparameter tuning can be computationally expensive, slow, and unintuitive even for experts. Databricks Runtime 5.4 and 5.4 ML (Azure | AWS) introduce new...
Enhanced Hyperparameter Tuning and Optimized AWS Storage with Databricks Runtime 5.4 ML
We are excited to announce the release of Databricks Runtime 5.4 ML (Azure | AWS). This release includes two Public Preview features to improve data science productivity, optimized storage in AWS for developing distributed applications, and a number of Python library upgrades. To get started, you simply select the Databricks Runtime 5.4 ML from the...
Introducing Databricks Runtime 5.4 with Conda (Beta)
We are excited to introduce a new runtime: Databricks Runtime 5.4 with Conda (Beta). This runtime uses Conda to manage Python libraries and environments. Many of our Python users prefer to manage their Python environments and libraries with Conda, which quickly is emerging as a standard. Conda takes a holistic approach to package management by...
Detecting Financial Fraud at Scale with Decision Trees and MLflow on Databricks
Detecting fraudulent patterns at scale is a challenge, no matter the use case. The massive amounts of data to sift through, the complexity of the constantly evolving techniques, and the very small number of actual examples of fraudulent behavior are comparable to finding a needle in a haystack while not knowing what the needle looks...
Understanding Dynamic Time Warping
This blog is part 1 of our two-part series Using Dynamic Time Warping and MLflow to Detect Sales Trends. To go to part 2, go to Using Dynamic Time Warping and MLflow to Detect Sales Trends. The phrase “dynamic time warping,” at first read, might evoke images of Marty McFly driving his DeLorean at 88...
Using Dynamic Time Warping and MLflow to Detect Sales Trends
This blog is part 2 of our two-part series Using Dynamic Time Warping and MLflow to Detect Sales Trends. The phrase “dynamic time warping,” at first read, might evoke images of Marty McFly driving his DeLorean at 88 MPH in the Back to the Future series. Alas, dynamic time warping does not involve time travel; instead, it’s...
Introducing MLflow Run Sidebar in Databricks Notebooks
At Spark+AI Summit 2019, we announced the GA of Managed MLflow on Databricks in which we take the latest and greatest of open source MLflow and make it easily accessible to all users of Databricks. In that blog post, we promised to build features which bridge Databricks and MLflow concepts to create a seamless integration...
Announcing General Availability of Managed MLflow on Databricks
MLflow is an open source platform to help manage the complete machine learning lifecycle. With MLflow, data scientists can track and share experiments locally or in the cloud, package and share models across frameworks, and deploy models virtually anywhere. Today at the Spark + AI Summit, we announced the General Availability of Managed MLflow on...
