MLflow On-Demand Webinar and FAQ Now Available!
On August 30th, our team hosted a live webinar—Introducing MLflow: Infrastructure for a complete Machine Learning lifecycle—with Matei Zaharia, Co-Founder and Chief Technologist at Databricks. In this webinar, we walked you through MLflow, a new open source project from Databricks that aims to design an open ML platform where organizations can use any ML library...
Introducing Flint: A time-series library for Apache Spark
This is a joint guest community blog by Li Jin at Two Sigma and Kevin Rasmussen at Databricks; they share how to use Flint with Apache Spark. Introduction The volume of data that data scientists face these days increases relentlessly, and we now find that a traditional, single-machine solution is no longer adequate to the demands...
Building the Fastest DNASeq Pipeline at Scale
In June, we announced the Unified Analytics Platform for Genomics with a simple goal: accelerate discovery with a collaborative platform for interactive genomic data processing, analytics and AI at massive scale. In this post, we’ll go into more detail about one component of the platform: a scalable DNASeq pipeline that is concordant with GATK4 at...
A Guide to Apache Spark Use Cases, Streaming, and Research Talks at Spark + AI Summit Europe
For much of Apache Spark’s history, its capacity to process data at scale and capability to unify disparate workloads has led Spark developers to tackle new use cases. Through innovation and extension of its ecosystem, developers combine data and AI to develop new applications. So it befits developers to come to this summit not just...
Introducing Cluster-scoped Init Scripts
This summer, I worked at Databricks as a software engineering intern on the Clusters team. As part of my internship project, I designed and implemented Cluster-scoped init scripts, improving scalability and ease of use. In this blog, I will discuss various benefits of Cluster-scoped init scripts, followed by my internship experience at Databricks, and the...
By Customer Demand: Databricks and Snowflake Integration
Today, we are proud to announce a partnership between Snowflake and Databricks that will help our customers further unify Big Data and AI by providing an optimized, production-grade integration between Snowflake’s built for the cloud-built data warehouse and Databricks’ Unified Analytics Platform. Over the course of the last year, our joint customers such as Rue...
5 Reasons to Attend Spark + AI Summit Europe 2018
The world’s largest event for the Apache Spark Community By Singh Garewal Posted in COMPANY BLOG August 27, 2018 Spark + AI Summit Europe will be held in London on October 2-4, 2018. Check out the full agenda and get your ticket before it sells out! Register today with the discount code 5Reasons and get...
Announcing Databricks Runtime 4.3
I'm pleased to announce the release of Databricks Runtime 4.3, powered by Apache Spark. We've packed this release with an assortment of new features, performance improvements, and quality improvements to the platform. We recommend moving to Databricks Runtime 4.3 in order to take advantage of these improvements. In our obsession to continually improve our platform’s...
How to Use MLflow to Experiment a Keras Network Model: Binary Classification for Movie Reviews
In the last blog post, we demonstrated the ease with which you can get started with MLflow, an open-source platform to manage machine learning lifecycle. In particular, we illustrated a simple Keras/TensorFlow model using MLflow and PyCharm. This time we explore a binary classification Keras network model. Using MLflow’s Tracking APIs, we will track metrics—accuracy...
New Features in MLflow v0.5.2 Release
Today, we’re excited to announce MLflow v0.5.0, MLflow v0.5.1, and MLflow v0.5.2, which were released last week with some new features. MLflow 0.5.2 is already available on PyPI and docs are updated. If you do pip install mlflow as described in the MLflow quickstart guide, you will get the recent release. In this post, we’ll...
