Writing a Faster Jsonnet Compiler
This blog post is part of our series of internal engineering blogs on the Databricks platform, infrastructure management, integration, tooling, monitoring, and provisioning. See our previous post on Declarative Infrastructure for background on some of the technologies discussed here. At Databricks, we use the Jsonnet data templating language to declaratively manage configurations for services running...
Databricks Engineering Interns & Impact in Summer 2018
Thanks to our awesome interns! This summer, our Engineering interns at Databricks did amazing work. Our interns, working on teams from Developer Tools to Machine Learning, built features and improvements which are already impacting our customers and the Apache Spark and AI communities. Spending a summer at Databricks Databricks Engineering internships are a mix of...
MLflow v0.7.0 Features New R API by RStudio
Today, we’re excited to announce MLflow v0.7.0, released with new features, including a new MLflow R client API contributed by RStudio. A testament to MLflow’s design goal of an open platform with adoption in the community, RStudio’s contribution extends the MLflow platform to a larger R community of data scientists who use RStudio and R...
Introduction to Neural Networks: On-Demand Webinar and FAQ Now Available!
On September 27th, we hosted a live webinar—Introduction to Neural Networks—with Denny Lee, Technical Product Marketing Manager at Databricks. This is the first webinar of a free deep learning fundamental series from Databricks. In this webinar, we covered the fundamentals of deep learning to better understand what gives neural networks their expressive power: the potential...
What’s New for Apache Spark on Kubernetes in the Upcoming Apache Spark 2.4 Release
This is a community blog from Yinan Li, a software engineer at Google, working in the Kubernetes Engine team. He is part of the group of companies that have contributed to Kubernetes support in the upcoming Apache Spark 2.4. Since the Kubernetes cluster scheduler backend was initially introduced in Apache Spark 2.3, the community has...
Databricks Delta: Now Available in Preview as Part of Microsoft Azure Databricks
Bringing unprecedented reliability and performance to cloud data lakes Designed by Databricks in collaboration with Microsoft, Azure Databricks combines the best of Databricks’ Apache SparkTM-based cloud service and Microsoft Azure. The integrated service provides the Databricks Unified Analytics Platform integrated with the Azure cloud platform, encompassing the Azure Portal; Azure Active Directory; and other data...
How to Use MLflow To Reproduce Results and Retrain Saved Keras ML Models
In part 2 of our series on MLflow blogs, we demonstrated how to use MLflow to track experiment results for a Keras network model using binary classification. We classified reviews from an IMDB dataset as positive or negative. And we created one baseline model and two experiments. For each model, we tracked its respective training...
Simplify Market Basket Analysis using FP-growth on Databricks
When providing recommendations to shoppers on what to purchase, you are often looking for items that are frequently purchased together (e.g. peanut butter and jelly). A key technique to uncover associations between different items is known as market basket analysis. In your recommendation engine toolbox, the association rules generated by market basket analysis (e.g. if...
New Features in MLflow v0.6.0
Today, we’re excited to announce MLflow v0.6.0, released early in the week with new features. Now available on PyPI and Maven, the docs are updated. You can install the recent release with pip install mlflow as described in the MLflow quickstart guide. MLflow v0.6.0 introduces a number of major features: A Java client API, available...
Identify Suspicious Behavior in Video with Databricks Runtime for Machine Learning
With the exponential growth of cameras and visual recordings, it is becoming increasingly important to operationalize and automate the process of video identification and categorization. Applications ranging from identifying the correct cat video to visually categorizing objects are becoming more prevalent. With millions of users around the world generating and consuming billions of minutes of...
