Introducing Built-in Image Data Source in Apache Spark 2.4
Introduction With recent advances in deep learning frameworks for image classification and object detection, the demand for standard image processing in Apache Spark has never been greater. Image handling and preprocessing have their specific challenges - for example, images come in different formats (eg., jpeg, png, etc.), sizes, and color schemes, and there is no...
CIO Survey: Top 3 Challenges Adopting AI and How to Overcome Them
We recently hosted the webinar — CIO Survey: Enterprise Challenges to AI and How to Overcome Them — featuring Jen Garofalo, Research Director at IDG, the parent company to CIO.com, and Pat McDonough, VP of Customer Success at Databricks. This webinar covered key findings from a recent CIO.com survey of 200 executives on the topic...
Apache Avro as a Built-in Data Source in Apache Spark 2.4
Apache Avro is a popular data serialization format. It is widely used in the Apache Spark and Apache Hadoop ecosystem, especially for Kafka-based data pipelines. Starting from Apache Spark 2.4 release, Spark provides built-in support for reading and writing Avro data. The new built-in spark-avro module is originally from Databricks’ open source project Avro Data...
Is your AI software vendor taking security shortcuts?
This is the final part of our blog series on security available here. In this blog, I will be talking about our culture of security at Databricks. Artificial intelligence software that can learn and improve human decision-making is transforming business. All sorts of companies are looking to AI to gain an edge over competitors. Unfortunately,...
Databricks Shows off AWS Competencies at AWS re:Invent
Powering innovation on AWS in life sciences, public sector, machine learning, and big data analytics. Databricks, founded by the original creators of Apache SparkTM, is proud to be a Platinum sponsor at AWS re:Invent. Stop by booth #416 to grab some cool schwag and see demos of MLflow, TensorFlow, Apache Spark, and other technologies that...
Introducing Databricks Runtime 5.0 for Machine Learning
Six months ago we introduced the Databricks Runtime for Machine Learning with the goal of making machine learning performant and easy on the Databricks Unified Analytics Platform. The Databricks Runtime for ML comes pre-packaged with many ML frameworks and enables distributed training and inference. Today we are excited to release the second iteration including Conda...
Preliminary Agenda Announced for Spark + AI Summit 2019
As part of the organizing committee and program chairs for this summit, we are delighted to share two achievements with you. First, we have published a preliminary agenda for the Spark + AI Summit in San Francisco on April 23-25, 2019, comprising of 12 technical tracks, with 100+ breakout sessions, on data and AI across...
MLflow v0.8.0 Features Improved Experiment UI and Deployment Tools
Last week we released MLflow v0.8.0 with multiple new features, including improved UI experience and support for deploying models directly via Docker containers to the Azure Machine Learning Service Workspace. Now available on PyPI and with docs online, you can install this new release with pip install mlflow as described in the MLflow quickstart guide....
Are you ready to scale your Data and AI initiatives? How will you scale your security?
This is Blog #3 in a series of blog posts about Databricks security. My colleagues David Cook (our CISO) and David Meyer (SVP products) laid out Databricks' approach to Security in blog #1 & blog #2. With this blog, I will be talking about deploying and operating Databricks at scale while minimizing human error. Democratize...
Introducing HorovodRunner for Distributed Deep Learning Training
Today, we are excited to introduce HorovodRunner in our Databricks Runtime 5.0 ML! HorovodRunner provides a simple way to scale up your deep learning training workloads from a single machine to large clusters, reducing overall training time. Motivated by the needs of many of our users who want to train deep learning models on datasets...
