Databricks Demonstrates AWS Platform Integrations at re:Invent 2019
Databricks was proud to be a Platinum sponsor at re:Invent. The past year has been an exciting one for our partnership with AWS, as we built new integrations and deepened existing ones with so many AWS services. re:Invent was a great opportunity to showcase how our joint customers have benefitted from those integrations and we...
Databricks, AWS, and SafeGraph Team Up For Easier Analysis of Consumer Behavior
This notebook was produced as a collaboration between SafeGraph and Databricks Ryan Fox Squire, Product Data Scientist @ SafeGraph Andrew Hutchinson, Solutions Architect @ Databricks Prasad Kona, Partner Solutions Architect @ Databricks We’ve created this Databricks notebook (.dbc download here), and published this blog, so that you can hit the ground running using SafeGraph Data...
Use Databricks Pools to Speed up your Data Pipelines and Scale Clusters Quickly
Data Engineering teams deploy short, automated jobs on Databricks. They expect their clusters to start quickly, execute the job, and terminate. Data Analytics teams run large auto-scaling, interactive clusters on Databricks. They expect these clusters to adapt to increased load and scale up quickly in order to minimize query latency. Databricks is pleased to announce...
Spark + AI in Amsterdam: European Summit Recap, Keynote Videos, & Announcements
Spark + AI Summit Europe 2019 came to Amsterdam this past week! Over 2,300 data scientists, data engineers, and global business leaders from 63 different countries descended upon the RAI Amsterdam Convention Centre, for the latest community and open source developments around Apache Spark™, Delta Lake, MLflow, Koalas, and more. Check out the keynote recordings...
Introducing Glow: an open-source toolkit for large-scale genomic analysis
The key to solving some of today’s most challenging medical problems lies in the analysis of genomics data. Understanding the impact of the minor changes in an individual’s genome on their overall health is fundamentally a data driven challenge that requires integration across hundreds of thousands of individuals. By analyzing genomes across large cohorts, researchers...
Introducing the MLflow Model Registry
Watch the announcement and demo At today’s Spark + AI Summit in Amsterdam, we announced the availability of the MLflow Model Registry, a new component in the MLflow open source ML platform. Since we introduced MLflow at Spark+AI Summit 2018, the project has gained more than 140 contributors and 800,000 monthly downloads on PyPI, making...
Managed MLflow Now Available on Databricks Community Edition
In February 2016, we introduced Databricks Community Edition, a free edition for big data developers to learn and get started quickly with Apache Spark. Since then our commitment to foster a community of developers remains steadfast: to date, we have over 150K registered Community Edition users; we have trained thousands of people at meetups and...
Delta Lake Now Hosted by the Linux Foundation to Become the Open Standard for Data Lakes
At today’s Spark + AI Summit Europe in Amsterdam, we announced that Delta Lake is becoming a Linux Foundation project. Together with the community, the project aims to establish an open standard for managing large amounts of data in data lakes. The Apache 2.0 software license remains unchanged. Delta Lake focuses on improving the reliability...
A Guide to MLflow Talks at Spark + AI Summit 2019 Europe
We are thrilled to see how well MLflow has been welcomed by the community since we launched it last summer. With now over 800K monthly downloads, 130 code contributors and dozens of contributing organizations including RStudio and Microsoft, it is one of the fastest growing open source projects in the field of Machine Learning, confirming...
A Guide to Developer, Deep Dive, and Apache Spark Tutorial Talks at Spark + AI Summit, Europe
You might have heard the famous saying, “Why software is eating the world.” But if software is eating the world, you may ask, where does software come from? Naturally, Developers! Some software developers advocate that the “Developers are eating the world.” A research report by Stripe indicates that “developers have the ability to raise global...