Scaling Financial Time Series Analysis Beyond PCs and Pandas: On-Demand Webinar and FAQ Now Available!
On Oct 9th, 2019, we hosted a live webinar —Scaling Financial Time Series Analysis Beyond PCs and Pandas — with Junta Nakai, Industry Leader Financial Services at Databricks, and Ricardo Portilla, Solution Architect at Databricks. This was a live webinar showcasing the content in this blog- Democratizing Financial Time Series Analysis with Databricks. Please find...
Introducing Glow: an open-source toolkit for large-scale genomic analysis
The key to solving some of today’s most challenging medical problems lies in the analysis of genomics data. Understanding the impact of the minor changes in an individual’s genome on their overall health is fundamentally a data driven challenge that requires integration across hundreds of thousands of individuals. By analyzing genomes across large cohorts, researchers...
Introducing the MLflow Model Registry
At today’s Spark + AI Summit in Amsterdam, we announced the availability of the MLflow Model Registry, a new component in the MLflow open source ML platform. Since we introduced MLflow at Spark+AI Summit 2018, the project has gained more than 140 contributors and 800,000 monthly downloads on PyPI, making MLflow one of the fastest...
Managed MLflow Now Available on Databricks Community Edition
In February 2016, we introduced Databricks Community Edition, a free edition for big data developers to learn and get started quickly with Apache Spark. Since then our commitment to foster a community of developers remains steadfast: to date, we have over 150K registered Community Edition users; we have trained thousands of people at meetups and...
Delta Lake Now Hosted by the Linux Foundation to Become the Open Standard for Data Lakes
At today’s Spark + AI Summit Europe in Amsterdam, we announced that Delta Lake is becoming a Linux Foundation project. Together with the community, the project aims to establish an open standard for managing large amounts of data in data lakes. The Apache 2.0 software license remains unchanged. Delta Lake focuses on improving the reliability...
How Informatica Data Engineering Goes Hadoop-less with Databricks
Back in May, we announced our partnership with Informatica to build out a rich set of integrations between our two platforms. It’s been exciting work for the team because of what we can do for joint customers that combine our Managed Delta Lake with Informatica’s Big Data Management and Enterprise Data Catalog. The vision led...
Democratizing Financial Time Series Analysis with Databricks
Introduction The role of data scientists, data engineers, and analysts at financial institutions includes (but is not limited to) protecting hundreds of billions of dollars worth of assets and protecting investors from trillion-dollar impacts, say from a flash crash. One of the biggest technical challenges underlying these problems is scaling time series manipulation. Tick data,...
A Guide to Women In Unified Analytics Events at Spark+AI Summit Europe
Spark + AI Summit is Europe’s largest data and machine learning conference, and the big news in 2019 is how many women are driving some of the greatest advances in big data, machine learning, and data science. This 15-17 October in Amsterdam, more than 1700 data scientists, engineers, and developers will gather to share best...
Simple, Reliable Upserts and Deletes on Delta Lake Tables using Python APIs
We are excited to announce the release of Delta Lake 0.4.0 which introduces Python APIs for manipulating and managing data in Delta tables. The key features in this release are: Python APIs for DML and utility operations (#89) - You can now use Python APIs to update/delete/merge data in Delta Lake tables and to run...
Analyzing Your MLflow Data with DataFrames
Max Allen interned with Databricks Engineering in the Summer of 2019. This blog post, written by Max, highlights the great work he did while on the team. Introduction to MLflow and the Machine Learning Development Lifecycle MLflow is an open source platform for the machine learning lifecycle, and many Databricks customers have been using it...
