Processing Geospatial Data at Scale With Databricks
The evolution and convergence of technology has fueled a vibrant marketplace for timely and accurate geospatial data. Every day billions of handheld and IoT devices along with thousands of airborne and satellite remote sensing platforms generate hundreds of exabytes of location-aware data. This boom of geospatial big data combined with advancements in machine learning is...
Streamlining Variant Normalization on Large Genomic Datasets with Glow
Cross posted from the Glow blog. Many research and drug development projects in the genomics world involve large genomic variant data sets, the volume of which has been growing exponentially over the past decade. However, the tools to extract, transform, load (ETL) and analyze these data sets have not kept pace with this growth. Single-node...
Databricks and Informatica Integration Simplifies Data Lineage and Governance for Cloud Analytics
In a rapidly evolving world of big data, data discovery, governance and data lineage is an essential aspect of data management. As organizations modernize their workloads into multi-cloud and hybrid environments, data starts to get distributed across cloud data lakes and SaaS applications. With that, organizations are trying to answer key questions: How do I...
New Databricks Integration for Jupyter Bridges Local and Remote Workflows
Introduction For many years now, data scientists have developed specific workflows on premises using local filesystem hierarchies, source code revision systems and CI/CD processes. On the other side, the available data is growing exponentially and new capabilities for data analysis and modeling are needed, for example, easily scalable storage, distributed computing systems or special hardware...
Migration from Hadoop to modern cloud platforms: The case for Hadoop alternatives
Companies rely on their big data and analytics platforms to support innovation and digital transformation strategies. However, many Hadoop users struggle with complexity, unscalable infrastructure, excessive maintenance overhead and overall, unrealized value. We help customers navigate their Hadoop migrations to modern cloud platforms such as Databricks and our partner products and solutions, and in this...
Brickster Spotlight: Meet Alex
At Databricks, we help data teams solve the world’s toughest problems and we couldn’t do that without our wonderful Databricks Team. “Teamwork makes the dream work” is not only a central function of our product but it is also central to our culture. Learn more about Alex, our Director of Resident Solution Architects in Databricks’...
How Databricks and Privacera Combine to Secure Data for Cloud Analytics
In their quest to anticipate customer needs, forward looking organizations are looking to use cloud-based analytics and AI to innovate. But we often hear from customers how challenging it is to manipulate large data volumes in a secure and compliant way. Databricks and Privacera have partnered to help customers address several key use cases for...
Deep Learning Tutorial Demonstrates How to Simplify Distributed Deep Learning Model Inference Using Delta Lake and Apache Spark™
On October 10th, our team hosted a live webinar—Simple Distributed Deep Learning Model Inference—with Xiangrui Meng, Software Engineer at Databricks. Model inference, unlike model training, is usually embarrassingly parallel and hence simple to distribute. However, in practice, complex data scenarios and compute infrastructure often make this "simple" task hard to do from data source to...
A Day at the 2019 Women in Product Conference
From left to right: Shveta, Julia, Yardley Pohl (a WIP board member and co-founder), Anna, Allie, Cyrielle, and Rani at the Databricks booth Databricks was a proud sponsor of the 2019 Women In Product conference which focuses on empowering women in product management and advocating for equal representation. We had a booth and happy hour...
Azure Databricks Highlights Adoption of Delta Lake, MLflow, and Integration with Azure Machine Learning at Microsoft Ignite 2019
At Microsoft Ignite 2019, thousands of attendees participated in hands-on workshops, breakout sessions, and theater presentations to learn how customers are achieving phenomenal results with Azure Databricks! It was an action-packed week of making new connections and learning about new innovation across data science, data engineering, and business analytics. We shared the news that over...