Enabling Massive Data Transformation Across Your Organization
The top four companies by market cap today (MSFT, APPL, AMZN, GOOG) are all software companies, running their businesses on data. This is a radical departure from a decade ago. Enterprises in every segment are beginning to establish dominance through data. As a bank executive recently told me: “We’re not a finance company, we are...
Productionize and Automate your Data Platform at Scale
Data-driven innovation is no longer optional to stay competitive in today's marketplace. Companies that can bring data, analytics, and ML-based products to market first will quickly beat their competition. While many companies have streamlined CI/CD (continuous integration and delivery) processes for application development, very few have well-defined processes for developing data and ML products. That’s...
Delivering and Managing a Cloud-Scale Enterprise Data Platform with Ease
Data is growing exponentially and organizations are building products to harness the data and provide services to their customers. However, this exponential growth cannot be sustained by an exponential growth in infrastructure spend or human capital cost. Today, there are over a hundred services available in each of the major clouds (AWS, Azure) that can...
Security that Unblocks the True Potential of your Data Lake
Over the last few years, Databricks has gained a lot of experience deploying data analytics at scale in the enterprise. In many cases, our customers have thousands of people using our product across different business units for a variety of different use cases —all of which involve accessing various data classifications from private and sensitive...
Solving the World’s Toughest Problems with the Growing Open Source Ecosystem and Databricks
We started Databricks in 2013 in a tiny little office in Berkeley with the belief that data has the potential to solve the world’s toughest problems. We entered 2020 as a global organization with over 1000 employees and a customer base spanning from two-person startups to Fortune 10s. In this blog post, let’s take a...
Make Your Data Lake CCPA Compliant with a Unified Approach to Data and Analytics
With more digital data being captured every day, there has been a rise of various regulatory standards such as the General Data Protection Regulation (GDPR) and recently the California Consumer Privacy Act (CCPA). These privacy laws and standards are aimed at protecting consumers from businesses that improperly collect, use, or share their personal information, and...
Deep Learning Tutorial Demonstrates How to Simplify Distributed Deep Learning Model Inference Using Delta Lake and Apache Spark™
On October 10th, our team hosted a live webinar—Simple Distributed Deep Learning Model Inference—with Xiangrui Meng, Software Engineer at Databricks. Model inference, unlike model training, is usually embarrassingly parallel and hence simple to distribute. However, in practice, complex data scenarios and compute infrastructure often make this "simple" task hard to do from data source to...
Delta Lake Now Hosted by the Linux Foundation to Become the Open Standard for Data Lakes
At today’s Spark + AI Summit Europe in Amsterdam, we announced that Delta Lake is becoming a Linux Foundation project. Together with the community, the project aims to establish an open standard for managing large amounts of data in data lakes. The Apache 2.0 software license remains unchanged. Delta Lake focuses on improving the reliability...
Productionizing Machine Learning: From Deployment to Drift Detection
Try this notebook to reproduce the steps outlined below and watch our on-demand webinar to learn more. In many articles and blogs the machine learning workflow starts with data prep and ends with deploying a model to production. But in reality, that's just the beginning of the lifecycle of a machine learning model. As they say,...
Diving Into Delta Lake: Unpacking The Transaction Log
The transaction log is key to understanding Delta Lake because it is the common thread that runs through many of its most important features, including ACID transactions, scalable metadata handling, time travel, and more. In this article, we’ll explore what the Delta Lake transaction log is, how it works at the file level, and how...