ACID Transactions on Data Lakes
As part of our Data + AI Online Meetup, we’ve explored topics ranging from genomics (with guests from Regeneron) to machine learning pipelines and GPU-accelerated ML to Tableau performance optimization. One key topic area has been an exploration of the Lakehouse. The rise of the Lakehouse architectural pattern is built upon tech innovations enabling the...
Enforcing Column-level Encryption and Avoiding Data Duplication With PII
This is a guest post by Keyuri Shah, lead software engineer, and Fred Kimball, software engineer, Northwestern Mutual. Protecting PII (personally identifiable information) is very important as the number of data breaches and records with sensitive information exposed every day are trending upwards. To avoid becoming the next victim and protect users from identity...
Delta vs. Lambda: Why Simplicity Trumps Complexity for Data Pipelines
“Everything should be as simple as it can be, but not simpler” - Albert Einstein Generally, a simple data architecture is preferable to a complex one. Code complexity increases points of failure, requires more compute to run jobs, adds latency, and increases the need for support. As a result, data pipeline performance degrades over time,...
How Scribd Uses Delta Lake to Enable the World’s Largest Digital Library
Scribd uses Delta Lake to enable the world’s largest digital library. Watch this discussion with QP Hou, Senior Engineer at Scribd and an Airflow committer, and R Tyler Croy, Director of Platform Engineering at Scribd to learn how they transitioned from legacy on-premises infrastructure to AWS and how they utilized, implemented, and optimized Delta tables...
MLflow Model Registry on Databricks Simplifies MLOps With CI/CD Features
MLflow helps organizations manage the ML lifecycle through the ability to track experiment metrics, parameters, and artifacts, as well as deploy models to batch or real-time serving systems. The MLflow Model Registry provides a central repository to manage the model deployment lifecycle, acting as the hub between experimentation and deployment. A critical part of MLOps,...
New Features to Accelerate the Path to Production With the Next Generation Data Science Workspace
Today, at the Data + AI Summit Europe 2020, we shared some exciting updates on the next generation Data Science Workspace - a collaborative environment for modern data teams - originally unveiled at Spark + AI Summit 2020. The power of data and artificial intelligence is already disrupting many industries, yet we've only scratched the...
Databricks and Coursera Launch Data Science Specialization for Data Analysts
Earlier this year, Databricks made a massive investment in training by providing free self-paced courses to all of our customers. Databricks furthers this investment by partnering with Coursera to provide Massive Open Online Courses (MOOC) training to the larger data community. Together we launched a new three-course specialization, Data Science with Databricks for Data Analysts,...
Databricks Partner Executive Summit at Data + AI Summit 2020 Europe
This week’s Partner Executive Summit, held in concert with Data + AI Summit 2020 Europe, is a feature event for our 500+ partners globally, and we love to share how partners are critical to making a positive impact on our joint customers with their solutions and integrations. Databricks success simply would not and could not...
Key Sessions for Microsoft Azure Customers at Data + AI Summit Europe 2020
Databricks, diamond sponsor Microsoft and Azure Databricks customers to present keynotes and breakout sessions at Data + AI Summit Europe. Data + AI Summit Europe is the free virtual event for data teams — data scientists, engineers and analysts — who will tune in from all over the world to share best practices, discover new...
Key Sessions for AWS Customers at Data + AI Summit Europe 2020
Databricks and Summit Gold Sponsor AWS Present on a wide variety of topics at this year’s premier data and AI event. Amazon Web Services (AWS) is sponsoring Data + AI Summit Europe 2020 and our work with AWS continues to make Databricks better integrated with other AWS services, making it easier for our customers to...
