Enforcing Column-level Encryption and Avoiding Data Duplication With PII
This is a guest post by Keyuri Shah, lead software engineer, and Fred Kimball, software engineer, Northwestern Mutual. Protecting PII (personally identifiable information) is very important as the number of data breaches and records with sensitive information exposed every day are trending upwards. To avoid becoming the next victim and protect users from identity...
Delta vs. Lambda: Why Simplicity Trumps Complexity for Data Pipelines
“Everything should be as simple as it can be, but not simpler” - Albert Einstein Generally, a simple data architecture is preferable to a complex one. Code complexity increases points of failure, requires more compute to run jobs, adds latency, and increases the need for support. As a result, data pipeline performance degrades over time,...
How Scribd Uses Delta Lake to Enable the World’s Largest Digital Library
Scribd uses Delta Lake to enable the world’s largest digital library. Watch this discussion with QP Hou, Senior Engineer at Scribd and an Airflow committer, and R Tyler Croy, Director of Platform Engineering at Scribd to learn how they transitioned from legacy on-premises infrastructure to AWS and how they utilized, implemented, and optimized Delta tables...
New Features to Accelerate the Path to Production With the Next Generation Data Science Workspace
Today, at the Data + AI Summit Europe 2020, we shared some exciting updates on the next generation Data Science Workspace - a collaborative environment for modern data teams - originally unveiled at Spark + AI Summit 2020. The power of data and artificial intelligence is already disrupting many industries, yet we've only scratched the...
Databricks and Coursera Launch Data Science Specialization for Data Analysts
Earlier this year, Databricks made a massive investment in training by providing free self-paced courses to all of our customers. Databricks furthers this investment by partnering with Coursera to provide Massive Open Online Courses (MOOC) training to the larger data community. Together we launched a new three-course specialization, Data Science with Databricks for Data Analysts,...
Databricks Partner Executive Summit at Data + AI Summit 2020 Europe
This week’s Partner Executive Summit, held in concert with Data + AI Summit 2020 Europe, is a feature event for our 500+ partners globally, and we love to share how partners are critical to making a positive impact on our joint customers with their solutions and integrations. Databricks success simply would not and could not...
Key Sessions for Microsoft Azure Customers at Data + AI Summit Europe 2020
Databricks, diamond sponsor Microsoft and Azure Databricks customers to present keynotes and breakout sessions at Data + AI Summit Europe. Data + AI Summit Europe is the free virtual event for data teams — data scientists, engineers and analysts — who will tune in from all over the world to share best practices, discover new...
Key Sessions for AWS Customers at Data + AI Summit Europe 2020
Databricks and Summit Gold Sponsor AWS Present on a wide variety of topics at this year’s premier data and AI event. Amazon Web Services (AWS) is sponsoring Data + AI Summit Europe 2020 and our work with AWS continues to make Databricks better integrated with other AWS services, making it easier for our customers to...
Fatal Force: Exploring Police Shootings With SQL Analytics
Introduction Data has shown that police in the United States kill civilians at a rate far higher than police in other wealthy countries.1 In 2019, law enforcement in the U.S. killed 33.5 civilians per 10 million people, compared to 1.3 in Germany and 0.5 in the U.K.2 This use of deadly force falls disproportionately on...
How to Evaluate Data Pipelines for Cost to Performance
Learn best practices for designing and evaluating cost-to-performance benchmarks from Germany’s #1 weather portal. While we certainly conduct several benchmarks, we know the best benchmark is your queries running on your data. But what are you benchmarking against in your evaluation? The answer seems obvious - cost and integration with your cloud architecture roadmap. We...
