Bucket Brigade — Securing Public S3 Buckets
Are your Amazon S3 buckets secure? Do you know which ones are public or private? Do you even know which ones are supposed to be? Data breaches are expensive. Facebook notoriously exposed 540 million records of its users recently, which was a contributing factor of the $5 billion fine they were issued by the FTC....
Azure Databricks Now Available in Azure Government (Public Preview)
We are excited to announce that Azure Databricks is now in Microsoft’s Azure Government region, enabling new data and AI use cases for federal agencies, state and local governments, public universities, and government contractors to enable faster decisions, more accurate predictions, and unified and collaborative data analytics. More than a dozen federal agencies are building...
Fighting Cyber Threats in the Public Sector with Scalable Analytics and AI
Join our webinar Real-time Threat Detection, Analytics and AI in the Public Sector to learn more and see a live demo. In 2019, there were 7,098 data breaches exposing over 15.1 billion records. That equates to a cyber incident every hour and fifteen minutes. The Public Sector is a prime target with cyber criminals and...
Azure Databricks Security Best Practices
Azure Databricks is a Unified Data Analytics Platform that is a part of the Microsoft Azure Cloud. Built upon the foundations of Delta Lake, MLflow, Koalas and Apache SparkTM, Azure Databricks is a first party PaaS on Microsoft Azure cloud that provides one-click setup, native integrations with other Azure cloud services, interactive workspace, and enterprise-grade...
Security that Unblocks the True Potential of your Data Lake
Over the last few years, Databricks has gained a lot of experience deploying data analytics at scale in the enterprise. In many cases, our customers have thousands of people using our product across different business units for a variety of different use cases —all of which involve accessing various data classifications from private and sensitive...
Securely Accessing Azure Data Sources from Azure Databricks
Azure Databricks is a Unified Data Analytics Platform that is a part of the Microsoft Azure Cloud. Built upon the foundations of Delta Lake, MLFlow , Koalas and Apache Spark, Azure Databricks is a first party service on Microsoft Azure cloud that provides one-click setup, native integrations with other Azure services, interactive workspace, and enterprise-grade...
Make Your Data Lake CCPA Compliant with a Unified Approach to Data and Analytics
With more digital data being captured every day, there has been a rise of various regulatory standards such as the General Data Protection Regulation (GDPR) and recently the California Consumer Privacy Act (CCPA). These privacy laws and standards are aimed at protecting consumers from businesses that improperly collect, use, or share their personal information, and...
Adventures in the TCP stack: Uncovering performance regressions in the TCP SACKs vulnerability fixes
Last month, we announced that the Databricks platform was experiencing network performance regressions due to Linux patches for the TCP SACKs vulnerabilities. The regressions were observed in less than 0.2% of cases when running the Databricks Runtime (DBR) on the Amazon Web Services (AWS) platform. In this post, we will dive deeper into our analysis that determined the TCP stack was the source of the degradation. We will discuss the symptoms we were seeing, walk through how we debugged the TCP connections, and explain the root cause in the Linux source. As a quick note before we jump in, Canonical is working on an Ubuntu 16.04 image that resolves these performance regressions. We plan to update the Databricks platform once that image is available and has passed our regression tests.
A failing benchmark
We were first alerted when one of our benchmarks became 6x slower. The regression appeared after upgrading the Amazon Machine Image (AMI) we use to incorporate Ubuntu’s fixes for the TCP SACKs vulnerabilities.Network performance regressions from TCP SACK vulnerability fixes
Update on Aug 2, 2019: Added to the end explaining our kernel patch and additional details we found. On June 17, three vulnerabilities in Linux’s networking stack were published. The most severe one could allow remote attackers to impact the system’s availability. We believe in offering the most secure image available to our customers, so...
How Databricks IAM Credential Passthrough Solves Common Data Authorization Problems
In our first blog post, we introduced Databricks IAM Credential Passthrough as a secure, convenient way for customers to manage access to their data. In this post, we'll take a closer look at how passthrough compares to other Identity and Access Management (IAM) systems. If you’re not familiar with passthrough, we suggest reading the first...
