The three biggest security challenges facing AI and data initiatives
In today’s business climate, the ability to anticipate and meet customer needs is central to success. Forward looking business leaders are looking to unleash the power of Artificial Intelligence (AI) to drive innovation, but this requires bringing together diverse teams and large volumes of data. With attackers getting more sophisticated, securing these complex data workflows...
Securely Managing Credentials in Databricks
Customers today leverage a variety of data sources to perform complex queries to drive business insight. For example, to better understand user behavior, an analyst might correlate click-through analytics streams from their data warehouse with customer metadata from Salesforce. The Databricks Runtime provides a variety of built-in Apache Spark™ data source connectors (AWS | Azure),...
Improving Threat Detection in a Big Data World
High-profile cybersecurity breaches dominated headlines in 2017. In the first half of the year, over 1.9B records were stolen. That’s more than 7,000 records breached every minute. And the fallout from a single event can be staggering. Customer attrition, negative PR and regulatory fines amount to millions in financial losses. In fact, according to recent...
Best Practices for Coarse Grained Data Security in Databricks
At Databricks, we work with hundreds of companies, all pushing the bleeding edge in their respective industries. We want to share patterns for securing data so that your organization can leverage best practices as opposed to recreating the wheel when you on -board to Databricks’ Unified Analytics Platform. This post is primarily aimed at those...
Managing and Securing Credentials in Databricks for Apache Spark Jobs
Since Apache Spark separates compute from storage, every Spark Job requires a set of credentials to connect to disparate data sources. Storing those credentials in the clear can be a security risk if not stringently administered. To mitigate that risk, Databricks makes it easy and secure to connect to S3 with either Access Keys via...
Entropy-based Log Redaction for Apache Spark on Databricks
This blog post is part of our series of internal engineering blogs on Databricks platform, infrastructure management, tooling, monitoring, and provisioning. We love logs at Databricks. And we want to provide users the best experience to access Apache Spark logs. On Databricks, Spark logs are available via built-in Spark UI for clusters. Databricks also supports...
